跪拜 Guibai
← Back to the summary

Camera2 to NV21: The Silent Face-Detection Failure in ArcSoft's Android SDK

APP Face Recognition Value-Added SDK: Camera2 Seamless Integration Guide

1. Background

Camera2 is Google's camera API introduced in Android 5.0. It offers richer functionality but also increases integration complexity. Many users report that when trying the APP Face Recognition Value-Added Android Platform V5.0 SDK (hereafter APPAndroidV5.0), converting images captured by Camera2 to NV21 still fails to detect faces.

This article uses the Camera2 implementation in the APP Android V5.0 Demo as an example and explains the responsibilities of relevant files, method call relationships, and key points for image conversion, detection parameter configuration, and resource release, following the order: open camera → get image → convert to NV21 → perform detection → display results. It serves as a reference for integration and troubleshooting.

1.1 Preparation

Before reading and practicing, you need to prepare the corresponding SDK version and APPAndroidV5.0 Demo, and complete SDK activation. Camera permission declaration and request are described in section 2.4. The image size and orientation used for preview and detection should match; the configuration will be explained with code later.

Sample project environment: The APPAndroidV5.0 Demo referenced in this article uses Gradle 5.4.1, Android Gradle Plugin (AGP) 3.5.3, compile SDK version 29, and native library packaging target arm64-v8a. These are the Demo's configurations; when integrating into other projects, confirm compatibility with the SDK version you use.

1.2 Reading Notes

Section 1 explains the integration background, preparation items, and overall flow. Section 2 introduces Camera2-related files, call order, and page initialization and authorization startup. Section 3 explains size and coordinate configuration, preview output, and image acquisition. Section 4 explains image conversion, SDK algorithm integration, and result drawing. Section 5 explains camera pause, resume, and resource release. Section 6 summarizes integration points.

Excerpts are labeled with the corresponding file and method; class and method names follow the actual names in the Demo. Code trimmed to this article's scope is marked as flow illustration; integration requires configuring detection branches based on business needs. All Java paths are relative to app/src/main/java/com/arcsoft/arcfacedemo/.

1.3 Flow Overview

Camera2 provides two outputs in one capture session: one for preview via TextureView, and the other for ImageReader to acquire images, which are converted to NV21 for face detection and liveness processing. In the diagram, blue indicates camera capture, green indicates image conversion and detection.

flowchart TD
    A["Preparation: Complete SDK activation, obtain camera permission, initialize detection engine"] --> C2["Camera2: Open camera and create capture session"]
    C2 --> P2["TextureView displays camera preview"]
    C2 --> I2["ImageReader obtains Image"]
    I2 --> N["Read image data, convert to NV21"]
    N --> F["Detect face, get face position, etc."]
    F --> L["Perform liveness detection as needed"]
    L --> R["Draw face frame and detection result on preview"]

    style C2 fill:#e8f2ff,stroke:#4777b3
    style N fill:#e9f6ec,stroke:#4e8b5c
    style F fill:#e9f6ec,stroke:#4e8b5c
    style L fill:#e9f6ec,stroke:#4e8b5c

2. Camera Integration

2.1 File Responsibilities

Camera2-related code is mainly distributed in the following files:

File Responsibility in this flow
ui/activity/LivenessInteractiveGlareCamera2Activity.java Initializes page, connects camera callbacks, draws results
util/camera2/Camera2Helper.java Opens camera, creates preview session and ImageReader, retrieves and releases Image
util/camera2/Camera2Listener.java Notifies page of camera open, image arrival, and close
ui/viewmodel/InteractiveGlareLivenessCamera2ViewModel.java Initializes engine, processes frame data, connects liveness business
util/ImageUtil.java Converts Image to NV21
util/face/FaceHelper.java Calls face detection and organizes face information
util/FaceRectTransformer.java, widget/FaceRectView.java Handle face frame coordinates and page drawing

2.2 Method Call Order

From starting the camera to processing one frame, the main call order is:

  1. Start camera. The Activity calls Camera2Helper.resumeCamera(TextureView), which opens the camera and creates a capture session, outputting preview and detection images to TextureView and ImageReader respectively.
  2. Receive image. Camera2Listener.onImageAvailable(Image) notifies the Activity, which then calls viewModel.onPreviewFrame(image) to pass the current frame to the ViewModel.
  3. Convert format. Call ImageUtil.getNV21FromImage2(Image) to convert the camera image to NV21.
  4. Detect face. Call faceHelper.onPreviewFrame(rgbNv21, null, false) to get face information for the current frame.
  5. Handle liveness business. Pass the same frame's NV21 and face information to processLiveness() for subsequent processing according to business configuration. See section 4.3.
  6. Display results. The Activity calls drawPreviewInfo() to display the face frame and detection state.
  7. Release image. After the current frame is processed and the callback returns, Camera2Helper calls image.close() to release the image.

Preview and detection are provided by two outputs of the capture session; the above describes the per-frame processing order. Image release requirements are in section 3.4, and thread arrangement in section 4.1.

2.3 Page Initialization

LivenessInteractiveGlareCamera2Activity.onCreate() calls initView(), initViewModel(), and initData() in order:

Excerpt: LivenessInteractiveGlareCamera2Activity.initCamera2Helper().

camera2Helper = new Camera2Helper();
Point point = viewModel.loadPreviewSize();
camera2Helper.init(this, String.valueOf(previewConfig.getRgbCameraId()), new Size(point.x, point.y), point);

Here, loadPreviewSize() reads the width and height from configuration and passes them to Camera2Helper. The Activity then registers Camera2Listener to receive camera open and image arrival notifications. init() only saves parameters and prepares the CameraManager; the camera is not actually opened yet.

2.4 Authorization and Startup

SDK activation and camera authorization are two independent steps: after SDK activation, you must still obtain Android system camera permission to capture images via Camera2.

2.4.1 Permission Declaration

Add the following inside the <manifest> node and outside the <application> node in app/src/main/AndroidManifest.xml:

<uses-permission android:name="android.permission.CAMERA" />

Android 6.0 (API 23) and above also require runtime permission request. Check authorization status when the user enters the camera detection feature, and confirm again before opening the camera; do not rely only on the last authorization result. See the Android runtime permission guide.

2.4.2 Permission Request

The following is the permission flow you need to add when integrating the page; it is not an excerpt from the Demo source:

  1. Check whether permission is granted. Use ContextCompat.checkSelfPermission(this, Manifest.permission.CAMERA). If the result is PackageManager.PERMISSION_GRANTED, continue checking whether the page and preview control are ready.
  2. Request when not granted. First use the result of ActivityCompat.shouldShowRequestPermissionRationale() to decide whether to explain the purpose, e.g., "Used to display camera preview and perform face detection". After the user proceeds, send the system authorization request.
  3. Receive the result. If using the project's existing ActivityCompat.requestPermissions(), check the request code in onRequestPermissionsResult() and verify that grantResults is non-empty and camera permission is granted. Alternatively, use ActivityResultContracts.RequestPermission when supported by the dependency version; choose one method to avoid duplicate requests.
  4. Keep the page operable after denial. Pause the camera preview and detection, display the reason permission is not enabled, and allow the user to return or actively retry. When the system no longer shows the authorization dialog, provide a user-clickable "Go to Settings" entry; do not loop dialogs or auto-navigate. Do not assume permanent denial just because shouldShowRequestPermissionRationale() returns false, because it may also return that value before the first request.

After returning from system settings or re-entering the page, re-check permission before deciding whether to start the camera. Also avoid duplicate requests during permission requests.

2.4.3 Starting the Camera

The authorization callback, page resume, and preview control readiness may occur in different orders. It is recommended to have these entry points uniformly check the following conditions before starting the camera:

The Demo's onGlobalLayout() removes this layout listener, so on first denial, do not wait only for that listener to fire again. After successful authorization, also proactively perform the above startup condition checks; if the page is paused, wait until it resumes. Confirm permission before each actual device open, and handle SecurityException caused by permission changes.

Excerpt: Startup invocation in LivenessInteractiveGlareCamera2Activity.onGlobalLayout().

camera2Helper.resumeCamera(binding.cameraTexturePreviewRgb);

Camera2Helper.resumeCamera(TextureView) first calls startHandlerThread(), then checks whether the TextureView's SurfaceTexture is available: if already available, it opens the camera; otherwise it waits for onSurfaceTextureAvailable() before opening.

The actual camera open happens in openCamera(): checks permissions, configures outputs, and calls cameraManager.openCamera(). After receiving CameraDevice.StateCallback.onOpened(), it saves the CameraDevice and starts creating the preview session, notifying the Activity via onCamera2Opened().

Here, onCamera2Opened() indicates the device is open, not that the session is necessarily configured successfully; that depends on the subsequent onConfigured() or onConfigureFailed().

3. Image Acquisition

3.1 Size and Coordinates

In the Activity's onCamera2Opened(), the configuration related to image size and display includes:

  1. Adjust the preview control and overlay size according to the preview size and orientation.
  2. Create a FaceRectTransformer, configuring size, orientation, and mirroring.
  3. Call viewModel.onRgbCameraOpened(previewSizeRgb) and then pass the coordinate transformer to the ViewModel.

The ViewModel's init() creates ftEngine and flEngine for face detection and this page's liveness processing respectively; onRgbCameraOpened(Size) then constructs FaceHelper with the preview width and height and configures it to keep only the largest face. The algorithm input width and height depend on the size saved here, so if you change the capture resolution, you must also update the corresponding FaceHelper configuration.

3.2 Creating ImageReader

setUpCameraOutputs() iterates over the device camera IDs, finds a camera matching specifiedCameraId, reads its output configurations, and creates an ImageReader.

Excerpt: Camera2Helper.setUpCameraOutputs().

previewSize = getBestPreviewSize(map.getOutputSizes(SurfaceTexture.class));
imageReader = ImageReader.newInstance(previewSize.getWidth(), previewSize.getHeight(), IMAGE_READER_FORMAT, 5);
imageReader.setOnImageAvailableListener(onImageAvailableListener, cameraHandler);

IMAGE_READER_FORMAT is defined in the class as ImageFormat.YUV_420_888. The final 5 is the maximum number of images that can be acquired but not yet closed, not the capture frame rate.

When integrating, choose a resolution based on the target camera's supported output configurations and confirm that preview and YUV images can be output at the chosen size simultaneously. Pass the chosen size to Camera2Helper and initialize FaceHelper with the corresponding image size.

3.3 Configuring the Capture Session

Excerpt: Output configuration in Camera2Helper.createCameraPreviewSession().

previewRequestBuilder = cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
Surface imageReaderSurface = imageReader.getSurface();
previewRequestBuilder.addTarget(imageReaderSurface);
if (surfaceTexture != null) {
    surfaceTexture.setDefaultBufferSize(previewSize.getWidth(), previewSize.getHeight());
    Surface surface = new Surface(surfaceTexture);
    previewRequestBuilder.addTarget(surface);
    outPuts = Arrays.asList(surface, imageReaderSurface);
} else {
    outPuts = Collections.singletonList(imageReaderSurface);
}

The ImageReader's Surface is always added to the request; if a SurfaceTexture is passed, the preview Surface is added as well. Then createCaptureSession(outPuts, ...) creates the session with these outputs.

In the session's onConfigured(), the Demo configures continuous auto-focus, sets auto-flash related parameters when flash is supported, then calls setRepeatingRequest() for continuous capture. Preview display and algorithm frame acquisition share the same camera session but consume their own outputs.

3.4 Frame Retrieval and Release

Excerpt: Camera2Helper.onImageAvailableListener.

private final ImageReader.OnImageAvailableListener onImageAvailableListener = new ImageReader.OnImageAvailableListener() {
    @Override
    public void onImageAvailable(ImageReader reader) {
        final Image image = reader.acquireLatestImage();
        if (image == null) {
            return;
        }
        if (camera2Listener != null) {
            camera2Listener.onImageAvailable(image);
        }
        image.close();
    }
};

acquireLatestImage() gets the latest available image. The Helper hands the Image to the page listener and closes it after the listener returns, so the current call is synchronous: the page calls the ViewModel, the ViewModel converts and processes the image, and only after all returns does image.close() execute.

When integrating your own page, put the listener call in try/finally and have the Helper close the image in finally, so it is released regardless of whether processing completes normally. The page and ViewModel finish reading before the Image is closed, and the Helper is solely responsible for closing.

4. Image Processing and Detection

4.1 Frame Processing Entry

After the Activity receives the onImageAvailable(Image) callback, it first clears the previous frame's display information, then passes the image to viewModel.onPreviewFrame(image) to complete format conversion, face detection, and liveness processing. If the returned result is non-empty and the coordinate transformer is ready, it calls drawPreviewInfo() to draw results; see section 4.4.

Flow illustration: trimmed from InteractiveGlareLivenessCamera2ViewModel.onPreviewFrame(Image). Liveness business entry configuration is in section 4.3.

public List<FacePreviewInfo> onPreviewFrame(Image image) {
    // Convert image and detect face.
    byte[] rgbNv21 = ImageUtil.getNV21FromImage2(image);
    List<FacePreviewInfo> facePreviewInfoList =
            faceHelper.onPreviewFrame(rgbNv21, null, false);
    clearLeftFace(facePreviewInfoList);

    // Pass the same frame image and face information to liveness business.
    return processLiveness(rgbNv21, facePreviewInfoList);
}

Two points when integrating:

4.2 Converting to NV21

This step prepares image data for the face algorithm. The camera provides YUV_420_888 format, but the current face detection call expects NV21, so the Demo uses ImageUtil.getNV21FromImage2() to convert, returning a byte[] array for subsequent detection.

Excerpt: ImageUtil.getNV21FromImage2().

public static byte[] getNV21FromImage2(Image image) {
    byte[] nv21 = ArcSoftImageUtil.yuvToNv21(
            ArcSoftImageUtil.getBufferAddress(image.getPlanes()[0].getBuffer()),
            image.getPlanes()[0].getRowStride(),
            ArcSoftImageUtil.getBufferAddress(image.getPlanes()[1].getBuffer()),
            image.getPlanes()[1].getRowStride(),
            image.getPlanes()[1].getPixelStride(),
            ArcSoftImageUtil.getBufferAddress(image.getPlanes()[2].getBuffer()),
            image.getPlanes()[2].getRowStride(),
            image.getPlanes()[2].getPixelStride(),
            image.getWidth(),
            image.getHeight()
    );
    return nv21;
}

This method passes the Image's full width and height; it does not pass a crop region, nor does it perform rotation or horizontal mirroring. Converting to NV21 only means the data format is rearranged; it does not mean the picture orientation is adjusted. For example, even if the preview appears mirrored, this method will not automatically flip the image bytes left-right.

4.3 SDK Algorithm Integration

After completing NV21 conversion, first detect a face, then connect liveness detection or registration/recognition as needed. The following explains integration points through the Demo's business encapsulation.

1. Face Detection

Call faceHelper.onPreviewFrame(rgbNv21, null, false) to get face information. null means no IR image is passed, and false means the identity recognition branch is skipped. FaceHelper internally calls the SDK's ftEngine.detectFaces(), with input format FaceEngine.CP_PAF_NV21, and width/height using the previewSize saved at initialization, which must match the current image. The returned result is wrapped as FacePreviewInfo, containing face information and a tracking ID.

2. Liveness Detection

When action liveness is needed, pass the NV21 image and its face detection result to the Demo's processLiveness() business entry, and configure the internal processing logic according to the selected business. The image and face information must come from the same frame, to avoid using a previous frame's detection result for the current image.

3. Registration and Recognition

For face registration, feature extraction, and identity comparison, refer to the business logic in RecognizeViewModel and connect the Camera2 NV21 data to the corresponding flow. The false parameter above skips the identity recognition branch, so registration/recognition business also needs the corresponding processing entry.

4.4 Result Drawing

Excerpt: LivenessInteractiveGlareCamera2Activity.drawPreviewInfo().

private void drawPreviewInfo(List<FacePreviewInfo> facePreviewInfoList) {
    if (rgbFaceRectTransformer != null) {
        List<FaceRectView.DrawInfo> rgbDrawInfoList = viewModel.getDrawInfo(facePreviewInfoList, LivenessType.RGB);
        binding.cameraFaceRectView.drawRealtimeFaceInfo(rgbDrawInfoList);
    }
}

The ViewModel's getDrawInfo() sets display text and colors based on the liveness result, such as "Detecting", "Detection Complete: Liveness", "Detection Complete: Non-liveness". The page then passes the drawing information to FaceRectView.

Coordinate handling depends on the FaceRectTransformer established when the camera opened. Preview orientation, algorithm byte orientation, and detection box coordinates are different issues: getNV21FromImage2() does not rotate the image, so a normal preview does not mean the algorithm input has been rotated.

5. Lifecycle Management

1. Pause Capture

When the page goes into the background, call pauseCamera() in onPause(). The following is a call illustration; refer to the Helper implementation in your project for specific stop and close logic.

@Override
protected void onPause() {
    super.onPause();
    if (camera2Helper != null) {
        camera2Helper.pauseCamera();
    }
}

2. Resume Preview

When the page resumes, in onResume() check camera permission, preview control, and initialization state as in section 2.4. Confirm the previous close is complete and the camera is not started or in the process of opening, then call resumeCamera().

3. Cleanup on Exit

When cleaning up, first stop new tasks and camera capture, wait for ongoing image processing to finish, then release images and detection resources no longer used. The Helper should coordinate the closing of the camera session, device, and ImageReader; the detection engine is released by its owning Activity or ViewModel, and finally stop the background thread that is no longer used.

Pause and destruction may trigger cleanup in sequence; avoid duplicate releases. Put waiting operations for tasks or threads in the background to avoid blocking the main thread.

6. Summary

Camera2 and the SDK connect through NV21 images: the camera side handles preview and capture, and the business layer handles format conversion, algorithm calls, and result display. For integration, refer to the Demo's call flow and connect face detection, liveness detection, or registration/recognition business as needed.

The key to integration is ensuring that image format, size, and orientation match the algorithm configuration, and that the image and face information come from the same frame. At the same time, clarify the separation between background processing and main-thread UI updates, and coordinate camera start/stop and resource release during page pause, resume, and exit, avoiding duplicate opens, premature image closes, or releasing detection resources while still in use.