AOSP Contemplation | Android Multimedia: Why Does a Video Need So Many Services

Author: 哲鲤

Tags: Android

In an Android app, playing a video with MediaPlayer usually starts with setting the data source, preparing, and calling start(). The interface looks simple, but internally the system has to handle container parsing, audio/video decoding, playback synchronization, audio output, and visual display. These tasks are carried out by different components working together, spread across the app process and multiple system services.

This division of labor also explains common phenomena during development: being able to read the video duration does not mean the decoder can handle the video track; having decoded video frames does not mean the frames have been displayed on screen. When encountering playback issues, first figure out where the data has reached and which component is responsible for that step, then you can find the appropriate debugging entry point.

This article starts from the AOSP source code, uses MediaPlayer's playback path to string together the multimedia framework, then expands to camera and recording chains, explaining what responsibilities MediaCodec, audio services, the display system, media copyright protection, and resource management each take on. After reading, hopefully you can place familiar application APIs back into the system architecture, and also follow the flow of control requests and media data to understand how the modules coordinate.

Looking at the Multimedia Framework's Division of Labor Through a Video

First, look at the playback architecture. Along the flow of media data, you can find the data source, demuxing, decoding, audio/video synchronization, and device output; along the control requests, you can find the components responsible for receiving calls, organizing playback, and managing resources.

Figure 1: Blue dashed lines represent control requests; green solid lines represent media data transfer or processing results. Audio and video separately enter output paths below; the sidebar covers media copyright protection, resource management, and codec adaptation notes. Gray groups are divided by responsibility; process locations follow the annotations in the nodes.

Module	Main Responsibility	Representative Components
App Interface	Set data source, configure parameters, initiate playback or recording	MediaPlayer / MediaRecorder / MediaCodec
Playback Service & Player	Receive playback requests, coordinate data source, decoding, and output sync	MediaPlayerService / NuPlayerDriver / NuPlayer
Container Parsing	Identify container, read track info and track data	MediaExtractorFactory / Extractor
Codec Adaptation	Configure components, submit input and receive output	MediaCodec / ACodec / CCodec
Audio Policy & Output	Select output device, organize audio data processing, interface with HAL	AudioPolicyService / AudioFlinger
Camera Capture	Configure camera session, hand captured data to output targets	CameraService / Camera3Device / Camera HAL
Media Copyright Protection	Manage DRM sessions and key exchange, provide content decryption interface	DrmHal / CryptoHal
Media Resource Management	Record resource usage, coordinate client release	ResourceManagerService
Video Display	Carry video buffer, manage windows, compose picture and output	Surface / WMS / SurfaceFlinger

The table lists different responsibilities; do not treat each row as a separate process, nor interpret components in the same row as a serial call chain. For example, Surface carries video output, WMS manages window layout and visibility, and SurfaceFlinger handles composition; video pixels do not need to go to WMS first then to SurfaceFlinger.

This article chooses MediaPlayer's NuPlayer playback path. When an app directly uses MediaCodec to implement a player, it can organize data reading, decoding, and output itself, without going through MediaPlayerService.

How Playback Requests Enter the System

MediaPlayerService Receives Requests, NuPlayer Organizes Playback

In this path, MediaPlayerService is the service entry. Its Client object holds the player instance for the corresponding client and forwards requests like prepareAsync() and start() to it. The specific player type is selected by a factory based on input and other information.

For the NuPlayer path discussed here, NuPlayerDriver interfaces with the service-side player API, and internally creates NuPlayer. The latter coordinates Source, Decoder, and Renderer: Source provides media data, Decoder interfaces with codec components, and Renderer handles output timing and audio/video synchronization.

Thus, in the architecture diagram, the "Playback Service & Player Coordination" box lists the service entry, player implementation, and internal responsibilities. They should be understood based on their relationships; do not interpret Source, Decoder, and Renderer as services that sequentially receive playback control commands.

Source: Client selects player, Client forwards playback request.

Source: NuPlayerDriver creates NuPlayer, NuPlayer requests Source preparation.

Service Division and Process Deployment Are Two Different Things

Here we look at three entries specifically: init configuration specifies which process to start, main_mediaserver.cpp explains which services mediaserver registers, and main_audioserver.cpp explains how audio services are created and registered. The version used in this article is the android17-release baseline of Android 17, with source links pinned to the corresponding repository commits.

First confirm the process. Two init configurations start different executables:

Source location: frameworks/av/media/mediaserver/mediaserver.rc.

service media /system/bin/mediaserver
    class main
    user media
    group audio camera inet net_bt net_bt_admin net_bw_acct drmrpc mediadrm

Source location: frameworks/av/media/audioserver/audioserver.rc.

service audioserver /system/bin/audioserver
    class core
    user audioserver

Here service media and service audioserver are init service names; the paths after them are the programs each starts. Just based on class names like MediaPlayerService and AudioFlinger, you cannot tell if they are in the same process; these two configurations are the direct entry to the deployment relationship.

Then look at mediaserver's service registration. In its main(), you can directly find:

Source location: frameworks/av/media/mediaserver/main_mediaserver.cpp.

MediaPlayerService::instantiate();
ResourceManagerService::instantiate();

This shows that when mediaserver starts, it registers the playback service and resource management service. Next, move to audioserver's main(), where creation and publication of audio services are done in two steps:

Source location: frameworks/av/media/audioserver/main_audioserver.cpp.

const auto af = sp<AudioFlinger>::make();
const auto afAdapter = sp<AudioFlingerServerAdapter>::make(af);

// …省略与当前结论无关的代码…

const auto aps = sp<AudioPolicyService>::make();
af->initAudioPolicyLocal(aps);

// …省略与当前结论无关的代码…

aps->onAudioSystemReady();

// …省略与当前结论无关的代码…

// Add AudioFlinger and AudioPolicy to ServiceManager.
sp<IServiceManager> sm = defaultServiceManager();
sm->addService(String16(IAudioFlinger::DEFAULT_SERVICE_NAME), afAdapter,
        false /* allowIsolated */, IServiceManager::DUMP_FLAG_PRIORITY_DEFAULT);
sm->addService(String16(AudioPolicyService::getServiceName()), aps,
        false /* allowIsolated */, IServiceManager::DUMP_FLAG_PRIORITY_DEFAULT);

First create AudioFlinger and AudioPolicyService, establish local associations and complete relevant initialization, then register with ServiceManager via addService(). Here afAdapter is AudioFlinger's service adapter object, aps is the audio policy service object. The source comments explain this order: if AudioFlinger were published first, remote calls might arrive before AudioPolicy is fully initialized.

From these code snippets, we get a clear deployment conclusion: playback service and resource management reside in mediaserver; the audio execution and audio policy services here reside in audioserver. How they pass control requests and media data requires further reading of interface and Buffer-related implementations. Service registration code itself cannot prove that "every frame is transported via Binder," so you cannot substitute process division for data path analysis.

Source: mediaserver registers player and resource management, audioserver's local initialization and registration.

What Processing Does Media Data Go Through?

Demux First, Then Hand to Decoder

MP4, WebM, etc., are container formats; H.264, HEVC, AAC, etc., are the encoding formats used by tracks. The Extractor is responsible for reading track information and data from the container; the decoder then processes that data according to the encoding format and configuration.

MediaExtractorFactory is responsible for selecting the extractor, MediaCodec is responsible for configuring the codec component. Therefore, being able to read duration and track information only means some container parsing work is done; it does not prove the decoder supports the track's encoding profile, resolution, or other parameters.

Source location: frameworks/av/media/libstagefright/MediaExtractorFactory.cpp.

sp<IMediaExtractor> MediaExtractorFactory::Create(
        const sp<DataSource> &source, const char *mime) {
    ALOGV("MediaExtractorFactory::Create %s", mime);

    if (!property_get_bool("media.stagefright.extractremote", true)) {
        // local extractor
        ALOGW("creating media extractor in calling process");
        return CreateFromService(source, mime);

// …省略与当前结论无关的代码…

sp<IBinder> binder = defaultServiceManager()->getService(String16("media.extractor"));

if (binder != 0) {
    sp<IMediaExtractorService> mediaExService(
            interface_cast<IMediaExtractorService>(binder));
    sp<IMediaExtractor> ex;
    mediaExService->makeExtractor(
            CreateIDataSourceFromDataSource(source),
            mime ? std::optional<std::string>(mime) : std::nullopt,
            &ex);
    return ex;

// …省略与当前结论无关的代码…

sp<IMediaExtractor> MediaExtractorFactory::CreateFromService(
        const sp<DataSource> &source, const char *mime) {

// …省略与当前结论无关的代码…

void *meta = nullptr;
void *creator = NULL;
FreeMetaFunc freeMeta = nullptr;
float confidence;
sp<ExtractorPlugin> plugin;
uint32_t creatorVersion = 0;
creator = sniff(source, &confidence, &meta, &freeMeta, plugin, &creatorVersion);
if (!creator) {

There are two layers of judgment. Create() first chooses between local creation or requesting the media.extractor service based on media.stagefright.extractremote; the actual extractor creation logic then uses sniff() to identify the input and select a plugin. When creator is null, it enters error handling. Thus "finding the extraction service" and "finding a plugin that recognizes this container" are also two different outcomes.

Source: MediaExtractorFactory::Create, MediaCodec::configure.

When discussing this path, several terms need to be distinguished:

Term	Meaning in this article
Encoded data	Data represented by audio/video encoding formats; the input before decoding compressed tracks
Container track sample (Sample)	Samples organized by container per track, along with timing and format info
Access Unit (AU)	Processing unit defined by encoding format and interface; NuPlayer's Source/Decoder interfaces use this name
PCM audio data	Typically the output of audio decoding or audio capture; can enter mixing
Video frames and Surface Buffer	Image content and the Buffers carrying them; concrete pixel format depends on components and config
Container muxing (Muxing)	Organizing audio/video track data and timing info into a container like MP4

Sample is a container-level concept; AU is a codec interface concept; they cannot always be equated across all formats. Containers can also contain PCM tracks, so "Extractor output" should not be generalized as "compressed stream."

In NuPlayer, GenericSource::dequeueAccessUnit() provides track data, the Decoder fetches input and submits it to MediaCodec, and passes the corresponding output to the Renderer. The Renderer schedules output based on media time; it does not decode again.

Source location: frameworks/av/media/libmediaplayerservice/nuplayer/NuPlayerDecoder.cpp.

status_t NuPlayer::Decoder::fetchInputData(sp<AMessage> &reply) {
    sp<ABuffer> accessUnit;
    bool dropAccessUnit = true;
    do {
        status_t err = mSource->dequeueAccessUnit(mIsAudio, &accessUnit);

        if (err == -EWOULDBLOCK) {
            return err;

// …省略与当前结论无关的代码…

err = mCodec->queueInputBuffer(
        bufferIx,
        codecBuffer->offset(),
        codecBuffer->size(),
        timeUs,
        flags,
        &errorDetailMsg);

// …省略与当前结论无关的代码…

mRenderer->queueBuffer(mIsAudio, buffer, reply);

The three snippets come from input fetching, input submission, and output processing paths: mSource provides Access Units, mCodec receives input Buffers, mRenderer receives corresponding outputs. timeUs is the media timestamp carried when submitting input; output synchronization later needs media time information.

Source: GenericSource::dequeueAccessUnit(), Decoder fetches Access Unit, Decoder submits input Buffer, Renderer::queueBuffer().

ACodec and CCodec Are Two Codec Adaptation Implementations

MediaCodec provides a unified codec control interface, adapting to specific components through adapters: ACodec interfaces with OMX, CCodec with Codec2. Both belong to the component control and adaptation layer; the components themselves can be software or hardware implementations.

At creation, GetCodecBase() first checks the component's ownership flag owner. If the ownership flag doesn't determine the adaptation implementation, it then checks the component name. The order in the source code is as follows:

Order	Condition	Result
1	owner is default	Create ACodec
2	owner starts with codec2	Create CCodec
3	Neither of the above, component name starts with c2.	Create CCodec
4	Neither of the above, component name starts with omx.	Create ACodec
5	None matched	Return null object

Component name prefix matching is case-insensitive. This table describes the selection rule at creation time; it cannot be used to infer an automatic retry strategy after decoding failure.

graph TD
    M[MediaCodec creates codec adaptation object] --> S[Check component ownership flag owner]
    S -->|default| A[ACodec]
    S -->|codec2 prefix| C[CCodec]
    S -->|other or absent| N[Check component name]
    N -->|c2. prefix| C
    N -->|omx. prefix| A
    N -->|other| E[Return null object]
    A --> O[Interface with OMX component]
    C --> K[Interface with Codec2 component]

Source location: frameworks/av/media/libstagefright/MediaCodec.cpp.

sp<CodecBase> MediaCodec::GetCodecBase(const AString &name, const char *owner) {
    if (owner) {
        if (strcmp(owner, "default") == 0) {
            return new ACodec;
        } else if (strncmp(owner, "codec2", 6) == 0) {
            return CreateCCodec();
        }
    }

    if (name.startsWithIgnoreCase("c2.")) {
        return CreateCCodec();
    } else if (name.startsWithIgnoreCase("omx.")) {
        // at this time only ACodec specifies a mime type.
        return new ACodec;
    } else {
        return NULL;
    }
}

Once the ownership flag matches, the function immediately returns the corresponding adapter; only when the ownership flag doesn't match does it continue to check the component name.

Source: MediaCodec::GetCodecBase(), ACodec::initiateAllocateComponent(), ACodec's OMX node allocation, CCodec::initiateAllocateComponent().

Audio Policy Determines Output, AudioFlinger Organizes Execution

Audio playback needs to answer two questions: which output device to select, and how to send audio data to that device.

AudioPolicyManager selects the output based on audio attributes, device, policy configuration, etc.; AudioFlinger creates a Track and processes audio data in the corresponding output thread, interfacing with the audio HAL. In the createTrack path, AudioFlinger obtains the output selected by policy, so the two are in a collaborative relationship.

The regular mixing path handles PCM. Direct and Offload use different output paths; you cannot draw all audio playback as "decode to PCM first, then enter the normal mixing thread." Which path is taken depends on format, requested parameters, and device capabilities.

Source location: frameworks/av/services/audioflinger/AudioFlinger.cpp.

lStatus = AudioSystem::getOutputForAttr(&localAttr, &output.outputId, sessionId, &streamType,
                                        adjAttributionSource, &input.config, input.flags,
                                        &selectedDeviceIds, &portId, &secondaryOutputs,
                                        &isSpatialized, &isBitPerfect);
output.selectedDeviceIds = selectedDeviceIds;

if (lStatus != NO_ERROR || output.outputId == AUDIO_IO_HANDLE_NONE) {
    ALOGE("createTrack() getOutputForAttr() return error %d or invalid output handle", lStatus);
    goto Exit;
}

// …省略与当前结论无关的代码…

{
    audio_utils::lock_guard _l(mutex());
    IAfPlaybackThread* thread = checkPlaybackThread_l(output.outputId);
    if (thread == NULL) {
        ALOGE("no playback thread found for output handle %d", output.outputId);
        lStatus = BAD_VALUE;
        goto Exit;
    }

getOutputForAttr() returns the selected output handle output.outputId, then checkPlaybackThread_l() looks up the playback thread based on that handle. If the policy fails to provide a valid output, or the corresponding thread is not found, it enters the failure path. This is the concrete connection point between "selecting output" and "organizing execution"; subsequent Track creation and data processing have other steps.

Source: AudioPolicyManager::getOutputForAttr, createTrack gets policy output, audioserver's local initialization and registration.

Between Decoding Complete and Picture Display There Is Output Synchronization

NuPlayer's Renderer schedules video output based on media time. Getting a decoded video frame only means the decoding stage has a result; that frame still needs to be handed to the Surface chain at the right output moment, then participate in SurfaceFlinger's composition and display.

So, for black screen issues, besides checking track and decoder, also check whether the output Surface is valid, window visibility, and whether Buffers are continuously delivered. WMS and SurfaceFlinger serve different roles here; don't lump their behaviors under just "display module."

Source location: frameworks/av/media/libmediaplayerservice/nuplayer/NuPlayerRenderer.cpp.

int64_t nowUs = ALooper::GetNowUs();
int64_t realTimeUs;
int64_t mediaTimeUs = -1;
if (mFlags & FLAG_REAL_TIME) {
    CHECK(entry->mBuffer->meta()->findInt64("timeUs", &realTimeUs));
} else {
    CHECK(entry->mBuffer->meta()->findInt64("timeUs", &mediaTimeUs));

    realTimeUs = getRealTimeUs(mediaTimeUs, nowUs);
}
realTimeUs = mVideoScheduler->schedule(realTimeUs * 1000) / 1000;

In the normal media time path, timeUs is extracted from the Buffer, converted to an output moment via getRealTimeUs(), then handed to the video scheduler. This reflects the responsibility of "when to output"; between the decoder providing a frame and that frame entering the display chain, there is still timing processing.

Source: Renderer's video output scheduling, WMS executes window Surface layout, SurfaceFlinger composition entry.

Why Copyright Protection and Resource Management Are in the Sidebar

Media DRM: Sessions, Key Exchange, and Content Decryption

Here Media DRM refers to Digital Rights Management, a different concept from Direct Rendering Manager (DRM) in the Linux graphics display subsystem.

DrmHal interfaces with DRM plugins, manages sessions, and handles key requests and responses; CryptoHal provides the decryption interface for encrypted media data. They work in conjunction with protected content playback; they cannot be drawn as a step executed after all video decoding.

For protected content, obtaining a license does not mean the device can necessarily play it. Secure decoding and output protection requirements also need to be confirmed with content policy and device capabilities. The architecture diagram therefore lists copyright protection separately, reminding readers to check these conditions during configuration and playback.

Source location: frameworks/av/drm/libmediadrm/DrmHal.cpp.

DrmStatus DrmHal::openSession(DrmPlugin::SecurityLevel securityLevel, Vector<uint8_t>& sessionId) {
    if (mDrmHalAidl->initCheck() == OK) return mDrmHalAidl->openSession(securityLevel, sessionId);
    return mDrmHalHidl->openSession(securityLevel, sessionId);
}

// …省略与当前结论无关的代码…

DrmStatus DrmHal::provideKeyResponse(Vector<uint8_t> const& sessionId,
                                     Vector<uint8_t> const& response, Vector<uint8_t>& keySetId) {
    if (mDrmHalAidl->initCheck() == OK)
        return mDrmHalAidl->provideKeyResponse(sessionId, response, keySetId);
    return mDrmHalHidl->provideKeyResponse(sessionId, response, keySetId);
}

Source location: frameworks/av/drm/libmediadrm/CryptoHal.cpp.

ssize_t CryptoHal::decrypt(const uint8_t key[16], const uint8_t iv[16], CryptoPlugin::Mode mode,
                           const CryptoPlugin::Pattern& pattern, const ::SharedBuffer& source,
                           size_t offset, const CryptoPlugin::SubSample* subSamples,
                           size_t numSubSamples, const ::DestinationBuffer& destination,
                           AString* errorDetailMsg) {
    // This requires plugin to be created.
    if (mCryptoHalAidl->initCheck() == OK)
        return mCryptoHalAidl->decrypt(key, iv, mode, pattern, source, offset, subSamples,
                                       numSubSamples, destination, errorDetailMsg);
    return mCryptoHalHidl->decrypt(key, iv, mode, pattern, source, offset, subSamples,
                                   numSubSamples, destination, errorDetailMsg);
}

The DrmHal snippet shows session creation and key response handling; the CryptoHal snippet shows decryption calls. Both select implementations based on the initialization state of the AIDL implementation; the parameters also reflect the responsibility difference: the former handles sessionId and key responses, the latter handles key identifier, initialization vector iv, source and destination Buffers. The parameter named key in the source code cannot be simply understood as the app having the plaintext key.

Source: DrmHal session and key exchange, CryptoHal decryption interface.

ResourceManagerService: Records Occupancy, Coordinates Resource Reclamation

ResourceManagerService records resource occupancy registered by clients. In codec scenarios, these records help the system determine which clients hold relevant resources.

Upon receiving a reclamation request, the service selects target clients according to resource policy, then calls the corresponding client's reclaimResource() to ask it to release resources. The service is responsible for selection and coordination; actual release is done by the client, and reclamation may also fail.

This is more specific than "associating component lifecycle": when a component uses a resource, it needs to update occupancy records; when reclamation occurs, the service and client collaborate. When analyzing a second video failing to start, you should place component errors, resource requests, and reclamation results on the same timeline.

Source location: frameworks/av/services/mediaresourcemanager/ResourceManagerService.cpp.

std::vector<ClientInfo> targetClients;
if (getTargetClients(clientInfo, resources, targetClients)) {
    // Reclaim all the target clients.
    *_aidl_return = reclaimUnconditionallyFrom(targetClients);
} else {
    // No clients to reclaim from.
    ALOGI("%s: There aren't any clients to reclaim from", __func__);
    // We need to log this failed reclaim as "no clients to reclaim from".
    targetClients.clear();
}

// …省略与当前结论无关的代码…

for (const ClientInfo& targetClient : targetClients) {
    std::shared_ptr<IResourceManagerClient> client = nullptr;
    {
        std::scoped_lock lock{mLock};
        client = getClient_l(targetClient.mPid, targetClient.mClientId);
    }
    if (client == nullptr) {
        // skip already released clients.
        continue;
    }
    String8 log = String8::format("reclaimResource from client %p", client.get());
    mServiceLog->add(log);
    bool success;
    Status status = client->reclaimResource(&success);
    if (!status.isOk() || !success) {
        failedClientId = targetClient.mClientId;
        failedClientPid = targetClient.mPid;

The first section selects reclamation targets; the second comes from reclaimUnconditionallyFrom(): after finding a client, it calls client->reclaimResource(&success), then checks the call status and release result. Therefore, initiating reclamation, selecting targets, and successful client release are three observation points; a service completing one request doesn't guarantee reclamation succeeded.

Source: Register resource, Remove resource record, Select reclamation target, Callback client to request release.

Recording and Playback Share Capabilities, But Data Directions Differ

Typical file playback reads track data from a container, then decodes and outputs; recording gets data from camera, audio capture, or input Surface, encodes it, and muxes to a file. Both paths can use MediaCodec and resource management capabilities, but playback and recording are organized by their respective implementations.

Figure 2: Above is the recording control request, marking objects in the app process and mediaserver; below is the data path for capture, encoding, muxing, and file writing. Blue dashed lines indicate control requests; green solid lines indicate media data transfer and file writing. Below is grouped by responsibility; specific inputs depend on recording configuration.

Where Do Recording Control Objects Run?

MediaRecorder is the app-side recording interface; MediaRecorderClient and StagefrightRecorder reside in mediaserver. The app's control requests reach MediaRecorderClient via Binder, which then calls StagefrightRecorder within the process.

First look at the Binder proxy's start(). Source location: frameworks/av/media/libmedia/IMediaRecorder.cpp, BpMediaRecorder::start().

status_t start()
{
    ALOGV("start");
    Parcel data, reply;
    data.writeInterfaceToken(IMediaRecorder::getInterfaceDescriptor());
    remote()->transact(START, data, &reply);
    return reply.readInt32();
}

remote()->transact(START, ...) sends the start request to the server and reads the result from the reply. This connection passes the start request, not the captured audio/video frames.

The server-side MediaRecorderClient creates StagefrightRecorder in its constructor. Source location: frameworks/av/media/libmediaplayerservice/MediaRecorderClient.cpp.

MediaRecorderClient::MediaRecorderClient(const sp<MediaPlayerService>& service,
        const AttributionSourceState& attributionSource)
{
    ALOGV("Client constructor");
    // attribution source already validated in createMediaRecorder
    mAttributionSource = attributionSource;
    mRecorder = new StagefrightRecorder(attributionSource);
    mMediaPlayerService = service;
}

The start() in the same file then calls this recorder object:

status_t MediaRecorderClient::start()
{
    ALOGV("start");
    Mutex::Autolock lock(mLock);
    if (mRecorder == NULL) {
        ALOGE("recorder is not initialized");
        return NO_INIT;
    }
    return mRecorder->start();

}

new StagefrightRecorder(...) creates the object in the process where MediaRecorderClient runs; mRecorder->start() is an in-process call. This is the relationship between the two mediaserver nodes in the diagram.

StagefrightRecorder resides in mediaserver and is responsible for organizing capture sources, encoders, and Writer; camera capture, audio capture, and specific codec processing may still involve other services or HALs. The "configure capture, encode, and Writer" in the diagram indicates coordination responsibility; you cannot infer that the entire recording chain runs within mediaserver.

Source: Server creates recording session, Binder start request, Create StagefrightRecorder, Server forwards start request.

How Encoder Output Is Handed to Writer

Taking StagefrightRecorder's MP4/WebM recording configuration as an example, the system creates a Writer based on output format; when video input is configured, it sets up the media source and video encoder, then adds the encoder output to the Writer; when audio input is configured and conditions are met, it adds an audio encoded track. Recording with only audio or only video need not include the other branch.

When recording ends, StagefrightRecorder also needs to stop the encoding source and Writer, completing writing and muxing finalization. After the app sends a stop request, you should continue to monitor the call result and whether the subsequent file is valid.

Source location: frameworks/av/media/libmediaplayerservice/StagefrightRecorder.cpp.

sp<MediaWriter> writer;
sp<MPEG4Writer> mp4writer;
if (mOutputFormat == OUTPUT_FORMAT_WEBM) {
    writer = new WebmWriter(mOutputFd);
} else {
    writer = mp4writer = new MPEG4Writer(mOutputFd);
}

if (mVideoSource < VIDEO_SOURCE_LIST_END) {
    setDefaultVideoEncoderIfNecessary();

    sp<MediaSource> mediaSource;
    err = setupMediaSource(&mediaSource);
    if (err != OK) {
        return err;
    }

    sp<MediaCodecSource> encoder;
    err = setupVideoEncoder(mediaSource, &encoder);
    if (err != OK) {
        return err;
    }

    writer->addSource(encoder);
    mVideoEncoderSource = encoder;
    mTotalBitRate += mVideoBitRate;
}

// …省略与当前结论无关的代码…

// disable audio for time lapse recording
const bool disableAudio = mCaptureFpsEnable && mCaptureFps < mFrameRate;
if (!disableAudio && mAudioSource != AUDIO_SOURCE_CNT) {
    err = setupAudioEncoder(writer);
    if (err != OK) return err;
    mTotalBitRate += mAudioBitRate;
}

What Cleanup Work Does Stopping Recording Include?

Source location: frameworks/av/media/libmediaplayerservice/StagefrightRecorder.cpp, stop().

int64_t stopTimeUs = systemTime() / 1000;
for (const auto &source : { mAudioEncoderSource, mVideoEncoderSource }) {
    if (source != nullptr && OK != source->setStopTimeUs(stopTimeUs)) {
        ALOGW("Failed to set stopTime %lld us for %s",
                (long long)stopTimeUs, source->isVideo() ? "Video" : "Audio");
    }
}

if (mWriter != NULL) {
    err = mWriter->stop();
    mLastSeqNo = mWriter->getSequenceNum();
    mWriter.clear();
}

In the configuration snippet, writer->addSource(encoder) adds the video encoder output to the container writing flow; whether audio is added depends on input and time-lapse related conditions. The stop snippet first sets the stop time for encoding sources, then calls mWriter->stop(). The result of mWriter->stop() is stored in err, then the Writer reference is cleared. To judge whether finalization succeeded, check the return status; seeing the object cleared is not sufficient.

Source: Configure MP4/WebM recording, Start recording, Stop recording and finalize.

Distinguish Request vs. Completion with Flow and States

Playback and Recording Separately

Below are the data directions for file playback and recording:

graph LR
    P0[Playback data source] --> P1[Demux and track read]
    P1 --> P2[Decode or applicable audio direct path]
    P2 --> P3[Output synchronization]
    P3 --> P4[Audio device or video display]

graph LR
    R0[Camera or audio capture] --> R1[Audio/video encoding]
    R1 --> R2[Container muxing]
    R2 --> R3[Write recording file]

Figures 3, 4: Playback reads existing media; recording generates a media file. The two diagrams indicate data direction, not that all nodes start in strict serial timing.

Playback States: Taking NuPlayerDriver as an Example

prepareAsync() returning success only means the preparation request was accepted. In the normal asynchronous preparation path, NuPlayerDriver goes from STATE_UNPREPARED to STATE_PREPARING; after the preparation completion callback succeeds, it enters STATE_PREPARED. Then calling start() enters STATE_RUNNING.

graph LR
    U[STATE_UNPREPARED not prepared] -->|prepareAsync| P[STATE_PREPARING preparing]
    P -->|preparation complete and success| D[STATE_PREPARED prepared]
    D -->|start| R[STATE_RUNNING running]
    R -->|pause| A[STATE_PAUSED paused]
    A -->|start| R

Figure 5: A subset of states for NuPlayerDriver's normal prepare, play, and pause paths. Data source setting, stop, reset, and error branches are not expanded here. State names come from that object's implementation, not equivalent to application API or other components' states.

This state diagram describes the playback object, separately managed from MediaRecorder's recording state. Even in STATE_RUNNING, you cannot prove based solely on the state value that the user has seen the first frame or heard sound. Player state, data output, and device presentation are different observation points; combine callbacks and output records to judge.

Source location: frameworks/av/media/libmediaplayerservice/nuplayer/NuPlayerDriver.cpp.

prepareAsync() accepts the request:

case STATE_UNPREPARED:
    mState = STATE_PREPARING;
    mIsAsyncPrepare = true;
    mPlayer->prepareAsync();
    return OK;

When returning OK, the state has been set to STATE_PREPARING, and preparation is still ongoing.

notifyPrepareCompleted() handles the preparation result:

mAsyncResult = err;

if (err == OK) {
    // update state before notifying client, so that if client calls back into NuPlayerDriver
    // in response, NuPlayerDriver has the right state
    mState = STATE_PREPARED;
    if (mIsAsyncPrepare) {
        notifyListener_l(MEDIA_PREPARED);
    }
} else {
    mState = STATE_UNPREPARED;
    if (mIsAsyncPrepare) {
        notifyListener_l(MEDIA_ERROR, MEDIA_ERROR_UNKNOWN, err);
    }
}

On success, it enters STATE_PREPARED and sends a preparation complete notification; on failure, it returns to STATE_UNPREPARED and sends an error notification.

start_l() starts playback:

case STATE_PAUSED:
case STATE_STOPPED_AND_PREPARED:
case STATE_PREPARED:
{
    mPlayer->start();

    FALLTHROUGH_INTENDED;
}
// …
mState = STATE_RUNNING;

return OK;

These states represent request acceptance, preparation completion, and start handling. First frame and audio output need to be observed with their respective output records.

Source: Async preparation, Preparation complete callback, Start and running state, Pause.

From Architecture Division to Development Debugging

From the architecture perspective, the multimedia framework separates container compatibility, codec components, device routing, display, and resource constraints. When analyzing collaboration, besides finding the caller and callee, you also need to confirm who provides the data, who holds the resource, and who notifies the completion result.

From the app development perspective, record time points you can observe: data source setting, preparation callback, start request, playback errors, and stop request. For a self-built MediaCodec player, you can also record track discovery, component configuration, input/output Buffers, etc. Platform debugging continues to correlate audio Track, Surface output, and resource reclamation records, choosing available records based on the API used.

In automotive systems, also record user, display, and audio device context. For example, video may be decoded but displayed on another target; audio may be played but routed to a different device. Such phenomena require checking along the output path.

Phenomenon	Priority Checks	Results to Distinguish
Can read duration but playback black screen	Video track parameters, component configuration errors, output Surface and window visibility	Container info reading, decode output, and actual display
Video fine but no audio	Whether audio track exists, output device selection, Track continuously processing data	Routing selection and audio data output
Second video fails to start	Component concurrency capability, configuration errors, resource occupancy and reclamation results	Format support and resource availability
Recording fails to start	Capture source configuration, permission errors, encoder configuration and resource request results	Capture input availability and encoding resource availability
Recording stopped but file abnormal	Stop return value, Writer errors, encoder output and file write records	Data already written vs. container finalization success
Protected content playback fails	DRM session, key response, decryption errors, secure decode and output protection conditions	Content authorization and subsequent playback conditions

These checks provide a locating order. The final judgment still requires correlating parameters, time points, error codes, and component records from the same operation.

Audio Focus, Media Sessions, and Native Audio Interfaces

Beyond the audio execution chain, there are also app-to-app playback coordination and system media controls. AudioService provides audio focus entry points, MediaSession connects system media controls with app callbacks; AAudio is another native audio stream interface. Do not treat all three as necessary paths for NuPlayer to process data.

For example, the playback state in MediaSession is reported by the app, suitable for driving the system control interface, but it cannot directly prove that AudioFlinger has output sound. AAudio's MMAP or Legacy path selection is also affected by stream parameters, system policy, and device support.

Source location: frameworks/base/services/core/java/com/android/server/media/MediaSessionRecord.java, setPlaybackState().

synchronized (mLock) {
    mPlaybackState = state;
    updateUserEngagedStateIfNeededLocked(
            /* isTimeoutExpired= */ false, isGlobalPrioritySessionActive);
}
// …省略调用身份处理等逻辑…
mService.onSessionPlaybackStateChanged(
        MediaSessionRecord.this, shouldUpdatePriority, mPlaybackState);
// …省略调用身份恢复…
mHandler.post(MessageHandler.MSG_UPDATE_PLAYBACK_STATE);

This stores the passed state, notifies the media session service, and schedules a state update message. This method does not read AudioFlinger's device output results, so the media control interface's "playing" state should be observed separately from actual audio output.

Source: Audio focus request, App reports media session state, AAudio's MMAP conditions and Legacy path.

Contemplation and Insights

Same Audio/Video Processing, Different Problems Solved

Placing the Android multimedia framework alongside FFmpeg and GStreamer clarifies the module division above. All three can participate in audio/video processing, but they provide different ranges of capability: FFmpeg provides libraries for formats, codecs, filters, etc.; GStreamer provides a composable media pipeline framework; Android must integrate media processing into an operating system where multiple apps share the device.

This comparison is between FFmpeg's libraries, GStreamer framework, and Android's platform media stack. The ffmpeg command-line program is one usage of these libraries; it is not equivalent to the entire runtime structure of an app integrating FFmpeg.

Comparison Dimension	FFmpeg	GStreamer	Android Platform Media Stack
Core organization	Libraries like libavformat, libavcodec, libavfilter divide tasks; app organizes calls; filters can form processing graphs	Elements connect through Pads to form a Pipeline; plugins combine processing capabilities	App API, player/recorder implementation, system services, and HALs cooperate
Data processing interface	Data structures like Packet, Frame, and codec input/output interfaces	Buffers flow between Elements; Caps describe and negotiate media formats	Source, MediaCodec Buffer, audio Track, Surface, etc. together form the path
Control and feedback	App manages library calls, error returns, and its own scheduling	Pipeline states, events, and Bus messages participate in control and feedback	API requests, Binder calls, component states, and async notifications reflect processing progress
Format extension	Modules for codecs, container formats, protocols, etc., selectable at build time	Plugins provide source, processing, encoding, output Elements	Extractor plugins, OMX/Codec2 components, and device HALs carry extension
Relationship with system	Can integrate platform hardware acceleration and output capabilities; multi-app device policies are typically borne by the host system	Can interface with device and hardware via platform plugins; system-level policies are provided by the integration environment	Audio policy, window display, permissions, and media resource management are themselves part of platform collaboration

FFmpeg's library interfaces reflect the idea of dividing by processing capability. For example, libavcodec's avcodec_send_packet() and avcodec_receive_frame() separate input submission from output retrieval: submitting one Packet does not guarantee an immediate corresponding Frame. This interface constraint is not the same implementation as Android's async processing, but it reminds developers of the same issue: input being accepted and output being produced are two different results.

GStreamer emphasizes composition of media components. Elements connect through Pads, Caps express format requirements, Pipeline organizes runtime states, and Bus delivers messages to the app. This design lets apps change processing by adjusting pipeline composition, but also requires attention to format negotiation, state transitions, and behavior of each Element.

Android's MediaPlayer/NuPlayer path organizes common playback processes for apps; typically apps don't need to connect extractors, decoders, and output components one by one. When directly using MediaCodec, the app must handle data reading, Buffer submission and retrieval, timestamps, and output coordination. The platform offers different abstraction levels, letting developers choose between convenience and control.

References: FFmpeg library responsibilities, libavcodec input/output interface, GStreamer basics, GStreamer Caps and format negotiation. These design ideas are generalizations based on responsibility division and interface constraints, not references to designers' personal motivations.

Where Android's Complexity Comes From: System Constraints

If only one app handles one media file, format parsing, codecs, and output could be organized fairly centrally. Android must also handle multiple apps running simultaneously, shared hardware, app exits, and changes in output devices and window states. These constraints explain why the framework cannot be just a single "read, decode, play" pipeline.

Resource availability depends on the entire system. A component supporting a particular encoding format does not mean an instance is currently available. ResourceManagerService records occupancy, selects reclamation targets, and asks clients to release resources; it handles cross-client resource contention. FFmpeg or GStreamer integrated with hardware components may also face resource shortages, but the operating system's reclamation rules cannot be decided solely by one app's processing pipeline.

Output devices are determined by system policy collectively. Apps provide audio attributes, AudioPolicy participates in output selection, and AudioFlinger organizes execution; video also needs coordination with Surface, windows, and the display system. Successful data processing is only one step; what the user hears and on which screen they see it is also affected by device routing and display state.

Protected content requires more than just decoding capability. Media DRM's session and key exchange, content decryption, and corresponding secure decoding and output protection requirements need to work with device implementation. Being able to integrate a library that supports the target format does not imply the device meets protected content playback requirements.

Service publication and internal initialization have ordering constraints. audioserver first creates and associates AudioFlinger and AudioPolicyService, then registers with ServiceManager, reflecting another responsibility of system services: when externally callable, internal dependencies must be in a usable state. This issue goes beyond the correctness of a single codec function and cannot be explained just by drawing a data flow.

These constraints can all be found in the service initialization, output selection, DRM interface, and resource reclamation code discussed earlier. Process deployment, component responsibilities, and media data flow thus need to be analyzed on different dimensions, then relationships established among them.

Inspiration for Android Development

Introducing FFmpeg or GStreamer can change how an app reads media, organizes processing, and combines codec capabilities, but how it connects with Android's device, display, permissions, and resource management still depends on the specific integration approach. When choosing a media framework, first clarify whether the problem to solve is format compatibility, pipeline composition, processing control, or system device coordination.

Looking back at Android's design, it organizes common media processing into easy-to-call interfaces while delegating shared devices and system policies to corresponding services. This reduces the coordination burden on typical apps, but also means failures can occur among multiple components. A takeaway worth keeping: check both the media processing itself and the system conditions it relies on; understanding a framework's capabilities also means understanding what responsibilities it takes on and what it leaves to the caller.