③NuPlayer播放框架之类NuPlayer源码分析

[时间：2016-10] [状态：Open]
[关键词：android，nuplayer，开源播放器，播放框架]

0 引言

差不多一个月了，继续分析AOSP的播放框架的源码。这次我们需要深入分析的是NuPlayer类，相比于NuPlayerDriver的接口功能，NuPlayer继承自AHandler类，是AOSP播放框架中连接Source、Decoder、Render的纽带。
我希望读完本文大家可以对NuPlayer的源码结构有一定了解。

本文是我的NuPlayer播放框架的第三篇。

1 主要接口和核心的类成员

NuPlayer类被NuPlayerDriver直接调用，其主要接口如下：

// code frome NuPlayer.h (~/frameworks/av/media/libmediaplayerservice/nuplayer/)
struct NuPlayer : public AHandler {
    NuPlayer(pid_t pid);
    void setUID(uid_t uid);
    void setDriver(const wp<NuPlayerDriver> &driver);
    void setDataSourceAsync(...);
    void prepareAsync();
    void setVideoSurfaceTextureAsync(const sp<IGraphicBufferProducer> &bufferProducer);
    void start();
    void pause();

    // Will notify the driver through "notifyResetComplete" once finished.
    void resetAsync();

    // Will notify the driver through "notifySeekComplete" once finished
    // and needNotify is true.
    void seekToAsync(int64_t seekTimeUs, bool needNotify = false);

    status_t setVideoScalingMode(int32_t mode);
    status_t getTrackInfo(Parcel* reply) const;
    status_t getSelectedTrack(int32_t type, Parcel* reply) const;
    status_t selectTrack(size_t trackIndex, bool select, int64_t timeUs);
    status_t getCurrentPosition(int64_t *mediaUs);

    sp<MetaData> getFileMeta();
    float getFrameRate();

protected:
    virtual ~NuPlayer();
    virtual void onMessageReceived(const sp<AMessage> &msg);
}

接口分类下，无外乎几个分类：

• 用于初始化的（比如构造函数、setDriver/setDataSourceAsync/prepareAsync/setVideoSurfaceTextureAsync）
• 用于销毁的（比如析构函数、resetAsync）
• 用于播放控制的（比如start/pause/seekToAsync）
• 用于状态获取的（比如getCurrentPosition/getFileMeta）

下面是主要的类成员部分

wp<NuPlayerDriver> mDriver; // 接口调用方
sp<Source> mSource; // 相当于FFmpeg中的demuxer
sp<Surface> mSurface; // 显示用的Surface
sp<DecoderBase> mVideoDecoder; // 视频解码器
sp<DecoderBase> mAudioDecoder; // 音频解码器
sp<CCDecoder> mCCDecoder; 
sp<Renderer> mRenderer; // 渲染器
sp<ALooper> mRendererLooper;

2 setDataSourceAsync实现分析

这个函数有多重不同的重载形式，如下：

void setDataSourceAsync(const sp<IStreamSource> &source);
void setDataSourceAsync(const sp<IMediaHTTPService> &httpService, const char *url,
            const KeyedVector<String8, String8> *headers);
void setDataSourceAsync(int fd, int64_t offset, int64_t length);
void setDataSourceAsync(const sp<DataSource> &source);

需要根据实际情况选择，这里以第三个接口为例，说明下多本地媒体文件是如何处理的。
下面是这个函数的实现代码：

void NuPlayer::setDataSourceAsync(int fd, int64_t offset, int64_t length) {
    sp<AMessage> msg = new AMessage(kWhatSetDataSource, this);

    sp<AMessage> notify = new AMessage(kWhatSourceNotify, this);
	// 创建对象用于读取本地文件
    sp<GenericSource> source =
            new GenericSource(notify, mUIDValid, mUID);
	// 实际干活的的代码
    status_t err = source->setDataSource(fd, offset, length);

    if (err != OK) {
        ALOGE("Failed to set data source!");
        source = NULL;
    }

    msg->setObject("source", source);
    msg->post();
}

看实现很简单，创建GenericSource对象，并调用其setDataSource接口，然后发送kWhatSetDataSource消息。
我们看看如何处理然后发送kWhatSetDataSource消息呢？代码如下：

case kWhatSetDataSource:
{
    CHECK(mSource == NULL);

    status_t err = OK;
    sp<RefBase> obj;
    CHECK(msg->findObject("source", &obj));
    if (obj != NULL) {
        Mutex::Autolock autoLock(mSourceLock);
        mSource = static_cast<Source *>(obj.get());
    } else {
        err = UNKNOWN_ERROR;
    }
	// 通知Driver函数调用完成
    CHECK(mDriver != NULL);
    sp<NuPlayerDriver> driver = mDriver.promote();
    if (driver != NULL) {
        driver->notifySetDataSourceCompleted(err);
    }
    break;
}

看到这里发现，其实没做什么就是直接通知NuPlayerDriver。我们还注意到这里构建了一个特殊消息（AMessage）notify，这个消息用于在Source和NuPlayer直接传递。下面这是消息循环中的处理函数：

case kWhatSourceNotify:
{
    onSourceNotify(msg);
    break;
}

在后续讨论Source的时候详细说明这个消息通知的意义。

3 prepareAsync

这个函数实现的功能对应于MediaPlayerBase::prepare/prepareAsync接口，实现异步的prepare功能，一般就是做一些额外的初始化工作。那么直接看一下实现：

void NuPlayer::prepareAsync() {
    (new AMessage(kWhatPrepare, this))->post();
}

代码就是发了一个kWhatPrepare的消息。接下来是如何处理这个消息。

case kWhatPrepare:
{
    mSource->prepareAsync();
    break;
}

最终还是调用了Source::prepareAsync接口。后面会解释其功能。（这里面可能会解析下码流，读取音频、视频、字幕流信息，读取时长、元数据等）。

4 setVideoSurfaceTextureAsync

调用这个接口主要为了设置视频渲染窗口。其实现相对简单，创建一个Surface，然后发送异步的kWhatSetVideoSurface消息。代码如下：

void NuPlayer::setVideoSurfaceTextureAsync( const sp<IGraphicBufferProducer> &bufferProducer) {
    sp<AMessage> msg = new AMessage(kWhatSetVideoSurface, this);

    if (bufferProducer == NULL) {
        msg->setObject("surface", NULL);
    } else {
        msg->setObject("surface", new Surface(bufferProducer, true /* controlledByApp */));
    }

    msg->post();
}

那么看看如何处理kWhatSetVideoSurface消息呢？

case kWhatSetVideoSurface: {
    sp<RefBase> obj;
    CHECK(msg->findObject("surface", &obj));
    sp<Surface> surface = static_cast<Surface *>(obj.get());

    // Need to check mStarted before calling mSource->getFormat because NuPlayer might
    // be in preparing state and it could take long time.
    // When mStarted is true, mSource must have been set.
    if (mSource == NULL || !mStarted || mSource->getFormat(false /* audio */) == NULL
            // NOTE: mVideoDecoder's mSurface is always non-null
            || (mVideoDecoder != NULL && mVideoDecoder->setVideoSurface(surface) == OK)) {
        performSetSurface(surface); // 通知NuPlayerDriver设置完成
        break;
    }
	// 清空音频、视频缓冲
    mDeferredActions.push_back(
            new FlushDecoderAction(FLUSH_CMD_FLUSH /* audio */,FLUSH_CMD_SHUTDOWN /* video */));
	// 最终调用NuPlayer::performSetSurface接口
    mDeferredActions.push_back(new SetSurfaceAction(surface));

    if (obj != NULL || mAudioDecoder != NULL) {
        if (mStarted) {
            // Issue a seek to refresh the video screen only if started otherwise
            // the extractor may not yet be started and will assert.
            // If the video decoder is not set (perhaps audio only in this case)
            // do not perform a seek as it is not needed.
            int64_t currentPositionUs = 0;
            if (getCurrentPosition(&currentPositionUs) == OK) {
                mDeferredActions.push_back(
                        new SeekAction(currentPositionUs));
            }
        }

        // 对于新的surface设置，重置下解码器
        mDeferredActions.push_back(new SimpleAction(&NuPlayer::performScanSources));
    }

    // After a flush without shutdown, decoder is paused.
    // Don't resume it until source seek is done, otherwise it could
    // start pulling stale data too soon.
    mDeferredActions.push_back(
            new ResumeDecoderAction(false /* needNotify */));
	// 把上面mDeferredActions中缓存的所有Action处理下，并清空
    processDeferredActions();
    break;
}

这里的代码相对复杂点，涉及到很多，其实主要是为了设置Surface之后，可以正常解码显示，因为某些情况下解码器初始化需要依赖于具体的Surface。当然，里边还涉及到NuPlayer状态及初始化判断。

5 start/pause

start函数实现很简单，实际就发送了kWhatStart消息。

void NuPlayer::start() {
    (new AMessage(kWhatStart, this))->post();
}

在消息处理函数中的处理如下：

case kWhatStart:
{
    if (mStarted) {
        // do not resume yet if the source is still buffering
        if (!mPausedForBuffering) {
            onResume();
        }
    } else {
        onStart();
    }
    mPausedByClient = false;
    break;
}

直接调用了OnStart/OnResume函数。
pause函数实现类似，只是发送的是kWhatPause消息。在消息处理函数中的代码如下：

case kWhatPause:
{
    onPause();
    mPausedByClient = true;
    break;
}

直接调用的onPause函数。下面单独分析下这三个函数。先从简单的函数开始OnPause/onResume

NuPlayer::onPause

这个函数实现暂停功能，总体来说就是把Source和Render暂停就可以了，代码如下：

void NuPlayer::onPause() {
    if (mPaused) {
        return;
    }
    mPaused = true;
    if (mSource != NULL) {
        mSource->pause();
    }
    if (mRenderer != NULL) {
        mRenderer->pause();
    }
}

NuPlayer::onResume

这个函数实现恢复功能，代码逻辑跟onPause差不多，把Source和Render恢复，还可能涉及其它操作。代码如下：

void NuPlayer::onResume() {
    if (!mPaused || mResetting) {
        return;
    }
    mPaused = false;
    if (mSource != NULL) {
        mSource->resume();
    }
    // |mAudioDecoder| may have been released due to the pause timeout, so re-create it if
    // needed.
    if (audioDecoderStillNeeded() && mAudioDecoder == NULL) {
        instantiateDecoder(true /* audio */, &mAudioDecoder);
    }
    if (mRenderer != NULL) {
        mRenderer->resume();
    }
}

NuPlayer::onStart

这个接口实现启动的操作，相对复杂点，需要初始化解码器、初始化Render、设置Source状态，并将三者关联起来。代码如下：

void NuPlayer::onStart(int64_t startPositionUs) {
    if (!mSourceStarted) {
        mSourceStarted = true;
        mSource->start(); // 设置Source状态
    }
    
	// ... (省略部分代码)

    sp<AMessage> notify = new AMessage(kWhatRendererNotify, this);
    ++mRendererGeneration; // 创建Render和RenderLooper，属性设置、与解码器关联
    notify->setInt32("generation", mRendererGeneration);
    mRenderer = new Renderer(mAudioSink, notify, flags);
    mRendererLooper = new ALooper;
    mRendererLooper->setName("NuPlayerRenderer");
    mRendererLooper->start(false, false, ANDROID_PRIORITY_AUDIO);
    mRendererLooper->registerHandler(mRenderer);

    status_t err = mRenderer->setPlaybackSettings(mPlaybackSettings);

    float rate = getFrameRate();
    if (rate > 0) {
        mRenderer->setVideoFrameRate(rate);
    }

    if (mVideoDecoder != NULL) {
        mVideoDecoder->setRenderer(mRenderer);
    }
    if (mAudioDecoder != NULL) {
        mAudioDecoder->setRenderer(mRenderer);
    }

    postScanSources();
}

上面代码中没有解码器的初始化，那只能继续看看postScanSources代码了。看实现发现就是发送了kWhatScanSources消息。那么消息循环里边是怎么处理的呢？

case kWhatScanSources:
{
    int32_t generation;
    CHECK(msg->findInt32("generation", &generation));
    if (generation != mScanSourcesGeneration) {
        // Drop obsolete msg.
        break;
    }

    mScanSourcesPending = false;
    bool mHadAnySourcesBefore = (mAudioDecoder != NULL) || (mVideoDecoder != NULL);
    bool rescan = false;

    // initialize video before audio because successful initialization of
    // video may change deep buffer mode of audio.
    if (mSurface != NULL) { // 初始化视频解码器
        if (instantiateDecoder(false, &mVideoDecoder) == -EWOULDBLOCK) {
            rescan = true;
        }
    }

    // Don't try to re-open audio sink if there's an existing decoder.
    if (mAudioSink != NULL && mAudioDecoder == NULL) { // 初始化音频解码器
        if (instantiateDecoder(true, &mAudioDecoder) == -EWOULDBLOCK) {
            rescan = true;
        }
    }

    if (!mHadAnySourcesBefore && (mAudioDecoder != NULL || mVideoDecoder != NULL)) {
        // This is the first time we've found anything playable.
		// 设置定期查询时长
        if (mSourceFlags & Source::FLAG_DYNAMIC_DURATION) {
            schedulePollDuration();
        }
    }

    status_t err; // 一些异常处理逻辑
    if ((err = mSource->feedMoreTSData()) != OK) {
        if (mAudioDecoder == NULL && mVideoDecoder == NULL) {
            // We're not currently decoding anything (no audio or
            // video tracks found) and we just ran out of input data.

            if (err == ERROR_END_OF_STREAM) {
                notifyListener(MEDIA_PLAYBACK_COMPLETE, 0, 0);
            } else {
                notifyListener(MEDIA_ERROR, MEDIA_ERROR_UNKNOWN, err);
            }
        }
        break;
    }
	// 如果需要的话，重新扫描Source
    if (rescan) {
        msg->post(100000ll);
        mScanSourcesPending = true;
    }
    break;
}

6 seekToAsync

这个函数完成seek操作，其实现比较简单直接发送kWhatSeek消息，代码如下：

void NuPlayer::seekToAsync(int64_t seekTimeUs, bool needNotify) {
    sp<AMessage> msg = new AMessage(kWhatSeek, this);
    msg->setInt64("seekTimeUs", seekTimeUs);
    msg->setInt32("needNotify", needNotify);
    msg->post();
}

在消息循环里边的处理代码如下：

case kWhatSeek:
{
    int64_t seekTimeUs;
    int32_t needNotify;
    if (!mStarted) {
        // Seek before the player is started. In order to preview video,
        // need to start the player and pause it. This branch is called
        // only once if needed. After the player is started, any seek
        // operation will go through normal path.
        // Audio-only cases are handled separately.
        onStart(seekTimeUs);
        if (mStarted) {
            onPause();
            mPausedByClient = true;
        }
        if (needNotify) {
            notifyDriverSeekComplete();
        }
        break;
    }

    mDeferredActions.push_back(
            new FlushDecoderAction(FLUSH_CMD_FLUSH /* audio */,
                                   FLUSH_CMD_FLUSH /* video */));
	// 真正做seek事情的在这里
    mDeferredActions.push_back(new SeekAction(seekTimeUs));

    // After a flush without shutdown, decoder is paused.
    // Don't resume it until source seek is done, otherwise it could
    // start pulling stale data too soon.
    mDeferredActions.push_back(new ResumeDecoderAction(needNotify));

    processDeferredActions();
    break;
}

实际代码中SeekAction最终调用performSeek接口，其实现如下：

void NuPlayer::performSeek(int64_t seekTimeUs) {
    if (mSource == NULL) {
        // This happens when reset occurs right before the loop mode
        // asynchronously seeks to the start of the stream.
        LOG_ALWAYS_FATAL_IF(mAudioDecoder != NULL || mVideoDecoder != NULL,
                "mSource is NULL and decoders not NULL audio(%p) video(%p)",
                mAudioDecoder.get(), mVideoDecoder.get());
        return;
    }
    mPreviousSeekTimeUs = seekTimeUs;
    mSource->seekTo(seekTimeUs); // 直接调用Source对应接口
    ++mTimedTextGeneration;

    // everything's flushed, continue playback.
}

7 resetAsync

重置函数实现逻辑相对简单，直接重置下，代码如下：

void NuPlayer::resetAsync() {
    sp<Source> source;
    {
        Mutex::Autolock autoLock(mSourceLock);
        source = mSource;
    }

    if (source != NULL) {
        // During a reset, the data source might be unresponsive already, we need to
        // disconnect explicitly so that reads exit promptly.
        // We can't queue the disconnect request to the looper, as it might be
        // queued behind a stuck read and never gets processed.
        // Doing a disconnect outside the looper to allows the pending reads to exit
        // (either successfully or with error).
        source->disconnect();
    }

    (new AMessage(kWhatReset, this))->post();
}

消息循环中对于kWhatReset处理如下：

case kWhatReset:
{
    mResetting = true;

    mDeferredActions.push_back(
            new FlushDecoderAction(
                FLUSH_CMD_SHUTDOWN /* audio */,
                FLUSH_CMD_SHUTDOWN /* video */));

    mDeferredActions.push_back(new SimpleAction(&NuPlayer::performReset));

    processDeferredActions();
    break;
}

上面的SimpleAction是直接调用接口的，其实现如下：

void NuPlayer::performReset() {
    cancelPollDuration();

    ++mScanSourcesGeneration;
    mScanSourcesPending = false;
	// 销毁Render
    if (mRendererLooper != NULL) {
        if (mRenderer != NULL) {
            mRendererLooper->unregisterHandler(mRenderer->id());
        }
        mRendererLooper->stop();
        mRendererLooper.clear();
    }
    mRenderer.clear();
    ++mRendererGeneration;
	// 销毁Source
    if (mSource != NULL) {
        mSource->stop();

        Mutex::Autolock autoLock(mSourceLock);
        mSource.clear();
    }
	// 通知Reset完成
    if (mDriver != NULL) {
        sp<NuPlayerDriver> driver = mDriver.promote();
        if (driver != NULL) {
            driver->notifyResetComplete();
        }
    }

    mStarted = false;
    mPrepared = false;
    mResetting = false;
    mSourceStarted = false;
}

8 getCurrentPosition/getFileMeta

getCurrentPosition用于获取当前播放位置，直接通过Render的对应接口获取的。实现代码如下：

status_t NuPlayer::getCurrentPosition(int64_t *mediaUs) {
    sp<Renderer> renderer = mRenderer;
    if (renderer == NULL) {
        return NO_INIT;
    }

    return renderer->getCurrentPosition(mediaUs);
}

getFileMeta获取媒体的元数据信息，直接通过Source的对应接口获取。实现代码如下：

sp<MetaData> NuPlayer::getFileMeta() {
    return mSource->getFileFormatMeta();
}

9 总结和疑问

到这里，我们已经把NuPlayer主要的函数分析完了，但是问题依旧在。比如下面几个：

1. 不同格式的多媒体文件如何探测并解析的？音视频数据缓冲区在哪里？（Source）
2. 视频如何显示的？音频如何播放的？音视频同步在哪里？（Renderer）
3. 音频解码线程、视频解码线程在哪里？ （DecoderBase）

我想接下来几个主题就是解决这些疑问的。

当然总结下本文的内容。
主要参考AOSP 7.0的源码，结合代码分析了NuPlayer主要对外接口的实现，并简单总结了各部分的功能。