[Android] Graphic Buffer 的申请

前言：

MediaCodec 支持 texture mode，即MediaCodec解码video完毕后把 yuv 数据填入 GPU 共享出来的 graphic buffer 里面，app 会把 video 的 yuv数据和 ui 的数据通过通过软件渲染组件(opengl等)发送给GPU 进行一并渲染。这样做的效率较低，但是稳定性较好。且性能取决于GPU的性能。

另外一种是 surface mode，就是常用的模式，这种模式下 app 不再获取 yuv 数据，而是只负责输送 es 数据并决定 render 还是 discard 解码后的数据，且是根据 bufferid 来判断的，根本没有机会接触解码后的 yuv 数据，所有的一切都在平台完成。这种办法效率搞且功耗也很低，因为不需要GPU的参与。大部分情况下都是这种模式。

本文简单理清一下 Graphic Buffer 的框架，以对 texture mode 中的 Graphic Buffer 申请流程有一个大致的概念。

Graphic Buffer 的申请

始于 ANativeWindow 类

frameworks/native/libs/nativewindow/include/system/window.h


struct ANativeWindow
{
#ifdef __cplusplusANativeWindow(): flags(0), minSwapInterval(0), maxSwapInterval(0), xdpi(0), ydpi(0){common.magic = ANDROID_NATIVE_WINDOW_MAGIC;common.version = sizeof(ANativeWindow);memset(common.reserved, 0, sizeof(common.reserved));}/* Implement the methods that sp<ANativeWindow> expects so that itcan be used to automatically refcount ANativeWindow's. */void incStrong(const void* /*id*/) const {common.incRef(const_cast<android_native_base_t*>(&common));}void decStrong(const void* /*id*/) const {common.decRef(const_cast<android_native_base_t*>(&common));}
#endifstruct android_native_base_t common;/* flags describing some attributes of this surface or its updater */const uint32_t flags;/* min swap interval supported by this updated */const int   minSwapInterval;/* max swap interval supported by this updated */const int   maxSwapInterval;/* horizontal and vertical resolution in DPI */const float xdpi;const float ydpi;/* Some storage reserved for the OEM's driver. */intptr_t    oem[4];/** Set the swap interval for this surface.** Returns 0 on success or -errno on error.*/int     (*setSwapInterval)(struct ANativeWindow* window,int interval);/** Hook called by EGL to acquire a buffer. After this call, the buffer* is not locked, so its content cannot be modified. This call may block if* no buffers are available.** The window holds a reference to the buffer between dequeueBuffer and* either queueBuffer or cancelBuffer, so clients only need their own* reference if they might use the buffer after queueing or canceling it.* Holding a reference to a buffer after queueing or canceling it is only* allowed if a specific buffer count has been set.** Returns 0 on success or -errno on error.** XXX: This function is deprecated.  It will continue to work for some* time for binary compatibility, but the new dequeueBuffer function that* outputs a fence file descriptor should be used in its place.*/int     (*dequeueBuffer_DEPRECATED)(struct ANativeWindow* window,struct ANativeWindowBuffer** buffer);/** hook called by EGL to lock a buffer. This MUST be called before modifying* the content of a buffer. The buffer must have been acquired with* dequeueBuffer first.** Returns 0 on success or -errno on error.** XXX: This function is deprecated.  It will continue to work for some* time for binary compatibility, but it is essentially a no-op, and calls* to it should be removed.*/int     (*lockBuffer_DEPRECATED)(struct ANativeWindow* window,struct ANativeWindowBuffer* buffer);/** Hook called by EGL when modifications to the render buffer are done.* This unlocks and post the buffer.** The window holds a reference to the buffer between dequeueBuffer and* either queueBuffer or cancelBuffer, so clients only need their own* reference if they might use the buffer after queueing or canceling it.* Holding a reference to a buffer after queueing or canceling it is only* allowed if a specific buffer count has been set.** Buffers MUST be queued in the same order than they were dequeued.** Returns 0 on success or -errno on error.** XXX: This function is deprecated.  It will continue to work for some* time for binary compatibility, but the new queueBuffer function that* takes a fence file descriptor should be used in its place (pass a value* of -1 for the fence file descriptor if there is no valid one to pass).*/int     (*queueBuffer_DEPRECATED)(struct ANativeWindow* window,struct ANativeWindowBuffer* buffer);/** hook used to retrieve information about the native window.** Returns 0 on success or -errno on error.*/int     (*query)(const struct ANativeWindow* window,int what, int* value);/** hook used to perform various operations on the surface.* (*perform)() is a generic mechanism to add functionality to* ANativeWindow while keeping backward binary compatibility.** DO NOT CALL THIS HOOK DIRECTLY.  Instead, use the helper functions* defined below.** (*perform)() returns -ENOENT if the 'what' parameter is not supported* by the surface's implementation.** See above for a list of valid operations, such as* NATIVE_WINDOW_SET_USAGE or NATIVE_WINDOW_CONNECT*/int     (*perform)(struct ANativeWindow* window,int operation, ... );/** Hook used to cancel a buffer that has been dequeued.* No synchronization is performed between dequeue() and cancel(), so* either external synchronization is needed, or these functions must be* called from the same thread.** The window holds a reference to the buffer between dequeueBuffer and* either queueBuffer or cancelBuffer, so clients only need their own* reference if they might use the buffer after queueing or canceling it.* Holding a reference to a buffer after queueing or canceling it is only* allowed if a specific buffer count has been set.** XXX: This function is deprecated.  It will continue to work for some* time for binary compatibility, but the new cancelBuffer function that* takes a fence file descriptor should be used in its place (pass a value* of -1 for the fence file descriptor if there is no valid one to pass).*/int     (*cancelBuffer_DEPRECATED)(struct ANativeWindow* window,struct ANativeWindowBuffer* buffer);/** Hook called by EGL to acquire a buffer. This call may block if no* buffers are available.** The window holds a reference to the buffer between dequeueBuffer and* either queueBuffer or cancelBuffer, so clients only need their own* reference if they might use the buffer after queueing or canceling it.* Holding a reference to a buffer after queueing or canceling it is only* allowed if a specific buffer count has been set.** The libsync fence file descriptor returned in the int pointed to by the* fenceFd argument will refer to the fence that must signal before the* dequeued buffer may be written to.  A value of -1 indicates that the* caller may access the buffer immediately without waiting on a fence.  If* a valid file descriptor is returned (i.e. any value except -1) then the* caller is responsible for closing the file descriptor.** Returns 0 on success or -errno on error.*/int     (*dequeueBuffer)(struct ANativeWindow* window,struct ANativeWindowBuffer** buffer, int* fenceFd);/** Hook called by EGL when modifications to the render buffer are done.* This unlocks and post the buffer.** The window holds a reference to the buffer between dequeueBuffer and* either queueBuffer or cancelBuffer, so clients only need their own* reference if they might use the buffer after queueing or canceling it.* Holding a reference to a buffer after queueing or canceling it is only* allowed if a specific buffer count has been set.** The fenceFd argument specifies a libsync fence file descriptor for a* fence that must signal before the buffer can be accessed.  If the buffer* can be accessed immediately then a value of -1 should be used.  The* caller must not use the file descriptor after it is passed to* queueBuffer, and the ANativeWindow implementation is responsible for* closing it.** Returns 0 on success or -errno on error.*/int     (*queueBuffer)(struct ANativeWindow* window,struct ANativeWindowBuffer* buffer, int fenceFd);/** Hook used to cancel a buffer that has been dequeued.* No synchronization is performed between dequeue() and cancel(), so* either external synchronization is needed, or these functions must be* called from the same thread.** The window holds a reference to the buffer between dequeueBuffer and* either queueBuffer or cancelBuffer, so clients only need their own* reference if they might use the buffer after queueing or canceling it.* Holding a reference to a buffer after queueing or canceling it is only* allowed if a specific buffer count has been set.** The fenceFd argument specifies a libsync fence file decsriptor for a* fence that must signal before the buffer can be accessed.  If the buffer* can be accessed immediately then a value of -1 should be used.** Note that if the client has not waited on the fence that was returned* from dequeueBuffer, that same fence should be passed to cancelBuffer to* ensure that future uses of the buffer are preceded by a wait on that* fence.  The caller must not use the file descriptor after it is passed* to cancelBuffer, and the ANativeWindow implementation is responsible for* closing it.** Returns 0 on success or -errno on error.*/int     (*cancelBuffer)(struct ANativeWindow* window,struct ANativeWindowBuffer* buffer, int fenceFd);
};

上述代码中有很多函数指针，比如

    int     (*dequeueBuffer)(struct ANativeWindow* window,struct ANativeWindowBuffer** buffer, int* fenceFd);

这些指针的具体实现在其子类中有实现和设置，这里是一种 C 的 hook 写法，而不是 c++ 的虚函数写法，可能出于兼容性考虑。

\frameworks\native\libs\nativewindow\ANativeWindow.cpp 里面并不是 ANativeWindow的实现，而是对其封装，比如：

int ANativeWindow_dequeueBuffer(ANativeWindow* window, ANativeWindowBuffer** buffer, int* fenceFd) {return window->dequeueBuffer(window, buffer, fenceFd);
}

可以看到，这里调用了 ANativeWindow的dequeueBuffer函数，而这个函数在 ANativeWindow 结构里是一个函数指针，而这个函数指针又指向哪里呢？

Hook的实现者Surface

\frameworks\native\libs\gui\include\gui\Surface.h

\frameworks\native\libs\gui\Surface.cpp

class Surface: public ANativeObjectBase<ANativeWindow, Surface, RefBase>
{}

Surface 是 ANativeWindow的子类，但是考虑到 ANativeWindow通过函数指针的方式实现多态，所以Surface里面一定有设置Hook的地方，那就是在构造函数里。

Surface::Surface(const sp<IGraphicBufferProducer>& bufferProducer, bool controlledByApp): mGraphicBufferProducer(bufferProducer),mCrop(Rect::EMPTY_RECT),mBufferAge(0),mGenerationNumber(0),mSharedBufferMode(false),mAutoRefresh(false),mSharedBufferSlot(BufferItem::INVALID_BUFFER_SLOT),mSharedBufferHasBeenQueued(false),mQueriedSupportedTimestamps(false),mFrameTimestampsSupportsPresent(false),mEnableFrameTimestamps(false),mFrameEventHistory(std::make_unique<ProducerFrameEventHistory>()) {// Initialize the ANativeWindow function pointers.ANativeWindow::setSwapInterval  = hook_setSwapInterval;ANativeWindow::dequeueBuffer    = hook_dequeueBuffer;ANativeWindow::cancelBuffer     = hook_cancelBuffer;ANativeWindow::queueBuffer      = hook_queueBuffer;ANativeWindow::query            = hook_query;ANativeWindow::perform          = hook_perform;ANativeWindow::dequeueBuffer_DEPRECATED = hook_dequeueBuffer_DEPRECATED;ANativeWindow::cancelBuffer_DEPRECATED  = hook_cancelBuffer_DEPRECATED;ANativeWindow::lockBuffer_DEPRECATED    = hook_lockBuffer_DEPRECATED;ANativeWindow::queueBuffer_DEPRECATED   = hook_queueBuffer_DEPRECATED;

至此，所有调用 ANativeWindow_xxx 的入口都会被导入 Surface 里的相应接口中，进入 hook_xxx。

int Surface::hook_dequeueBuffer(ANativeWindow* window,ANativeWindowBuffer** buffer, int* fenceFd) {Surface* c = getSelf(window);return c->dequeueBuffer(buffer, fenceFd);
}

进而在进入 Surface 的 xxx 函数。从 xxx 函数开始，和 graphic service 打交道就正式开始了。

int Surface::dequeueBuffer(android_native_buffer_t** buffer, int* fenceFd) {ATRACE_CALL();status_t result = mGraphicBufferProducer->dequeueBuffer(&buf, &fence, reqWidth, reqHeight,reqFormat, reqUsage, &mBufferAge,enableFrameTimestamps ? &frameTimestamps: nullptr);}

Graphic Buffer 服务

Surface 的所有和 Graphic Buffer 相关的接口都会使用到成员 mGraphicBufferProducer

sp<IGraphicBufferProducer> mGraphicBufferProducer;

这个成员的类型一看就是一个 binder Interface，继续看 IGraphicBufferProducer。

\frameworks\native\libs\gui\include\gui\IGraphicBufferProducer.h

class IGraphicBufferProducer : public IInterface
{
public:using HGraphicBufferProducerV1_0 =::android::hardware::graphics::bufferqueue::V1_0::IGraphicBufferProducer;using HGraphicBufferProducerV2_0 =::android::hardware::graphics::bufferqueue::V2_0::IGraphicBufferProducer;。。。virtual status_t dequeueBuffer(int* slot, sp<Fence>* fence, uint32_t w, uint32_t h,PixelFormat format, uint64_t usage, uint64_t* outBufferAge,FrameEventHistoryDelta* outTimestamps) = 0;。。。}

至此，一个binder 模型已经形成。

Surface::mGraphicBufferProducer as Bp <--IGraphicBufferProducer BINDER--> ? as Bn

那么这里的Bn是什么呢？只需要在源码里搜一下哪些类继承IGraphicBufferProducer 即可。其中一个常用的类就是 BufferQueueProducer。

\frameworks\native\libs\gui\include\gui\BufferQueueProducer.h

class BufferQueueProducer : public BnGraphicBufferProducer,private IBinder::DeathRecipient {。。。virtual status_t queueBuffer(int slot,const QueueBufferInput& input, QueueBufferOutput* output);
。。。}

继承自BnGraphicBufferProducer，Bn开头，一看就知道是server端。

status_t BufferQueueProducer::dequeueBuffer(int* outSlot, sp<android::Fence>* outFence,uint32_t width, uint32_t height, PixelFormat format,uint64_t usage, uint64_t* outBufferAge,FrameEventHistoryDelta* outTimestamps) {sp<GraphicBuffer> graphicBuffer = new GraphicBuffer(width, height, format, BQ_LAYER_COUNT, usage,{mConsumerName.string(), mConsumerName.size()});status_t error = graphicBuffer->initCheck();}

在上面的代码里有 new GraphicBuffer 和 initCheck() 这两个动作，这里就是在创建管理对象并进行初始化检查。

内存管理

\frameworks\native\libs\ui\include\ui\GraphicBuffer.h

class GraphicBuffer: public ANativeObjectBase<ANativeWindowBuffer, GraphicBuffer, RefBase>,public Flattenable<GraphicBuffer>
{GraphicBuffer(uint32_t inWidth, uint32_t inHeight, PixelFormat inFormat,uint32_t inUsage, std::string requestorName = "<Unknown>");}

\frameworks\native\libs\ui\GraphicBuffer.cpp

GraphicBuffer::GraphicBuffer(uint32_t inWidth, uint32_t inHeight, PixelFormat inFormat,uint32_t inLayerCount, uint64_t inUsage, std::string requestorName): GraphicBuffer() {mInitCheck = initWithSize(inWidth, inHeight, inFormat, inLayerCount, inUsage,std::move(requestorName));
}

status_t GraphicBuffer::initWithSize(uint32_t inWidth, uint32_t inHeight,PixelFormat inFormat, uint32_t inLayerCount, uint64_t inUsage,std::string requestorName)
{GraphicBufferAllocator& allocator = GraphicBufferAllocator::get();uint32_t outStride = 0;status_t err = allocator.allocate(inWidth, inHeight, inFormat, inLayerCount,inUsage, &handle, &outStride, mId,std::move(requestorName));if (err == NO_ERROR) {mBufferMapper.getTransportSize(handle, &mTransportNumFds, &mTransportNumInts);width = static_cast<int>(inWidth);height = static_cast<int>(inHeight);format = inFormat;layerCount = inLayerCount;usage = inUsage;usage_deprecated = int(usage);stride = static_cast<int>(outStride);}return err;
}

上面创建 Graphic Buffer 的过程中会有一个 initWithSize ，这是初始化的过程，在初始化过程里又又 allocator.allocate，可见是一个内存分配的过程。

内存分配 GraphicBufferAllocator

\frameworks\native\libs\ui\include\ui\GraphicBufferAllocator.h

\frameworks\native\libs\ui\GraphicBufferAllocator.cpp

class GraphicBufferAllocator : public Singleton<GraphicBufferAllocator>
{}

allocate 方法如下

status_t GraphicBufferAllocator::allocate(uint32_t width, uint32_t height,PixelFormat format, uint32_t layerCount, uint64_t usage,buffer_handle_t* handle, uint32_t* stride,uint64_t /*graphicBufferId*/, std::string requestorName)
{。。。status_t error =mAllocator->allocate(width, height, format, layerCount, usage, 1, stride, handle);
。。。}

mAllocator 是什么？

sp<hardware::graphics::allocator::V2_0::IAllocator> mAllocator;

又是一个 Interface ，这次是 hardware 的，因此其实现是由硬件完成的。具体细节参考 /hardware/interfaces/graphics/allocator/2.0/