Android Camera原理之camera service与camera provider session会话与capture request轮转
上层调用CameraManager.openCamera的时候,会触发底层的一系列反应,之前我们分享过camera framework到camera service之间的调用,但是光看这一块还不够深入,接下来我们讨论一下camera service与camera provider之间在openCamera调用的时候做了什么事情。
status_t Camera3Device::initialize(sp<CameraProviderManager> manager, const String8& monitorTags) {
sp<ICameraDeviceSession> session;
status_t res = manager->openSession(mId.string(), this,
/*out*/ &session);
//......
res = manager->getCameraCharacteristics(mId.string(), &mDeviceInfo);
//......
std::shared_ptr<RequestMetadataQueue> queue;
auto requestQueueRet = session->getCaptureRequestMetadataQueue(
[&queue](const auto& descriptor) {
queue = std::make_shared<RequestMetadataQueue>(descriptor);
if (!queue->isValid() || queue->availableToWrite() <= 0) {
ALOGE("HAL returns empty request metadata fmq, not use it");
queue = nullptr;
// don't use the queue onwards.
}
});
//......
std::unique_ptr<ResultMetadataQueue>& resQueue = mResultMetadataQueue;
auto resultQueueRet = session->getCaptureResultMetadataQueue(
[&resQueue](const auto& descriptor) {
resQueue = std::make_unique<ResultMetadataQueue>(descriptor);
if (!resQueue->isValid() || resQueue->availableToWrite() <= 0) {
ALOGE("HAL returns empty result metadata fmq, not use it");
resQueue = nullptr;
// Don't use the resQueue onwards.
}
});
//......
mInterface = new HalInterface(session, queue);
std::string providerType;
mVendorTagId = manager->getProviderTagIdLocked(mId.string());
mTagMonitor.initialize(mVendorTagId);
if (!monitorTags.isEmpty()) {
mTagMonitor.parseTagsToMonitor(String8(monitorTags));
}
return initializeCommonLocked();
}
上面camera service中执行openCamera 中的核心步骤,可以看出,第一步执行的就是 manager->openSession(mId.string(), this, /out/ &session);
本文就是通过剖析openSession的执行流程来 还原 camera service 与camera provider 的执行过程。
为了让大家看得更加清楚,列出各个文件的位置:
CameraProviderManager : frameworks/av/services/camera/libcameraservice/common/CameraServiceProvider.cpp
CameraDevice : hardware/interfaces/camera/device/3.2/default/CameraDevice.cpp
CameraModule : hardware/interfaces/camera/common/1.0/default/CameraModule.cpp
CameraHAL : hardware/libhardware/modules/camera/3_0/CameraHAL.cpp
Camera : hardware/libhardware/modules/camera/3_0/Camera.cpp
CameraDeviceSession : hardware/interfaces/camera/device/3.2/default/CameraDeviceSession.cpp
中间涉及到一些指针函数的映射,如果看不明白,可以参考:《Android Camera原理之底层数据结构总结》,具体的调用流程就不说了,按照上面的时序图走,都能看明白的。
一些回调关系还是值得说下的。我们看下CameraProviderManager::openSession调用的地方:
status_t CameraProviderManager::openSession(const std::string &id,
const sp<hardware::camera::device::V3_2::ICameraDeviceCallback>& callback,
/*out*/
sp<hardware::camera::device::V3_2::ICameraDeviceSession> *session) {
std::lock_guard<std::mutex> lock(mInterfaceMutex);
auto deviceInfo = findDeviceInfoLocked(id,
/*minVersion*/ {3,0}, /*maxVersion*/ {4,0});
if (deviceInfo == nullptr) return NAME_NOT_FOUND;
auto *deviceInfo3 = static_cast<ProviderInfo::DeviceInfo3*>(deviceInfo);
Status status;
hardware::Return<void> ret;
ret = deviceInfo3->mInterface->open(callback, [&status, &session]
(Status s, const sp<device::V3_2::ICameraDeviceSession>& cameraSession) {
status = s;
if (status == Status::OK) {
*session = cameraSession;
}
});
if (!ret.isOk()) {
ALOGE("%s: Transaction error opening a session for camera device %s: %s",
__FUNCTION__, id.c_str(), ret.description().c_str());
return DEAD_OBJECT;
}
return mapToStatusT(status);
}
我们看下IPC调用的地方:
ret = deviceInfo3->mInterface->open(callback, [&status, &session]
(Status s, const sp<device::V3_2::ICameraDeviceSession>& cameraSession) {
status = s;
if (status == Status::OK) {
*session = cameraSession;
}
});
传入两个参数,一个是 const sp<hardware::camera::device::V3_2::ICameraDeviceCallback>& callback, 另一个是open_cb _hidl_cb
callback 提供了 camera HAL层到 camera service的回调。
open_cb _hidl_cb 是硬件抽象层提供了一种IPC间回传数据的方式。就本段代码而言,需要传回两个数据,一个status:表示当前openSession是否成功;另一个是session:表示camera session会话创建成功之后返回的session数据。
CameraDevice::open(...)函数
{
session = createSession(
device, info.static_camera_characteristics, callback);
if (session == nullptr) {
ALOGE("%s: camera device session allocation failed", __FUNCTION__);
mLock.unlock();
_hidl_cb(Status::INTERNAL_ERROR, nullptr);
return Void();
}
if (session->isInitFailed()) {
ALOGE("%s: camera device session init failed", __FUNCTION__);
session = nullptr;
mLock.unlock();
_hidl_cb(Status::INTERNAL_ERROR, nullptr);
return Void();
}
mSession = session;
IF_ALOGV() {
session->getInterface()->interfaceChain([](
::android::hardware::hidl_vec<::android::hardware::hidl_string> interfaceChain) {
ALOGV("Session interface chain:");
for (auto iface : interfaceChain) {
ALOGV(" %s", iface.c_str());
}
});
}
mLock.unlock();
}
_hidl_cb(status, session->getInterface());
最后执行的代码 _hidl_cb(status, session→getInterface()); 当前session创建成功之后,回调到 camera service 中。
const sp<hardware::camera::device::V3_2::ICameraDeviceCallback>& callback 设置到什么地方?这个问题非常重要的,camera 上层很依赖底层的回调,所以我们要搞清楚底层的回调被设置到什么地方,然后在搞清楚在合适的时机触发这些回调。
执行CameraDeviceSession构造函数的时候,传入了这个callback。
CameraDeviceSession::CameraDeviceSession(
camera3_device_t* device,
const camera_metadata_t* deviceInfo,
const sp<ICameraDeviceCallback>& callback) :
camera3_callback_ops({&sProcessCaptureResult, &sNotify}),
mDevice(device),
mDeviceVersion(device->common.version),
mIsAELockAvailable(false),
mDerivePostRawSensKey(false),
mNumPartialResults(1),
mResultBatcher(callback) {
mDeviceInfo = deviceInfo;
camera_metadata_entry partialResultsCount =
mDeviceInfo.find(ANDROID_REQUEST_PARTIAL_RESULT_COUNT);
if (partialResultsCount.count > 0) {
mNumPartialResults = partialResultsCount.data.i32[0];
}
mResultBatcher.setNumPartialResults(mNumPartialResults);
camera_metadata_entry aeLockAvailableEntry = mDeviceInfo.find(
ANDROID_CONTROL_AE_LOCK_AVAILABLE);
if (aeLockAvailableEntry.count > 0) {
mIsAELockAvailable = (aeLockAvailableEntry.data.u8[0] ==
ANDROID_CONTROL_AE_LOCK_AVAILABLE_TRUE);
}
// Determine whether we need to derive sensitivity boost values for older devices.
// If post-RAW sensitivity boost range is listed, so should post-raw sensitivity control
// be listed (as the default value 100)
if (mDeviceInfo.exists(ANDROID_CONTROL_POST_RAW_SENSITIVITY_BOOST_RANGE)) {
mDerivePostRawSensKey = true;
}
mInitFail = initialize();
}
CameraDeviceSession 中的 mResultBatcher 类构造中传入了这个 callback,现在由 CameraDeviceSession::ResultBatcher 来持有 callback了。看下ResultBatcher全局代码,在CameraDeviceSession.h中。
那以后底层要回调到上层必定要经过 CameraDeviceSession::ResultBatcher的mCallback来完成了。
class ResultBatcher {
public:
ResultBatcher(const sp<ICameraDeviceCallback>& callback);
void setNumPartialResults(uint32_t n);
void setBatchedStreams(const std::vector<int>& streamsToBatch);
void setResultMetadataQueue(std::shared_ptr<ResultMetadataQueue> q);
void registerBatch(uint32_t frameNumber, uint32_t batchSize);
void notify(NotifyMsg& msg);
void processCaptureResult(CaptureResult& result);
protected:
struct InflightBatch {
// Protect access to entire struct. Acquire this lock before read/write any data or
// calling any methods. processCaptureResult and notify will compete for this lock
// HIDL IPCs might be issued while the lock is held
Mutex mLock;
bool allDelivered() const;
uint32_t mFirstFrame;
uint32_t mLastFrame;
uint32_t mBatchSize;
bool mShutterDelivered = false;
std::vector<NotifyMsg> mShutterMsgs;
struct BufferBatch {
BufferBatch(uint32_t batchSize) {
mBuffers.reserve(batchSize);
}
bool mDelivered = false;
// This currently assumes every batched request will output to the batched stream
// and since HAL must always send buffers in order, no frameNumber tracking is
// needed
std::vector<StreamBuffer> mBuffers;
};
// Stream ID -> VideoBatch
std::unordered_map<int, BufferBatch> mBatchBufs;
struct MetadataBatch {
// (frameNumber, metadata)
std::vector<std::pair<uint32_t, CameraMetadata>> mMds;
};
// Partial result IDs that has been delivered to framework
uint32_t mNumPartialResults;
uint32_t mPartialResultProgress = 0;
// partialResult -> MetadataBatch
std::map<uint32_t, MetadataBatch> mResultMds;
// Set to true when batch is removed from mInflightBatches
// processCaptureResult and notify must check this flag after acquiring mLock to make
// sure this batch isn't removed while waiting for mLock
bool mRemoved = false;
};
// Get the batch index and pointer to InflightBatch (nullptrt if the frame is not batched)
// Caller must acquire the InflightBatch::mLock before accessing the InflightBatch
// It's possible that the InflightBatch is removed from mInflightBatches before the
// InflightBatch::mLock is acquired (most likely caused by an error notification), so
// caller must check InflightBatch::mRemoved flag after the lock is acquried.
// This method will hold ResultBatcher::mLock briefly
std::pair<int, std::shared_ptr<InflightBatch>> getBatch(uint32_t frameNumber);
static const int NOT_BATCHED = -1;
// move/push function avoids "hidl_handle& operator=(hidl_handle&)", which clones native
// handle
void moveStreamBuffer(StreamBuffer&& src, StreamBuffer& dst);
void pushStreamBuffer(StreamBuffer&& src, std::vector<StreamBuffer>& dst);
void sendBatchMetadataLocked(
std::shared_ptr<InflightBatch> batch, uint32_t lastPartialResultIdx);
// Check if the first batch in mInflightBatches is ready to be removed, and remove it if so
// This method will hold ResultBatcher::mLock briefly
void checkAndRemoveFirstBatch();
// The following sendXXXX methods must be called while the InflightBatch::mLock is locked
// HIDL IPC methods will be called during these methods.
void sendBatchShutterCbsLocked(std::shared_ptr<InflightBatch> batch);
// send buffers for all batched streams
void sendBatchBuffersLocked(std::shared_ptr<InflightBatch> batch);
// send buffers for specified streams
void sendBatchBuffersLocked(
std::shared_ptr<InflightBatch> batch, const std::vector<int>& streams);
// End of sendXXXX methods
// helper methods
void freeReleaseFences(hidl_vec<CaptureResult>&);
void notifySingleMsg(NotifyMsg& msg);
void processOneCaptureResult(CaptureResult& result);
void invokeProcessCaptureResultCallback(hidl_vec<CaptureResult> &results, bool tryWriteFmq);
// Protect access to mInflightBatches, mNumPartialResults and mStreamsToBatch
// processCaptureRequest, processCaptureResult, notify will compete for this lock
// Do NOT issue HIDL IPCs while holding this lock (except when HAL reports error)
mutable Mutex mLock;
std::deque<std::shared_ptr<InflightBatch>> mInflightBatches;
uint32_t mNumPartialResults;
std::vector<int> mStreamsToBatch;
const sp<ICameraDeviceCallback> mCallback;
std::shared_ptr<ResultMetadataQueue> mResultMetadataQueue;
// Protect against invokeProcessCaptureResultCallback()
Mutex mProcessCaptureResultLock;
} mResultBatcher;
到这里,openSession工作就完成了,这个主要是设置了上层的回调到底层,并且底层返回可用的camera session到上层来,实现底层和上层的交互通信。
1.获取的session是什么?为什么这个重要?
此session是 ICameraDeviceSession 对象,这个对象是指为了操作camera device,camera provider与 camera service之间建立的一个会话机制,可以保证camera service IPC调用到camera provider进程中的代码。
1.1.获取session当前请求原数组队列
auto requestQueueRet = session->getCaptureRequestMetadataQueue(
[&queue](const auto& descriptor) {
queue = std::make_shared<RequestMetadataQueue>(descriptor);
if (!queue->isValid() || queue->availableToWrite() <= 0) {
ALOGE("HAL returns empty request metadata fmq, not use it");
queue = nullptr;
// don't use the queue onwards.
}
});
到HAL层的 CameraDeviceSession.cpp中调用 getCaptureRequestMetadataQueue
Return<void> CameraDeviceSession::getCaptureRequestMetadataQueue(
ICameraDeviceSession::getCaptureRequestMetadataQueue_cb _hidl_cb) {
_hidl_cb(*mRequestMetadataQueue->getDesc());
return Void();
}
这个mRequestMetadataQueue 是在 CameraDeviceSession::initialize 执行的时候初始化的。
int32_t reqFMQSize = property_get_int32("ro.camera.req.fmq.size", /*default*/-1);
if (reqFMQSize < 0) {
reqFMQSize = CAMERA_REQUEST_METADATA_QUEUE_SIZE;
} else {
ALOGV("%s: request FMQ size overridden to %d", __FUNCTION__, reqFMQSize);
}
mRequestMetadataQueue = std::make_unique<RequestMetadataQueue>(
static_cast<size_t>(reqFMQSize),
false /* non blocking */);
if (!mRequestMetadataQueue->isValid()) {
ALOGE("%s: invalid request fmq", __FUNCTION__);
return true;
}
首先读取 ro.camera.req.fmq.size 属性,如果没有找到,则直接赋给一个 1M 大小的 请求原数组队列。这个队列很重要,后续的camera capture请求都是通过这个队列处理的。
1.2.获取session 当前结果原数组队列
这个和 请求原数组队列相似,不过结果原数组中保留的是 camera capture的结果数据。大家可以看下源码,这儿就不贴源码了
2.开始运转capture request线程
camera service 与 camera provider 建立session 会话之后,开始运转capture request请求线程,之后发送的capture request都会到这个线程中执行,这就是熟知的capture request轮转。
在Camera3Device::initializeCommonLocked 中执行了 capture request轮转。
/** Start up request queue thread */
mRequestThread = new RequestThread(this, mStatusTracker, mInterface, sessionParamKeys);
res = mRequestThread->run(String8::format("C3Dev-%s-ReqQueue", mId.string()).string());
if (res != OK) {
SET_ERR_L("Unable to start request queue thread: %s (%d)",
strerror(-res), res);
mInterface->close();
mRequestThread.clear();
return res;
}
开始启动当前的capture request 队列,放在 RequestThread线程中执行,这个线程会一直执行,当有新的capture request发过来,会将capture request放进当前会话的请求队列中,继续执行。这个轮转很重要,这是camera能正常工作的前提。
轮转的主要工作在Camera3Device::RequestThread::threadLoop 函数中完成,这是native中定义的一个 线程执行函数块。
bool Camera3Device::RequestThread::threadLoop() {
ATRACE_CALL();
status_t res;
// Handle paused state.
if (waitIfPaused()) {
return true;
}
// Wait for the next batch of requests.
waitForNextRequestBatch();
if (mNextRequests.size() == 0) {
return true;
}
//......
// Prepare a batch of HAL requests and output buffers.
res = prepareHalRequests();
if (res == TIMED_OUT) {
// Not a fatal error if getting output buffers time out.
cleanUpFailedRequests(/*sendRequestError*/ true);
// Check if any stream is abandoned.
checkAndStopRepeatingRequest();
return true;
} else if (res != OK) {
cleanUpFailedRequests(/*sendRequestError*/ false);
return false;
}
// Inform waitUntilRequestProcessed thread of a new request ID
{
Mutex::Autolock al(mLatestRequestMutex);
mLatestRequestId = latestRequestId;
mLatestRequestSignal.signal();
}
//......
bool submitRequestSuccess = false;
nsecs_t tRequestStart = systemTime(SYSTEM_TIME_MONOTONIC);
if (mInterface->supportBatchRequest()) {
submitRequestSuccess = sendRequestsBatch();
} else {
submitRequestSuccess = sendRequestsOneByOne();
}
//......
return submitRequestSuccess;
}
waitForNextRequestBatch() 不断去轮训底层是否有InputBuffer数据,获取的inputBuffer数据放在request中,这些数据会在之后被消费。
这儿先列个调用过程:对照着代码看一下,camera的producer与consumer模型之后还会详细讲解的。
这儿也为了大家快速进入代码,也列出来代码的对应位置:
Camera3Device::RequestThread : frameworks/av/services/camera/libcameraservice/device3/Camera3Device.cpp 中有内部类 RequestThread,这是一个线程类。
Camera3Stream : frameworks/av/services/camera/libcameraservice/device3/Camera3Stream.cpp
Camera3InputStream : frameworks/av/services/camera/libcameraservice/device3/Camera3InputStream.cpp
Camera3IOStreamBase : frameworks/av/services/camera/libcameraservice/device3/Camera3IOStreamBase.cpp
BufferItemConsumer : frameworks/native/libs/gui/BufferItemConsumer.cpp
ConsumerBase : frameworks/native/libs/gui/ConsumerBase.cpp
BnGraphicBufferConsumer : frameworks/native/libs/gui/IGraphicBufferConsumer.cpp
上层发过来来的capture request,手下到底层申请Consumer buffer,这个buffer数据存储在capture request缓存中,后期这些buffer数据会被复用,不断地生产数据,也不断地被消费。
capture request开启之后,camera hal层也会受到capture request批处理请求,让camera hal做好准备,开始和camera driver层交互。hal层的请求下一章讲解。
作者:木子一秋
链接:https://www.jianshu.com/p/c1f75c48ed7c
来源:简书
简书著作权归作者所有,任何形式的转载都请联系作者获得授权并注明出处。