ais_server 学习笔记
- 一前序
- 二、ais init
- 1、时序图如下
- 2. 初始化一共分为以下几个重要步骤:
- 2.1.1、在ais_server中启动main函数,然后创建AisEngine,接着初始化AisEngine
- 2.1.2、解析/var/camera_config.xml 文件,获取相关配置参数。这个camera_config.xml 后面会细聊
- 2.1.3、启动EventHandler线程,会启动四个EventHandler,来承接外面调回来的通知,比如FRAME_DONE、FATAL_ERROR等等, 至于为什么是四个EventHandler,我猜测可能是一个CSIPHY 对应一个 Thread handler吧。
- 2.1.4、"加载" sensor so、CSIPHY 、IFE
- 2.1.5、接着会根据前面解析的camera_config.xml 里面的配置来给input、ife、csiphy 里面的参数进行配置,在csiphy里面会进行mipi协议的一些配置, 配置好mipiconfig 后就会去 start mipi
- 2.1.6、启动平台图像处理等功能
- 三、amera_client -> qcarcam_initialize
- 四、camera_client -> qcarcam_open
- 五、camera_client -> qcarcam_s_param
- 六、camera_client -> qcarcam_start
- 七、camera_client -> qcarcam_get_frame
一前序
ais server 的主要功能,简单理解就是高通平台用于使能相机模块,处理camera 数据,并在camera client连接的时候将camera数据顺利给到client。
我们还是从ais的init,然后client调用接口,init、open、s_param、start、get_frame、release_frame、stop、close、deinit聊起, 每个调用都会有对应的时序图。
二、ais init
1、时序图如下
2. 初始化一共分为以下几个重要步骤:
ais_servermainAisEngineAisEngine_9">2.1.1、在ais_server中启动main函数,然后创建AisEngine,接着初始化AisEngine
camera_configxml_camera_configxml__10">2.1.2、解析/var/camera_config.xml 文件,获取相关配置参数。这个camera_config.xml 后面会细聊
<?xml version="1.0" encoding="UTF-8"?><!--Copyright (c) 2020 Qualcomm Technologies, Inc.All Rights Reserved.Confidential and Proprietary - Qualcomm Technologies, Inc.--><!-- THIS IS A SAMPLE CAMERA CONFIG XML TO ILLUSTRATE HOW IT CAN BE USED --><!--SA8155 SAMPLE
=========================================================================== */
/* CSI lanes IFEs I2C Port Sensors* max96716 2 4 2 /dev/cci1 2 x 96717 YUV SENSORS
*/
--><CameraConfig version="0x403"><board name="SA8155_ADP" boardType="CAMERA_HW_BOARD_ADPAIR_V2_PL195" multiSocEnable="0"><i2cDevs><i2cDev name="cci0"><properties i2ctype = "CAMERA_I2C_TYPE_CCI" device_id = "0" port_id = "0"/><sda_pin gpio = "17" function = "1"/><scl_pin gpio = "18" function = "1"/></i2cDev><i2cDev name="cci1"><properties i2ctype = "CAMERA_I2C_TYPE_CCI" device_id = "0" port_id = "1"/><sda_pin gpio = "19" function = "1"/><scl_pin gpio = "20" function = "1"/></i2cDev><i2cDev name="cci2"><properties i2ctype = "CAMERA_I2C_TYPE_CCI" device_id = "1" port_id = "0"/><sda_pin gpio = "31" function = "1"/><scl_pin gpio = "32" function = "1"/></i2cDev></i2cDevs><inputDevs><inputDev devId = "0"><properties><!-- 2 AR0231 Bayer Sensor --><config subdevId = "0" type="1" numSensors="1" pmasterSocId="0" socMap="1" opMode="0" /><sensor link="0" type="1" /></properties><driverInfo devCategory = "CAMERA_DEVICE_CATEGORY_SENSOR" libName="ais_nio_max96716f" openFcn="CameraSensorDevice_Open_max96716"/><i2cPort soc_id = "0" i2ctype = "CAMERA_I2C_TYPE_CCI" device_id = "0" port_id = "1" /><csiInfo csiId = "0" numLanes = "2" laneAssign = "0x10" numIfeMap="1" ifeMap="0x0" /><gpio id = "CAMERA_GPIO_RESET" num = "22" config = "0x31"/><intr id = "CAMERA_GPIO_INTR" pin_id = "106" intr_type = "CAMERA_GPIO_INTR_TLMM" trigger = "CAMERA_GPIO_TRIGGER_FALLING" gpio_cfg = "0x30"/></inputDev><inputDev devId = "1"><properties><!-- 2 AR0231 Bayer Sensor --><config subdevId = "1" type="1" numSensors="4" pmasterSocId="0" socMap="1" opMode="0" /><sensor link="0" type="1" /><sensor link="1" type="1" /><sensor link="2" type="1" /><sensor link="3" type="1" /></properties><driverInfo devCategory = "CAMERA_DEVICE_CATEGORY_SENSOR" libName="ais_nio_max96722" openFcn="CameraSensorDevice_Open_max96722"/><i2cPort soc_id = "0" i2ctype = "CAMERA_I2C_TYPE_CCI" device_id = "1" port_id = "0" /><csiInfo csiId = "3" numLanes = "4" laneAssign = "0x3210" ifeMap="0x32" numIfeMap="2"/><gpio id = "CAMERA_GPIO_RESET" num = "25" config = "0x31"/><intr id = "CAMERA_GPIO_INTR" pin_id = "13" intr_type = "CAMERA_GPIO_INTR_TLMM" trigger = "CAMERA_GPIO_TRIGGER_FALLING" gpio_cfg = "0x30"/><intr id = "CAMERA_GPIO_LOCK" pin_id = "15" intr_type = "CAMERA_GPIO_INTR_TLMM" trigger = "CAMERA_GPIO_TRIGGER_FALLING" gpio_cfg = "0x30"/><intr id = "CAMERA_GPIO_PASS" pin_id = "90" intr_type = "CAMERA_GPIO_INTR_TLMM" trigger = "CAMERA_GPIO_TRIGGER_EDGE" gpio_cfg = "0x10"/><intr id = "CAMERA_GPIO_CUSTOM1" pin_id = "149" intr_type = "CAMERA_GPIO_INTR_TLMM" trigger = "CAMERA_GPIO_TRIGGER_FALLING" gpio_cfg = "0x30"/><intr id = "CAMERA_GPIO_CUSTOM2" pin_id = "150" intr_type = "CAMERA_GPIO_INTR_TLMM" trigger = "CAMERA_GPIO_TRIGGER_FALLING" gpio_cfg = "0x30"/><intr id = "CAMERA_GPIO_CUSTOM_REG1" pin_id = "151" intr_type = "CAMERA_GPIO_INTR_TLMM" trigger = "CAMERA_GPIO_TRIGGER_FALLING" gpio_cfg = "0x30"/><intr id = "CAMERA_GPIO_CUSTOM_REG2" pin_id = "152" intr_type = "CAMERA_GPIO_INTR_TLMM" trigger = "CAMERA_GPIO_TRIGGER_FALLING" gpio_cfg = "0x30"/></inputDev><inputDev devId = "2"><properties><!-- 2 AR0231 Bayer Sensor --><config subdevId = "0" type="1" numSensors="3" pmasterSocId="0" socMap="1" opMode="0" /><sensor link="0" type="1" /><sensor link="1" type="1" /><sensor link="2" type="1" /></properties><driverInfo devCategory = "CAMERA_DEVICE_CATEGORY_SENSOR" libName="ais_nio_svc_fy" openFcn="CameraSensorDevice_Open_max96712"/><i2cPort soc_id = "0" i2ctype = "CAMERA_I2C_TYPE_CCI" device_id = "0" port_id = "0" /><csiInfo csiId = "2" numLanes = "4" laneAssign = "0x3210" ifeMap="0x1" numIfeMap="1"/></inputDev></inputDevs><engineSettings numBufMax="12" powerManagementPolicy="CAMERA_PM_POLICY_LPM_EVENT_ALL" latencyMeasurementMode="CAMERA_LM_MODE_DISABLE"recoveryRestartDelay="33" recoveryTimeoutAfterUsrCtxtRestart="500" recoveryRetryDelay="300" recoveryMaxNumAttempts="1"><inputFatalError type="MATCH_INPUT_DEVICE" severity="0"/><csidFatalError type="MATCH_IFE_DEVICE" severity="0"/><ifeOutputError type="MATCH_IFE_DEVICE" severity="0"/></engineSettings></board><inputMapping><inputMap qcarcamId = "1" opMode = "QCARCAM_OPMODE_RAW_DUMP"><inputSrc devId = "0" srcId = "0"/></inputMap><inputMap qcarcamId = "2" opMode = "QCARCAM_OPMODE_RAW_DUMP"><inputSrc devId = "2" srcId = "2"/></inputMap><inputMap qcarcamId = "3" opMode = "QCARCAM_OPMODE_RAW_DUMP"><inputSrc devId = "2" srcId = "0"/></inputMap><inputMap qcarcamId = "4" opMode = "QCARCAM_OPMODE_RAW_DUMP"><inputSrc devId = "2" srcId = "1"/></inputMap><inputMap qcarcamId = "5" opMode = "QCARCAM_OPMODE_RAW_DUMP"><inputSrc devId = "1" srcId = "0"/></inputMap><inputMap qcarcamId = "6" opMode = "QCARCAM_OPMODE_RAW_DUMP"><inputSrc devId = "1" srcId = "1"/></inputMap><inputMap qcarcamId = "7" opMode = "QCARCAM_OPMODE_RAW_DUMP"><inputSrc devId = "1" srcId = "2"/></inputMap><inputMap qcarcamId = "8" opMode = "QCARCAM_OPMODE_RAW_DUMP"><inputSrc devId = "1" srcId = "3"/></inputMap></inputMapping>
</CameraConfig>
2.1.3、启动EventHandler线程,会启动四个EventHandler,来承接外面调回来的通知,比如FRAME_DONE、FATAL_ERROR等等, 至于为什么是四个EventHandler,我猜测可能是一个CSIPHY 对应一个 Thread handler吧。
2.1.4、“加载” sensor so、CSIPHY 、IFE
分别会调用RegisterDeviceFromLib()、RegisterDevice()将libais_nio_*.so 和 IFE 、CSIPHY open
static CameraDeviceManagerRegisteredType sStaticDevicesSA8155[] =
{
#ifndef AIS_WITH_HAL_CAMERA{{CAMERADEVICEID_IFE_0, CAMERA_DEVICE_CATEGORY_IFE}, ife_device_open},{{CAMERADEVICEID_IFELITE_1, CAMERA_DEVICE_CATEGORY_IFE}, ifelite_device_open},
#endif{{CAMERADEVICEID_IFE_1, CAMERA_DEVICE_CATEGORY_IFE}, ife_device_open},{{CAMERADEVICEID_IFELITE_0, CAMERA_DEVICE_CATEGORY_IFE}, ifelite_device_open},{{CAMERADEVICEID_CSIPHY_0, CAMERA_DEVICE_CATEGORY_CSIPHY}, csiphy_device_open},{{CAMERADEVICEID_CSIPHY_1, CAMERA_DEVICE_CATEGORY_CSIPHY}, csiphy_device_open},{{CAMERADEVICEID_CSIPHY_2, CAMERA_DEVICE_CATEGORY_CSIPHY}, csiphy_device_open},{{CAMERADEVICEID_CSIPHY_3, CAMERA_DEVICE_CATEGORY_CSIPHY}, csiphy_device_open},
};
这里继续调用到RegisterDevice,将
libais_nio_max96722.so -> CameraSensorDevice_Open_max96722
libais_nio_svc_fy.so -> CameraSensorDevice_Open_max96712
libais_nio_max96716f.so -> CameraSensorDevice_Open_max96716
保存在registeredDevices中
CameraSensorDevice_Open_max96722 -> max96722_sensor_open_lib()
CameraSensorDevice_Open_max96712 -> max96712_sensor_open_lib()
CameraSensorDevice_Open_max96716 -> max96716_sensor_open_lib()
创建AisInputConfigurer、AisIFEConfigurer、AisCSIConfigurer,然后init他们三个,在init中分别调用DeviceOpen方法,最终会调用到CameraSensorDevice_Open_max967*、 ife_device_open、csiphy_device_open。
ife_device_open 和 ifelite_device_open 分别会被调用两次,每次都会创建一个VFE/CSID(IFE持有VFE和CSID) 对象,ife_device_open 和 ifelite_device_open的取别主要在于RDI的个数,前者会有四个RDI通道,后者会有三个RDI通道。
CSID和IFE分别在创建的时候对应一个线程,分别open四次就会有四个线程,这个四个线程用于监听中断和其他消息通知,后面会用到
camera_configxml_inputifecsiphy_csiphymipi_mipiconfig__start_mipi_173">2.1.5、接着会根据前面解析的camera_config.xml 里面的配置来给input、ife、csiphy 里面的参数进行配置,在csiphy里面会进行mipi协议的一些配置, 配置好mipiconfig 后就会去 start mipi
2.1.6、启动平台图像处理等功能
当ais完全启动后,后续将从client端使用的角度来分析,ais_server对应的每一步处理
三、amera_client -> qcarcam_initialize
Camera client和ais_server是通过libais_client进行通信的,如果是client是在qnx 则通过socket,如果client是在
android侧,则通过hab(也是基于共享内存封装的类socket机制)
Camera client 开始使用的时候首先要调用接口qcarcam_initialize, 首先会调用到ais_client中,会为这个camera分配一个s_ais_client变量,然后再建立该camera client和ais_server(ais_client 是camera client和ais_server之间的桥梁)之间的心跳机制
这里注意sgs_ais_client数组最大值是64,意味着最多同时可以连接64个client,但是这里有个奇怪的点,就是我们可以open的camera个数其实和rdi有关系?
–> 所以答案是:
每个camera client qcarcam_initialize之后都会有个心跳线程
camera_client__qcarcam_open_190">四、camera_client -> qcarcam_open
camera_client 调用qcarcam_open到AisEngine 中这里并没有对IFE、INPUT、CSI做什么操作,只是为这个open的camera分配了一个属于,camera_client和ais_server 识别这个camera的唯一标识句柄qcarcam_hndl
camera_client__qcarcam_s_param_193">五、camera_client -> qcarcam_s_param
这里client会为每个camera申请5个buffer(MIN_USR_NUMBER_OF_BUFFERS 3 <buffer < QCARCAM_MAX_NUM_BUFFERS 12)
将client传过来的buffer地址和ais_server中的buffer做内存映射
smmu_map_pt_v2 函数可能用于将物理地址(DA, Device Address)映射到虚拟地址空间,具体是通过系统的 SMMU(System Memory Management Unit,系统内存管理单元)进行的,这里就简单理解camera_client buffer和ais_server的buffer做了映射,然后ais_server的buffer又绑定了一块专门用于camera data传输的内存。
如上log分析,handle 就是camera_client buffer的地址,DA就是做完映射后,ais_server使用的buffer地址,我们打开了四个camera,每个camera申请了5个buffer。
根据环视的camera_config.xml 配置,当有camera_client连接ais_server的时候,会优先给该camera分配IFE2,然后当IFE2的四个RDI都占用后,再连接的时候就分配IFE3的RDI
如下图,CSID2 对应的 IFE就可以理解为IFE2、CSID3 对应的IFE就可以理解为IFE3,他们IFE都是Lite模式四路数据传输线。
这个后面camera data传递的时候会深入讨论
camera_client__qcarcam_start_211">六、camera_client -> qcarcam_start
如上时序图,camera_client的start操作,其实主要针对的就是ais_input_configurer, CISPHY已经在ais_server初始化的时候已经start过了,然后根据先后顺序会start IFE 中的VFE/CISD,然后会start camera sensor, 这里我们用的摄像头加串器是96716/解串器是96722。
在96722sensor中 ,max96722_sensor_open_lib的时候,
memcpy(&pCtxt->sensor_lib, &sensor_lib_ptr, sizeof(pCtxt->sensor_lib));memcpy(&pCtxt->max96722_sensors, max96722_sensors_init_table, sizeof(pCtxt->max96722_sensors));
static max96722_sensor_info_t max96722_sensors_init_table[] =
{{.state = SENSOR_STATE_INVALID,.serializer_addr = MSM_DES_0_ADDR_CAM_SER_0,.sensor_addr = MSM_DES_0_ADDR_CAM_SNSR_0,},{.state = SENSOR_STATE_INVALID,.serializer_addr = MSM_DES_0_ADDR_CAM_SER_1,.sensor_addr = MSM_DES_0_ADDR_CAM_SNSR_1,},{.state = SENSOR_STATE_INVALID,.serializer_addr = MSM_DES_0_ADDR_CAM_SER_2,.sensor_addr = MSM_DES_0_ADDR_CAM_SNSR_2,},{.state = SENSOR_STATE_INVALID,.serializer_addr = MSM_DES_0_ADDR_CAM_SER_3,.sensor_addr = MSM_DES_0_ADDR_CAM_SNSR_3,},
};
static sensor_lib_t sensor_lib_ptr =
{.sensor_slave_info ={.sensor_name = SENSOR_MODEL,.slave_addr = MSM_DES_0_SLAVEADDR,.i2c_freq_mode = SENSOR_I2C_MODE_CUSTOM,.addr_type = CAMERA_I2C_WORD_ADDR,.data_type = CAMERA_I2C_BYTE_DATA,.sensor_id_info ={.sensor_id_reg_addr = 0x00,.sensor_id = MSM_DES_0_SLAVEADDR,.sensor_id_mask = 0xff00,},.power_setting_array ={.power_up_setting_a ={{.seq_type = CAMERA_POW_SEQ_VREG,.seq_val = CAMERA_VDIG,.config_val = 0,.delay = 1,},{.seq_type = CAMERA_POW_SEQ_GPIO,.seq_val = CAMERA_GPIO_RESET,.config_val = GPIO_OUT_LOW,.delay = 10,},{.seq_type = CAMERA_POW_SEQ_GPIO,.seq_val = CAMERA_GPIO_RESET,.config_val = GPIO_OUT_HIGH,.delay = 100,},},.size_up = 3,.power_down_setting_a ={{.seq_type = CAMERA_POW_SEQ_GPIO,.seq_val = CAMERA_GPIO_RESET,.config_val = GPIO_OUT_LOW,.delay = 0,},},.size_down = 1,},.is_init_params_valid = 1,},.csi_params ={{.lane_cnt = 4, /*4 lane mode is selected */.settle_cnt = 0xE,.lane_mask = 0x1F,.combo_mode = 0,.is_csi_3phase = 0,}},.sensor_close_lib = max96722_sensor_close_lib,.sensor_capability = 0,/*custom functions that were defined in the driver */.sensor_custom_func ={.sensor_set_platform_func_table = &max96722_sensor_set_platform_func_table,.sensor_power_suspend = &max96722_sensor_power_suspend,.sensor_power_resume = &max96722_sensor_power_resume,.sensor_detect_device = &max96722_sensor_detect_device,.sensor_detect_device_channels = &max96722_sensor_detect_device_channels,.sensor_init_setting = &max96722_sensor_init_setting,.sensor_set_channel_mode = &max96722_sensor_set_channel_mode,.sensor_start_stream = &max96722_sensor_start_stream,.sensor_stop_stream = &max96722_sensor_stop_stream,.sensor_s_param = &max96722_des_sensor_s_param,.sensor_g_param = &max96722_des_sensor_g_param,//.sensor_power_on = &max96722_sensor_power_on,},.use_sensor_custom_func = TRUE,
};
如上代码所示,在open的时候就将sensor_lib_ptr和max96722_sensors_init_table copy给了sensor_lib和max96722_sensors,后面对于camera寄存器的读写都操作sensor_lib_ptr,对于加/解串操作max96722_sensors,
max96722_sensor_start_stream() -> ser96717_start_link() -> i2c_slave_write_array(max96722_sensors->serializer_addr, cam_ser_reg_setting), 这里96722/96716对应的解串器都是96717,这里做了统一化处理,包括来自J5的摄像头也是定义了一个96717的解串器名字
camera_client__qcarcam_get_frame_337">七、camera_client -> qcarcam_get_frame
当ais_server 准备好camera data后就会通知camera_client去使用camera data,大致时序如下:
如上时序图,VFE中的camera data流转过程:
1、在VEF模块初始化的时候,会注册函数ifeIST()到CameraPlatformISTInit中,在CameraPlatformISTInit中会启动一个线程CameraIST,会专门用于监听中断, 然后回调ifeIST().
2、ifeIST() -> ProcessIST() -> CameraSetSignal(m_ifeIstProcThreadContext.evIfeIstNotification) -> ISTWorkerThread(), 回调到 ifeIST() 之后会间接的唤醒等待的函数ISTWorkerThread ->
CameraWaitOnSignal(pIfeCtx->m_ifeIstProcThreadContext.evIfeIstNotification), 然后会紧接着调用
ProcessISTQueueItem(), 将填充好sQueueItem的根据中断类型执行相应的操作:
ProcessResetIRQ(): 触发重启VFE中断
ProcessSofIRQ(): camera data从硬件缓冲区到内存,将client申请的buffer映射出来的软件缓冲区提交到硬件缓冲区,去填充camera data
ProcessWrBusIRQ(): 软件缓冲区数据填满后就会触发中断,通知去消费数据
ProcessOverflowIRQ(): 将IFE、MIPI相关的error, waring 通知到camera client端。
如上都是基于学习的精神做的笔记,若有错误的地方请大佬们多多指正。