Halcon 3D Vision
3D Reconstruction Methods
两个3D坐标系的位置关系：旋转分量矩阵的三个列向量为新坐标系的三个轴向相对于旧坐标系的坐标。
Coordinate System Rotation Chains
• When reading a chain from the left to right, rotations are performed around the “new” axes.
• When reading a chain from the right to left, rotations are performed around the “old” axes.
The camera coordinate system is defined so that its x and y axes are parallel to the column and row axes of the image respectively, and the z axis is perpendicular to the image plane.
Measurement in a specified plane with a single camera
In halcon,the measurement plane is defined as the plane z=0 with respect to the world coordinate system.
确定测量平面位姿方法：
- 其中一个标定板位置位于测量平面上
get_calib_data (CalibDataID, 'calib_obj_pose', [0,1], 'pose', Pose) //[CalibObjIdx=0, CalibObjPoseIdx=1]
To compensate for the thickness of the calibration plate, the pose is moved by the corresponding value.
set_origin_pose( : : PoseIn, DX, DY, DZ : PoseNewOrigin) //the translation is performed relative to the local coordinate system of the pose itself
set_origin_pose（: :PoseIn, 0, 0, d: PoseNewOrigin）//d为标定板厚度，PoseNewOrigin为真实测量平面姿态
- 标定板不位于测量平面上
首先对相机进行标定，标定时相机可以先不固定。可以标定完成后将相机安装到测量位置，标定板放置到测量平面上，相机拍摄板定板图像，从而获取测量平面位姿。通过get_calib_data_observ_points（）获取位姿。
- 使用3D点和对应的图像点坐标点对
vector_to_pose( : : WorldX, WorldY, WorldZ, ImageRow, ImageColumn, CameraParam, Method, QualityType : Pose, Quality)
Use halcon calibration plate,the whole process of finding the calibration plate,extract the marks and determine the corresponddence between the marks and the respective 3D world coordinates can be carried automatically.Even more important, these calibration plates are highly accurate, down to 1 µm or below, which is a prerequisite for high accuracy applications.
Calibration plates with hexagonally arranged marks and calibration marks with rectangularly arranged marks are available. The calibration plates with hexagonally arranged marks are introduced with HALCON 12 and are recommended for most applications, as they provide the following advantages compared to the calibration plates with rectangularly arranged marks:
• The calibration plates with rectangularly arranged marks should fill a fourth of the image area. To
cover the whole field of view including different tilts, many images (at least 10 to 20) are needed.
As calibration plates with hexagonally arranged marks contain a significantly larger number of
calibration marks and therefore can and should cover the whole image area with a single image,
less images (6 to 7) are needed to get a comparable calibration result.
• The calibration plates with rectangularly arranged marks must be completely visible in the im-
ages whereas plates with hexagonally arranged marks may protude beyond the rim of the image.
Thus, with the latter less care must be taken when placing the calibration plate in the images.
The acquisition of the calibration images becomes faster and more convenient without a loss of
robustness.
标定板描述文件
The description files for calibration plates with hexagonally arranged marks have the file extension ’.cpd’ and those of the calibration plates with rectangularly arranged marks have the file extension ’.descr’.
Image coordinate --> World coordinate
In halcon,the region cannot be transformed directly into the world coordinate system.you should first convert them into XLD contours by gen_contour_region_xld(),then use the operator contour_to_world_plane_xld().
World coordinate --> Image coordinate
pose_to_hom_mat3d() affine_trans_point_3d() project_3d_point()
3D Position Recognition of Known Object
3D-2D points pair(mono 3D)
Shape-based 3D matching
Surface-based 3D matching (Volume matching)
3D Primitives Fitting
Perspective matching (calibrated perspective deformable matching,calibrated descriptor-based matching)
circle/rectangle pose estimation
Pose Estimation from Points
disp_3d_coord_system( : : WindowHandle, CamParam, Pose, CoordAxesLength : ) //显示3D坐标系
vector_to_pose( : : WorldX, WorldY, WorldZ, ImageRow, ImageColumn, CameraParam, Method, QualityType : Pose, Quality) //通过点对计算姿态
find_caltab(Image : CalPlate : CalPlateDescr, SizeGauss, MarkThresh, MinDiamMarks : ) //从图片中找出标定板区域
find_marks_and_pose() //Extract rectangularly arranged 2D calibration marks from the image and calculate initial values for the external camera parameters.
Pose Estimation Using Shape-based 3D Matching
The 3D shape model is generated by computing different views of the 3D object model within a user-specified pose range. The views are obtained by placing virtual cameras around the object model and projecting the 3D object model into the image plane of each camera position. The resulting 2D shape representations of all views are stored in the 3D shape model.
An important task is to specify the pose range. To ease this task, imagine a sphere that surrounds the object. On the surface of the sphere, a camera is placed that looks at the object. Now, the pose range can be defined by restricting the position of the camera to a part of the sphere’s surface. Additionally, the
minimum and maximum distance of the camera to the object, i.e., the radii of different spheres, must be specified.
物体坐标系的xz面定义为赤道面，N极为负y轴方向，零经线与赤道交点位于z轴上。
read_shape_model_3d（），create_shape_model_3d（），find_shape_model_3d（）
Pose Estimation Using Surface-Based 3D Matching
The surface model consists of a set of 3D points and the points' normal vectors.In contrast to shape-based matching,the instances of object are not located in images but in a 3D scene.
create_surface_model(),find_surface_model(),get_surface_matching_result()
Pose Estimation Using Deformable Surface-Based 3D Matching
create_deformable_surface_model()
find_deformable_surface_model()
add_deformable_surface_model_reference_point() //定义参考点，可位于任何位置，可用于确定抓取物体时的抓取点
get_deformable_surface_matching_result()
Pose Estimation Using 3D Primitives Fitting
fit_primitives_object_model_3d()
segment_object_model_3d()
Pose Estimation Using Calibrated Perspective Deformable Matching
create_planar_calib_deformable_model()
find_planar_calib_deformable_model()
Pose Estimation Using Calibrated Descriptor-Based Matching
Note that the calibrated descriptor-based matching is suitable mainly to determine the 3D pose of planar objects with characteristic texture and distinctive object points. For low-textured objects with rounded edges you should select one of the other pose estimation approaches. Further, thedescriptor-basedmatch-
ing is less accurate than the perspective deformable matching. But on the other hand, it is significantly faster if a large search space, e.g., caused by a large scale range, is used.
Pose Estimation for Cirlces
It is based on the known geometrical behavior of perspectively distorted circles. In particular, 3D circles are represented as ellipses in the image. Using the extracted 2D ellipse of a 3D circle together with the internal camera parameters and the known radius of the circle, the two possible 3D poses of the circle (having the same position but opposite orientations) can be obtained easily using the operator get_circle_pose.
Pose Estimation for Rectangles
In particular, a contour is segmented into four line segments and their intersections are considered as the corners of a quadrangular contour.Using the extracted 2D quadrangle of the 3D rectangle together with the internal camera parameters and the known size of the rectangle, the four (or eight in case of a square) possible 3D poses of the rectangle can be obtained easily using the operator get_rectangle_pose.
3D Vison with a Stereo System
双目
立体匹配方法有：
correlation-based stereo matching：fast,can be parallelized and is invariant against gray-value chages,works good only for significantly textured areas.
multigrid stereo matching:interpolates 3D information for areas without texture based on the surrounding areas, in particular for edges, the accuracy in general is higher than for correlation-based stereo, and the resolution is higher than for correlation-based stereo, i.e., smaller objects can be reconstructed.
multi-scanline stereo matching:determines 3D information for areas with little texture,runtime increases significantly with image size and disparity search range and high memory consumption.
计算视差：binocular_disparity(),binocular_disparity_mg(),binocular_disparity_ms(),计算结果以第一个rectified image为参考，the disparity for the point (r,c) of the first rectified image is the gray value at the position (r,c) of the disparity image.
计算距离：The distance of an object point from the stereo camera system is defined as its distance from the x-y-plane of the coordinate system of the first rectified camera.
binoclular_distance(), binocular_distance_mg(), binocular_distance_ms(),disparity_to_distance(),
计算3D坐标：disparity_to_point_3d(),disparity_image_to_xyz()
Laser Triangulation with Sheet of Light
A sheet of light can be calibrated in two different ways:using a standard HALCON calibration plate or using a special 3D calibration object.
The calibration of the sheet-of-light setup is applied to get the internal and external camera parameters,the orientation of the light plane in the WCS, and the relative movement of the object between two successive measurements.
Depth from Focus
The setup requires only a single camera, therefore, it is possibly more compact than, e.g., a stereo setup. DFF requires, however, cameras with telecentric or microscope lenses in order to achieve a (nearly) parallel projection. Therefore, DFF is only suitable for small objects.
图像锐度评价：hdevelop\Applications\General\determine_sharpness.hdev
Robot Vision
典型的关节型机器人有6个自由度，SCARA机器人有4个自由度（3平移+1旋转）
For articulated robots（关节型机器人） that are used together with one camera, it is possible to calibrate the camera together with the hand-eye setup. For SCARA robots and all systems that use stereo or multi-view setups, the camera(s) must be calibrated in advance.
对于关节型机器人，如果手眼标定前，相机已经经过标定，则相机内参参数应该从优化参数中排除：
set_calib_data (CalibDataID, 'camera', 'general', 'excluded_settings', 'params')
对于SCARA机器人，相机需要提前经过标定，相机内参参数会自动从待优化参数中排除，无需再手动设置。
手眼标定时，如果使用相机+标定板时，create_calib_data（）中，NumCameras和NumCalibObjects参数指定为1，使用3D传感器时，这两个参数指定为0.
Please note that for the hand-eye calibration we strongly recommend to use the calibration plate with hexagonally arranged calibration marks.
使用3D sensor时，直接将姿态设置到标定数据模型中：set_calib_data_observ_pose (CalibDataID, 0, 0, I, ObjInCamPose)
Determine Translation in Z Direction for SCARA Robots
When calibrating SCARA robots, it is not possible to determine the Z translation of ’obj_in_base_pose’ (moving camera) or ’obj_in_tool_pose’ (stationary camera). To eliminate this ambiguity the Z translation is internally set to 0.0 in these poses and the poses ’tool_in_cam_pose’ (moving camera) and
’base_in_cam_pose’ (stationary camera), respectively, are calculated accordingly. It is necessary to determine the true translation in Z after the calibration by moving the robot to a pose of known height in the camera coordinate system. For this, the following approach can be applied：
