ros图像处理相关
1. image_transport
ros规定了多种基本的数据结构，用于node之间的传输。图像也是一种常用的数据，image_transport package就是用来处理图像数据传输的，它本身是个框架，只提供最基本的原始图像数据(raw)，如果需要降低传输时的带宽，还需要压缩格式，由框架下集成的各种插件来完成。
image_transport 会发布 sensor_msgs/Image 格式的数据，其他压缩格式由插件提供。
ROS中原始图像格式"sensor_msgs/Image"如下：
# This message contains an uncompressed image
# (0, 0) is at top-left corner of image
#
Header header # std_msgs/Header
uint32 height # image height, that is, number of rows
uint32 width # image width, that is, number of columns
# The legal values for encoding are in file src/image_encodings.cpp
# If you want to standardize a new string format, join
# ros-users@lists.sourceforge.net and send an email proposing a new encoding.
string encoding # Encoding of pixels -- channel meaning, ordering, size
# taken from the list of strings in include/sensor_msgs/image_encodings.h
uint8 is_bigendian # is this data bigendian?
uint32 step # Full row length in bytes
uint8[] data # actual matrix data, size is (step * rows)
其中 header 是标准结构:
# Standard metadata for higher-level stamped data types.
# This is generally used to communicate timestamped data
# in a particular coordinate frame.
#
# sequence ID: consecutively increasing ID
uint32 seq
#Two-integer timestamp that is expressed as:
# * stamp.sec: seconds (stamp_secs) since epoch (in Python the variable is called 'secs')
# * stamp.nsec: nanoseconds since stamp_secs (in Python the variable is called 'nsecs')
# time-handling sugar is provided by the client library
time stamp
#Frame this data is associated with
string frame_id
一个具体例子:
root@suntus:~# rostopic echo -n 1 /cv_camera/image_raw --noarr
header:
seq: 293 # 当前帧序号
stamp: # 获取图像的时间戳
secs: 1636868180
nsecs: 53476744
frame_id: "camera" # frame_id
height: 480 # 行数
width: 640 # 列数
encoding: "bgr8" # 像素点的压缩格式，通道数3，顺序是BGR，深度是24。按照这个可以知道data数组中值的意义
is_bigendian: 0 # 是否大端排序
step: 1920 # 步长，一行的字节数，宽度(640) x 像素点字节数(3) = 1920bytes。图像总共 480行，也就是 1920x480=921600字节，也是下边data数组长度
data: "<array type: uint8, length: 921600>"
使用上的例子
不使用 image_transport 的订阅发布：
// Do not communicate images this way!
#include <ros/ros.h>
void imageCallback(const sensor_msgs::ImageConstPtr& msg)
{
// ...
}
ros::NodeHandle nh;
ros::Subscriber sub = nh.subscribe("in_image_topic", 1, imageCallback);
ros::Publisher pub = nh.advertise<sensor_msgs::Image>("out_image_topic", 1);
使用 image_transport 的订阅发布:
// Use the image_transport classes instead.
#include <ros/ros.h>
#include <image_transport/image_transport.h>
void imageCallback(const sensor_msgs::ImageConstPtr& msg)
{
// ...
}
ros::NodeHandle nh;
image_transport::ImageTransport it(nh); // 相当于将原始的nh封装了一次
image_transport::Subscriber sub = it.subscribe("in_image_base_topic", 1, imageCallback);
image_transport::Publisher pub = it.advertise("out_image_base_topic", 1);
提供的东西:
API
publisher
image_transport::Publisher
image_transport本身只在基础topic: <base topic> 上发布 sensor_msgs/Image 原始格式的图像，如果有其他插件，会自动在基础topic下的其他topic发布:<base topic>/<transport name>，比如基础topic是 "/cv_camera",
image_transport 会发布 "/cv_camera/image_raw" 原始格式的topic，插件compressed_image_transport 会在 "/cv_camera/image_raw/compressed" 发布压缩格式的图像。
subscriber
image_transport::Subscriber
订阅基础topic后，会自动订阅其他相关topic。
parameter
image_transport 本身不发布参数，但插件会发布各种参数，比如帧率、压缩等级等，需要看插件本身。
发布者应该使用动态参数机制，让参数更容易被订阅者使用。
参数格式一般是: <base topic>/<transport name>/<parameter name>
比如: /cv_camera/image_raw/compressed/parameter_descriptions
node
$ rosrun image_transport republish [in_transport] in:=<in_base_topic> [out_transport] out:=<out_base_topic>
比如将 theora 格式的topic图像解压，重新发布到另一个topic，可以这样写:
$ rosrun image_transport republish theora in:=camera/image raw out:=camera/image_decompressed
如果一个node只发布原始的图像格式，可以用 image_transport 重新发布成多种格式的topic。
命令行工具
$ rosrun image_transport list_transports
可以查看当前所有的package，看看是否有可用的图像处理插件，以及插件状态。
下边的插件介绍
使用 image_transport 提供的一个node命令： list_transports，可以查看当前有的topic和对应的插件：
root@suntus:~# rosrun image_transport list_transports
Declared transports:
image_transport/compressed
image_transport/compressedDepth
image_transport/ffmpeg (*): Not available. Try 'catkin_make --pkg ffmpeg_image_transport'.
image_transport/raw
image_transport/theora
Details:
----------
"image_transport/compressed"
- Provided by package: compressed_image_transport
- Publisher:
This plugin publishes a CompressedImage using either JPEG or PNG compression.
- Subscriber:
This plugin decompresses a CompressedImage topic.
----------
"image_transport/compressedDepth"
- Provided by package: compressed_depth_image_transport
- Publisher:
This plugin publishes a compressed depth images using PNG compression.
- Subscriber:
This plugin decodes a compressed depth images.
----------
"image_transport/ffmpeg"
*** Plugins are not built. ***
- Provided by package: ffmpeg_image_transport
- Publisher:
This plugin encodes frame into ffmpeg compressed packets
- Subscriber:
This plugin decodes frame from ffmpeg compressed packets
----------
"image_transport/raw"
- Provided by package: image_transport
- Publisher:
This is the default publisher. It publishes the Image as-is on the base topic.
- Subscriber:
This is the default pass-through subscriber for topics of type sensor_msgs/Image.
----------
"image_transport/theora"
- Provided by package: theora_image_transport
- Publisher:
This plugin publishes a video packet stream encoded using Theora.
- Subscriber:
This plugin decodes a video packet stream encoded using Theora.
compressed_image_transport("compressed"):
发布和订阅 JPEG 或 PNG 压缩格式的图像，支持运行时改变图片质量。
压缩格式:
# This message contains a compressed image
Header header # std_msgs/Header
string format # Specifies the format of the data
# Acceptable values:
# jpeg, png
uint8[] data # Compressed image buffer
一个例子:
root@suntus:~# rostopic echo -n 1 /cv_camera/image_raw/compressed --noarr
header:
seq: 0
stamp:
secs: 1636880329
nsecs: 751462995
frame_id: "camera"
format: "bgr8; jpeg compressed bgr8"
data: "<array type: uint8, length: 43548>"
---
提供的东西：
发布插件
发布topic:
<base_topic>/compressed (sensor_msgs/CompressedImage)
参数
- <base_topic>/compressed/format (string, default: jpeg) Compression format to use, "jpeg" or "png".
- <base_topic>/compressed/jpeg_quality (int, default: 80) JPEG quality percentile, in the range [1, 100]. Lower values trade image quality for space savings.
- <base_topic>/compressed/png_level (int, default: 9) PNG compression level, in the range [1, 9]. Higher values trade computation time for space savings.
订阅插件
订阅topic:
<base_topic>/compressed (sensor_msgs/CompressedImage)
compressed_depth_image_transport
提供深度图像(depth image),主要用于3D领域，带距离的图像格式。
theora_image_transport
发布成视频流
theora视频编码格式的视频流数据
theora是个视频编码格式，类似H264这种
对应的声音编码技术是vorbis
对应的文件封装格式是ogg
发布topic
<base topic>/theora (theora_image_transport/Packet)
参数
- <base topic>/theora/optimize_for (int, default: Quality (1))
Controls whether to use constant bitrate (CBR) encoding, aiming for ~theora/target_bitrate; or variable bitrate (VBR) encoding, aiming for ~theora/quality. Values are Bitrate (0) and Quality (1). - <base topic>/theora/target_bitrate (int, default: 800000)
Target bitrate. Used if optimize_for is set to Bitrate. - <base topic>/theora/quality (int, default: 31)
Encoding quality level, in the range [0, 63]. Used if optimize_for is set to Quality. - <base topic>/theora/keyframe_frequency (int, default: 64)
Maximum distance between key frames. If set to 1, every frame is a keyframe.
订阅topic:
<base topic>/theora (theora_image_transport/Packet)
提供有一个node: ogg_saver，可以保存成 .ogv 格式的封装文件，用其他播放器播放。
ffmpeg_image_transport
使用ffmpe将图像转换成h264或h265格式，并且支持nvidia GPU硬件加速。
https://github.com/daniilidis-group/ffmpeg_image_transport
cv_bridge
在ROS Image messages 和 OpenCV images 格式之间进行转换。
ros图像格式基础的是 sensor_msgs/Image, 还有 sensor_msgs/CompressedImage, opencv中用到的格式为 cv::Mat 矩阵，需要进行转换，才能放到opencv中使用。
opencv中支持的像素编码格式有：
8UC[1-4]
8SC[1-4]
16UC[1-4]
16SC[1-4]
32SC[1-4]
32FC[1-4]
64FC[1-4]
cv_bridge有时候会进行必要的像素格式转换，可以使用如下的字符串来表示目标格式：
- mono8: CV_8UC1, grayscale image
- mono16: CV_16UC1, 16-bit grayscale image
- bgr8: CV_8UC3, color image with blue-green-red color order
- rgb8: CV_8UC3, color image with red-green-blue color order
- bgra8: CV_8UC4, BGR color image with an alpha channel
- rgba8: CV_8UC4, RGB color image with an alpha channel
其中 mono8和bgr8是opencv中常用的格式。
roscpp
初始化和停止
ros::init(argc, argv, "my_node_name");
ros::shutdown();
ros::ok() 用于检查node是否shutdown
timer
// 定时器，周期性执行回调。
ros::Timer ros::NodeHandle::createTimer(ros::Duration period, <callback>, bool oneshot = false);
ros::Timer timer = nh.createTimer(ros::Duration(0.1), timerCallback); // 创建timer
// 回调签名:
void callback(const ros::TimerEvent&);
// 其中 ros::TimerEvent:
// ros::Time last_expected -- 理想中前一个回调应该被执行的时间
// ros::Time last_real -- 实际上前一个回调执行的时间
// ros::Time current_expected -- 理想中当前回调应该被执行的时间
// ros::Time current_real -- 实际上当前回调执行的时间
// ros::WallTime profile.last_duration -- 前一个回调执行的耗时
线程
单线程
// 单线程调用
ros::init(argc, argv, "my_node");
ros::NodeHandle nh;
ros::Subscriber sub = nh.subscribe(...);
...
ros::spin();
// 自己实现 spin()
#include <ros/callback_queue.h>
ros::NodeHandle n;
while (ros::ok())
{
ros::getGlobalCallbackQueue()->callAvailable(ros::WallDuration(0.1));
}
// 另一种形式的单线程调用
ros::Rate r(10); // 10 hz
while (should_continue)
{
... do some work, publish some messages, etc. ...
ros::spinOnce();
r.sleep();
}
// 自己实现 spinOnce()
#include <ros/callback_queue.h>
ros::getGlobalCallbackQueue()->callAvailable(ros::WallDuration(0));
多线程
// 同步spin，阻塞
ros::MultiThreadedSpinner spinner(4); // Use 4 threads
spinner.spin(); // spin() will not return until the node has been shutdown
// 异步spin
ros::AsyncSpinner spinner(4); // Use 4 threads
spinner.start();
ros::waitForShutdown(); // 这里会阻塞
调用队列
// ros有个全局默认队列, 所有回调都会挂到这个上边
ros::getGlobalCallbackQueue()
// 创建自己的调用队列，有两层次:
#include <ros/callback_queue.h>
ros::CallbackQueue my_queue;
// 层次1: 每个 subscribe(), advertise(), advertiseService()
// 使用 Options 结构体传入自己的队列
// 层次2: 每个 NodeHandle，更常用. ros::spin() 不会自动触发这些回调，需要手动去触发
ros::NodeHandle nh;
nh.setCallbackQueue(&my_callback_queue);
// 一次调用所有的回调
callAvailable()
// 一次调用一个回调
callOne()
