cyberRT消息解析
Rosbag的包 的解析工具
即只通过bag包数据直接反序列化，不用添加消息的descriptor相关信息。
CyberRecord解析工具
需要添加消息的descriptor相关信息，进行解析
https://zhuanlan.zhihu.com/p/499516617
https://github.com/daohu527/record_msg/tree/main/modules/drivers/lidar/proto
###https://apollo.baidu.com/community/article/148
cyber_launch主要用来启动cyber模块，
其中一个launch文件可以有一个或者多个module，
每个module 包含一个dag文件，而一个dag文件则对应一个或者多个components。
launch文件中有几个module则会启动几个进程，每个进程有单独的内存空间，比如静态变量等都不会共享。
一个dag文件中可以有一个或者多个components，
一个components对应一个协程。协程中的静态变量是共享的，并且全局唯一。
DAG依赖配置文件 (例如common.dag)中配置
cyber_recorder工具
01.record 结构
cyber/proto/record.proto
cyber/proto/proto_desc.proto
cyber/record/file/section.h
02.record 读写
cyber/record/file/record_file_base.h
cyber/record/file/record_file_reader
cyber/record/file/record_file_writer
cyber/record/[header_build record_base]
cyber/record/record_reader
cyber/record/record_viewer
cyber/record/record_writer
03.record-python读写
cyber/python/cyber_py3/record.py
cyber/python/cyber_py3/examples/py_record.py
04.cyber_recorder 工具 tools cyber_recorder
cyber/tools/cyber_recorder/main.cc
cyber/tools/cyber_recorder/recorder.cc
改写
1.cyber_record/common.py包含了section的定义以及 header的一些参数
section.h
struct Section {
proto::SectionType type;
int64_t size;
};
header_builder.h
private:
static const uint32_t MAJOR_VERSION_ = 1;
static const uint32_t MINOR_VERSION_ = 0;
static const proto::CompressType COMPRESS_TYPE_ = proto::CompressType::COMPRESS_NONE;
static const uint64_t CHUNK_INTERVAL_ = 20 * 1000 * 1000 * 1000ULL; // 20s
static const uint64_t SEGMENT_INTERVAL_ = 60 * 1000 * 1000 * 1000ULL; // 60s
static const uint64_t CHUNK_RAW_SIZE_ = 16 * 1024 * 1024ULL; // 16MB
static const uint64_t SEGMENT_RAW_SIZE_ = 2048 * 1024 * 1024ULL; // 2GB
header_build.cc
proto::Header header;
header.set_major_version(MAJOR_VERSION_);
header.set_minor_version(MINOR_VERSION_);
header.set_compress(COMPRESS_TYPE_);
header.set_chunk_interval(CHUNK_INTERVAL_);
header.set_segment_interval(SEGMENT_INTERVAL_);
header.set_chunk_raw_size(CHUNK_RAW_SIZE_);
header.set_segment_raw_size(SEGMENT_RAW_SIZE_);
GetHeader GetHeaderWithSegmentParams GetHeaderWithChunkParams
apollo是整个chunk序列化
proto
proto2
each class is __metaclass__ = reflection.GeneratedProtocolMessageType
proto3
Python is a little different —
the Python compiler generates a module with a static descriptor of each message type in your .proto,
which is then used with a metaclass to create the necessary Python data access class at runtime
When you run the protocol buffer compiler on a .proto
that uses an enum, the generated code will have a corresponding enum for Java, Kotlin, or C++,
or a special EnumDescriptor class for Python
that’s used to create a set of symbolic constants with integer values
in the runtime-generated class
In Python, the package directive is ignored,
since Python modules are organized according to their location in the file system.
In Python, it is still strongly recommended to specify the package for the .proto file,
as otherwise it may lead to naming conflicts
in descriptors and make the proto not portable for other languages.
protobuf2
SerializeToString(): serializes the message and returns it as a string.
Note that the bytes are binary, not text; we only use the str type as a convenient container.
ParseFromString(data)
python3
序列化的时候， python3 使用 protobuf3 的接口变了。
encode_to_bytes 替换了 SerializeToString
parse_from_bytes 替换了 ParseFromString
说明
ParseFromString是一种方法-它不返回任何内容，而是填充self已解析的内容
from google.protobuf import message_factory, descriptor_pb2, descriptor_pool
file_desc_proto = descriptor_pb2.FileDescriptorProto()
file_desc_proto.ParseFromString(proto_desc.desc)
版本
Protobufer 协议版本-文件版本 .proto files
1.0.1 - December 07, 2011 (79.5 KB)
1.0.0 - November 07, 2011 (79.0 KB)
2.0.0 - October 23, 2012 (359.0 KB)
2.8.13 - March 13, 2014 (115.0 KB)
3.0.0 - February 26, 2014 (118.5 KB)
3.8.0 - September 06, 2017 (199.5 KB)
3.10.0 - March 18, 2019 (229.5 KB)
3.10.9 - January 12, 2024 (236.0 KB)
proto文件定义结构化数据
protoc 版本
Protocol Buffers的编译器 Protobuf是小端编码
“3.21.7,” we say that “3” is the major version,
“21” is the minor version, and “7” is the micro or patch number.
each language has its own major version that can be incremented independently of other languages
The minor and patch versions, however, remain coupled.
每种语言都有自己的主版本
when Python 4.21.0 was released in May of 2022,
that set the end of public support of Python 3.20.1 at the end of 2023 Q2.
###参考 https://protobuf.dev/support/version-support/
This table provides specific dates for support duration.
Branch Initial Release Public Support Until
3.20.x 25 Mar 2022 30 Jun 2023
4.21.x 25 May 2022 16 Feb 2023
4.22.x 16 Feb 2023 8 May 2023
4.23.x 8 May 2023 8 Aug 2023
4.24.x 8 Aug 2023 1 Nov 2023
4.25.x 1 Nov 2023 31 Mar 2025
5.26.x 13 Mar 2024 TBD
版本now
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import symbol_database as _symbol_database
from google.protobuf.internal import builder as _builder
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\)
_globals = globals()
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'record_pb2', _globals)
if not _descriptor._USE_C_DESCRIPTORS:
DESCRIPTOR._loaded_options = None
_globals['_SECTIONTYPE']._serialized_start=1593
protoc-3.20.1-win64
版本old
from google.protobuf.internal import enum_type_wrapper
from google.protobuf import descriptor as _descriptor
from google.protobuf import message as _message
from google.protobuf import reflection as _reflection
from google.protobuf import symbol_database as _symbol_database
DESCRIPTOR = _descriptor.FileDescriptor()
_SECTIONTYPE = _descriptor.EnumDescriptor( )
_sym_db.RegisterEnumDescriptor(_SECTIONTYPE)
SectionType = enum_type_wrapper.EnumTypeWrapper(_SECTIONTYPE)
版本说明
In Python, the compiler only outputs code to build descriptors for the generated classes,
Currently both proto2 and proto3 use _pb2.py for their generated files.
protocol-buffers-in-python-no-classes-generated
新版本的 grpc_tools 不直接生成定义了，而是运行中动态生成
grpcio-tools 最后一个能生成方法和类的定义的版本是 1.43.0 ，
从 1.44 开始就没有定义内容了，开始动态生成
Protoc can generate Python stubs (.pyi) using the --pyi_out parameter
Just add to protoc command this arg: --pyi_out=. 可以通过添加pyi
*.pyi文件
后缀为.pyi的stub存根文件 里面的函数没有具体的实现
(stub file
enerating type stubs via .pyi files has been available for a few months
The python_stubs file is an
automatically generated file that contains dummy definitions for built-in functions.
It is used by PyCharm to infer the types of the built-in functions
in the case that they weren't hardcoded for the given version.
版本说明
Mar 5, 2022
Protocol Buffers v3.20.0
Protoc is now able to generate python stubs (.pyi) with --pyi_out
Protobuf python generated codes are simplified.
Descriptors and message classes' definitions are now dynamic created in internal/builder.py.
https://github.com/protocolbuffers/protobuf/releases?page=7
google.protobuf.internal import builder as _builder
site-packages/google/protobuf/internal/builder.p
注意 低版本的是缺少buidler.py这个文件，从此版本开始生成的pb_2.py变得简单了
Jan 29, 2022
Protocol Buffers v3.19.4
应用
msg.DESCRIPTOR.fields_by_name.keys()
file_des.enum_types_by_name.items()
BuildMessageAndEnumDescriptors
BuildTopDescriptorsAndMessages
参考
https://protobuf.dev/reference/python/python-generated/
https://developers.google.com/protocol-buffers/docs/reference/python-generated
解决python grpcio.protoc生成的pb文件里面没有类和方法定义的问题
https://apollo.baidu.com/docs/apollo/latest/recorder_8cc_source.html
cyber_recorder报文录制简介 https://cloud.tencent.com/developer/article/2348431
https://protobuf.dev/programming-guides/proto3/
https://googleapis.dev/python/protobuf/latest/google/protobuf/message.html#google.protobuf.message.Message
https://googleapis.dev/python/protobuf/latest/google/protobuf/message.html#google.protobuf.message.Message
https://protobuf.dev/getting-started/pythontutorial/
