基于protobuf的Go服务快速开发框架
设想
假设某种IDL
- 可以快速序列化与反序列化,说白了支持binary的序列化方案,也支持json/text的序列化方案.
- 可以描述数据结构与服务接口,一键生成常用的访问方式(rpc, http, websocket...)
- 可以被大多数项目接受, 且被验证是成熟可行方案
- 可以与Go/C/C++/Java/Python主流语言快速集成,实现跨语言方案!
调研现有的流行IDL:
- xml/json/yaml/toml: 这些都是特定的序列化/反序列化技术, 算不上IDL, 特别是rpc的描述更加不到位
- thrift: 没用过不是很熟悉, 但在Go世界貌似没见到太多采用, 毕竟Go是姓google, thrift是姓facebook!
- protobuf: 想来想去, 也就有protobuf天生迎合需要. 特别是Go服务, 以前一致性是restful api, 但现在却常常要求grpc接口.
困难
- protobuf描述message/service常常用grpc来实现, 如何用http来实现?
- protobuf的option扩展机制需要import自定义的descriptor, 这个表面看起来没啥问题! 但实现第一版后却发现是个累赘, 因为调用者主要用protobuf生成相应的客户端, 这些自定义的服务端option却影响客户端的protoc生成过程, 经常性报错促使调用者望而生畏! 如何用其他机制来扩展?
曙光
深入接触golang的build constraints概念后, 觉得上述的难点通过引入"proto constraints"来解决! 原因:
- proto constatins与build constraints一样, 实质是符合特定规范comments. 其对protoc是透明的, 也就不会影响调用者!
- proto constatins可以更直接地补充http定义. 例如: http上传文件的情景明确需要POST + multipart/form-data. 使用下述comments可以很好地定义提取过程!
// +http-form
message UploadRequest{
// +http-file
string test_file = 1;
}
message UploadResponse{
}
service UploadService{
// +http-path /demo/upload
rpc Upload(UploadRequest) returns(UploadResponse);
}
1. +http-form: 一般地从request提取message都是认为Content-Type: application/json, 通过http-form约束表示从request提取message要按照Content-Type: multipart/form-data.
2. +http-file: 表示message特定的字段是form-file, 后面可以通过某种辅助API获取到File, 例如: protokit.File(ctx, UploadRequest.test_file) => name, size, reader
3. +http-path: 表示method的访问URI, 一般是<package>/<service>/<method>全部转小写, 或者package, service, method去掉相应的Prefix/Suffix后再行拼凑得到结果, 具体实现TODO.
实践
- 定义某个数据结构, 从proto提取Package/Messages(Fields)/Services(Methods)的metadata及constratins.
- 通过template或programa的形式渲染自己需要的代码
具体过程https://github.com/fasgo/protoc-gen-go-http
最后
对于Go服务端服务API:
- 根据需求定义protobuf
- 根据protobuf一键生成grpc, http(post), websocket等访问代码
- 根据生成的service接口实现业务逻辑
这样就形成了一套基于protobuf的"3步走"Go服务快速开发框架!
下面是在protobuf的github上面看到的一段话:
This project hosts the Go implementation for protocol buffers, which is a language-neutral, platform-neutral, extensible mechanism for serializing structured data. The protocol buffer language is a language for specifying the schema for structured data. This schema is compiled into language specific bindings. This project provides both a tool to generate Go code for the protocol buffer language, and also the runtime implementation to handle serialization of messages in Go. See the protocol buffer developer guide for more information about protocol buffers themselves.
This project is comprised of two components:
Code generator: The protoc-gen-go tool is a compiler plugin to protoc, the protocol buffer compiler. It augments the protoc compiler so that it knows how to generate Go specific code for a given .proto file.
Runtime library: The protobuf module contains a set of Go packages that form the runtime implementation of protobufs in Go. This provides the set of interfaces that define what a message is and functionality to serialize message in various formats (e.g., wire, JSON, and text).
See the developer guide for protocol buffers in Go for a general guide for how to get started using protobufs in Go.
This project is the second major revision of the Go protocol buffer API implemented by the google.golang.org/protobuf module. The first major version is implemented by the github.com/golang/protobuf module.
...
Over the next 10 years, use of Go would skyrocket and use of protobufs in Go would skyrocket as well. With increased popularity also came more diverse usages and requirements for Go protobufs and an increased number of custom proto.Message implementations that were not generated by protoc-gen-go
简单地说, 就是google.golang.org/protobuf从v2版本起,针对protobuf的扩展重构了v1的API, 明确地将protobuf与grpc的"Code Generator + Runtime library"定位区分, 摆脱github.com/golang/protobuf的臃肿与不合理! 从而为GO使用protobuf的IDL铺平了道路!
