go语言中protobuf以及grpc的使用
首先定义数据结构，保存为.proto文件
syntax = "proto3";
// The protocol compiler generates a class from the following .proto file with
// methods for writing and reading the message fields. The class is called
// "Person" and is in a package called "tutorial". You can specify a different
// class name using the java_outer_classname option.
message Person {
string name = 1;
int32 id = 2; // Unique ID number for this person.
string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
string number = 1;
PhoneType type = 2;
}
repeated PhoneNumber phones = 4;
}
// Our address book file is just one of these.
message AddressBook {
repeated Person people = 1;
}
然后，用protocol buffer编译器以生成go语言源代码。没想多遇见了好几次问题。当然，已经有现成且有效的解决方案了。
编译指令：
protoc -I=$SRC_DIR --go_out=$DST_DIR $SRC_DIR/addressbook.proto
首先报错的是，没有protoc-gen-go。于是百度，搜到了下面这篇博客。
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
没有程序，首先就要安装。使用上条指令安装后，需要手动复制protoc-gen-go.exe到proto的bin路径下。
随后，又执行上面的编译指令，就出现了下面的问题。
protoc-gen-go: unable to determine Go import path for "person.proto"
Please specify either:
• a "go_package" option in the .proto source file, or
• a "M" argument on the command line.
See https://protobuf.dev/reference/go/go-generated#package for more information.
解决方案是在proto文件中添加下面这行
option go_package="/person";也就是需要自己设定生成的pb.go的包名，下文再详细说明
解决方案，这篇博客非常详细，清楚，也可以直接查看。
最终的proto文件：
person.proto
syntax = "proto3";
option go_package = "/person";//生成的pb文件的包名，会自动创建该文件夹，前面必须有/
// The protocol compiler generates a class from the following .proto file with
// methods for writing and reading the message fields. The class is called
// "Person" and is in a package called "tutorial". You can specify a different
// class name using the java_outer_classname option.
message Person {
string name = 1;
int32 id = 2; // Unique ID number for this person.
string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
string number = 1;
PhoneType type = 2;
}
repeated PhoneNumber phones = 4;
}
// Our address book file is just one of these.
message AddressBook {
repeated Person people = 1;
}
当前目录和项目结构：
yyjeqhc@yyjeqhc MINGW64 ~/Desktop/proto
$ tree
.
`-- protobuf
`-- person.proto
1 directory, 1 file
编译指令：
protoc -I=./protobuf --go_out=./ person.proto
这个就很通俗易懂了。输入文件是protobuf文件夹中的person.proto，输出目录是当前目录。由于person.proto中go_package为/person，所以会创建person目录，生成的文件也会保存在person文件夹下
编译后文件结构：
yyjeqhc@yyjeqhc MINGW64 ~/Desktop/proto
$ tree
.
|-- person
| `-- person.pb.go
`-- protobuf
`-- person.proto
2 directories, 2 files
main.go
package main
import (
"fmt"
"google.golang.org/protobuf/proto"
"io/ioutil"
"log"
"proto/person"
)
//因为go run会有临时目录，所以使用相对路径没有那么方便，这里就使用绝对路径了，因为使用的是readfile，所以需要自己先创建这个文件。当然，修改代码，使用其他文件的API也可以的。
const fileName = "C:\\Users\\yyjeqhc\\Desktop\\proto\\addressbook.data"
func testProc() {
in, err := ioutil.ReadFile(fileName)
if err != nil {
log.Fatalln("Error reading file:", err)
}
book := &person.AddressBook{}
if err := proto.Unmarshal(in, book); err != nil {
log.Fatalln("Failed to parse address book:", err)
}
// Add an address.
book.People = append(book.People, &person.Person{
Name: "Alice",
Id: 100,
Email: "alice@example.com",
})
// Write the new address book back to disk.
out, err := proto.Marshal(book)
if err != nil {
log.Fatalln("Failed to encode address book:", err)
}
if err := ioutil.WriteFile(fileName, out, 0644); err != nil {
log.Fatalln("Failed to write address book:", err)
}
fmt.Println("that is ok")
}
func main() {
testProc()
}
代码编写完毕，接下来需要使用module解决包的引入。
go mod init proto //以项目名称作为模块名称
go mod tidy
剩下的，直接运行即可。
最后，上述整个的文件夹结构：
yyjeqhc@yyjeqhc MINGW64 ~/Desktop/proto
$ tree
.
|-- addressbook.data
|-- go.mod
|-- go.sum
|-- main.go
|-- person
| `-- person.pb.go
`-- protobuf
`-- person.proto
2 directories, 6 files
通过简短的例子和bing的解释，大致了解了protobuf的优势。以二进制的方式压缩数据，在传输过程中就能节省资源，减轻网络负荷。
例子比较简易，然后就想试一下protobuf在grpc中的使用，就有了下面的例子。
calculator.proto
syntax = "proto3";
option go_package="/calculator";
// The calculator service definition.
service Calculator {
// Sends two integers and returns their sum
rpc Add (AddRequest) returns (AddResponse) {}
// Sends two integers and returns their difference
rpc Subtract (SubtractRequest) returns (SubtractResponse) {}
// Sends two integers and returns their product
rpc Multiply (MultiplyRequest) returns (MultiplyResponse) {}
// Sends two integers and returns their quotient
rpc Divide (DivideRequest) returns (DivideResponse) {}
rpc Hello(HelloRequest) returns (HelloResponse) {}
//使用流来多次传输数据
rpc Channel(stream String) returns (stream String) {}
}
// The request message containing two integers
message AddRequest {
int32 a = 1;
int32 b = 2;
}
// The response message containing the sum
message AddResponse {
int32 result = 1;
}
// The request message containing two integers
message SubtractRequest {
int32 a = 1;
int32 b = 2;
}
// The response message containing the difference
message SubtractResponse {
int32 result = 1;
}
// The request message containing two integers
message MultiplyRequest {
int32 a = 1;
int32 b = 2;
}
// The response message containing the product
message MultiplyResponse {
int32 result = 1;
}
// The request message containing two integers
message DivideRequest {
int32 a = 1;
int32 b = 2;
}
// The response message containing the quotient
message DivideResponse {
int32 result = 1;
}
message HelloRequest {
string name = 1;
}
message HelloResponse {
string str = 1;
}
message String {
string str = 1;
}
protoc -I=D:\Git\protoTest\protobuf --go_out=D:\Git\protoTest calculator.proto
没注意，以为都是用protoc-gen-go.exe就好了呢。直接编译，还是生成了一堆文件，没有报错。
但是，发觉和官方给的例子差异很大。比如，官方的helloworld例子，我直接看的在线代码，里面proto生成的pb.go包含客户端和服务器的部分，但是我的pb.go文件就没有。我还觉得很奇怪呢，但还是有样学样的，添加一些代码，终于改好了client。于是开始改server端，但是太麻烦了，越改越觉得离谱。于是重新看了下bing给我的编译指令，和我这个可差太远了。才意识到问题所在。
protoc calculator.proto --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative
使用这个指令编译，遇见的问题，就是类似于上面的，没有protoc-gen-go-grpc。
于是有样学样，安装就好了。
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
同理，安装后需要复制protoc-gen-go-grpc.exe到protoc的bin路径下。
然后就可以正常编译了。
这是当前路径以及编译前的项目结构
yyjeqhc@yyjeqhc MINGW64 /d/Git/go/examples/calc (master)
$ tree
.
`-- protobuf
`-- calculator.proto
编译指令：
protoc --go_out=./ --go-grpc_out=./ ./protobuf/calculator.proto 咱也不知道，为什么不能像上面那样-I参数
经过本人尝试，go_out和go-grpc_out都是必须的参数
编译后的项目结构
yyjeqhc@yyjeqhc MINGW64 /d/Git/go/examples/calc (master)
$ tree
.
|-- calculator
| |-- calculator.pb.go
| `-- calculator_grpc.pb.go
`-- protobuf
`-- calculator.proto
2 directories, 3 files
然后，仍然是一样的
go mod init calc
go mod tidy
下面就编写具体的调用代码
server.go
package main
import (
"calc/calculator"
"context"
"io"
"strings"
)
type Server struct {
calculator.UnimplementedCalculatorServer
}
// Add implements the Add rpc method
func (s *Server) Add(ctx context.Context, in *calculator.AddRequest) (*calculator.AddResponse, error) {
// Get the two integers from the request
a := in.GetA()
b := in.GetB()
// Compute the sum
result := a + b
// Return the response with the sum
return &calculator.AddResponse{Result: result}, nil
}
func (s *Server) Subtract(ctx context.Context, in *calculator.SubtractRequest) (*calculator.SubtractResponse, error) {
// Get the two integers from the request
a := in.GetA()
b := in.GetB()
// Compute the sum
result := a - b
// Return the response with the sum
return &calculator.SubtractResponse{Result: result}, nil
}
func (s *Server) Multiply(ctx context.Context, in *calculator.MultiplyRequest) (*calculator.MultiplyResponse, error) {
// Get the two integers from the request
a := in.GetA()
b := in.GetB()
// Compute the sum
result := a * b
// Return the response with the sum
return &calculator.MultiplyResponse{Result: result}, nil
}
func (s *Server) Divide(ctx context.Context, in *calculator.DivideRequest) (*calculator.DivideResponse, error) {
// Get the two integers from the request
a := in.GetA()
b := in.GetB()
// Compute the sum
result := a / b
// Return the response with the sum
return &calculator.DivideResponse{Result: result}, nil
}
func (s *Server) Hello(ctx context.Context, in *calculator.HelloRequest) (*calculator.HelloResponse, error) {
name := in.GetName()
str := "hello," + name
return &calculator.HelloResponse{Str: str}, nil
}
//打开一个流，可以不停的传输数据，直到退出
func (s *Server) Channel(stream calculator.Calculator_ChannelServer) error {
for {
// 接收客户端发送的数据
request, err := stream.Recv()
if err == io.EOF {
return nil
}
if err != nil {
return err
}
// 处理客户端发送的数据
message := request.GetStr()
// 假设将接收到的字符串转为大写并返回给客户端
response := &calculator.String{
Str: strings.ToUpper(message),
}
// 发送返回的数据给客户端
if err := stream.Send(response); err != nil {
return err
}
}
}
client.go（这个文件并不需要）
package main
import (
"context"
"google.golang.org/grpc"
"calc/calculator"
)
type Client struct {
conn *grpc.ClientConn
}
// NewClient creates a new Client with the given connection
func NewClient(conn *grpc.ClientConn) *Client {
return &Client{conn: conn}
}
// Add calls the Add rpc method with two integers and returns their sum
func (c *Client) Add(a, b int32) (int32, error) {
// Create a new CalculatorClient with the connection
//client := NewCalculatorClient(c.conn)
client := calculator.NewCalculatorClient(c.conn)
// Create a new AddRequest with the two integers
req := &calculator.AddRequest{A: a, B: b}
// Call the Add method with the request and get the response
res, err := client.Add(context.Background(), req)
if err != nil {
return 0, err
}
// Return the result from the response
return res.GetResult(), nil
}
func (c *Client) Subtract(a, b int32) (int32, error) {
// Create a new CalculatorClient with the connection
//client := NewCalculatorClient(c.conn)
client := calculator.NewCalculatorClient(c.conn)
// Create a new AddRequest with the two integers
req := &calculator.SubtractRequest{A: a, B: b}
// Call the Add method with the request and get the response
res, err := client.Subtract(context.Background(), req)
if err != nil {
return 0, err
}
// Return the result from the response
return res.GetResult(), nil
}
func (c *Client) Multiply(a, b int32) (int32, error) {
// Create a new CalculatorClient with the connection
//client := NewCalculatorClient(c.conn)
client := calculator.NewCalculatorClient(c.conn)
// Create a new AddRequest with the two integers
req := &calculator.MultiplyRequest{A: a, B: b}
// Call the Add method with the request and get the response
res, err := client.Multiply(context.Background(), req)
if err != nil {
return 0, err
}
// Return the result from the response
return res.GetResult(), nil
}
func (c *Client) Divide(a, b int32) (int32, error) {
// Create a new CalculatorClient with the connection
//client := NewCalculatorClient(c.conn)
client := calculator.NewCalculatorClient(c.conn)
// Create a new AddRequest with the two integers
req := &calculator.DivideRequest{A: a, B: b}
// Call the Add method with the request and get the response
res, err := client.Divide(context.Background(), req)
if err != nil {
return 0, err
}
// Return the result from the response
return res.GetResult(), nil
}
// Close closes the connection
func (c *Client) Close() error {
return c.conn.Close()
}
main.go
package main
import (
"calc/calculator"
"context"
"fmt"
"google.golang.org/grpc"
"io"
"log"
"net"
"os"
"os/signal"
"time"
)
const address = "localhost:50051"
const (
// Port on which the server listens.
port = ":50051"
)
func testServer(c chan bool) {
// Listen on a port
lis, err := net.Listen("tcp", port)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
// Create a new gRPC server
s := grpc.NewServer()
// Register a new Calculator service with the server
calculator.RegisterCalculatorServer(s, &Server{})
c <- true
log.Println("server listening at :50051")
// Serve gRPC requests on the port
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
func main() {
ch := make(chan bool)
go testServer(ch)
<-ch //确保服务器启动完成再发出请求
conn, err := grpc.Dial(address, grpc.WithInsecure(), grpc.WithBlock())
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
// Create a calculator client
c := calculator.NewCalculatorClient(conn)
// Call the Add method with two integers
r, err := c.Add(context.Background(), &calculator.AddRequest{A: 3, B: 5})
if err != nil {
log.Fatalf("could not add: %v", err)
}
// Print the result
log.Printf("The sum is: %d", r.GetResult())
r1, err := c.Subtract(context.Background(), &calculator.SubtractRequest{A: 3, B: 5})
if err != nil {
log.Fatalf("could not sub: %v", err)
}
// Print the result
log.Printf("The sum is: %d", r1.GetResult())
r2, err := c.Multiply(context.Background(), &calculator.MultiplyRequest{A: 3, B: 5})
if err != nil {
log.Fatalf("could not mul: %v", err)
}
// Print the result
log.Printf("The sum is: %d", r2.GetResult())
r3, err := c.Divide(context.Background(), &calculator.DivideRequest{A: 13, B: 5})
if err != nil {
log.Fatalf("could not div: %v", err)
}
// Print the result
log.Printf("The sum is: %d", r3.GetResult())
rh, err := c.Hello(context.Background(), &calculator.HelloRequest{Name: "Jack"})
if err != nil {
log.Fatalf("could not hello: %v", err)
}
log.Printf("the response is %s", rh.GetStr())
stream, err := c.Channel(context.Background())
if err != nil {
log.Fatal(err)
}
go func() {
for {
if err := stream.Send(&calculator.String{
Str: "hi",
}); err != nil {
log.Fatal(err)
}
time.Sleep(time.Second)
}
}()
go func() {
for {
reply, err := stream.Recv()
if err != nil {
if err == io.EOF {
break
}
log.Fatal(err)
}
fmt.Println(reply.GetStr())
}
}()
quit := make(chan os.Signal, 1)
signal.Notify(quit, os.Interrupt)
<-quit
}
过程中缺什么包啥的，直接go get就好了。一执行，还是没有任何问题的。
总的目录结构：
$ tree
.
|-- calculator
| |-- calculator.pb.go
| `-- calculator_grpc.pb.go
|-- client.go
|-- go.mod
|-- go.sum
|-- main.go
|-- protobuf
| `-- calculator.proto
`-- server.go
执行结果：
2023/07/20 10:51:04 server listening at :50051
2023/07/20 10:51:04 The sum is: 8
2023/07/20 10:51:04 The sum is: -2
2023/07/20 10:51:04 The sum is: 15
2023/07/20 10:51:04 The sum is: 2
2023/07/20 10:51:04 the response is hello,Jack
HI
HI
HI
HI
HI
ctrl+c
于是又想着，protobuf是可以支持不同平台的。所以还想弄个python请求go的例子
没想到处处碰壁，python/Java/C++/node。都很困难。暂时就先不弄了。
grpcurl工具的使用
安装：
go get github.com/fullstorydev/grpcurl
go install github.com/fullstorydev/grpcurl/cmd/grpcurl
go install的都在user/go/bin路径下，可以直接把这个bin加到系统的path环境变量下
grpcurl localhost:50051 list
Failed to dial target host "localhost:50051": tls: first record does not look like a TLS handshake
没有配置证书就会这样。
grpcurl -plaintext localhost:50051 list
Failed to list services: server does not support the reflection API
服务器没有注册反射服务就会这样。
注册反射：
calculator.RegisterCalculatorServer(s, &Server{})
reflection.Register(s)//添加这一句
在注册服务器的地方，添加下面那一句就好了。
添加以后：
C:\Users\yyjeqhc>grpcurl -plaintext localhost:50051 list
Calculator
grpc.reflection.v1alpha.ServerReflection
C:\Users\yyjeqhc>grpcurl -plaintext localhost:50051 list Calculator
Calculator.Add
Calculator.Channel
Calculator.Divide
Calculator.Hello
Calculator.Multiply
Calculator.Subtract
C:\Users\yyjeqhc>grpcurl -plaintext localhost:50051 describe Calculator
Calculator is a service:
service Calculator {
rpc Add ( .AddRequest ) returns ( .AddResponse );
rpc Channel ( stream .String ) returns ( stream .String );
rpc Divide ( .DivideRequest ) returns ( .DivideResponse );
rpc Hello ( .HelloRequest ) returns ( .HelloResponse );
rpc Multiply ( .MultiplyRequest ) returns ( .MultiplyResponse );
rpc Subtract ( .SubtractRequest ) returns ( .SubtractResponse );
}
有啥问题，grpcurl -h，看看帮助文档就好了。
grpc-gateway的使用
helloworld.proto
syntax = "proto3";
option go_package="/helloworld";
import "google/api/annotations.proto";
// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {
option (google.api.http) = {
get: "/v1/example/echo/{name}"
};
}
}
// The request message containing the user's name.
message HelloRequest {
string name = 1;
}
// The response message containing the greetings
message HelloReply {
string message = 1;
}
https://github.com/googleapis/googleapis/tree/master/google
C:\protoc-23.2-win64\include\google
protoc --go_out=./ --go-grpc_out=./ --grpc-gateway_out=./ ./protobuf/helloworld.proto
'protoc-gen-grpc-gateway' 不是内部或外部命令，也不是可运行的程序
使用指令编译：
D:\Git\go\examples\gateway>protoc --go_out=./ --go-grpc_out=./ --grpc-gateway_out=./ ./protobuf/helloworld.proto
'protoc-gen-grpc-gateway' 不是内部或外部命令，也不是可运行的程序
或批处理文件。
--grpc-gateway_out: protoc-gen-grpc-gateway: Plugin failed with status code 1.
和上面的一样处理，进行安装
D:\Git\go\examples\gateway>go install google.golang.org/grpc/cmd/protoc-gen-grpc-gateway@latest
go: downloading google.golang.org/grpc v1.57.0
go: google.golang.org/grpc/cmd/protoc-gen-grpc-gateway@latest: module google.golang.org/grpc@latest found (v1.57.0), but does not contain package google.golang.org/grpc/cmd/protoc-gen-grpc-gateway
下载失败了。
原来：从gRPC-Gateway v2开始，protoc-gen-grpc-gateway的安装路径已经改变。
你应该使用以下命令来安装：go install github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway@latest
D:\Git\go\examples\gateway>go install github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway@latest
go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0
go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.16.2
go: downloading github.com/golang/glog v1.1.0
go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20230726155614-23370e0ffb3e
go: downloading google.golang.org/genproto v0.0.0-20230706204954-ccb25ca9f130
同样的，复制protoc-gen-grpc-gateway.exe到protoc的bin目录下。继续编译。
D:\Git\go\examples\gateway>protoc --go_out=./ --go-grpc_out=./ --grpc-gateway_out=./ ./protobuf/helloworld.proto
google/api/annotations.proto: File not found.
protobuf/helloworld.proto:5:1: Import "google/api/annotations.proto" was not found or had errors.
这就需要去https://github.com/googleapis/googleapis/tree/master/google，把google目录完整的下载到protoc的include目录了。
下载项目完整压缩包，然后复制过去即可。
最后，终于编译成功了。
D:\Git\go\examples\gateway>protoc --go_out=./ --go-grpc_out=./ --grpc-gateway_out=./ ./protobuf/helloworld.proto
helloworld目录和之前grpc相比，多了一个gw.go文件
main.go
package main
import (
"context"
"github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
"google.golang.org/grpc"
"google.golang.org/grpc/reflection"
"log"
"net"
"net/http"
gw "gateway/helloworld" // replace with your package location
)
type server struct {
gw.UnimplementedGreeterServer
}
func (s *server) SayHello(ctx context.Context, in *gw.HelloRequest) (*gw.HelloReply, error) {
return &gw.HelloReply{Message: "Hello " + in.Name}, nil
}
func main() {
// Start gRPC server
go func() {
lis, err := net.Listen("tcp", ":50051")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
gw.RegisterGreeterServer(s, &server{})
// Register reflection service on gRPC server.
reflection.Register(s)
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}()
// Start gateway
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
defer cancel()
mux := runtime.NewServeMux()
opts := []grpc.DialOption{grpc.WithInsecure()}
err := gw.RegisterGreeterHandlerFromEndpoint(ctx, mux, "localhost:50051", opts)
if err != nil {
log.Fatalf("failed to start HTTP gateway: %v", err)
}
log.Fatal(http.ListenAndServe(":8080", mux))
}
运行程序，浏览器访问http://localohost:8080/v1/example/echo/admin
{"message":"Hello admin"}
这样，就把grpc服务转换成了http请求了。
