Protobuf - FIELD NUMBERS

Required fields in a message can be thought of as frequently used fields since you cannot skip them as you can for optional fields. It is a best practice to reserve some numbers between 1 and 15 for the fields that can be frequently used since the numbers take 1 byte to encode in that range. For example, if you introduce a field with the name correlation_id, and it is used in almost all types of requests, you can assign one of the pre-reserved numbers for this new field. In the same way, it takes 2 bytes to encode numbers from 16 to 2,047. Giving frequently used fields numbers between 1 and 15 will increase performance quality.

 

// order.proto
message CreateOrderRequest {
    int64 user_id = 1;
}

// main.go
request := CreateOrderRequest{
    UserId: 65
}
// send a request via gRPC

 

The request object is marshalled by the protocol buffer (http://mng.bz/D49n) into []byte to be able to be sent over gRPC. Marshalling results in some bytes containing encoding information of the metadata and the data itself (see figure 3.1):
1 The metadata section is expressed with 1 byte and has the first three bits for denoting the wire type: 000, which is type 0 (Varint) since our data type is int. (You can see the whole list here: http://mng.bz/QPV6.)
2 The first bit of the data section is called the most significant bit (MSB), and its value is 0 when there is no additional byte. Its value becomes 1 if more bytes come to encode the remaining data.
3 The remaining bits of the metadata section contain the field value.
4 The data section contains the MSB (i.e., a continuation bit) to state whether there are more bytes.
5 The remaining seven bits are used for the data itself.

 

A field’s value can be anything based on your needs, and thus cannot affect performance. However, we can affect performance by following some rules for field numbers. For example, you can use numbers less than or equal to 15 for field numbers since that is the maximum number a metadata block can store. More metadata blocks are needed to express a specified field number. In the same way, if you want to store a data value greater than 127 (the maximum capacity of a data block), you need more bytes to fit that value in those data blocks.

 

Let’s see how to encode an object with a value greater than 127. Let’s say that the CreatePaymentRequest message has only one field, user_id, with type int, and field number 2. We compiled this message and used it in our production code:

// order.proto
message CreatePaymentRequest {
    int64 user_id = 2;
}
// main.go
request := CreatePaymentRequest {
    UserId: 21567
}
// send a request via gRPC

The Protobuf compiler will marshal the request object into a []byte, and the metadata section will be encoded, just like the previous example and as visualized in figure 3.2.

The data section will be handled this way:
1 Convert the decimal value 21567 to a binary value: 101010000111111.
2 Split the binary value into seven-bit blocks: 0000001-0101000-0111111.
3 The seven-bit block is for data, and the last bit will be used to store MSB.
4 Reverse the order of data parts (https://betterexplained.com/articles/understanding-big-and-little-endian-byte-order/), which will result in 0111111-0101000-0000001.
5 Since there are three data parts here, the first will have the MSB as 1, the second as 1, and the 3rd as 0 since no more bytes come after that.