Go - Encoding Data to a Customized Binary Format

Problem: You want to encode struct data to a customized binary format.

Solution: Design your customized format and use the encoding/binary package to write data in structs to it.

 

Using gob has a couple of drawbacks. First, gob is supported by Go only and works best if both sender and receiver are written in Go. Second, gob stores the whole struct, labels and all, which makes the encoded binary data relatively large. In fact, there is no difference between the size of a piece of JSON data compared to the size of a piece of gob data when they have the same content!
An alternative is to strip away the labels; for example, the Meter struct can be stored this way. Figure 10-1 shows how data for the Meter struct will be stored.

Remember uint8 is 1 byte, uint16 is 2 bytes, uint32 is 4 bytes, float32 is 4 bytes, float64 is 8 bytes and uint64 is 8 bytes. You don’t need the labels if you know the positions of the values.

Writing this format is surprisingly easy. You simply use binary.Write to write the data from the struct instance into this format:

 

The first parameter is the writer you want to write to; in this case, it’s a file. The second parameter is the byte order for format. The encoding/binary supports both big-endian and little-endian, and in this case, you are using big-endian. The last parameter is the struct instance you’re taking the data from.

If you look at the file that’s created, it’s just 24 bytes, as opposed to the earlier gob format, which turned out to be 110 bytes. That’s a significant reduction if you’re moving smaller packets of data over a low-bandwidth network.

 

You might have thought it would encode faster but it’s only slightly better than encoding in JSON, and gob encoding beats it by quite a bit. In addition, it takes up more memory doing the job.

 

func main() {
    var reading Meter = Meter{
        Id:        123456,
        Voltage:   229.5,
        Current:   1.3,
        Energy:    4321,
        Timestamp: uint64(time.Now().UnixNano()),
    }

    file, err := os.Create("data.bin")
    if err != nil {
        log.Println("Cannot  create  file:", err)
    }
    defer file.Close()

    err = binary.Write(file, binary.BigEndian, reading)
    if err != nil {
        log.Println("Cannot  write  to  file:", err)
    }
}

While binary.Write is the easiest way to encode customized binary data, you can also use the encoding/binary package to encode a struct instance manually. Here’s how this can be done:

func main() {
    var reading Meter = Meter{
        Id:        123456,
        Voltage:   229.5,
        Current:   1.3,
        Energy:    4321,
        Timestamp: uint64(time.Now().UnixNano()),
    }

    file, err := os.Create("data.bin")
    if err != nil {
        log.Println("Cannot  create  file:", err)
    }
    defer file.Close()

    buf := make([]byte, 24)
    binary.BigEndian.PutUint32(buf[0:], reading.Id)
    binary.BigEndian.PutUint32(buf[4:], math.Float32bits(reading.Voltage))
    binary.BigEndian.PutUint32(buf[8:], math.Float32bits(reading.Current))
    binary.BigEndian.PutUint32(buf[12:], reading.Energy)
    binary.BigEndian.PutUint64(buf[16:], reading.Timestamp)
    file.Write(buf)
}

It seems like a lot of work, and the file size remains the same, so check out the performance:

It’s a world of difference! The performance is significantly better, and it uses less memory than the Write alone.