Go - Avoiding Test Fixtures in Performance Tests

Problem: You want to customize the performance tests to avoid benchmarking test fixtures.

Solution: You can start, stop, and reset the benchmark timers using the StartTimer , StopTimer , and ResetTimer , respectively. This will allow you the flexibility to avoid test fixtures.

 

As with functional tests, you will sometimes need to set up test fixtures before running the performance test. Here is an example where you want to take an image file and flip it. In this example, you want to flip a PNG-format image of the Mona Lisa.

The algorithm is easy, and you first load the image file into a 2D grid of pixels (where a pixel is represented by a color.Color struct).

The specific code you want to test is the flip function that takes the grid and flips the pixels in them:

func flip(grid [][]color.Color) {
    for x := 0; x < len(grid); x++ {
        col := grid[x]
        for y := 0; y < len(col)/2; y++ {
            k := len(col) - y - 1
            col[y], col[k] = col[k], col[y]
        }
    }
}

The test fixture you need to set up is to take a PNG file and load it into a grid to get it ready for testing:

func BenchmarkFlip(b *testing.B) {
    grid := load("monalisa.png")
    for i := 0; i < b.N; i++ {
        flip(grid)
    }
}

Let’s run this benchmark function:
% go test -v -bench=Flip -run=XXX

goos: darwin

goarch: arm64

pkg: github.com/sausheong/gocookbook/ch19_benchmarking

BenchmarkFlip

BenchmarkFlip - 10 6492 184067 ns/op

PASS

ok github.com/sausheong/gocookbook/ch19_benchmarking 1.538s

 

You can see that it takes around 184,067 nanoseconds (or around 184 microseconds) on average to do this. But wait, there is an issue here because the benchmark timer starts at the beginning of the benchmark function, which means this timing includes the setup activity of loading up the file. To overcome this, you need to reset the timer after doing the setup activities by calling b.ResetTimer() after loading the PNG file:

func BenchmarkFlip(b *testing.B) {
    grid := load("monalisa.png")
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        flip(grid)
    }
}

Run it again and see what happens:
% go test -v -bench=Flip -run=XXX

goos: darwin

goarch: arm64

pkg: github.com/sausheong/gocookbook/ch19_benchmarking

BenchmarkFlip

BenchmarkFlip - 10 6618 181478 ns/op

PASS

ok github.com/sausheong/gocookbook/ch19_benchmarking 2.338s

The timing is now about 181 microseconds instead, about 3 microseconds difference.

This is great for setting up before you get into the benchmarking loop. For example, you need to actually do something every iteration and not just once before the start of the loop:

func BenchmarkLoadAndFlip(b *testing.B) {
    for i := 0; i < b.N; i++ {
        grid := load("monalisa.png")
        flip(grid)
    }
}

Take a look at the timing:
% go test -v -bench=LoadAndFlip -run=XXX

goos: darwin

goarch: arm64

pkg: github.com/sausheong/gocookbook/ch19_benchmarking

BenchmarkLoadAndFlip

BenchmarkLoadAndFlip - 10 69 14613379 ns/op

PASS

ok github.com/sausheong/gocookbook/ch19_benchmarking 1.232s

 

This took about 14.6 milliseconds. To ignore the timing for loading the image file, you need to stop the timer before calling load and start the timer afterward. For this, you can use the StopTimer and StartTimer to control which parts of the iteration you don’t want to benchmark:

func BenchmarkLoadAndFlip(b *testing.B) {
    for i := 0; i < b.N; i++ {
        b.StopTimer()
        grid := load("monalisa.png")
        b.StartTimer()
        flip(grid)
    }
}

Run it again and see the results:

% go test -v -bench=LoadAndFlip -run=XXX

goos: darwin

goarch: arm64

pkg: github.com/sausheong/gocookbook/ch19_benchmarking

BenchmarkLoadAndFlip

BenchmarkLoadAndFlip - 10 1540 672674 ns/op

PASS

ok github.com/sausheong/gocookbook/ch19_benchmarking 23.672s

The timing for the flip function is about 0.67 milliseconds. However, the entire performance test took a lot longer, almost 24 seconds! This is because the flip function is much faster than the load function, and therefore you could do many more iterations (1,540 compared with 69 earlier). However, even though you are not considering the timing for the load function in the performance test, it still executes and takes much longer.