Recently, I compared a few methods for compressing generic unit vectors.
The method used in Cry Engine 3 for compressing normal vectors is irrelevant here, because it has a limitation that the z component cannot be -1. Since it's only designed for view space normal vectors, those vectors should not point away from the view. However, in our system, we need to handle compression of generic unit vectors, whose directions can be arbitrary.
Below are results generated from a unit test with 10,000,000 random unit vectors for each:
Quantized Spherical Coordinates in Jensen's Photon Map (16 bits):
Max. Error: 1.57047
Avg. Error: 0.711124
Optimized Spherical Coordinates (16 bits)
Max. Error: 0.562332
Avg. Error: 0.30557
Half Precision Spherical Coordinates (32 bits):
Max. Error: 0.118694
Avg. Error: 0.0324938
Optimized Spherical Coordinates (24 bits):
Max. Error: 0.0484566
Avg. Error: 0.0158454
In conclusion, the method described in "Encoding Normal Vectors using Optimized Spherical Coordinates" provides the best precision with the same number of bytes.
