winterhell's picture

[Solved] Precision issues with HalfFloat attribites

I'm running out of vram for geometry which results in heavy streaming via PCI Express, slowing each frame with 10ms on average.
So I changed the vertex position and normal format to Vector3h. While it improves the streaming issue, it also introduces unexpectedly high errors.

I'm generating them with new Vector3h(oldVector3), and after that accordingly changing the sizes for making the VBOs and Atrribute Bindings.
The shaders have in vec3 at all times regardless of atributes precision( dont know if this has any effect)
On the pic, the 1st torus is with full precision for both normals and positions, the second one is with half for both normals and positions, and the third is half for normals, full for position.
The torus itself has an 8 unit radius.

Halfs are not supposed to have precision loss higher than 1 unit until their value is higher than 256.0 or so.

Inline Images


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Frassle's picture

Half only have 11 bits of precision compared to singles with 24 bits. They're less precise at all magnitudes not just above 256 (if you only care about integers they can represent all integers exactly up to 2048).

You may have to reduce your vertex count so that you can use singles if half's aren't precise enough.

winterhell's picture

The error I'm getting is orders of magnitude higher than what one would expect.
Input values are in range of -10 to +10(thats how big the image is)
For many of the values, indeed the error is less than 1% difference between the 32 and 16 bit precision.

Tried converting to half with both Half and Vector3h and then back/rendering with the new 32-bit values but the same errors persisted, thus the problem seems to be in the float to half conversion inside the OpenTK libraries.

I've outputted to file the values whose' conversions result in higher than 1 unit error
for reference the code is

   for (int i = 0; i < numVerts; i++)
                if (Math.Abs(temp_vert[i].Position.Y - new Half(temp_vert[i].Position.Y)) > 1)
                    Log.Write("" + temp_vert[i].Position.Y + "  " + (new Half(temp_vert[i].Position.Y)) + "  " + (temp_vert[i].Position.Y - new Half(temp_vert[i].Position.Y)));

Here is an extract from the log, left column is the original 32bit float, the next is the new half float, and the 3rd is the difference

7.9989  4  3.9989
7.999022  4  3.999022
-7.998439  -4  -3.998439
-7.998956  -4  -3.998956
-7.999104  -4  -3.999104
-7.999101  -4  -3.999101
-7.999967  -4  -3.999967
-8  -4  -4
-7.999992  -4  -3.999992

And with some bigger numbers

128  64  63.99999
31.99999  16  15.99999
127.9786  64  63.97855

How does 7.9 round to 4.0 ?

Edit: It rounds to half the number even smaller ones like -0.4998922 to -0.25 and -1.999558 to -1.

Is there an exponent conversion problem?

the Fiddler's picture

The relevant code is here:

And the original C++ code:

The original code is returning a short, while OpenTK is returning a ushort. It is unclear if this makes a difference - we need a unit test for this.

winterhell's picture

Changing the type thorough the source does not seem to affect the end result.

Frassle's picture

I'm fairly convinced this is an issue in your code. I just ran tests to check the following:

Every possible half value when converted to a float and back results in the same value.
Every single half value when converted to a float prints the same value (i.e. half.ToString and ((float)half).ToString() are the same)
Every integer between -2048 and +2048 when converted to a float (float A) and then to a half and float again (float B), A and B are equal.

Certainly -8 as a float does roundtrip through half back to -8, I'm not sure how your getting -4.

@Fiddler, is there a preference on how to do unit tests? Currently I'm just using MSUnit as its built into Visual Studio, but if you want to mandate a test suite to use I can clean this up and pull req it.

winterhell's picture

Can you try with new Half(7.9989f) ?
I did mention most of the values are transformed correctly. 7.9989 here goes to 4, but 7.99 to 7.988281(correct) and 8.0 to 8
Inputting doubles instead of singles gets the same error.
I'm using the 2014-05-19 stable release, 64-bit Windows 7.

the Fiddler's picture

@Frassle, no preference as long as the unit-test framework is cross-platform. NUnit or NUnitLite probably fit the bill.

Edit: NUnitLite can be included as source in the test project, so that's probably preferable to adding another binary to the Dependencies/ folder.

Frassle's picture

Ah yup found it! Nice spot on finding this! Pull request opened to fix it (

winterhell's picture

Now it renders correctly. Thanks.

the Fiddler's picture

Cheers, merged.