## DEV Community π©βπ»π¨βπ» is a community of 963,864 amazing developers

We're a place where coders share, stay up-to-date and grow their careers.

Thibaut Andrieu

Posted on • Originally published at kissyagni.com

# Floating Point Precision in Opengl, Vulkan and 3D in General β Part 1

When zooming a lot on 3D objects or when having small object lost in a very large scene, it can happen that your objects βjiggleβ from one position to another, making rendering really shaky. We will see in this series of article where does it comes from and how to workaround it without using double precision everywhere.

## What we would like

Lets think in 2D. We have an X/Y coordinate system. When we specify the (x, y) coordinate of some point, we would like this point to be located at (x, y):

## What we have using integer coordinate

What would happen if we were using integer coordinate system ? When locating a point in (1.5, 1.3), its position would be rounded to (1, 1). And if we move this point, it will jump from position to position:

## What we have using float coordinates

Floating points suffer from the same issue than integer values. Only a set of position are accessible, the difference is that, near to 0, a lot of position are possible. But the further away from zeros, the further the possible positions are. Instead of having regular grid of position like for integers, we have a logarithmic grid:

What does it means ? It means that for small scene you wonβt see your point jumping from one position to another. There are enough precision, enough possible position so that your point correspond to what you expect. But for large scene, object far from the origin will not have a lot of possible position and will jump from one point to another.

## What does βlarge sceneβ means ?

Iβm talking about βlarge sceneβ, but what does it means practically ? As a first approximation, you can consider your scene as large if you need more that 6 digit precision to locate your objects. I will detail why 6 digit later.

So a large scene does not depend on the absolute size of your scene but the ratio between your smallest object and the whole scene. A solar system can be consider as a small scene: Distance from Earth to the Sun is around 150 millions km, or 150.10e6 km. Meaning with 6 digit precision, assuming heliocentric coordinates, you can locate the earth position +/- 100km in space. If your goal is just to have an overview of the main planets in space, then this precision is enough. No one will notice earth is actually 100km to the left when looking at the whole system.

However, if you want to localize the International Space Station which orbit at ~400km altitude, and zooming on it, then 100km precision is not enough. You cannot just use heliocentric coordinates with single floating point precision. When animating your station, it will jump from one position to another.

## Where does these β6 digitsβ comes from ?

Computers does not have infinite amount of memory, and thus cannot store infinite amount of digits. Furthermore, they work only with power of 2, but lets thinks with decimal system for now.

Base 10 precision

To represent a floating value using finite precision, you can store a certain amount of digit times a power of 10. For example, to store the value β1.5β, you will store the value β000015 * 10^-01β. In this case, 000015 is called βthe fractionβ and ^-1 is called βthe exponentβ.

Here is a few other example:

• 150 => 000015 * 10^01
• 150 000 000 000 => 000015 * 10^10
• 0.15 => 000015 * 10^-02

I think you get the point. Values are stored as M * 10^n, with M having a fixed size of 6 digits and n an exponent of size 2.

But what happen when using more than 6 digit precision ?

• 123 456 789 => 123456 * 10^3 => 123 456 000
• 0.123 456 789 => 123456 * 10^-9 => 0.123 456 000

You will just βdropβ them, as it cannot store them in our representation. Note that exponent is also clamped. As we cannot store more than 2 digits for the exponent, we cannot store value higher than 999999 * 10^99.

Base 2 precision

Now, lets go back to power of 2. As you know, computer use base 2 representation, not base 10. Number of base 2 digit they can store is defined by the IEEE 754 standard, which gives for 32bit floating point type:

1 bit for sign (>0 or <0), 8 bits for exponent (so up to ^255) and 23 digit. But where does the β6 digit in base 10β comes from ? To understand it, you have to ask βhow many base 2 digits is required to represent a base 10 digitβ ? The answer is:

So, 23 base 2 digits can represent ~6.9 base 10 digits:

It is not easy to understand what is β6.9β digits after all. A method I use is that, as a very unprecise approximation, you need 3 bits to count up to 10 (111 = 7 ~=10). 3 binary digits represent 1 decimal digit. So 23 binary digit represent 23/3=7.6 decimal digit. As we know this is a very rough approximation, lets get safe and take only 6 decimal digit precision.

## Using Double precision

Double precision are like single Float precision, except they use 11 bits for exponent and 52 bits for fraction, which can represent ~15 decimal digits precision instead of 6.

But using double precision has a cost regarding memory consumption, performances and legacy code refactoring effort.

I hope you better understand where does this problem comes from. I will develop in next articles the problem of floating precision when accumulating transformation matrices.