Your opinion on my precision number handling.

calin92540842 profile image PDS OWNER CALIN (Calin Baenen) ・1 min read

When calculating big, precise numbers, I handle them using special methods to break them up and make them human readable. But I also apply math. Though the method I have is sloppy, so would it matter if a precise number like 75^5.275 (~7779436933) had one (1) subtracted from it? Since the method I'm using is flooring the numbers, and is subtracting ~0.295039 almost every time.

Is this ok, or should I try to improve the accuracy rate?


markdown guide

Well.. depends on where you plan to use those numbers. I work in statistics so in my case it would be a "No. You need to be more accurate, just randomly subtracting a person would be wrong".
Does your use-case even need the higher precision? If it's about human readability you could also leave one or two positions after the dot: 7779436933.29 is perfectly readable. If you still want to go to whole numbers you could just leave a note somewhere in your user interface that numbers have been rounded and may not be 100% accurate.


I was planning on using this in a project that involves the human population. Thanks for your response, Alexander, it's a great help.