Quantcast
Channel: Hot Weekly Questions - Web Applications Stack Exchange
Viewing all articles
Browse latest Browse all 9788

Google Sheet: calculation/math inaccuracy [duplicate]

$
0
0

I found weird math error. I was doing calculation that comes down to this:=TRUNC(253-0.81*300) This comes back as 9, but should be 10. For some reason, when you increase the visible decimal places, this number =253-0.81*300 comes as 9.9999999999999700. Did some more testing and =0.81*300 is exactly 243, only after subtracting that number from 253 it gives 9.999s instead of 10. When subtracting from 254 it gives clean 11. It seems the wrong results come down to a range from 10 to -9, so from 253-0.81*300 to 234-0.81*300.

Is this a known problem?


Viewing all articles
Browse latest Browse all 9788

Latest Images

Trending Articles

<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>