Hacker News new | past | comments | ask | show | jobs | submit login

2 with 64 bit floats, 0 with 32 bit floats.



I could understand 0, but how does it get 2?


FP numbers are (roughly) stored in the form m×2^e (m = mantissa, e = exponent). When numbers can't be represented exactly, m is rounded. My guess is that these numbers end up being encoded as 4999999999999999×2 and 4999999999999999.5×2, where the latter is rounded up to 5000000000000000×2.


The nearest doubles to each of these two decimal constants end up being roughly 2 apart. Whereas for fp32 both decimal constants are stored as the same float.


Because 9999999999999999 is rounded to 10000000000000000 before any math even happens. Precision != order of magnitude.


These integers are so large that they cannot be precisely represented by 32 bit or 64 bit floats. So there's a rounding effect. https://stackoverflow.com/a/1848953


because 2 is the interval of precision at that scale. In floating point loss of precision scales with magnitude.


x999 rounds up when stored, x998 stays the same.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: