On 13 Jan 2020, at 00:28, Paul DeBruicker <pdebruic@gmail.com> wrote:
Whats wrong with #roundedTo: ?
If you actually need it calculated to 8 significant digits then you can use scaled decimals e.g.
(0.09560268 asScaledDecimal - 0.005 asScaledDecimal)
The weird thing is though, that is still does not behave 100% as one would expect (i.e. it does not seem exact). 0.09560268 asScaledDecimal - 0.005 asScaledDecimal. "0.09060268s8" 0.09560268 asScaledDecimal - 0.005 asScaledDecimal = 0.09060268 asScaledDecimal. "false" 0.09560268 asScaledDecimal - 0.005 asScaledDecimal = 0.09060268s8. "false" The internal fractions are not the same ... While (9560268 / 1e8) - (5 / 1e3) = (9060268 / 1e8). "true" So, is the equality test of ScaledDecimal correct ?
see https://0.30000000000000004.com for more about why floating point math does this sometimes
Donald Howard wrote
Hello everyone,
This is an issue I've had awareness of going way back but it's the first time I've had to personally deal with it. I've done some research and found it addressed in a number of ways but given current time pressures and a huge list of to-dos, I thought I'd turn to the community for rapid advice to get me over the hump. The circumstances are summarized below in code and comments suitable for a playground.
"This is the actual arithmetic I want to do... 0.09560268 - 0.005 = 0.09060268"
"Evaluate this in a playground and you'll see this is what floating point arithmetic produces --the first two are wrong, every one after is correct" 0.09560268 - 0.005 "=> 0.09060267999999999". 0.0956026 - 0.005 "=> 0.09060259999999999". 0.095602 - 0.005 "=> 0.090602". 0.09560 - 0.005 "=> 0.0906". 0.0956 - 0.005 "=> 0.0906". 0.095 - 0.005 "=> 0.09".
"This workaround works" 0.09560268 - 0.005 roundTo: 0.00000001 "=> 0.09060268". "but one has to compute number of digits of precision and according to study above, it's not necessary at precisions >= 0.000001"
"What are other suggestions, workarounds or approaches community has?" "THANKS!"
Don Howard
<https://objectguild.com>; *Founding Member*
dhoward@
+31653139744 (US) 651-253-7024
-- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html