FORUMS: list search recent posts

Strange maths result..

COW Forums : Adobe Flash

<< PREVIOUS   •   FAQ   •   VIEW ALL   •   PRINT   •   NEXT >>
Nick Rosier
Strange maths result..
on Feb 10, 2011 at 1:45:14 am

So got a very odd result from a maths operation:

var myNumber = 6.3 + 9.1;
trace(myNumber);


returns not 15.4, as you would expect, but 15.399999999999999

any ideas?


Return to posts index

demetri tashie
Re: Strange maths result..
on Feb 10, 2011 at 2:45:25 am

i'm not going to pretend to really understand why, but it is a common phenomena not just in Actionscript, but in ALL computer languages. it will show up when you least expect it.

it is a well documented mathematical phenomena involving Bytes and floating decimals - with everything actually taken out to the 16th place.

run something like this and see what happens:
var i:Number=0.1;

function addI():void {
i=i+.1
trace(i);
addI();
}

addI();


most often, this is not a problem because 15.399999999999999 really IS 15.4, for that .0000000000001 is SO infinitely small ( think about it).

if it is critical to present the result of these calculations, always round to whatever decimal place you need


Return to posts index

Nick Rosier
Re: Strange maths result..
on Feb 10, 2011 at 11:34:34 am

thanks for getting back to me - would agree that it's irrelevant in most scenarios, but unfortunately I'm coding a scientific calculator. shall find a way to count decimal places in a sum and round accordingly


Return to posts index



<< PREVIOUS   •   VIEW ALL   •   PRINT   •   NEXT >>
© 2018 CreativeCOW.net All Rights Reserved
[TOP]