Posts
301
Comments
28
Trackbacks
210
God made integers, all the rest is the work of the devil

So, working with Yukon at the moment and wasn't having a very good week. Finally got some code working , but some of my conversions weren't giving the right values. This is GIS stuff, so I'm dealing with Latitude & Longitude, converting to decimal values, calculating distances etc. I couldn't work out why things weren't right, and spent hours debugging. Finally, with Alex's help, we realised I was using the wrong type - I should have been using decimal to preserve accuracy in calculations.

Now I understand rounding and the instrinsic problems of storing floating point numbers, but it's just so painful having to do lots of type conversions/casting just so you can get accurate numbers. I mean I'm only dealing with a few decimal places so you'd kinda expect things to be accurate, but oh no. As a good example, start a project in VS. Doesn't matter what type, but break into the debugger. View the immediate window and type 9.2-9 - what do you expect? By and large I'm a fairly optimistic guy, and although my maths skills are pretty poor, even I knew it should be 0.2. But I was wrong. Now call me a pedant. Call me stupid. Call me naive, but don't call me wrong for wanting to believe that such a simple calculation should give an incorrect answer. At what level should we expect rounding errors to occur?

On a side note (and perhaps not seriously, but then again perhaps I am serious), why is it that we have rounding errors at all? Why is there any need to store floating point numbers as actual floating point numbers? After all, they could be stored as integers, all calculations could be done on intergers and accuracy would be preserved. The decimal point is really only needed for display purposes. Of course, it would mean radical changes to every computing platform, but heck, there's no gain without large restructing of the world as we know it.

posted on Sunday, July 18, 2004 3:04 PM Print
Comments have been closed on this topic.