Unexpected Modulo Results with Length Values

Dr_Monster

When performing modulo operations with length values (e.g., in millimeters), I'm encountering unexpected results for certain input combinations. For example, 300mm % 50mm (when represented as floating-point numbers) sometimes returns 50mm instead of the expected 0mm. This issue doesn't seem to occur with integer calculations.

I'm trying to determine if this behavior is related to the imprecision of floating-point representation, or if other factors are at play. I'd like to perform modulo operations directly on length values without converting them to integers or using rounding (which could introduce other inaccuracies), while minimizing these errors.

Are there any recommended approaches or best practices for handling modulo operations with length values in a way that avoids or minimizes these inaccuracies? Any insights would be greatly appreciated.

Caden_Armstrong

One way you could do it would be to cast to an integer, but on a much different scale - like a nanometer scale. Scale up the float to x 1e9 and any loss of accuracy is at the sub-nano meter scale, which should be fine.

var result = ((((x.value*1e9 ) as number) % ((y.value * 1e9) as number))/1e9)*meter;

I did a bunch of tests and it seems to work just fine, but then I remembered that casting involves rounding - so this is just rounding with extra steps. Which is what Ilya suggested. Rounding at floating point accuracy scale is fine, no engineering work is done at that level of accuracy. No CAD works at that level of accuracy. Rounding to the nearest whole number is very different to rounding to 1e-12.

When you learn how floating point numbers work you'll find out that some numbers just cannot be represented in a computer. There is always rounding going on, you just aren't aware of it. Theres a reason you have an issue with 300 and probably not one with 200.

tl;dr just round to 1e-12

ilya_baran

Unfortunately, it is floating point math. Onshape uses meters internally, so the expression gets evaluated as (300 * .001) % (50 * .001) — if you, for example, type this into your browser console, you'll see 0.049999999999999975 instead of 0.

The only ways I know to deal with this are rounding (which you said you want to avoid) or adding a small tolerance to the numerator (finding the right one is tricky and, like with rounding, you won't get a "perfect" result). Maybe an option is to compute the quotient using a tolerance — floor((300mm + 1e-8 m) / (50mm)) and then compute the remainder without a tolerance: 300mm - #quotient * 50mm but then there is a risk that the result is a very small negative number.

I'd be curious if others have good suggestions here…

Caden_Armstrong

One way you could do it would be to cast to an integer, but on a much different scale - like a nanometer scale. Scale up the float to x 1e9 and any loss of accuracy is at the sub-nano meter scale, which should be fine.

var result = ((((x.value*1e9 ) as number) % ((y.value * 1e9) as number))/1e9)*meter;

I did a bunch of tests and it seems to work just fine, but then I remembered that casting involves rounding - so this is just rounding with extra steps. Which is what Ilya suggested. Rounding at floating point accuracy scale is fine, no engineering work is done at that level of accuracy. No CAD works at that level of accuracy. Rounding to the nearest whole number is very different to rounding to 1e-12.

When you learn how floating point numbers work you'll find out that some numbers just cannot be represented in a computer. There is always rounding going on, you just aren't aware of it. Theres a reason you have an issue with 300 and probably not one with 200.

tl;dr just round to 1e-12

Dr_Monster

https://forum.onshape.com/discussion/comment/111701#Comment_111701

The solutions you suggested were all helpful and appropriate, and I was able to get the results I wanted. I really appreciate your help.

However, there were some additional things to consider during this process, which made the user experience a little less convenient. I'm taking this opportunity to learn more about this discipline.

I think if these things were made more user-friendly, Onshape could become even better. I've been consistently impressed with Onshape's convenience and usability, and I sincerely hope it continues to grow and improve.