Fix Mod and Div precision/scale issues in ColumnStore (MCOL-4601) #3850
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Root Cause:
• The divisor was truncated to an integer (div), losing its scale. May not work is the interger part is zero. (e.g: 0.5)
• Modulus computation used integer arithmetic instead of scaled decimal arithmetic.
• Precision and scale were not correctly aligned between the dividend and divisor.
Solution:
• Retrieve both operands as IDB_Decimal values using getDecimalVal().
• Normalize the scales of both operands using max(d.scale, div.scale).
• Perform modulus on properly scaled 128-bit integer representations.
• Return a result with consistent scale and divisor precision.
Root Cause:
• Operands were converted to double, losing precision for DECIMAL values.
• Scale differences between dividend and divisor were not handled.
• Also, double cannot exactly represent 1.1 in binary.
1.1 in base 10 equals:
1 + 1/10 = 1 + 0.0001100110011001100...(in binary)
That repeating binary fraction never ends.
When stored in a finite 64-bit double, it gets rounded to the closest representable number, which is actually: 1.1000000000000001
Solution:
• For exact decimal display, Its better we use the DECIMAL, not double.
• When printing doubles, format to a fixed precision: 1.1000000000000001
• Retrieve operands as IDB_Decimal instead of double.
• Normalize scales between dividend (dec1) and divisor (dec2) before performing division.
• Use 128-bit integer arithmetic for precise scaled division.