Repository navigation
[fxp] Round half-even values below one unit to ±1, not 0. - #7796
Conversation
|
LGTM. The one-line fix makes What I ran
Missing (medium): no SQL-level regression test. The description says this changes |
mihaibudiu
left a comment
There was a problem hiding this comment.
Does this affect gen2?
I assume so, since I imagine it uses fxp |
i128_mul_pow10_round_even() stepped away from zero by the sign of the truncated quotient. When the quotient was 0, the step was 0, so values such as 0.6 at scale 0 rounded to 0 instead of 1. Step by the sign of the value instead, as round_inner() does. Add tests in fxp and in the SQL compiler, for CAST of VARCHAR to DECIMAL and of DECIMAL to TIMESTAMP. Fixes #7717. Signed-off-by: Ben Pfaff <blp@feldera.com>
daba26f to
7f9fee6
Compare
|
Thanks for the review. I added the SQL-level regression test in
With only the fix line reverted, the test fails: ±0.6 give -- Claude |
i128_mul_pow10_round_even()(crates/fxp/src/lib.rs) stepped away from zero by the sign of the truncated quotient. When the quotient was 0, the step was 0, so values such as 0.6 at scale 0 rounded to 0 instead of 1. This change steps by the sign of the value instead, asround_inner()already does.This affects every half-even path in fxp:
Fixed::from_str,convert_round_even,new_round_evenandDynamicDecimal::into_fixed_round_even. In SQL, it affectsCAST('0.6' AS DECIMAL(5, 0))and DECIMAL to TIMESTAMP casts.The new test
round_even_below_onefails without the fix and passes with it.Fixes #7717.