← 输入框|github.com|刷新
Skip to content

[adapters] Keep the JSON text of floats stable - #7735

Merged
swanandx merged 1 commit into
mainfrom
json-float-ryu
Oct 31, 2026
Merged

swanandx merged 1 commit into
mainfrom
json-float-ryu

Conversation

@swanandx

@swanandx swanandx commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

serde_json 1.0.149 and later write floats differently. Any update of serde_json brings this in.

serde_json 1.0.145 (main) 1.0.149+ 1.0.149+ with this PR
JSON output of 1e20 1e20 1e+20 1e20
Delete after resume from a checkpoint Correct Old row stays, plus a negative row Correct

TO_JSON text can be stored in pipeline state. If the text changes, a delete does not match the stored row. This PR writes floats with ryu, as earlier serde_json and CAST to VARCHAR do.

Describe Manual Test Plan

I measured both rows of the table on real pipelines. The new tests fail without this PR on serde_json 1.0.154.

Breaking Changes?

None.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM. Here is what I ran:

Check Result
cargo test -p feldera-types float_format, -p feldera-sqllib --lib variant (97 tests), -p static-connectors json_output_keeps_float_text, with the locked serde_json 1.0.145 pass
Same tests after cargo update -p serde_json --precise 1.0.149 pass
JsonSerializer reverted to serde_json::Serializer::new on 1.0.149 fails as it should ([1e+20,3.4e+38] vs [1e20,3.4e38])
pre-commit run --files <changed> pass

I also traced the paths. Row output for every JSON-encoded sink goes through seroutput::JsonSerializer. TO_JSON and VARIANT JsonString serialization go through Variant/FlatVariant::to_json_string. All of these now use the new formatter. ryu is already a workspace dependency and is licensed Apache-2.0/BSL-1.0 (Boost), so that's fine.

  • tech-debt (medium): Nothing stops new code from serializing SQL values with plain serde_json::to_string or Serializer::new, which would bring the checkpoint-divergence bug back after the iceberg bump. Consider a clippy.toml disallowed-methods entry for sqllib and static-connectors, or at least a note in the JsonFloatFormatter doc comment.
  • missing (low-medium): No end-to-end SQL test checks TO_JSON(CAST(1e20 AS DOUBLE)) or a DOUBLE/REAL column in JSON sink output. The current tests are unit-level only, and they only bite once serde_json reaches ≥1.0.149. They also don't cover -0.0 or a REAL value through JsonSerializer.

serde_json 1.0.149 writes 1e20 as 1e+20. Write floats with ryu, as
earlier serde_json and CAST to VARCHAR do, in JSON output and TO_JSON.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM. Here is what I ran:

Check Result
cargo test -p feldera-types float_format, -p feldera-sqllib --lib variant::test, -p static-connectors json_output_keeps (lock at serde_json 1.0.145) pass
Same tests after a local cargo update -p serde_json to 1.0.154, which pulls in zmij (I restored the lockfile afterwards) pass, so the formatter does override the new float output
pre-commit run --files <changed> (clippy, fmt, machete, crypto check) pass
  • I looked for other places that write floats with plain serde_json into state or keys and found none outside tests. The JSON encoder's key path re-parses with arbitrary_precision, which keeps the original text. Number::from_f64 only shows up in tests.
  • The dependency is fine: ryu is Apache-2.0 OR BSL-1.0 (Boost) and was already a transitive dependency.
  • Missing (medium): no automated checkpoint→resume→delete test covers the failure the PR describes. The unit tests guard the text only, and the delete behaviour rests on the manual test. A TO_JSON(CAST(1e20 AS VARIANT)) row in a pipeline test that is deleted after a resume would lock it down.

@swanandx
swanandx requested review from blp and mihaibudiu and removed request for blp October 30, 2026 14:41

@blp blp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like Fred's idea of adding a way to warn if the normal serde_json gets used (I haven't tried anything like that before though).

I hadn't known about serde_json::ser::Formatter. Good to know.

@swanandx
swanandx added this pull request to the merge queue Oct 30, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 30, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Verdict: infrastructure, not this PR. Fairly confident.

The merge queue run's Runtime Integration Tests (gke-runners-amd64) job dropped the PR, but all 9 in-flight failures are unrelated tests (adhoc queries, adaptive joins, clock advance/timezone, checkpoint runtime upgrade, lateness) that failed at the same instant with the same error:

feldera.rest.errors.FelderaAPIError: FelderaError. Error message: upstream connect error or disconnect/reset before headers. reset reason: remote connection failure
Response Status: 503
  • Job: https://github.com/feldera/feldera/actions/runs/41137819254/job/124345935880
  • Tests started normally at 22:28:43, then every worker's request hung until 22:34:31 (~5:48 later) and all failed together — an ingress/control-plane connectivity blip on the shared amd64 test cluster, not something any test or the pipeline manager did.
  • This PR only touches crates/feldera-types, crates/sqllib, crates/static-connectors (stable JSON float formatting) — nothing related to pipeline provisioning or the test infra.

This is the same failure class as #7465 (two prior occurrences); I added this run as a third data point there.

Next step: re-queue — this should pass on a retry.

@mihaibudiu
mihaibudiu added this pull request to the merge queue Oct 30, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 31, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Verdict: CI infrastructure failure, not this PR. Confident.

The only job GitHub reported as failure — Unit Tests / Rust Unit Tests (gke-runners-amd64, x86_64, x86_64-unknown-linux-gnu) — never reached any test. It was mid-download of build artifacts when the runner itself died:

##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

Filed #7761 to track the runner eviction.

Next step: re-queue — this should pass on a retry.

@swanandx
swanandx added this pull request to the merge queue Oct 31, 2026
Merged via the queue into main with commit dbb5357 Oct 31, 2026
1 check passed
@swanandx
swanandx deleted the json-float-ryu branch October 31, 2026 15:46
swanandx added a commit that referenced this pull request Nov 2, 2026
The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its scans now also prune Parquet pages with the page index.
Nested columns in files without field ids now read as Spark reads them.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of #7735.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
swanandx added a commit that referenced this pull request Nov 2, 2026
The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its snapshot reads now run as a DataSource, prune Parquet pages with
the page index, and unescape LIKE prefixes before pushdown. A local
check of 22 snapshot filters, with LIKE, ranges, IN, NOT and OR, gave
the right rows.
Nested columns in files without field ids now read as Spark reads them.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of #7735.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
swanandx added a commit that referenced this pull request Nov 2, 2026
The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its snapshot reads now run as a DataSource, prune Parquet pages with
the page index, and unescape LIKE prefixes before pushdown. A local
check of 22 snapshot filters, with LIKE, ranges, IN, NOT and OR, gave
the right rows.
Nested columns in files without field ids now read as Spark reads them.
A new test reads a table made with add_files, and the changelog
notes the change.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of #7735.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
swanandx added a commit that referenced this pull request Nov 2, 2026
The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its snapshot reads now run as a DataSource, prune Parquet pages with
the page index, and unescape LIKE prefixes before pushdown. A local
check of 22 snapshot filters, with LIKE, ranges, IN, NOT and OR, gave
the right rows.
Nested columns in files without field ids now read their real values.
A new test reads a table made with add_files, and the changelog
notes the change.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of #7735.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
swanandx added a commit that referenced this pull request Nov 2, 2026
The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its snapshot reads now run as a DataSource, prune Parquet pages with
the page index, and unescape LIKE prefixes before pushdown. A local
check of 22 snapshot filters, with LIKE, ranges, IN, NOT and OR, gave
the right rows.
Nested columns in files without field ids now read their real values.
A new test reads a table made with add_files, and the changelog
notes the change.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of #7735.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
swanandx added a commit that referenced this pull request Nov 2, 2026
The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its snapshot reads now run as a DataSource, prune Parquet pages with
the page index, and unescape LIKE prefixes before pushdown.
Nested columns in files without field ids now read their real values.
A new test reads such a table, with a page index, under 8 filters.
The changelog notes the change.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of #7735.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
swanandx added a commit that referenced this pull request Nov 2, 2026
The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its snapshot reads now run as a DataSource, prune Parquet pages with
the page index, and unescape LIKE prefixes before pushdown.
Nested columns in files without field ids now read their real values.
A new test reads such a table, with a page index, under 9 filters.
The changelog notes the change.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of #7735.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
swanandx added a commit that referenced this pull request Nov 3, 2026
The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its snapshot reads now run as a DataSource, prune Parquet pages with
the page index, and unescape LIKE prefixes before pushdown.
Nested columns in files without field ids now read their real values.
A new test reads such a table, with a page index, under 9 filters.
The changelog notes the change.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of #7735.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
swanandx added a commit that referenced this pull request Nov 3, 2026
The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its snapshot reads now run as a DataSource, prune Parquet pages with
the page index, and unescape LIKE prefixes before pushdown.
Nested columns in files without field ids now read their real values.
A new test reads such a table, with a page index, under 9 filters.
The changelog notes the change.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of #7735.
serde_json now writes 1e40 as 1e+40 in parsed number text, so one
test error text changes.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
swanandx added a commit that referenced this pull request Nov 4, 2026
The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its snapshot reads now run as a DataSource, prune Parquet pages with
the page index, and unescape LIKE prefixes before pushdown.
Nested columns in files without field ids now read their real values.
A new test reads such a table, with a page index, under 9 filters.
The changelog notes the change.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of #7735.
serde_json now writes 1e40 as 1e+40 in parsed number text, so one
test error text changes.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
swanandx added a commit that referenced this pull request Nov 4, 2026
The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its snapshot reads now run as a DataSource, prune Parquet pages with
the page index, and unescape LIKE prefixes before pushdown.
Nested columns in files without field ids now read their real values.
A new test reads such a table, with a page index, under 9 filters.
The changelog notes the change.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of #7735.
serde_json now writes 1e40 as 1e+40 in parsed number text, so one
test error text changes.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
Karakatiza666 pushed a commit to AnnaZhand/feldera that referenced this pull request Nov 5, 2026
The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its snapshot reads now run as a DataSource, prune Parquet pages with
the page index, and unescape LIKE prefixes before pushdown.
Nested columns in files without field ids now read their real values.
A new test reads such a table, with a page index, under 9 filters.
The changelog notes the change.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of feldera#7735.
serde_json now writes 1e40 as 1e+40 in parsed number text, so one
test error text changes.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants