Repository navigation
[adapters] Keep the JSON text of floats stable - #7735
Conversation
|
LGTM. Here is what I ran:
I also traced the paths. Row output for every JSON-encoded sink goes through
|
serde_json 1.0.149 writes 1e20 as 1e+20. Write floats with ryu, as earlier serde_json and CAST to VARCHAR do, in JSON output and TO_JSON. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
f84a56a to
6303596
Compare
|
LGTM. Here is what I ran:
|
blp
left a comment
There was a problem hiding this comment.
I like Fred's idea of adding a way to warn if the normal serde_json gets used (I haven't tried anything like that before though).
I hadn't known about serde_json::ser::Formatter. Good to know.
|
Verdict: infrastructure, not this PR. Fairly confident. The merge queue run's
This is the same failure class as #7465 (two prior occurrences); I added this run as a third data point there. Next step: re-queue — this should pass on a retry. |
|
Verdict: CI infrastructure failure, not this PR. Confident. The only job GitHub reported as
Filed #7761 to track the runner eviction. Next step: re-queue — this should pass on a retry. |
The fork now matches upstream, which merged our reader stall fix. The DataFusion integration crate is renamed back to iceberg-datafusion. Its scans now also prune Parquet pages with the page index. Nested columns in files without field ids now read as Spark reads them. apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of floats does not change because of #7735. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
The fork now matches upstream, which merged our reader stall fix. The DataFusion integration crate is renamed back to iceberg-datafusion. Its snapshot reads now run as a DataSource, prune Parquet pages with the page index, and unescape LIKE prefixes before pushdown. A local check of 22 snapshot filters, with LIKE, ranges, IN, NOT and OR, gave the right rows. Nested columns in files without field ids now read as Spark reads them. apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of floats does not change because of #7735. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
The fork now matches upstream, which merged our reader stall fix. The DataFusion integration crate is renamed back to iceberg-datafusion. Its snapshot reads now run as a DataSource, prune Parquet pages with the page index, and unescape LIKE prefixes before pushdown. A local check of 22 snapshot filters, with LIKE, ranges, IN, NOT and OR, gave the right rows. Nested columns in files without field ids now read as Spark reads them. A new test reads a table made with add_files, and the changelog notes the change. apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of floats does not change because of #7735. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
The fork now matches upstream, which merged our reader stall fix. The DataFusion integration crate is renamed back to iceberg-datafusion. Its snapshot reads now run as a DataSource, prune Parquet pages with the page index, and unescape LIKE prefixes before pushdown. A local check of 22 snapshot filters, with LIKE, ranges, IN, NOT and OR, gave the right rows. Nested columns in files without field ids now read their real values. A new test reads a table made with add_files, and the changelog notes the change. apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of floats does not change because of #7735. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
The fork now matches upstream, which merged our reader stall fix. The DataFusion integration crate is renamed back to iceberg-datafusion. Its snapshot reads now run as a DataSource, prune Parquet pages with the page index, and unescape LIKE prefixes before pushdown. A local check of 22 snapshot filters, with LIKE, ranges, IN, NOT and OR, gave the right rows. Nested columns in files without field ids now read their real values. A new test reads a table made with add_files, and the changelog notes the change. apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of floats does not change because of #7735. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
The fork now matches upstream, which merged our reader stall fix. The DataFusion integration crate is renamed back to iceberg-datafusion. Its snapshot reads now run as a DataSource, prune Parquet pages with the page index, and unescape LIKE prefixes before pushdown. Nested columns in files without field ids now read their real values. A new test reads such a table, with a page index, under 8 filters. The changelog notes the change. apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of floats does not change because of #7735. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
The fork now matches upstream, which merged our reader stall fix. The DataFusion integration crate is renamed back to iceberg-datafusion. Its snapshot reads now run as a DataSource, prune Parquet pages with the page index, and unescape LIKE prefixes before pushdown. Nested columns in files without field ids now read their real values. A new test reads such a table, with a page index, under 9 filters. The changelog notes the change. apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of floats does not change because of #7735. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
The fork now matches upstream, which merged our reader stall fix. The DataFusion integration crate is renamed back to iceberg-datafusion. Its snapshot reads now run as a DataSource, prune Parquet pages with the page index, and unescape LIKE prefixes before pushdown. Nested columns in files without field ids now read their real values. A new test reads such a table, with a page index, under 9 filters. The changelog notes the change. apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of floats does not change because of #7735. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
The fork now matches upstream, which merged our reader stall fix. The DataFusion integration crate is renamed back to iceberg-datafusion. Its snapshot reads now run as a DataSource, prune Parquet pages with the page index, and unescape LIKE prefixes before pushdown. Nested columns in files without field ids now read their real values. A new test reads such a table, with a page index, under 9 filters. The changelog notes the change. apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of floats does not change because of #7735. serde_json now writes 1e40 as 1e+40 in parsed number text, so one test error text changes. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
The fork now matches upstream, which merged our reader stall fix. The DataFusion integration crate is renamed back to iceberg-datafusion. Its snapshot reads now run as a DataSource, prune Parquet pages with the page index, and unescape LIKE prefixes before pushdown. Nested columns in files without field ids now read their real values. A new test reads such a table, with a page index, under 9 filters. The changelog notes the change. apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of floats does not change because of #7735. serde_json now writes 1e40 as 1e+40 in parsed number text, so one test error text changes. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
The fork now matches upstream, which merged our reader stall fix. The DataFusion integration crate is renamed back to iceberg-datafusion. Its snapshot reads now run as a DataSource, prune Parquet pages with the page index, and unescape LIKE prefixes before pushdown. Nested columns in files without field ids now read their real values. A new test reads such a table, with a page index, under 9 filters. The changelog notes the change. apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of floats does not change because of #7735. serde_json now writes 1e40 as 1e+40 in parsed number text, so one test error text changes. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
The fork now matches upstream, which merged our reader stall fix. The DataFusion integration crate is renamed back to iceberg-datafusion. Its snapshot reads now run as a DataSource, prune Parquet pages with the page index, and unescape LIKE prefixes before pushdown. Nested columns in files without field ids now read their real values. A new test reads such a table, with a page index, under 9 filters. The changelog notes the change. apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of floats does not change because of feldera#7735. serde_json now writes 1e40 as 1e+40 in parsed number text, so one test error text changes. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
serde_json 1.0.149 and later write floats differently. Any update of serde_json brings this in.
1e201e+201e20TO_JSON text can be stored in pipeline state. If the text changes, a delete does not match the stored row. This PR writes floats with ryu, as earlier serde_json and CAST to VARCHAR do.
Describe Manual Test Plan
I measured both rows of the table on real pipelines. The new tests fail without this PR on serde_json 1.0.154.
Breaking Changes?
None.