Repository navigation
feat: partition arbitrary valid JSON and NDJSON files - #4391
Conversation
|
Note on the latest commit ( While validating this branch we found that serialized The fix intentionally uses only existing mechanisms:
cc @qued as the author of #4291 in case you want to sanity-check the reconstruction interplay. |
End-to-end validation evidenceValidated on an internal single-node platform deployment running this branch's library on the partition API, with the companion plugin change on the ingestion path. Shape suite (60 fixtures, 6 categories): pretty-printed/compact objects (up to 4.8 MB single-line), arrays (3 to 4,000 records), NDJSON (3 to 9,000 lines, crossing the 1 MiB detection bound), serialized-element files, and deliberately-malformed inputs. Result: 60/60 adjudicated PASS - 15,046 elements compared against this branch's expected output at type + full-text level; every valid file partitioned with order and content preserved; every malformed file failed per-file with a clear error and the batch completed. Metadata suite (11 fixtures): all 37 recognized element types round-trip; parent_id references stay consistent across re-hashed element ids (#3365 behavior); links/emphasis, languages, page metadata, ad-hoc metadata keys, CheckBox states, and image_base64 payloads survive byte-exact. Serialized chunked output containing split tables rehydrates fully - Known items observed (all pre-existing or documented): element ids are re-hashed on re-partition (#3365); metadata.coordinates is stripped unless requested via the API's coordinates parameter (verified preserved when requested); BOM-prefixed JSON still raises (documented deferral in this PR). Full fixture/output pack and evaluation reports are linked from the internal companion PR. |
Rerun validation (2026-07-10, stack @ library
|
There was a problem hiding this comment.
All reported issues were addressed across 17 files
Shadow auto-approve: would not auto-approve because issues were found.
Re-trigger cubic
|
Product scope approved for this approach. The two-mode I am also approving the explicit deviations: NDJSON in v1, deprecation warnings for the old processability helpers, and the documented one-record serialized-element routing limitation through Before merge, please address two small items:
With those fixed, I am comfortable with this as the library release basis for 0.25.0. |
Review feedback on #4391 (Sudarshan + cubic): - json.loads accepts NaN/Infinity/-Infinity by default, so those non-standard constants were partitioning into Text elements instead of being rejected as malformed. Add loads_strict_json() (shared by both partitioners) that passes parse_constant to raise, so such input surfaces the canonical 'Not a valid json'/'Not a valid ndjson' ValueError. Covers the top-level constant, a constant nested in an object/array, and NDJSON lines. - Fix the json_partitioning module docstring: it described the empty object as JSON-mode -> no elements, which is stale since {} now yields Text('{}') in both modes. The remaining mode difference is empty arrays only.
There was a problem hiding this comment.
1 issue found across 5 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="unstructured/partition/common/json_partitioning.py">
<violation number="1" location="unstructured/partition/common/json_partitioning.py:45">
P2: Custom agent: **Enforce Pragmatic Test Coverage**
The new `loads_strict_json()` parser is shared by both JSON and NDJSON partitioners and intentionally rejects non-standard constants (`NaN`, `Infinity`, `-Infinity`), but there are no tests verifying this rejection or confirming the resulting error path surfaces correctly through callers. Without coverage, a regression—such as accidentally dropping the `parse_constant` argument—would silently re-allow these constants past the new guard. Please add tests for both the normal parsing path and the rejection of non-standard constants through `partition_json()` and `partition_ndjson()`.</violation>
</file>
Shadow auto-approve: would not auto-approve because issues were found.
Re-trigger cubic
| raise json.JSONDecodeError(f"Non-standard JSON constant: {constant}", constant, 0) | ||
|
|
||
|
|
||
| def loads_strict_json(text: str) -> Any: |
There was a problem hiding this comment.
P2: Custom agent: Enforce Pragmatic Test Coverage
The new loads_strict_json() parser is shared by both JSON and NDJSON partitioners and intentionally rejects non-standard constants (NaN, Infinity, -Infinity), but there are no tests verifying this rejection or confirming the resulting error path surfaces correctly through callers. Without coverage, a regression—such as accidentally dropping the parse_constant argument—would silently re-allow these constants past the new guard. Please add tests for both the normal parsing path and the rejection of non-standard constants through partition_json() and partition_ndjson().
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At unstructured/partition/common/json_partitioning.py, line 45:
<comment>The new `loads_strict_json()` parser is shared by both JSON and NDJSON partitioners and intentionally rejects non-standard constants (`NaN`, `Infinity`, `-Infinity`), but there are no tests verifying this rejection or confirming the resulting error path surfaces correctly through callers. Without coverage, a regression—such as accidentally dropping the `parse_constant` argument—would silently re-allow these constants past the new guard. Please add tests for both the normal parsing path and the rejection of non-standard constants through `partition_json()` and `partition_ndjson()`.</comment>
<file context>
@@ -28,6 +33,20 @@
+ raise json.JSONDecodeError(f"Non-standard JSON constant: {constant}", constant, 0)
+
+
+def loads_strict_json(text: str) -> Any:
+ """`json.loads` that rejects the non-standard constants Python accepts by default."""
+ return json.loads(text, parse_constant=_reject_json_constant)
</file context>
|
Thanks @SudSampath - scope approval much appreciated. Both items done in
One design call worth flagging: I put the strict parse in the two partitioners (the validation gates) rather than in All affected suites green. Good to go as the 0.25.0 basis from my side once CI's clean. |
|
@cubic-dev-ai please re-review |
@simoncoombes I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Shadow auto-approve: would not auto-approve. Auto-approval blocked by 1 unresolved issue from previous reviews.
Re-trigger cubic
There was a problem hiding this comment.
2 issues found across 17 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="unstructured/partition/common/json_partitioning.py">
<violation number="1" location="unstructured/partition/common/json_partitioning.py:47">
P3: Large valid integer documents leak a raw `ValueError` instead of partitioning or returning the route's canonical parse error. Normalize the integer-conversion limit or explicitly handle this parser failure in the strict loader.</violation>
<violation number="2" location="unstructured/partition/common/json_partitioning.py:92">
P2: Exponent-form JSON that overflows a float is emitted as `Infinity`, so valid arbitrary JSON is silently changed into non-JSON text despite strict mode. Preserve numeric precision or reject non-finite parsed floats before formatting.</violation>
</file>
Shadow auto-approve: would not auto-approve because issues were found.
Re-trigger cubic
| # NOTE(simon): sort_keys=True gives deterministic, diffable output; it alphabetizes | ||
| # customer field order -- revisit if source-order fidelity is required. | ||
| try: | ||
| return json.dumps(value, indent=2, sort_keys=True) |
There was a problem hiding this comment.
P2: Exponent-form JSON that overflows a float is emitted as Infinity, so valid arbitrary JSON is silently changed into non-JSON text despite strict mode. Preserve numeric precision or reject non-finite parsed floats before formatting.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At unstructured/partition/common/json_partitioning.py, line 92:
<comment>Exponent-form JSON that overflows a float is emitted as `Infinity`, so valid arbitrary JSON is silently changed into non-JSON text despite strict mode. Preserve numeric precision or reject non-finite parsed floats before formatting.</comment>
<file context>
@@ -0,0 +1,114 @@
+ # NOTE(simon): sort_keys=True gives deterministic, diffable output; it alphabetizes
+ # customer field order -- revisit if source-order fidelity is required.
+ try:
+ return json.dumps(value, indent=2, sort_keys=True)
+ except RecursionError:
+ raise ValueError("JSON value is nested too deeply to partition")
</file context>
|
|
||
| def loads_strict_json(text: str) -> Any: | ||
| """`json.loads` that rejects the non-standard constants Python accepts by default.""" | ||
| return json.loads(text, parse_constant=_reject_json_constant) |
There was a problem hiding this comment.
P3: Large valid integer documents leak a raw ValueError instead of partitioning or returning the route's canonical parse error. Normalize the integer-conversion limit or explicitly handle this parser failure in the strict loader.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At unstructured/partition/common/json_partitioning.py, line 47:
<comment>Large valid integer documents leak a raw `ValueError` instead of partitioning or returning the route's canonical parse error. Normalize the integer-conversion limit or explicitly handle this parser failure in the strict loader.</comment>
<file context>
@@ -0,0 +1,114 @@
+
+def loads_strict_json(text: str) -> Any:
+ """`json.loads` that rejects the non-standard constants Python accepts by default."""
+ return json.loads(text, parse_constant=_reject_json_constant)
+
+
</file context>
|
Findings
Why: uploads often provide Relevant code, if result_file_type == FileType.JSON:
result_file_type = self._disambiguate_json_file_typeand try:
json.loads(file_text)
return FileType.JSONConcrete fix: in the JSON/NDJSON disambiguation path, use an explicit
Why: Relevant code, for line in file_text.splitlines():
if not line.strip():
continue
values.append(loads_strict_json(line))but detection still rejects scalar JSON starts in if text_head.lstrip()[0] not in "[{":
return FalseConcrete fix: update
Why: the PR rejects Relevant code, return json.loads(text, parse_constant=_reject_json_constant)but for line in lines:
json.loads(line)Concrete fix: use the same strict parser semantics in disambiguation, ideally from a shared lower-level utility to avoid awkward dependency direction. |
…dings) Addresses badGarnet's review on #4391 - the partitioners were upgraded but detect_filetype still used old heuristics, so router and partitioner disagreed in edge cases: - F1: an explicit .ndjson extension / application/x-ndjson content-type now wins over the whole-payload-JSON rule in _disambiguate_json_file_type, so a one-record .ndjson with an asserted application/json content-type rehydrates via partition_ndjson instead of becoming arbitrary Text. Compact .json objects still route to JSON (SAP path unchanged). - F2: scalar JSON/NDJSON is now reachable through partition(): .ndjson added to the _TextFileDifferentiator trusted-extension list (.json was already trusted) so scalar .json/.ndjson route to the JSON/NDJSON partitioners rather than TXT. Scope-limited to those extensions; a file misnamed .ndjson containing prose now raises (consistent with how .csv/.md/.tsv/.rst extension-trusting already behaves). - F3: loads_strict_json / _reject_json_constant moved to the low-level unstructured.utils module (re-exported from json_partitioning for existing callers) so file_utils.filetype can use strict parsing without an import cycle; disambiguation now rejects NaN/Infinity consistently with the partitioners. Guardrails re-verified: compact-object -> JSON, {} -> Text('{}'), NaN -> ValueError, FS-2017 gate untouched, #3365 xfail intact. Suites 394 passed + 1 xfailed; test_auto json/ndjson subset 11 passed + 1 xfailed.
There was a problem hiding this comment.
All reported issues were addressed across 5 files (changes from recent commits).
Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Shadow auto-approve: would not auto-approve. Auto-approval blocked by 3 unresolved issues from previous reviews.
Re-trigger cubic
|
All review feedback is addressed and the branch is green:
CI is green and cubic is at 0 issues. Guardrails re-verified: serialized-element arrays/TableChunks still rehydrate, compact-object routing unchanged, malformed input still raises. Could I get a formal approve so this can merge as the |
|
|
||
| if isinstance(value, list) and value and all(is_element_shaped_dict(i) for i in value): | ||
| # -- Branch A: rehydrate serialized Unstructured elements -- | ||
| elements = rehydrate_elements(value) |
There was a problem hiding this comment.
non-blocking: wonder if we should capture ValueError when value resembles elements but can't be parsed then fallback to elements_from_arbitrary_value method.
partition_json() and partition_ndjson() are now two-mode partitioners:
serialized Unstructured element JSON/NDJSON is rehydrated exactly as
before (Branch A), and any other valid JSON/NDJSON is converted to Text
elements containing the pretty-printed JSON (Branch B) instead of being
rejected with a schema error.
- Remove the is_json_processable / is_ndjson_processable pre-gates from
partition_json, partition_ndjson, and auto.partition dispatch; the
branch decision now lives inside the partitioners (elements_from_dicts
yielding at least one element selects rehydration).
- Fix _disambiguate_json_file_type so a payload that parses whole as a
single JSON value is FileType.JSON even on one line; previously a
compact single-line object was misrouted to NDJSON and rejected.
- Preserve error contracts: malformed JSON still raises
ValueError('Not a valid json'); malformed NDJSON lines still raise
ValueError('Not a valid ndjson'). Empty string / {} / [] yield [].
- Keep is_json_processable / is_ndjson_processable defined (legacy
helpers, no longer used as partitioning gates).
- Replace exception-catch mode selection with an explicit shape predicate
(is_serialized_element_dict): a list rehydrates only when every item has
a recognized str 'type', the type's required field ('text' as str /
'checked'), and dict-or-absent metadata. Consequences: corrupt
element-shaped payloads raise a chained ValueError instead of leaking
zlib.error/binascii.Error; customer dicts with a 'metadata' key no
longer crash; mixed element/arbitrary arrays partition whole as
arbitrary JSON (no more silent partial rehydration); non-str 'text'
and unhashable 'type' values route to arbitrary JSON.
- Extract shared unstructured/partition/common/arbitrary_json.py used by
both partitioners (predicate, rehydrate_elements, pretty_json_text,
elements_from_arbitrary_json).
- _disambiguate_json_file_type: bound the probe to 1 MiB (was unbounded
whole-file read), restore file position after reading (detect-then-
partition on the same handle no longer silently returns []), and treat
RecursionError as a parse failure. Deeply nested payloads now raise
ValueError('Not a valid json'/'Not a valid ndjson') from the
partitioners instead of RecursionError.
- Deprecate is_json_processable / is_ndjson_processable
(DeprecationWarning; no longer used for partitioning or detection
routing) rather than deleting them — downstream consumers still import
them.
- Merge the now-identical JSON/NDJSON dispatch blocks in auto.py.
- CHANGELOG: document the above plus the intentional asymmetries
(partition_json('{}') -> [] vs partition_ndjson('{}') -> [Text('{}')];
one-record serialized-element files now emit pretty-printed Text).
… naming, and test cleanups
Edge-case fixes:
- pretty_json_text(): json.dumps recurses far shallower than json.loads
parses, so deeply nested VALID JSON (~1k-10k levels) escaped as a raw
RecursionError; it now raises ValueError('JSON value is nested too
deeply to partition').
- JSON/NDJSON disambiguation: a valid NDJSON file with fewer than two
complete lines inside the 1 MiB read bound classified as JSON and then
failed partitioning; the truncated path now classifies NDJSON from one
or more complete parsing lines (restoring first-line semantics for
oversized records), and an exact-bound-size payload is no longer
treated as truncated.
Cleanups (selection):
- Rename partition/common/arbitrary_json.py -> json_partitioning.py and
is_serialized_element_dict -> is_element_shaped_dict; the arbitrary-
value formatter is now elements_from_arbitrary_value().
- CheckBox arm of the shape predicate type-checks 'checked'.
- Rehydration-failure message no longer uses internal jargon; both
DeprecationWarnings now name the replacement behavior.
- Timeless docstring voice; NOTE comment follows the repo's author-tag
convention; CHANGELOG folded into Enhancements and tightened.
- Tests: dedupe rewritten-legacy/BDD twins, confine exact pretty-printed
literals to one canonical test per output shape (structural assertions
elsewhere), rename mis-prefixed NDJSON tests, relocate stranded
detector tests, add coverage for the shape predicate (new
test_json_partitioning.py), CheckBox round-trip, file= routes,
arbitrary-NDJSON chunking, deep-nesting through both partitioners,
boundary-size disambiguation, and end-to-end NDJSON rehydration via
partition().
Serialized TableChunk dicts (the chunker's output for oversized tables) could not be deserialized: TYPE_TO_TEXT_ELEMENT_MAP has no TableChunk entry, so elements_from_dicts() silently dropped them, and on this branch one TableChunk flipped an entire serialized payload to arbitrary-JSON Text. This completes the table-reconstruction feature from #4291, whose reconstruct_table_from_chunks() explicitly expects 'user-provided/deserialized chunks' but had no deserialization path able to produce a TableChunk. Intent and mechanism: - Special-case 'TableChunk' in elements_from_dicts() exactly like the existing CheckBox special case, rather than adding it to TYPE_TO_TEXT_ELEMENT_MAP. The map also feeds the COCO category vocabulary (convert_to_coco derives positional category ids from its keys), so a map entry would renumber existing category ids on every export; the special case leaves the element-type vocabulary untouched and changes nothing for payloads without TableChunk dicts. - Accept 'TableChunk' (str text) in is_element_shaped_dict() symmetrically. - No new mechanisms: metadata (table_id, chunk_index, is_continuation, num_carried_over_header_rows) already round-trips via ElementMetadata to_dict/from_dict. Also: pin the exact-bound-size probe in JSON/NDJSON disambiguation with a test that fails if the probe is reverted; document the >1MiB first-record NDJSON degradation and the partition()-route qualifier in the CHANGELOG; test array-valued NDJSON lines stay one Text per line; test the predicate rejects unrecognized types; sweep em-dashes from branch-touched files.
Diff-line coverage audit of the branch found three reachable-but-untested paths: the empty-document guards in partition_json/partition_ndjson are only reachable via the file=/filename= routes (a pre-existing text='' guard catches the text route first), and the per-line NDJSON-check fallback in _disambiguate_json_file_type needs an asserted-JSON payload that neither whole-parses nor line-parses. Adds one test for each; branch diff coverage is now 100% and all three TableChunk rehydration branches are mutation-tested (removing or weakening any of them fails 2-4 tests).
Reverses an earlier deviation: the output contract specifies one Text
element with literal text {} for an empty object (an empty array still
yields no elements). This also removes the previously documented
empty-container asymmetry between partition_json and partition_ndjson;
the remaining intentional difference is empty arrays only ([] as an
NDJSON line yields Text("[]"), a JSON-mode [] document yields nothing).
Review feedback on #4391 (Sudarshan + cubic): - json.loads accepts NaN/Infinity/-Infinity by default, so those non-standard constants were partitioning into Text elements instead of being rejected as malformed. Add loads_strict_json() (shared by both partitioners) that passes parse_constant to raise, so such input surfaces the canonical 'Not a valid json'/'Not a valid ndjson' ValueError. Covers the top-level constant, a constant nested in an object/array, and NDJSON lines. - Fix the json_partitioning module docstring: it described the empty object as JSON-mode -> no elements, which is stale since {} now yields Text('{}') in both modes. The remaining mode difference is empty arrays only.
…ejection) Adds a Describe block in test_json_partitioning.py exercising the shared strict parser directly: valid JSON parses like json.loads, and NaN / Infinity / -Infinity (top-level and nested) raise. Complements the existing rejection tests through partition_json/partition_ndjson.
…dings) Addresses badGarnet's review on #4391 - the partitioners were upgraded but detect_filetype still used old heuristics, so router and partitioner disagreed in edge cases: - F1: an explicit .ndjson extension / application/x-ndjson content-type now wins over the whole-payload-JSON rule in _disambiguate_json_file_type, so a one-record .ndjson with an asserted application/json content-type rehydrates via partition_ndjson instead of becoming arbitrary Text. Compact .json objects still route to JSON (SAP path unchanged). - F2: scalar JSON/NDJSON is now reachable through partition(): .ndjson added to the _TextFileDifferentiator trusted-extension list (.json was already trusted) so scalar .json/.ndjson route to the JSON/NDJSON partitioners rather than TXT. Scope-limited to those extensions; a file misnamed .ndjson containing prose now raises (consistent with how .csv/.md/.tsv/.rst extension-trusting already behaves). - F3: loads_strict_json / _reject_json_constant moved to the low-level unstructured.utils module (re-exported from json_partitioning for existing callers) so file_utils.filetype can use strict parsing without an import cycle; disambiguation now rejects NaN/Infinity consistently with the partitioners. Guardrails re-verified: compact-object -> JSON, {} -> Text('{}'), NaN -> ValueError, FS-2017 gate untouched, #3365 xfail intact. Suites 394 passed + 1 xfailed; test_auto json/ndjson subset 11 passed + 1 xfailed.
…k (cubic P3) An asserted application/x-ndjson content-type is mapped to FileType.NDJSON by the content-type strategy before _disambiguate_json_file_type (which runs only for a FileType.JSON result) is reached, so that operand in _signals_ndjson was unreachable. Reduce it to the .ndjson extension tie-break and document why the content-type needs no handling here. No behavior change (x-ndjson still classifies NDJSON upstream; the .ndjson + application/json extension tie-break is unaffected). test_filetype.py 269 passed.
f982de8 to
183a660
Compare
Summary
partition_json()andpartition_ndjson()currently accept only serialized Unstructured element output - any other valid JSON is rejected with"Schema does not match the Unstructured schema"(and an array of scalars crashes with a rawAttributeError). This PR makes both partitioners two-mode:Textelements containing the pretty-printed JSON, instead of raising.It also fixes a file-type detection bug this feature exposed: a compact single-line JSON object was misrouted to
FileType.NDJSONand could never reachpartition_jsonat all.Behavior change
{"customer": "Acme", ...}ValueError: JSON cannot be partitioned. Schema does not match…Textelement, pretty-printed[{"id":"one"},{"id":"two"}]ValueErrorTextper object, array order[1,2,3]AttributeError: 'int' object has no attribute 'get'Textwith the whole arrayValueError/ crashText.jsonfileFileType.JSON→ 1TextValueErrorTextper line, line ordermetadata.coordinates, non-gziporig_elements)ValueError/binascii.Error/zlib.errorleaksValueError("Payload resembles serialized Unstructured elements but could not be reconstructed: …")TableChunkelements (chunked output with split tables)elements_from_dicts()TableChunk; whole chunked payloads round-trip and can feedreconstruct_table_from_chunks()(#4291){}(JSON route)Textcontaining{}(per the output contract)[]/ empty string (JSON route)[]/[][]{}/[]as an NDJSON lineText("{}")/Text("[]")(a line is a record)RecursionErrorescapesValueError("Not a valid json"/"…nested too deeply…")[{"hi":"there"}]]ValueError("Not a valid json")ValueError("Not a valid ndjson")Design notes
is_element_shaped_dictinpartition/common/json_partitioning.py) - a list rehydrates only when every item is a dict with a recognizedstrtype, the type's required field (strtext/boolcheckedfor CheckBox), and dict-or-absentmetadata. Branches are exclusive; no exception-based control flow. Prefix/schema pre-gates are removed frompartition_json,partition_ndjson, andauto.py;is_json_processable/is_ndjson_processableare deprecated (DeprecationWarning naming the replacement) but keep working for downstream importers.unstructured.file_utils.ndjson.loads/loadare intentionally retained undeprecated as generic utilities.ValueError- loud, never a leaked low-level error.staging/base.pydeliberately not modified: the shape predicate rejects payloads like{"type": "Title"}(nostrtext) beforeelements_from_dictsis ever called, so they partition as arbitrary JSON; hardeningitem["text"]→.get()in staging instead would silently rehydrate customer dicts as empty elements.json.loadssuccess →FileType.JSON; else ≥2 newline-delimited JSON values →NDJSON. The probe is bounded to 1 MiB, distinguishes an exact-bound-size payload from a truncated one, restores the file position (adetect_filetype(file=f)→partition_json(file=f)sequence works on the same handle), and treatsRecursionErroras a parse failure. For payloads exceeding the bound, one or more complete parsing lines classify as NDJSON (first-line semantics for oversized records); the residual degradation is an NDJSON file whose first record has no newline inside the bound, which classifies as JSON. One intentional flip: a one-line serialized-element object previously rehydrated via the NDJSON route; it now partitions as arbitrary JSON (rehydration applies only to arrays).pretty_json_text()-json.dumps(value, indent=2, sort_keys=True); stable, diffable output;sort_keysalphabetizes source field order (commented at the definition as the knob to revisit). NDJSON is strictly oneTextper line; the only empty-container divergence is[](an NDJSON line yieldsText("[]"), a JSON-mode[]document yields no elements).elements_from_arbitrary_value()(partition/common/json_partitioning.py) is the single swap-point for a future walker.TableChunk rehydration - intent
Review of this branch surfaced that serialized
TableChunkdicts could not be deserialized at all:TYPE_TO_TEXT_ELEMENT_MAPhas noTableChunkentry, soelements_from_dicts()silently dropped them onmain, and with this branch's shape predicate one TableChunk flipped an entire serialized payload to arbitrary-JSONText. The intent of the fix commit is narrow and explicit:reconstruct_table_from_chunks()(added in feat: tablechunks can reconstruct table #4291) filtersisinstance(e, TableChunk)and its docstring states reconstruction "can be called on user-provided/deserialized chunks" - but no deserializer could produce aTableChunkuntil now. The reconstruction metadata (table_id,chunk_index,is_continuation,num_carried_over_header_rows) already round-trips viaElementMetadata; only the element-class dispatch was missing.TYPE_TO_TEXT_ELEMENT_MAPalso feeds the COCO category vocabulary (convert_to_cocoderives positional category ids from its keys), so adding TableChunk there would renumber existing category ids on every export, even for data with no TableChunks. Insteadelements_from_dicts()special-cases"TableChunk"exactly like the existingCheckBoxspecial case. Verified strictly additive: a 40-dict all-type sweep shows byte-identical output vs the pre-fix function for every non-TableChunk payload;documents/elements.pyis untouched; COCO ids unchanged.type: "TableChunk"records already flow to destinations today whenever live chunking splits a table (library output, ingest, hosted API responses); this changes only the read-back side.Testing
test_auto.py -k "json or ndjson"7 passed, 1 xfailed (the bug(json): partition_json() does not preserve original element_id or metadata #3365 strict-xfail - unaffected and kept).last_modified, metadata stamping), including end-to-end NDJSON rehydration throughpartition().{}no longer raises, empty NDJSON container lines emitText, one-record payloads route JSON) - each renamed to describe the new behavior.partition_json-> byte-identical re-serialization ->reconstruct_table_from_chunksreturns the table), NDJSON TableChunk lines, predicate accept/reject, and staging serde tests. The exact-bound-size detection probe is pinned by a test proven to fail if the probe is reverted.text=/file=/filename=routes, corrupt-payload errors viafile=, deep-nesting through both partitioners, boundary-size (1 MiB ± 1) disambiguation, detect-then-partition on one file handle, and chunking over arbitrary JSON and NDJSON output.example-docs/arbitrary-records.json,single-line-object.json,arbitrary-records.ndjson.