← 输入框|github.com|刷新
Skip to content

[SQL] Support for new Calcite UNNEST operator - #6599

Merged
mihaibudiu merged 1 commit into
feldera:mainfrom
mihaibudiu:unnest
Nov 10, 2026
Merged

mihaibudiu merged 1 commit into
feldera:mainfrom
mihaibudiu:unnest

Conversation

@mihaibudiu

@mihaibudiu mihaibudiu commented Jul 6, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #7759 (by advancing Calcite version)

This will enable more correlated queries to compile correctly.

This needs apache/calcite#5031 to be merged first
apache/calcite#5086 is a fragment of this PR which could be useful too.

Checklist

  • Unit tests added/updated

@mihaibudiu
mihaibudiu marked this pull request as draft July 6, 2026 23:31

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Draft first-pass review — high-level only.

Nice cleanup: replacing the hand-rolled pattern match in visitCorrelate with a proper Calcite pipeline (Project extraction → CORRELATE_UNCOLLECT_MERGE → single visitUncollect) is the right direction and removes a lot of fragile RelNode sniffing.

A few high-level notes/questions:

  1. CorrelateProjectExtractor.java looks like a straight port of a Calcite class (naming, RelHomogeneousShuttle, @Override public RelNode style, doc format). If it's vendored from apache/calcite#5031 (or a companion CALCITE JIRA) pending release, a one-line pointer at the class-level Javadoc would save the next reader some archaeology. Also worth noting whether we plan to delete this file once we upgrade to a Calcite version that ships it.

  2. visitCorrelate now unconditionally throws decorrelateError. The invariant is now "by the time we compile, every LogicalCorrelate must have been merged with its Uncollect or otherwise decorrelated." The comment on visit() — "Give up if the plan is not decorrelated" — is a bit terse for a fairly load-bearing invariant. Consider spelling it out (what the optimizer step is responsible for producing, and that a surviving Correlate means the new merge rule didn't fire).

  3. compileModifyTable LogicalUnion fix looks like a genuine correctness bug fix — the old loop silently dropped LogicalValues children of a mixed union, so INSERT ... VALUES (expr_row), (lit_row) would lose the literal rows. Nice catch (and testUnnestCoalesce exercises exactly this shape). Semantically unrelated to UNNEST though; if it's easy to split, it would make bisection nicer, but not blocking on a draft.

  4. UNNEST semantics look right for the cases in scope:

    • Arrays and maps both flow through ICollectionType uniformly.
    • WITH ORDINALITY, the "Calcite forgets to mark inner fields nullable" workaround, and the ordinality index type are preserved verbatim from the old path.
    • getCollectionFieldIndices().cardinality() != 1 is a stricter and clearer guard than the old inputRowType.size() > 1 (which conflated passthrough fields with collection fields).
    • uncollect.isOuter is the new gate for LEFT JOIN UNNEST — correctly still unimplemented.
    • shuffleSize = type.size() with an IdShuffle is fine given the merged Uncollect already carries the full output row type.
  5. Test coverage for the new shape is present (nested field UNNEST, nested + COALESCE, ordinality, mixed VALUES rows) and the pre-existing testUnnest1/testDoubleUnnest1 cases remain, so the merged pipeline is exercised on both old and new plan shapes. Good.

  6. containsUncollect full-plan scan before every step is a minor efficiency wart — a Hep rule set could be made no-op automatically — but negligible on realistic plans. Not worth touching unless it shows up in profiling.

Multisets — not exercised explicitly in the new tests as far as I can see; they go through the same ICollectionType path so should be fine, but a one-liner test wouldn't hurt once the Calcite dep lands.

Marking as COMMENT — happy to do a proper pass once apache/calcite#5031 is merged and this leaves draft.

@mihaibudiu

Copy link
Copy Markdown
Contributor Author

I have reworked apache/calcite#5031. This PR needs that one to be merged into Calcite, but this code verifies that all tests pass if linked with a version of Calcite containing that change.

@mihaibudiu mihaibudiu added the Calcite Fix required in Calcite label Oct 14, 2026
@mihaibudiu

Copy link
Copy Markdown
Contributor Author

Unfortunately no one has reviewed the Calcite PR, so this is still blocked.

@mihaibudiu
mihaibudiu marked this pull request as ready for review November 9, 2026 22:54
@mihaibudiu

Copy link
Copy Markdown
Contributor Author

Finally, the Calcite PR has been merged, so this is ready to merge too.

@mihaibudiu
mihaibudiu added this pull request to the merge queue Nov 9, 2026
@mihaibudiu
mihaibudiu force-pushed the unnest branch 2 times, most recently from 3520543 to 062291b Compare November 10, 2026 00:33
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 10, 2026
Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
@mihaibudiu
mihaibudiu enabled auto-merge November 10, 2026 03:02
@mihaibudiu
mihaibudiu disabled auto-merge November 10, 2026 03:12
@mihaibudiu
mihaibudiu added this pull request to the merge queue Nov 10, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 10, 2026
@mihaibudiu
mihaibudiu added this pull request to the merge queue Nov 10, 2026
Merged via the queue into feldera:main with commit b77b338 Nov 10, 2026
1 check passed
@mihaibudiu
mihaibudiu deleted the unnest branch November 10, 2026 07:10
@mihaibudiu mihaibudiu mentioned this pull request Nov 10, 2026
2 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Calcite Fix required in Calcite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A comma join followed by an outer JOIN is evaluated as (a, b) JOIN c, unlike SQL:1999, PostgreSQL, MySQL and DuckDB

3 participants