← 输入框|github.com|刷新
Skip to content

[SQL] Lower a few selected operator representations into DBSP primitives in the JSON representation - #7891

Merged
mihaibudiu merged 7 commits into
mainfrom
primitives-lowering
Nov 10, 2026
Merged

mihaibudiu merged 7 commits into
mainfrom
primitives-lowering

Conversation

@mihaibudiu

Copy link
Copy Markdown
Contributor

There are 7 independent commits.

Commits 1, 2, 5, 7 are unrelated fixes discovered during this work.
Commit 3 adds a new Java interface IPrimitive which marks operators that correspond directly to DBSP primitives (cannot be broken down into simpler nodes). E.g., many existing linear operators are primitives.
Commit 4 adds a few new primitives: Accumulator, Delays, Integrals, chain aggregates kernels, join kernels.
Commit 6 expands the existing chain aggregates and joins into their DBSP decompositions directly. The expansion is generates persistent IDs and regions and includes a CSE pass which can e.g., share integrals.

Currently there is no Rust code generated for these new primitives (I filed issue #7890 for doing that). One can still emit JSON and PNG representations of the lowered representations.

Checklist

  • Unit tests added/updated

…t of merged operators

ILinearAggregate extended ILinear, but a linear aggregate is not a linear
operator. It now extends IIncremental, so it requires a DELTA input. A
ChainAggregate is built on a collection before incrementalization, so it
accepts a DELTA or a COLLECTION and produces the kind of its input.

RemoveCSE.replaceMultiOutput mapped a merged operator with several outputs
to the kept operator of the old circuit, and added that operator again. It
now maps each output to the output of the kept operator in the new circuit.

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
No code creates a DBSPWeighOperator since f9049f2 removed its only
creator from ExpandOperators.

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
…mitive

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
…, chain aggregate and join kernels

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
@mihaibudiu

Copy link
Copy Markdown
Contributor Author
joins-shared-integrals

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran: ./build.sh, then mvn test -pl SQL-compiler on PrimitivesTests, Gen2LoweringTests, TestTagRegions, StreamKindTests, PersistentIdTests, CircuitGraphsTests, IncrementalRegressionTests (87), ShareIndexesTests, ShareWindowIntegralsTests, FuseExpensiveMapsTests and IncrementalConstantTests. All passed. I also checked the default (non-gen2) path: ExpandAggregates wraps DBSPAggregateLinearPostprocessOperator in D/I, so making ILinearAggregate require a DELTA input does not break kind validation. pre-commit has no hooks for Java files.

QA tests I added locally (not pushed):

Case Result
Self-join a JOIN a: 1 accumulator, 1 integral, 2 kernels after CSE pass
JOIN … WHERE x > y (join filter-map) is lowered to kernels pass
Persistent id of a shared integral stays the same when an unrelated view is removed fail (see inline comment)

Missing (medium): uncovered cases. No test lowers a join where only one side is DELTA, which should leave the join as it is. No test merges two DBSPDelayInputOperators with CSE (setCanonical recursing into the output halves). No test round-trips the Gen2JsonVisitor output, which uses forward references.

Commit quality (medium, feldera-pr-checks gate): commits 3–7 have one-line messages and no body, including commit 6, the main lowering/CSE/split-operator change. They need a body that explains why. Several subjects are also past 72 characters and get truncated on GitHub.

Observation: @NeedsSupport is added but no test uses it yet. Once the Gen-2 runner is enabled, the join and MIN/MAX tests will emit the new primitive classes.

…-gen2

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
…are not supported yet

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
@mihaibudiu
mihaibudiu added this pull request to the merge queue Nov 10, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 10, 2026
@mihaibudiu
mihaibudiu added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit 47d926b Nov 10, 2026
1 check passed
@mihaibudiu
mihaibudiu deleted the primitives-lowering branch November 10, 2026 23:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants