Repository navigation
[SQL] Lower a few selected operator representations into DBSP primitives in the JSON representation - #7891
Conversation
…t of merged operators ILinearAggregate extended ILinear, but a linear aggregate is not a linear operator. It now extends IIncremental, so it requires a DELTA input. A ChainAggregate is built on a collection before incrementalization, so it accepts a DELTA or a COLLECTION and produces the kind of its input. RemoveCSE.replaceMultiOutput mapped a merged operator with several outputs to the kept operator of the old circuit, and added that operator again. It now maps each output to the output of the kept operator in the new circuit. Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
No code creates a DBSPWeighOperator since f9049f2 removed its only creator from ExpandOperators. Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
…mitive Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
…, chain aggregate and join kernels Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
|
What I ran: QA tests I added locally (not pushed):
Missing (medium): uncovered cases. No test lowers a join where only one side is DELTA, which should leave the join as it is. No test merges two Commit quality (medium, feldera-pr-checks gate): commits 3–7 have one-line messages and no body, including commit 6, the main lowering/CSE/split-operator change. They need a body that explains why. Several subjects are also past 72 characters and get truncated on GitHub. Observation: |
…-gen2 Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
…are not supported yet Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
d0c9636 to
f9ae32c
Compare

There are 7 independent commits.
Commits 1, 2, 5, 7 are unrelated fixes discovered during this work.
Commit 3 adds a new Java interface
IPrimitivewhich marks operators that correspond directly toDBSPprimitives (cannot be broken down into simpler nodes). E.g., many existing linear operators are primitives.Commit 4 adds a few new primitives: Accumulator, Delays, Integrals, chain aggregates kernels, join kernels.
Commit 6 expands the existing chain aggregates and joins into their DBSP decompositions directly. The expansion is generates persistent IDs and regions and includes a CSE pass which can e.g., share integrals.
Currently there is no Rust code generated for these new primitives (I filed issue #7890 for doing that). One can still emit JSON and PNG representations of the lowered representations.
Checklist