← 输入框|github.com|刷新
Skip to content

[DBSP][SQL] Incremental constant operators - #7886

Merged
mihaibudiu merged 2 commits into
mainfrom
incremental-constant
Nov 7, 2026
Merged

mihaibudiu merged 2 commits into
mainfrom
incremental-constant

Conversation

@mihaibudiu

Copy link
Copy Markdown
Contributor

Two commits: one adding an IncrementalConstantGenerator operator in DBSP, and the other adding support in the compiler. An incremental constant is a constant followed by a Differentiator, but it's more efficient to implement it as a single operator. Performance is probably not as important as uniformity and simplicity of the code generated.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

fred-review did not finish; see https://github.com/feldera/feldera/actions/runs/42325328478

Comment thread crates/dbsp/src/operator/generator.rs Resolved
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
cargo test -p dbsp --lib generator (2 new tests) ✅ pass
mvn test -Dtest=RecursiveTests,IncrementalUnboundedStateTests ✅ 62/62
mvn test -Dtest=InternalNowTests ❌ 3/3 fail: Circuit contains a differentiation operator (see inline comment)

Findings

  • bug (high): The new NoIntegralOrDifferentiatorVisitor breaks every NOW() query that still has a DBSPNowOperator → D pair. CI will go red.
  • missing (medium), gate "tests for changed behaviour": The compiler commit changes the generated circuit but adds no Java tests. Tests should cover these rewrites:
    • PullFilterVisitor folding a filter into an incremental constant.
    • RemoveIAfterD turning I(incremental constant) into a plain constant.
    • CSE keeping a ConstantOperator and an IncrementalConstantOperator with the same rows distinct.
    • PropagateEmptySources/RemoveConstantFilters producing an empty incremental constant.
    • A JSON round trip of the new operator (fromJson).
  • pitfall (medium), checkpoint compatibility: Existing pipelines compiled with the old ConstantGenerator+Differentiator pair will get new persistent IDs on upgrade. Everything downstream of a constant will then be treated as new and bootstrapped again. Please confirm that this is acceptable. If it is, note it in the PR description or release notes.
  • missing (medium), Rust test gap: No DBSP test covers a nested (recursive) scope, where clock_end resets emitted once per parent step and fixedpoint depends on it. No test covers bootstrapping a modified circuit either, which is what is_deterministic_source is for.

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
@mihaibudiu
mihaibudiu force-pushed the incremental-constant branch from fc26180 to 1e4f4ca Compare November 7, 2026 04:13
@mihaibudiu
mihaibudiu added this pull request to the merge queue Nov 7, 2026
Merged via the queue into main with commit 8b2f45b Nov 7, 2026
1 check passed
@mihaibudiu
mihaibudiu deleted the incremental-constant branch November 7, 2026 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants