← 输入框|github.com|刷新
Skip to content

[SQL] Rebuild a keyed table's whole row in an index closure under --gen2 - #7739

Merged
mihaibudiu merged 1 commit into
mainfrom
gz/gen2-window-keyed-row
Oct 31, 2026
Merged

mihaibudiu merged 1 commit into
mainfrom
gz/gen2-window-keyed-row

Conversation

@gz

@gz gz commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

With --gen2 the indexed value of a table with a primary key drops the key columns. An index closure that reads the whole row, such as the partitioning of a RANGE window over SELECT *, read that value as the row, and compilation failed with "Tuple field ... has type Tup3<..> that does not match the declared field type Tup4<..>". The closure now rebuilds the row from the key and the value. Adds Gen2JsonTests#gen2WindowReadsTheWholeKeyedRow.

With --gen2 the indexed value of a table with a primary key drops the key columns. An index closure
that reads the whole row, such as the partitioning of a RANGE window over SELECT *, read that value
as the row, and compilation failed with "Tuple field ... has type Tup3<..> that does not match the
declared field type Tup4<..>". The closure now rebuilds the row from the key and the value.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
@gz
gz requested a review from mihaibudiu October 30, 2026 14:31
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Step Result
./build.sh ✅
mvn test -Dtest=Gen2JsonTests ✅ 13/13
New test with the ExpandIndexedInputs.java change reverted ❌ fails with the reported Tup3 … does not match … Tup4 error, so the test reproduces the bug
QA: composite PK (k2, k1) at non-leading positions, window over SELECT * with PARTITION BY on a non-key column and on a key column (incremental) ✅ compiles
QA: single PK at a non-leading position, non-incremental ✅ compiles

missing (medium): the new test only checks that compilation succeeds. The type checker catches a wrong key/value interleaving only when the swapped columns have different types. Two same-typed key columns in the wrong order would still pass. The test also covers just one shape (a single leading VARCHAR key). Please add the composite/non-leading key case, ideally with an assertion on the field order of the rebuilt row:

+    /** A window reads the whole row of a table whose composite key is not leading. */
+    @Test
+    public void gen2WindowReadsTheWholeCompositeKeyedRow() {
+        String json = this.circuitJson(this.compile("""
+                CREATE TABLE t(a INT, k1 INT NOT NULL, x INT, k2 VARCHAR NOT NULL, ts INT, PRIMARY KEY (k2, k1));
+                CREATE VIEW v AS SELECT *, SUM(x) OVER (
+                    PARTITION BY a ORDER BY ts RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS s FROM t;""",
+                true, true, true));
+        Assert.assertTrue(json.contains("\"DBSPSourceMapOperator\""));
+    }

No other gates are hit: this is an internal compiler fix, it has a regression test, it adds no unsafe code or dependencies, and it needs no docs. Otherwise LGTM.

@mihaibudiu

Copy link
Copy Markdown
Contributor

I am confused, @ryzhyk made changes to this pass especially for gen2 in #6978
Are you trying to undo the changes we made by some compensating change?
Why did the tests stop working?

@gz

gz commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor Author
window.sql

  CREATE TABLE t(id VARCHAR NOT NULL PRIMARY KEY, a INT, ts INT, x INT);
  CREATE VIEW v AS SELECT *, SUM(x) OVER (
      PARTITION BY a ORDER BY ts RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS s FROM t;

  Compile it with the pipeline-manager's Gen2 flags

  cd feldera/sql-to-dbsp-compiler && ./build.sh
  java -jar SQL-compiler/target/sql2dbsp-jar-with-dependencies.jar \
      -i --alltables --ignoreOrder --gen2 window.sql -o out.rs

  Results

  ┌─────────────────────────────────┬───────────────┬────────────────┐
  │            Compiler             │    --gen2     │ Without --gen2 │
  ├─────────────────────────────────┼───────────────┼────────────────┤
  │ feldera main f5c129a88 (no fix) │ error, exit 1 │ compiles       │
  ├─────────────────────────────────┼───────────────┼────────────────┤
  │ submodule 9fcd6aa49 (has #7739) │ compiles      │ compiles       │
  └─────────────────────────────────┴───────────────┴────────────────┘

  The error on main:
  error: Compiler error: Tuple field ((*(t_20.1)).clone()) has type Tup3<i32?, i32?, i32?>
  that does not match the declared field type Tup4<s, i32?, i32?, i32?>

  Tup4 is the whole row (id, a, ts, x). Tup3 is the value with the primary-key column id removed.

@mihaibudiu
mihaibudiu added this pull request to the merge queue Oct 31, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 31, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Verdict: flaky test, not this PR. High confidence.

The merge queue run's only test failure was in Adapter Tests, not SQL compiler:

Job Test Adapters (arm64)
Failing test integrated::postgres::cdc_input::tests::a_panic_on_the_worker_task_still_shuts_the_runtime_down_in_time
Error the wrapper thread hung on the runtime after a panic: Timeout (log line 1705)

This PR only touches ExpandIndexedInputs.java and a Java test — nothing in crates/connectors. The test asserts a worker-panic triggers runtime shutdown within a fixed margin; under CI host contention that margin is occasionally missed.

This is a known, already-tracked flake: #7716, now on its 4th occurrence across unrelated PRs. I added this run as another data point there.

Next step: re-queue the PR — it should pass on retry since the failure is unrelated to this change.

@mihaibudiu
mihaibudiu added this pull request to the merge queue Oct 31, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 31, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Verdict: pre-existing bug on main, not caused by this PR (high confidence)

This PR only touches SQL compiler Java (ExpandIndexedInputs.java, Gen2JsonTests.java); the failing test is an unrelated Rust Kafka connector test.

Failing job Adapter Tests / Test Adapters (amd64)
Failing test transport::kafka::ft::output::tests::an_outage_aborts_and_replays_the_transaction
Assertion log:1690-1693 — assertion left == right failed: the spool must be released at commit (left: 190, right: 0)

Already tracked in #7760 (opened earlier today for the same assertion in a different merge-queue run, root-caused to commit 6ad915e99 on main). I added this run's evidence there.

Next step: requeue is likely to pass once #7760 is fixed — this failure is unrelated to your change and will keep reoccurring until then.

@mihaibudiu
mihaibudiu added this pull request to the merge queue Oct 31, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 31, 2026
@mihaibudiu
mihaibudiu added this pull request to the merge queue Oct 31, 2026
Merged via the queue into main with commit c38bc95 Oct 31, 2026
3 checks passed
@mihaibudiu
mihaibudiu deleted the gz/gen2-window-keyed-row branch October 31, 2026 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants