← 输入框|github.com|刷新
Skip to content

[adapters] Commit all of an API transaction's queued input with it. - #7583

Merged
blp merged 8 commits into
mainfrom
early-commit-keeps-transaction
Oct 31, 2026
Merged

blp merged 8 commits into
mainfrom
early-commit-keeps-transaction

Conversation

@blp

@blp blp commented Oct 22, 2026

Copy link
Copy Markdown
Member

Linked Issue

None. Found with Antithesis while testing /ingress with API transactions.

The docs say that a transaction contains the input that the pipeline receives between start_transaction and commit_transaction. In a single-host pipeline, three paths broke that promise, so an ad-hoc query or an /egress reader could see part of a transaction:

Case What went wrong
Rows queued when the commit arrives The commit step took at most one batch (max_batch_size, 10,000 x workers for /ingress) from each endpoint. Queued rows after that went into later transactions.
Commit before any step started the transaction The commit erased the transaction. Its rows went in as automatic transactions, one batch per step.
Transaction started and committed while another transaction commits The same erase.

What changed

advance_transaction_state() is now two rules, applied in order, in place of nested matches with four asserts. The replay branch does not change.

Rule Effect
Start In state None, start the transaction that the initiators ask for, or else an automatic transaction
Commit In state Started, start to commit when no initiator is still adding input and the input queued at the API's commit request is in the circuit

An automatic transaction has no initiators, so it commits in the step that starts it. A transaction that is committed early is not a special case now, so the two code paths that erased it are gone.

An API commit records how many records each input endpoint has received (CommitTarget). The wait always ends: an endpoint is done when it has fed those records, has nothing buffered, was removed, or gives no records to a step that polls all endpoints.

In a multihost pipeline, there is no commit target, because the coordinator commits with steps that feed no input. An early commit now starts and commits the real transaction in the coordinator's commit step, where it ran an automatic transaction before. The host reports the same status either way. Multihost commits still leave queued input for the next transaction; that needs a coordinator change and a separate fix.

Other commits:

  • The transactions docs now say that a transaction can also include some input that arrives while its commit starts.
  • DevTweaks::disable_auto_transaction() is now enable_auto_transaction(). The configuration key does not change.

Describe Manual Test Plan

New controller tests use a 10-record batch and a step trigger that ignores input, so only a commit causes steps:

Test Without the fix
api_transaction_committed_before_it_starts only 0 of 100 records committed
api_transaction_commit_includes_queued_records only 10 of 100 records committed
api_transaction_committed_during_another_commit only 10 of 200 records committed
multihost_api_transaction_committed_before_it_starts passes (no change expected); fails if a multihost host waits for a commit target
auto_transactions_are_enabled_unless_disabled fails if the accessor does not negate the field

The new tests passed 30 serial and 30 parallel runs. cargo test -p dbsp_adapters --lib passes (1058 tests), without the tests that need Kafka, Postgres, DynamoDB or Redis.

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

Mark if you think the answer is yes for any of these components:

Describe Incompatible Changes

None. A commit can now take more steps, because it waits for the queued input. The Python SDK's commit_transaction(wait=True) now waits for the queued rows after an early commit; before, the erase made it return at once.

@blp
blp requested a review from ryzhyk October 22, 2026 00:37
@blp blp self-assigned this Oct 22, 2026
@blp blp added bug Something isn't working connectors Issues related to the adapters/connectors crate rust Pull requests that update Rust code labels Oct 22, 2026
// The input queued now belongs to the transaction. In a multihost
// pipeline, the coordinator commits with steps that feed no input.
if !transaction_info.is_multihost {
transaction_info.commit_target = Some(CommitTarget::new(&statuses));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug (medium): removing the old !transaction_state.is_started() check here leaves TransactionState::is_started() (line 7212) used only by the new tests. cargo check -p dbsp_adapters now warns method is_started is never used, and cargo clippy --workspace -- -D warnings (from scripts/pre-push) fails on it. Mark it #[cfg(test)] or move it into the test module.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: TransactionState::is_started() is now #[cfg(test)] (1567002).

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Review of the API-transaction commit fix. The design looks right to me: the CommitTarget wait always ends, the lock order (input statuses, then transaction_info) matches input_step, and the docs and dev-tweak rename are fine. Apart from the dead-code warning in the inline comment, no feldera-pr-checks gate fails.

What I ran

Command Result
cargo test -p dbsp_adapters --lib api_transaction (the new tests), 20 runs in a row ✅ 5/5 passed every run
cargo test -p dbsp_adapters --lib controller:: ✅ 167 passed
cargo test -p feldera-types auto_transactions ✅
pre-commit run --files <changed files> ✅ all hooks passed, but cargo check shows a new dead-code warning (see inline comment)

missing (medium): tests for the cases where the wait ends early. The PR's tests don't cover the drained() early exits. I wrote these locally and all of them pass. Please consider adding them:

#[test]
fn qa_api_commit_with_paused_endpoint_terminates() {
    let (controller, feeder, before) = start_api_transaction_test(false);
    controller.start_transaction().unwrap();
    push_records(&feeder, 0..100);
    controller.pause_input_endpoint("feeder").unwrap();
    controller.start_commit_transaction().unwrap();
    wait(|| controller.inner.get_transaction_state() == TransactionState::None
            && controller.inner.get_transaction_number() > before,
         API_TRANSACTION_TIMEOUT_MS).expect("commit hung on a paused endpoint");
    controller.stop().unwrap();
}
// Same shape with `controller.disconnect_input(&id)` after the commit (removed endpoint),
// and with a second thread that keeps calling push_records (the commit must not starve).

I did not test the multihost path beyond the PR's own test. The PR description says multihost commits still leave queued input for the next transaction, to be fixed separately.

@blp
blp force-pushed the early-commit-keeps-transaction branch from 0e7ebae to 538cc1c Compare October 22, 2026 14:47
@blp

blp commented Oct 22, 2026

Copy link
Copy Markdown
Member Author

Thanks. 1567002 now adds tests for the three early exits: a connector that holds input back after it commits its part, a client that keeps sending input, and an endpoint that is removed. Each test fails if I remove the rule that covers its case. I left out the paused-endpoint test: the test endpoint still flushes its queue while paused, and the controller still polls paused endpoints, so that test would pass with every early exit removed.

@blp

blp commented Oct 26, 2026

Copy link
Copy Markdown
Member Author

@ryzhyk posted a review for this PR here: #7607 (comment)

blp and others added 7 commits October 27, 2026 17:05
The docs say that a transaction contains the input that the pipeline
received between `start_transaction` and `commit_transaction`.  In a
single-host pipeline, three paths broke that promise:

- A commit started in the next step, and that step took at most one
  batch (`max_batch_size`) from each input endpoint.  Input queued
  beyond that went into the transactions that followed.

- A commit that arrived before any step started the transaction
  erased the transaction.  Its input then went in as automatic
  transactions, one batch per step, and a query could see part of it.

- The same erase happened to a transaction that the API started and
  committed while another transaction committed.

This rewrites advance_transaction_state() as two rules, applied in
order, in place of nested matches with four asserts:

- Start: in state None, start the transaction that the initiators ask
  for, or else an automatic transaction.

- Commit: in state Started, start to commit once no initiator is still
  adding input and the input queued at the API's commit request is in
  the circuit.

An automatic transaction has no initiators, so it commits in the step
that starts it.  A transaction committed before a step started it, or
during another commit, is no longer a special case, so the two paths
that erased it are gone.

An API commit now records, for each input endpoint, how many records
it has received (CommitTarget).  The wait always ends: an endpoint
counts as done when it has fed its records, has none buffered, was
removed, or gives no records to a step that polls all endpoints.

In a multihost pipeline, there is no commit target, because the
coordinator commits with steps that feed no input.  The erase was
removed there too: an early commit now starts and commits the
transaction in the coordinator's commit step, where it used to run an
automatic transaction.  The host reports the same status either way.

The tests use a 10-record batch and a step trigger that ignores input,
and an hour between clock ticks, so only a commit makes steps.  Without
the fix, each single-host test commits only part of its records.  The
multihost test passes without the fix, and fails if the host waits for
a commit target.  Three more tests cover the cases that end the wait
early: a connector that holds input back after it commits its part,
input that clients keep sending, and a removed endpoint.  Each fails
if the rule that covers its case is removed.

TransactionState::is_started() has no caller outside the tests now, so
it is compiled only for them.

Signed-off-by: Ben Pfaff <blp@feldera.com>
A commit starts only after the pipeline ingests the input that was
queued at the commit request, and input that arrives meanwhile can
join the transaction.  The docs promised only the first part.

Signed-off-by: Ben Pfaff <blp@feldera.com>
…nsaction().

A positive name is easier to read where it is used.  The dev tweak
keeps its name, `disable_auto_transaction`, so configurations do not
change.

Signed-off-by: Ben Pfaff <blp@feldera.com>
An API commit waits for the input queued at the commit request, but a
step that fed no records from an endpoint ends the wait.  A commit
request that arrived after a step polled an endpoint, or listed the
endpoints, but before the step decided whether to commit, thus let the
transaction commit without the input queued meanwhile.

The controller now numbers each step's poll under the transaction lock,
and the commit target records the number at the request, so only a
poll that began after the request can end the wait.  This costs at most
one more step.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
The cutover of a concurrent bootstrap pauses ingestion and then waits
for the open transaction to commit.  An API commit in that window
waited for its queued input, which no step could ingest, so neither the
transaction nor the bootstrap ever finished.

During the cutover, the commit now starts without the queued input,
which goes into a later transaction, as it did before commits waited
for queued input.  The transaction docs say so.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
Regression tests for the previous two commits: a commit request that
arrives while a step waits for a slower endpoint after polling the one
that holds the input, before or after the transaction starts, or for an
endpoint created during the step; and a commit with input queued during
a concurrent bootstrap's cutover.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
test_concurrent_bootstrap_api_commit_with_queued_input checked only that
the commit and the bootstrap finish.  It now also checks that the input
queued during the cutover goes into a later transaction.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp

blp commented Oct 28, 2026

Copy link
Copy Markdown
Member Author

@ryzhyk Thanks for the review in #7607 (comment) and for the prototype. Both issues were real. I reviewed early-commit-keeps-transaction-fixes and took its three commits into this PR. I did not change the code in them:

  • Ignore polls that predate an API commit request.
  • Don't wait for queued input during a bootstrap cutover. (option (a))
  • Test API commits that race a step or a bootstrap cutover.

What I checked:

Check Result
New tests on the old PR head, 538cc1c05 All 4 fail. Each race test commits 0 of 100 records. The bootstrap test hangs in Synchronizing after 39,095 steps.
Poll fix alone The 3 race tests pass. The bootstrap test still fails.
All commits All 16 transaction and concurrent-bootstrap tests pass. 20 of 20 stress runs pass.
ingestion_paused Same condition as the early return in input_step. A stop-the-world bootstrap needs no change, because the API refuses to start or commit a transaction while one runs.

I added one commit, Check that input queued during a bootstrap cutover is kept. The bootstrap test now also checks that the 10 records queued during the cutover reach the view after the cutover. A mutant that drops force=true input during the pause makes this check fail.

I also rebased onto main, after the move to crates/connectors and the engine-neutral test suites:

  • The controller tests and the bootstrap test are now #[engine_test] cases. The multi-step commit test requires ChunkedCommit. I put each port into the commit that adds the test, so each commit builds with its tests.
  • TransactionState::is_started is now under cfg(any(test, feature = "testing")), like the other test-only items, because the suites build the tests through the testing feature.
  • Without the two fixes, the ported tests still fail.
  • The subject prefixes are now [connectors].

@blp
blp force-pushed the early-commit-keeps-transaction branch from 538cc1c to 6339223 Compare October 28, 2026 00:16
@blp

blp commented Oct 28, 2026

Copy link
Copy Markdown
Member Author

@ryzhyk This needs a (re)review now.

@blp
blp added this pull request to the merge queue Oct 29, 2026
@blp
blp removed this pull request from the merge queue due to a manual request Oct 29, 2026
@mihaibudiu
mihaibudiu added this pull request to the merge queue Oct 30, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 30, 2026
@blp
blp added this pull request to the merge queue Oct 30, 2026
…rgin.

`concurrent_bootstrap_api_commit_with_queued_input` failed in the merge
queue because the backfill finished before `/start_transaction` opened
the transaction.  With 2,000 records, the backfill took about 0.55 s and
the transaction opened after about 0.22 s.  With 20,000 records, the
backfill takes about 4.8 s.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp removed this pull request from the merge queue due to a manual request Oct 30, 2026
@blp
blp enabled auto-merge October 30, 2026 17:59
@blp
blp disabled auto-merge October 30, 2026 21:57
@blp
blp added this pull request to the merge queue Oct 30, 2026
Merged via the queue into main with commit 74cbae8 Oct 31, 2026
1 check passed
@blp
blp deleted the early-commit-keeps-transaction branch October 31, 2026 03:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working connectors Issues related to the adapters/connectors crate rust Pull requests that update Rust code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants