Repository navigation
[adapters] Commit all of an API transaction's queued input with it. - #7583
Conversation
| // The input queued now belongs to the transaction. In a multihost | ||
| // pipeline, the coordinator commits with steps that feed no input. | ||
| if !transaction_info.is_multihost { | ||
| transaction_info.commit_target = Some(CommitTarget::new(&statuses)); | ||
| } |
There was a problem hiding this comment.
bug (medium): removing the old !transaction_state.is_started() check here leaves TransactionState::is_started() (line 7212) used only by the new tests. cargo check -p dbsp_adapters now warns method is_started is never used, and cargo clippy --workspace -- -D warnings (from scripts/pre-push) fails on it. Mark it #[cfg(test)] or move it into the test module.
There was a problem hiding this comment.
Fixed: TransactionState::is_started() is now #[cfg(test)] (1567002).
|
Review of the API-transaction commit fix. The design looks right to me: the What I ran
missing (medium): tests for the cases where the wait ends early. The PR's tests don't cover the #[test]
fn qa_api_commit_with_paused_endpoint_terminates() {
let (controller, feeder, before) = start_api_transaction_test(false);
controller.start_transaction().unwrap();
push_records(&feeder, 0..100);
controller.pause_input_endpoint("feeder").unwrap();
controller.start_commit_transaction().unwrap();
wait(|| controller.inner.get_transaction_state() == TransactionState::None
&& controller.inner.get_transaction_number() > before,
API_TRANSACTION_TIMEOUT_MS).expect("commit hung on a paused endpoint");
controller.stop().unwrap();
}
// Same shape with `controller.disconnect_input(&id)` after the commit (removed endpoint),
// and with a second thread that keeps calling push_records (the commit must not starve).I did not test the multihost path beyond the PR's own test. The PR description says multihost commits still leave queued input for the next transaction, to be fixed separately. |
0e7ebae to
538cc1c
Compare
|
Thanks. 1567002 now adds tests for the three early exits: a connector that holds input back after it commits its part, a client that keeps sending input, and an endpoint that is removed. Each test fails if I remove the rule that covers its case. I left out the paused-endpoint test: the test endpoint still flushes its queue while paused, and the controller still polls paused endpoints, so that test would pass with every early exit removed. |
|
@ryzhyk posted a review for this PR here: #7607 (comment) |
The docs say that a transaction contains the input that the pipeline received between `start_transaction` and `commit_transaction`. In a single-host pipeline, three paths broke that promise: - A commit started in the next step, and that step took at most one batch (`max_batch_size`) from each input endpoint. Input queued beyond that went into the transactions that followed. - A commit that arrived before any step started the transaction erased the transaction. Its input then went in as automatic transactions, one batch per step, and a query could see part of it. - The same erase happened to a transaction that the API started and committed while another transaction committed. This rewrites advance_transaction_state() as two rules, applied in order, in place of nested matches with four asserts: - Start: in state None, start the transaction that the initiators ask for, or else an automatic transaction. - Commit: in state Started, start to commit once no initiator is still adding input and the input queued at the API's commit request is in the circuit. An automatic transaction has no initiators, so it commits in the step that starts it. A transaction committed before a step started it, or during another commit, is no longer a special case, so the two paths that erased it are gone. An API commit now records, for each input endpoint, how many records it has received (CommitTarget). The wait always ends: an endpoint counts as done when it has fed its records, has none buffered, was removed, or gives no records to a step that polls all endpoints. In a multihost pipeline, there is no commit target, because the coordinator commits with steps that feed no input. The erase was removed there too: an early commit now starts and commits the transaction in the coordinator's commit step, where it used to run an automatic transaction. The host reports the same status either way. The tests use a 10-record batch and a step trigger that ignores input, and an hour between clock ticks, so only a commit makes steps. Without the fix, each single-host test commits only part of its records. The multihost test passes without the fix, and fails if the host waits for a commit target. Three more tests cover the cases that end the wait early: a connector that holds input back after it commits its part, input that clients keep sending, and a removed endpoint. Each fails if the rule that covers its case is removed. TransactionState::is_started() has no caller outside the tests now, so it is compiled only for them. Signed-off-by: Ben Pfaff <blp@feldera.com>
A commit starts only after the pipeline ingests the input that was queued at the commit request, and input that arrives meanwhile can join the transaction. The docs promised only the first part. Signed-off-by: Ben Pfaff <blp@feldera.com>
…nsaction(). A positive name is easier to read where it is used. The dev tweak keeps its name, `disable_auto_transaction`, so configurations do not change. Signed-off-by: Ben Pfaff <blp@feldera.com>
An API commit waits for the input queued at the commit request, but a step that fed no records from an endpoint ends the wait. A commit request that arrived after a step polled an endpoint, or listed the endpoints, but before the step decided whether to commit, thus let the transaction commit without the input queued meanwhile. The controller now numbers each step's poll under the transaction lock, and the commit target records the number at the request, so only a poll that began after the request can end the wait. This costs at most one more step. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
The cutover of a concurrent bootstrap pauses ingestion and then waits for the open transaction to commit. An API commit in that window waited for its queued input, which no step could ingest, so neither the transaction nor the bootstrap ever finished. During the cutover, the commit now starts without the queued input, which goes into a later transaction, as it did before commits waited for queued input. The transaction docs say so. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
Regression tests for the previous two commits: a commit request that arrives while a step waits for a slower endpoint after polling the one that holds the input, before or after the transaction starts, or for an endpoint created during the step; and a commit with input queued during a concurrent bootstrap's cutover. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
test_concurrent_bootstrap_api_commit_with_queued_input checked only that the commit and the bootstrap finish. It now also checks that the input queued during the cutover goes into a later transaction. Signed-off-by: Ben Pfaff <blp@feldera.com>
|
@ryzhyk Thanks for the review in #7607 (comment) and for the prototype. Both issues were real. I reviewed
What I checked:
I added one commit, I also rebased onto
|
538cc1c to
6339223
Compare
|
@ryzhyk This needs a (re)review now. |
…rgin. `concurrent_bootstrap_api_commit_with_queued_input` failed in the merge queue because the backfill finished before `/start_transaction` opened the transaction. With 2,000 records, the backfill took about 0.55 s and the transaction opened after about 0.22 s. With 20,000 records, the backfill takes about 4.8 s. Signed-off-by: Ben Pfaff <blp@feldera.com>
Linked Issue
None. Found with Antithesis while testing
/ingresswith API transactions.The docs say that a transaction contains the input that the pipeline receives between
start_transactionandcommit_transaction. In a single-host pipeline, three paths broke that promise, so an ad-hoc query or an/egressreader could see part of a transaction:max_batch_size, 10,000 x workers for/ingress) from each endpoint. Queued rows after that went into later transactions.What changed
advance_transaction_state()is now two rules, applied in order, in place of nested matches with four asserts. The replay branch does not change.None, start the transaction that the initiators ask for, or else an automatic transactionStarted, start to commit when no initiator is still adding input and the input queued at the API's commit request is in the circuitAn automatic transaction has no initiators, so it commits in the step that starts it. A transaction that is committed early is not a special case now, so the two code paths that erased it are gone.
An API commit records how many records each input endpoint has received (
CommitTarget). The wait always ends: an endpoint is done when it has fed those records, has nothing buffered, was removed, or gives no records to a step that polls all endpoints.In a multihost pipeline, there is no commit target, because the coordinator commits with steps that feed no input. An early commit now starts and commits the real transaction in the coordinator's commit step, where it ran an automatic transaction before. The host reports the same status either way. Multihost commits still leave queued input for the next transaction; that needs a coordinator change and a separate fix.
Other commits:
DevTweaks::disable_auto_transaction()is nowenable_auto_transaction(). The configuration key does not change.Describe Manual Test Plan
New controller tests use a 10-record batch and a step trigger that ignores input, so only a commit causes steps:
api_transaction_committed_before_it_startsapi_transaction_commit_includes_queued_recordsapi_transaction_committed_during_another_commitmultihost_api_transaction_committed_before_it_startsauto_transactions_are_enabled_unless_disabledThe new tests passed 30 serial and 30 parallel runs.
cargo test -p dbsp_adapters --libpasses (1058 tests), without the tests that need Kafka, Postgres, DynamoDB or Redis.Checklist
Breaking Changes?
Mark if you think the answer is yes for any of these components:
Describe Incompatible Changes
None. A commit can now take more steps, because it waits for the queued input. The Python SDK's
commit_transaction(wait=True)now waits for the queued rows after an early commit; before, the erase made it return at once.