← 输入框|github.com|刷新
Skip to content

[connectors] Kafka test proxy: do not hold EndTxn error replies. - #7847

Merged
bestouff merged 1 commit into
mainfrom
kafka-proxy-endtxn-error
Nov 5, 2026
Merged

bestouff merged 1 commit into
mainfrom
kafka-proxy-endtxn-error

Conversation

@blp

@blp blp commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

The test proxy held every EndTxn reply as if the broker had committed the transaction. If the broker answered with an error, the client could not retry, the transaction stayed open, and a read_committed consumer never reached the high watermark.

Now the proxy passes EndTxn error replies to the client, prints proxy: passing EndTxn vN error E to the client, counts them, and holds only a reply that succeeds.

Might help with #7765. See #7765 (comment) for the analysis.

Testing

Check Result
cargo test -p dbsp_connectors --lib transport::kafka::ft::output::tests::real against Redpanda v24.2.4 5 passed
proxy_passes_end_txn_errors_and_holds_the_success (mock cluster, one injected EndTxn error) passed; fails (passed_end_txn_errors is 0, not 1) with the error branch disabled
end_txn_error_reads_every_header_version passed

If the flake happens again, the log line shows whether an EndTxn error reply caused it.

@blp
blp requested a review from bestouff November 4, 2026 22:45
@blp blp added connectors Issues related to the adapters/connectors crate ft Fault tolerant, distributed, and scale-out implementation QA Testing and quality assurance rust Pull requests that update Rust code labels Nov 4, 2026
@blp blp self-assigned this Nov 4, 2026
if error != 0 {
// The broker did not end the transaction. Let the client
// see the error and retry; hold the reply that succeeds.
eprintln!("proxy: passing EndTxn v{version} error {error} to the client");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing (medium): The PR says this path hasn't run, but you can hit it without Redpanda: put a BrokerProxy in front of a MockCluster and call cluster.request_errors(EndTxn, &[CONCURRENT_TRANSACTIONS]). I tried it (diff in the top-level comment). It passes, and the log shows proxy: passing EndTxn v1 error 51 to the client. One catch: that test also passes against the old code, because the held error goes out on release and the client retries. To make it a real regression test, count the errors that get passed through, e.g. a passed_end_txn_errors: AtomicUsize in State, and assert == 1. A log line alone is not enough.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, done. The proxy now counts the EndTxn errors that it passes through (passed_end_txn_errors). Your mock-cluster test is in as proxy_passes_end_txn_errors_and_holds_the_success and asserts that the count is 1. With the error branch disabled, it fails with left: 0, right: 1, so it now catches the old behavior. Your end_txn_error_reads_every_header_version unit test is in too. Both pass, and clippy is clean.

Agreed on #7765: it stays open until the merge queue has run clean for a while.

-- Claude Code, replying at @blp's request.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Review: test-proxy fix for the EndTxn flake (#7765)

The header parsing is correct for EndTxn v0–v2 and for v3+ with flexible-header tags. Factoring out skip_tagged_fields is a clean refactor. This PR changes only test code, so the SQL-type, docs and breaking-change gates do not apply.

Ran Result
cargo test --locked -p dbsp_connectors --lib -- …tests::proxy (with the two tests below added) 2 passed
pre-commit run --files proxy.rs passed
tests::real (needs Redpanda) not run
  • missing (medium), gate 2 (tests for changed behaviour): The new error branch and end_txn_error have no test, and the PR says they never ran. Both tests below pass. The mock-cluster test does reach the new branch: the log shows the line proxy: passing EndTxn v1 error 51 to the client. It needs the counter assertion from the inline comment, though; without it, the old code also passes.
  • pitfall (medium): The PR says it possibly fixes Kafka ft output: restart_skips_a_commit_whose_reply_was_lost times out waiting for high watermark #7765. Leave the issue open until the merge queue has run clean for a while. If the flake comes back without the new log line, the cause is somewhere else.
QA tests I added locally
--- a/crates/connectors/src/transport/kafka/ft/output/tests/proxy.rs
+++ b/crates/connectors/src/transport/kafka/ft/output/tests/proxy.rs
+#[test]
+fn end_txn_error_reads_every_header_version() {
+    // correlation id, throttle time, error code 51
+    let v0 = [0, 0, 0, 7, 0, 0, 0, 0, 0, 51];
+    assert_eq!(end_txn_error(&v0, 0), 51);
+    assert_eq!(end_txn_error(&v0, 2), 51);
+    // correlation id, no header tags, throttle time, error code 0, body tags
+    let v3 = [0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0];
+    assert_eq!(end_txn_error(&v3, 3), 0);
+    // correlation id, one 2-byte header tag, throttle time, error code 47
+    let tagged = [0, 0, 0, 7, 1, 0, 2, 9, 9, 0, 0, 0, 0, 0, 47, 0];
+    assert_eq!(end_txn_error(&tagged, 4), 47);
+}
--- a/crates/connectors/src/transport/kafka/ft/output/tests.rs
+++ b/crates/connectors/src/transport/kafka/ft/output/tests.rs
+#[test]
+fn proxy_passes_end_txn_errors_and_holds_the_success() {
+    let cluster = cluster();
+    let proxy = proxy::BrokerProxy::new(cluster.bootstrap_servers());
+    let config: KafkaOutputConfig = serde_json::from_value(json!({
+        "topic": TOPIC,
+        "bootstrap.servers": proxy.address,
+        "fault_tolerance": {
+            "producer_options": { "transaction.timeout.ms": "60000" }
+        }
+    }))
+    .unwrap();
+    let mut endpoint = KafkaOutputEndpoint::new(config, CancellationToken::new(), None).unwrap();
+    let callbacks = Arc::new(RecordingCallbacks::default());
+    endpoint.connect(callbacks.clone()).unwrap();
+    endpoint.batch_start(0, OutputBatchType::Delta).unwrap();
+    endpoint.push_buffer(b"retried commit").unwrap();
+    cluster.request_errors(
+        RDKafkaApiKey::EndTxn,
+        &[RDKafkaRespErr::RD_KAFKA_RESP_ERR_CONCURRENT_TRANSACTIONS],
+    );
+    proxy.hold_commit(true);
+    let worker = BatchWorker::new(endpoint);
+    wait(|| proxy.held_commits() > 0, 30_000).expect("the successful EndTxn must be held");
+    assert_eq!(proxy.held_commits(), 1);
+    // + assert_eq!(proxy.passed_end_txn_errors(), 1); once the counter exists
+    proxy.hold_commit(false);
+    let (result, _endpoint) = worker.finish();
+    result.unwrap();
+    assert_eq!(read_topic(&cluster).len(), 1);
+}

@blp
blp force-pushed the kafka-proxy-endtxn-error branch 2 times, most recently from ceccde0 to 77a20bb Compare November 5, 2026 00:42
The proxy held every EndTxn reply as if the broker had committed the
transaction. If the broker answered with an error, the client could not
retry, the transaction stayed open, and a read_committed consumer never
reached the high watermark. Now the proxy passes error replies to the
client, prints them, and holds only a reply that succeeds.

Might help with #7765.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@bestouff
bestouff force-pushed the kafka-proxy-endtxn-error branch from 77a20bb to 7155ceb Compare November 5, 2026 07:21
@bestouff
bestouff added this pull request to the merge queue Nov 5, 2026
Merged via the queue into main with commit 3585883 Nov 5, 2026
1 check passed
@bestouff
bestouff deleted the kafka-proxy-endtxn-error branch November 5, 2026 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

connectors Issues related to the adapters/connectors crate ft Fault tolerant, distributed, and scale-out implementation QA Testing and quality assurance rust Pull requests that update Rust code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants