Repository navigation
[connectors] Kafka test proxy: do not hold EndTxn error replies. - #7847
Conversation
| if error != 0 { | ||
| // The broker did not end the transaction. Let the client | ||
| // see the error and retry; hold the reply that succeeds. | ||
| eprintln!("proxy: passing EndTxn v{version} error {error} to the client"); |
There was a problem hiding this comment.
missing (medium): The PR says this path hasn't run, but you can hit it without Redpanda: put a BrokerProxy in front of a MockCluster and call cluster.request_errors(EndTxn, &[CONCURRENT_TRANSACTIONS]). I tried it (diff in the top-level comment). It passes, and the log shows proxy: passing EndTxn v1 error 51 to the client. One catch: that test also passes against the old code, because the held error goes out on release and the client retries. To make it a real regression test, count the errors that get passed through, e.g. a passed_end_txn_errors: AtomicUsize in State, and assert == 1. A log line alone is not enough.
There was a problem hiding this comment.
Thanks, done. The proxy now counts the EndTxn errors that it passes through (passed_end_txn_errors). Your mock-cluster test is in as proxy_passes_end_txn_errors_and_holds_the_success and asserts that the count is 1. With the error branch disabled, it fails with left: 0, right: 1, so it now catches the old behavior. Your end_txn_error_reads_every_header_version unit test is in too. Both pass, and clippy is clean.
Agreed on #7765: it stays open until the merge queue has run clean for a while.
-- Claude Code, replying at @blp's request.
|
Review: test-proxy fix for the EndTxn flake (#7765) The header parsing is correct for EndTxn v0–v2 and for v3+ with flexible-header tags. Factoring out
QA tests I added locally--- a/crates/connectors/src/transport/kafka/ft/output/tests/proxy.rs
+++ b/crates/connectors/src/transport/kafka/ft/output/tests/proxy.rs
+#[test]
+fn end_txn_error_reads_every_header_version() {
+ // correlation id, throttle time, error code 51
+ let v0 = [0, 0, 0, 7, 0, 0, 0, 0, 0, 51];
+ assert_eq!(end_txn_error(&v0, 0), 51);
+ assert_eq!(end_txn_error(&v0, 2), 51);
+ // correlation id, no header tags, throttle time, error code 0, body tags
+ let v3 = [0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0];
+ assert_eq!(end_txn_error(&v3, 3), 0);
+ // correlation id, one 2-byte header tag, throttle time, error code 47
+ let tagged = [0, 0, 0, 7, 1, 0, 2, 9, 9, 0, 0, 0, 0, 0, 47, 0];
+ assert_eq!(end_txn_error(&tagged, 4), 47);
+}
--- a/crates/connectors/src/transport/kafka/ft/output/tests.rs
+++ b/crates/connectors/src/transport/kafka/ft/output/tests.rs
+#[test]
+fn proxy_passes_end_txn_errors_and_holds_the_success() {
+ let cluster = cluster();
+ let proxy = proxy::BrokerProxy::new(cluster.bootstrap_servers());
+ let config: KafkaOutputConfig = serde_json::from_value(json!({
+ "topic": TOPIC,
+ "bootstrap.servers": proxy.address,
+ "fault_tolerance": {
+ "producer_options": { "transaction.timeout.ms": "60000" }
+ }
+ }))
+ .unwrap();
+ let mut endpoint = KafkaOutputEndpoint::new(config, CancellationToken::new(), None).unwrap();
+ let callbacks = Arc::new(RecordingCallbacks::default());
+ endpoint.connect(callbacks.clone()).unwrap();
+ endpoint.batch_start(0, OutputBatchType::Delta).unwrap();
+ endpoint.push_buffer(b"retried commit").unwrap();
+ cluster.request_errors(
+ RDKafkaApiKey::EndTxn,
+ &[RDKafkaRespErr::RD_KAFKA_RESP_ERR_CONCURRENT_TRANSACTIONS],
+ );
+ proxy.hold_commit(true);
+ let worker = BatchWorker::new(endpoint);
+ wait(|| proxy.held_commits() > 0, 30_000).expect("the successful EndTxn must be held");
+ assert_eq!(proxy.held_commits(), 1);
+ // + assert_eq!(proxy.passed_end_txn_errors(), 1); once the counter exists
+ proxy.hold_commit(false);
+ let (result, _endpoint) = worker.finish();
+ result.unwrap();
+ assert_eq!(read_topic(&cluster).len(), 1);
+} |
ceccde0 to
77a20bb
Compare
The proxy held every EndTxn reply as if the broker had committed the transaction. If the broker answered with an error, the client could not retry, the transaction stayed open, and a read_committed consumer never reached the high watermark. Now the proxy passes error replies to the client, prints them, and holds only a reply that succeeds. Might help with #7765. Signed-off-by: Ben Pfaff <blp@feldera.com>
77a20bb to
7155ceb
Compare
The test proxy held every EndTxn reply as if the broker had committed the transaction. If the broker answered with an error, the client could not retry, the transaction stayed open, and a
read_committedconsumer never reached the high watermark.Now the proxy passes EndTxn error replies to the client, prints
proxy: passing EndTxn vN error E to the client, counts them, and holds only a reply that succeeds.Might help with #7765. See #7765 (comment) for the analysis.
Testing
cargo test -p dbsp_connectors --lib transport::kafka::ft::output::tests::realagainst Redpanda v24.2.4proxy_passes_end_txn_errors_and_holds_the_success(mock cluster, one injected EndTxn error)passed_end_txn_errorsis 0, not 1) with the error branch disabledend_txn_error_reads_every_header_versionIf the flake happens again, the log line shows whether an EndTxn error reply caused it.