Repository navigation
[connectors] Judge Kafka test reads against the live high watermark - #7836
Conversation
| .unwrap_or_default(), | ||
| }), | ||
| Some(Err(KafkaError::PartitionEOF(_))) => { | ||
| let (_, high) = watermarks |
There was a problem hiding this comment.
pitfall / medium: A read_committed consumer stops at the LSO, and now it has to reach the uncommitted HWM. Suppose a partition still has a transaction that hasn't been decided yet, for example a producer that died mid-transaction and is waiting for transaction.timeout.ms. In that case this call waits the full 60 s and then panics with "must reach the high watermark", which says nothing about the open transaction. No current caller hits this, because every read happens after EndTxn has been decided. A future test that reads while a transaction is still open would hang, though. Could you document this precondition in the doc comment, or put high and the position in the expect message so the failure is easier to diagnose?
|
Review summary. This is a test-only fix for the flaky
missing / medium: regression test. The description mentions an "early-reader" test that starts // in real.rs: begin reading while EndTxn is still in flight, then release it
proxy.hold_commit(true);
let worker = BatchWorker::new(endpoint);
let reader = std::thread::spawn({ let t = topic.to_owned(); move || committed(&t, 1) });
wait(|| proxy.held_commits() > 0, 30_000).unwrap();
assert_eq!(reader.join().unwrap().len(), 1);
proxy.hold_commit(false);There are no other gate failures: there are no production code changes, no unsafe blocks and no dependency changes. |
88258f2 to
c31a690
Compare
A broker answers EndTxn before it writes the commit markers to the data partitions. The ft output recovery tests hold that reply and read the topic at once, so the read_committed watermark that read_records took before reading could be the last stable offset of the still open transaction. The consumer then passed that offset once the markers landed and librdkafka emits PartitionEOF only once per end offset, so the read waited 60 s and panicked with "the consumer must reach the high watermark". Query the high watermark with a read_uncommitted consumer at each EOF instead of trusting a snapshot taken before reading. Fixes #7765
…mmit lands Also name the topic, partition and the last EOF seen in the watermark panic, and document that read_records needs every transaction decided.
c31a690 to
6a63439
Compare
| let proxy = BrokerProxy::new(default_redpanda_server()); | ||
| let (mut endpoint, _) = endpoint(topic, &proxy, 60_000, 60_000); | ||
| endpoint.batch_start(0, OutputBatchType::Delta).unwrap(); | ||
| endpoint.push_buffer(b"read before its markers").unwrap(); |
There was a problem hiding this comment.
bug (medium, flaky test): push_buffer only queues the record in librdkafka. The transactional producer still has to do AddPartitionsToTxn, wait out linger.ms and send the Produce request. Meanwhile the reader thread may reach its first EOF at offset 0, where the read_uncommitted high watermark is still 0. In that case read_records returns [] and assert_eq!(…len(), 1) fails under CI load. This race could also explain part of the "fails without the fix" result. The fix is to wait for delivery before spawning the reader, so the test really checks "open transaction, record on the broker":
| endpoint.push_buffer(b"read before its markers").unwrap(); | |
| endpoint.push_buffer(b"read before its markers").unwrap(); | |
| wait( | |
| || callbacks.metric("kafka_output_delivered_messages_total") == 1.0, | |
| 30_000, | |
| ) | |
| .unwrap(); |
(Bind callbacks instead of _ from endpoint(...).)
|
Review summary. This change touches only tests. Re-querying the read_uncommitted high watermark at each EOF fixes the stale-snapshot timeout described in #7765, and I agree with the diagnosis.
|
|
fred-review did not finish; see https://github.com/feldera/feldera/actions/runs/41854460216 |
…st Fetch Wait for delivery before the reader starts, assert that the transaction is open, and route the reader through the proxy so the worker only starts after the reader has queried the watermark and sent its first Fetch.
|
LGTM. This PR only touches tests. The fix fits the root cause:
|
mihaibudiu
left a comment
There was a problem hiding this comment.
This is somewhat intricate, but I hope it's an improvement in flakiness
Fixes #7765.
Cause
Redpanda answers EndTxn as soon as the commit is decided and writes the commit markers to the data partitions afterwards (
do_commit_tm_tx: "release control to the client before committing transaction on data partitions"). Therealrecovery tests hold that reply and callcommitted()at once.read_recordssnapshotted the read_committed watermark before reading, so under CI load the snapshot was the last stable offset of the still open transaction. Once the markers landed, the consumer passed that offset. librdkafka emitsPartitionEOFonce per end offset, so the position never matched the stale snapshot and the read timed out after 60 s.The broker side
transaction.timeout.msis not involved. The 60 sREQTMOUTin the logs is librdkafka's socket timeout on the held reply.Reproduced locally against
ghcr.io/feldera/redpanda-ci:v24by snapshotting the watermark while the transaction is open:Fix
read_recordsre-queries the high watermark with a read_uncommitted consumer at each EOF instead of trusting a snapshot. The proxy comment that claimed the transaction is already committed when the broker answers is corrected.Validation
committed()while the transaction is open fails with the exact CI panic without the fix and passes with it.transport::kafka::ft::outputpassed 3 of 3 runs with the fix.