Repository navigation
[connectors] Retain Kafka output through broker outages - #7653
Conversation
| .expect("the sink must report itself healthy again"); | ||
| assert_eq!(callbacks.metric("kafka_output_delivery_errors_total"), 0.0); | ||
| assert!(callbacks.metric("kafka_output_queue_full_wait_seconds_total") > 0.0); | ||
| assert!( |
There was a problem hiding this comment.
bug, high: This assert fails every time locally (6 out of 6 runs). While the broker is down, ClientContext::error sends BrokerTransportFailure and AllBrokersDown to cb.error with the tag kafka_nonft_producer, so errors is never empty. The fix is to record the tag and assert only that no kafka_nonft_delivery error occurred. With that change the test passed 3 out of 3 runs:
- errors: Mutex<Vec<String>>,
+ errors: Mutex<Vec<(Option<&'static str>, String)>>,
...
- fn error(&self, _fatal: bool, error: AnyError, _tag: Option<&'static str>) {
- self.errors.lock().unwrap().push(error.to_string());
+ fn error(&self, _fatal: bool, error: AnyError, tag: Option<&'static str>) {
+ self.errors.lock().unwrap().push((tag, error.to_string()));
...
- assert!(
- callbacks.errors.lock().unwrap().is_empty(),
+ // The down broker raises client-level transport errors; only a delivery
+ // error means a lost message.
+ assert!(
+ !callbacks
+ .errors
+ .lock()
+ .unwrap()
+ .iter()
+ .any(|(tag, _)| *tag == Some("kafka_nonft_delivery")),| Ok(_) => { | ||
| self.metrics | ||
| .delivered_messages | ||
| .fetch_add(1, Ordering::Relaxed); |
There was a problem hiding this comment.
pitfall, medium: Every successful delivery clears the unhealthy state, even while the queue is still full. When the broker is slow but still up, a push can wait on a full queue past MAX_POLLING_INTERVAL while deliveries keep arriving. Each queue_full then sets Unhealthy and the next delivery sets Healthy again within milliseconds, so the status flaps every 5 s and the controller rarely shows the stall. The two flags can also race: swap and the callback are not atomic together. If a delivery's report_healthy lands between a report_unhealthy swap and its callback, the controller ends on Unhealthy while unhealthy == false, and the status stays stuck until the next unhealthy→healthy transition. Consider letting only enqueued() clear a queue-full stall, so deliveries clear only delivery errors. Alternatively, send the callback while holding a lock.
| /// controller's output thread, a DBSP aux thread that `RuntimeHandle::kill` | ||
| /// joins, and a queue that a down broker keeps full would otherwise hold the | ||
| /// teardown open for as long as the broker stays down. | ||
| /// Enqueues `record` in `producer`, waiting out a full producer queue. |
There was a problem hiding this comment.
tech-debt, low: This adds a second doc header under the existing "Hands record to producer…" paragraph, so rustdoc shows two summaries. Please merge them into one header.
|
What I ran
Gates
Tests not covered by this PR
Open question (default): I'm in favour of the unbounded default, given that |
5cc4833 to
e47572a
Compare
|
@bestouff If you like dividing changes into stacks, that's fine, but it would also be fine to submit PRs with more commits in them. |
|
Honestly it was also an excuse to try stacked PRs ... |
d68c243 to
73a3ecd
Compare
|
fred-review did not finish; see https://github.com/feldera/feldera/actions/runs/40752930124 |
| .client() | ||
| .fetch_metadata(Some("test_topic"), Duration::from_secs(10)) | ||
| .unwrap(); | ||
| endpoint.push_buffer(b"recovered").unwrap(); |
There was a problem hiding this comment.
bug / high — flaky test. It failed 5 times in 13 runs locally (output.rs:708, "a successful delivery must restore health"). I added a diagnostic and every failure showed delivered=0 errors=3: the recovered record also misses the 1 s message.timeout.ms, because the producer can still be reconnecting after fetch_metadata returns. Health then stays unhealthy for good. Retrying the push until one record is delivered passed 14 times in 14 runs:
| endpoint.push_buffer(b"recovered").unwrap(); | |
| // The reconnect can still outlast the 1 s deadline; retry until one | |
| // record is delivered. | |
| wait( | |
| || { | |
| let errors = callbacks.metric("kafka_output_delivery_errors_total"); | |
| endpoint.push_buffer(b"recovered").unwrap(); | |
| wait( | |
| || { | |
| callbacks.metric("kafka_output_delivered_messages_total") > 0.0 | |
| || callbacks.metric("kafka_output_delivery_errors_total") > errors | |
| }, | |
| 15_000, | |
| ) | |
| .unwrap(); | |
| callbacks.metric("kafka_output_delivered_messages_total") > 0.0 | |
| }, | |
| 90_000, | |
| ) | |
| .expect("a record must be delivered after the broker is back"); | |
| assert!(matches!( | |
| callbacks.health_status(), | |
| Some(ConnectorHealthStatus::Healthy) | |
| )); |
(Replace the push_buffer and the wait(...) that follows it. A 5 s inner wait was too short: an expiry sometimes took longer than 5 s to show up.)
| // This value was established empirically. | ||
| const MAX_MESSAGE_OVERHEAD: usize = 64; | ||
|
|
||
| const DELIVERY_STALL_TIMEOUT: Duration = Duration::from_secs(10); |
There was a problem hiding this comment.
pitfall / medium. The 10 s threshold is hard-coded and measures age in the queue, not lack of progress. A sink limited by broker throughput, with a large queue (the default queue.buffering.max.kbytes is about 1 GiB), can routinely hold messages for more than 10 s. It would then report unhealthy, or flap, while it is delivering normally. Options: make the threshold configurable, or count only "no delivery completed in N s while messages are pending" as a stall.
|
Review: unbounded retries and health/metrics for the non-FT Kafka sink What I ran (debug build, merge commit):
|
73a3ecd to
82c3f90
Compare
blp
left a comment
There was a problem hiding this comment.
I read all the Rust code changes, except the changes to the tests, and it looks good to me.
| /// Pending deliveries include the record currently waiting for queue space. | ||
| /// IDs preserve enqueue order even when acknowledgements arrive out of order. | ||
| #[derive(Default)] | ||
| struct DeliveryState { | ||
| next_id: usize, | ||
| pending: BTreeMap<usize, Instant>, | ||
| failure: Option<String>, | ||
| fatal_error: Option<String>, | ||
| unhealthy: bool, | ||
| } |
There was a problem hiding this comment.
Needs better comments (probably should have per-field comments).
82c3f90 to
e270e53
Compare
The non-fault-tolerant Kafka sink stopped reporting broker transport failures as connector errors, because librdkafka reconnects without losing queued records and the health status reports a sustained outage. It logged them at debug level, so the log at its default level did not say why a pipeline parked behind the sink. Log them as warnings under the endpoint's span, and say so in the connector docs and the changelog. Signed-off-by: Xavier Bestel <xavier.bestel@feldera.com>
e270e53 to
6d95c35
Compare
|
Verdict: flaky test, not this PR. High confidence. The merge queue run failed only one job, Adapter Tests, on: job: https://github.com/feldera/feldera/actions/runs/40986438451/job/123845019033 (log lines 1614-1696) This test deliberately panics a worker task and then expects Already tracked as a known flake in #7716, which has the same failure from an earlier, unrelated run. I added this run as a new occurrence there. Next step: re-queue — this should pass on a retry. |
The exactly-once Kafka sink stopped after five delivery retries. A failed commit then left its producer in an error state, and later batches failed while the controller continued processing them. A transaction cannot simply be retried: librdkafka fails it at its first failed delivery and purges its queue on abort, and the broker aborts any transaction older than transaction.max.timeout.ms (15 minutes by default). So keep each transaction's encoded records in a replay spool, with a 64 MiB memory threshold and a spill file in the pipeline's storage directory. Retry retriable commits as-is; abort and replay abortable transactions. Fatal and unclassified errors report the failure and hold output until shutdown, retaining the spool. Never recreate a fenced producer. A broker can fence an expired transaction too; restart then recovers from the last committed position. A record that librdkafka refuses for its own sake is reported once and dropped, also when a replay first discovers it. The broker refuses whole batches, so a broker refusal, a revoked permission, a missing topic or an expired transaction drops nothing: after three aborts in a row the sink reports a connector error that names the cause and keeps retrying. Check cancellation between transaction calls, during backoff, and before each replayed send. Expose health and queue, delivery, transaction, and spool metrics. Mock tests cover error classification, replay refusals, broker refusals, cancellation, and fencing. Real-broker tests use a connection proxy to exercise disk-backed replay, delayed and lost commit replies, restart deduplication, ownership fencing, and transaction expiry. Consumers read through the high watermark so assertions catch trailing duplicates as well as missing records. Third step for feldera#7578, stacked on feldera#7653. Signed-off-by: Xavier Bestel <xavier.bestel@feldera.com>
Linked Issue
Second step for #7578, stacked on #7639.
During a broker outage the non-fault-tolerant Kafka sink dropped messages after librdkafka's five-minute delivery timeout. The timeout now defaults to zero: the producer queue fills, the next push blocks, and the controller applies backpressure. Explicit
message.timeout.msanddelivery.timeout.msvalues are kept.The connector reports unhealthy after a delivery fails, or when a message has waited ten seconds with no delivery completed meanwhile, so a slow broker that keeps acknowledging is not a stall. It reports healthy only when nothing is overdue. Broker connection failures are logged as warnings, not counted as errors, and records purged at shutdown raise no error callbacks. New per-endpoint metrics cover queue size and limits, broker retries and timeouts, delivery results, and full-queue wait time. Docs and changelog updated.
Describe Manual Test Plan
feldera-typestimeout regression: default and both spellings.-D warnings,cargo fmt --check,git diff --check.Checklist
Breaking Changes?
Describe Incompatible Changes
The default
message.timeout.msof the non-fault-tolerant Kafka sink changes from 300000 to 0, so a pipeline waits during a long outage instead of dropping records. Setmessage.timeout.msordelivery.timeout.msto300000to keep the old deadline. Broker connection failures no longer count as transport errors.