Repository navigation
[connectors] Give transport output endpoints a health and metrics channel - #7639
Conversation
|
LGTM. This is a refactor that changes no connector's behaviour. The new channel to the controller is wired correctly: What I ran
I couldn't run the Kafka and NATS endpoint tests, because they need brokers. Their changes only rename things. Pitfall (medium): Missing test (low, follow-up is fine): No test calls the callbacks after the endpoint has been removed. That path should be a silent no-op, because |
The non-fault-tolerant Kafka sink dropped undelivered messages after librdkafka's five-minute default timeout while the pipeline kept running. Default to unbounded delivery retries so a sustained outage fills the producer queue and applies backpressure. Preserve either explicit timeout alias so deployments can retain a finite deadline. Report health from pending delivery age and delivery results. An enqueue can succeed because a failed delivery freed space, so it cannot establish recovery. A periodic task on the shared runtime detects quiet stalled sinks even with statistics disabled. Track delivery identities so progress on one partition cannot hide an older pending record on another. Expose queue, request, delivery, and wait metrics through the output callback API. Keep broker counters monotonic across resets and include active and interrupted queue waits. Log temporary disconnections without reporting lost output. Document the changed default and its limits. Validate timeout aliases, outage recovery and payloads, quiet sinks, expiry followed by enqueue, out-of-order delivery, fatal health, counter resets, and cancellation. Check controller callback forwarding and backpressure with the static engine. The alias and enqueue-health tests fail when their respective fixes are removed. Second step for #7578, stacked on #7639.
|
LGTM. This refactor keeps behaviour the same: the error path still goes through What I ran
Gates: tests ✅, flaky ✅, no Gaps (medium):
|
blp
left a comment
There was a problem hiding this comment.
Looks as I'd expect. I read the changes except that I just skimmed the test changes.
6a5b699 to
886f05b
Compare
…nnel A transport output endpoint got only an error closure from `OutputEndpoint::connect`. It had no way to report its health or to export its own metrics, which only integrated sinks could do because they hold the controller. Replace the closure with the trait object `OutputEndpointCallbacks`, the output counterpart of the reporting methods of `InputConsumer`: | Method | Controller side | |---------------------------|---------------------------------------| | `error` | `output_transport_error`, as before | | `update_connector_health` | `update_output_connector_health` | | `set_custom_metrics` | `ControllerStatus::set_output_custom_metrics` | The two Kafka sinks keep the trait object where they kept the closure. The other endpoints ignore it as they ignored the closure. Tests that drive an endpoint without a controller use `MockOutputCallbacks`. The new controller test connects an endpoint that reports a gauge, its health and an error through the channel, and checks the endpoint status and the Prometheus text. With the health and metrics forwarding stubbed out, it fails on the health assertion. First step for #7578: the Kafka sinks need this channel to show queue depth and to report themselves unhealthy while they retry. Signed-off-by: Xavier Bestel <xavier.bestel@feldera.com>
The non-fault-tolerant Kafka sink dropped undelivered messages after librdkafka's five-minute default timeout while the pipeline kept running. Default to unbounded delivery retries so a sustained outage fills the producer queue and applies backpressure. Preserve either explicit timeout alias so deployments can retain a finite deadline. Report health from pending delivery age and delivery results. An enqueue can succeed because a failed delivery freed space, so it cannot establish recovery. A periodic task on the shared runtime detects quiet stalled sinks even with statistics disabled. Track delivery identities so progress on one partition cannot hide an older pending record on another. Expose queue, request, delivery, and wait metrics through the output callback API. Keep broker counters monotonic across resets and include active and interrupted queue waits. Log temporary disconnections without reporting lost output. Document the changed default and its limits. Validate timeout aliases, outage recovery and payloads, quiet sinks, expiry followed by enqueue, out-of-order delivery, fatal health, counter resets, and cancellation. Check controller callback forwarding and backpressure with the static engine. The alias and enqueue-health tests fail when their respective fixes are removed. Second step for #7578, stacked on #7639. Signed-off-by: Xavier Bestel <xavier.bestel@feldera.com>
Linked Issue
First step for #7578. The Kafka sinks need to report queue depth and health, but a transport output endpoint only got an error closure from
OutputEndpoint::connect. This PR replaces it with the trait objectOutputEndpointCallbacks, the output counterpart ofInputConsumer:erroroutput_transport_error, as beforeupdate_connector_healthupdate_output_connector_healthset_custom_metricsControllerStatus::set_output_custom_metricsNo connector changes behavior. Next PRs: #7653, #7682, #7683.
Describe Manual Test Plan
Engine test
output_endpoint_callbacks_reach_status_and_metricspushes a gauge, health and an error through the channel and checks the endpoint status and the Prometheus text. It then removes the endpoint and stops the controller, and checks that late reports through the same callbacks do nothing. Clippy-D warnings,cargo fmtandcargo check --all-targetspass. The broker tests were not run; their changes only rename an argument.Checklist
Breaking Changes?
feldera-adapterlibcrate APIDescribe Incompatible Changes
OutputEndpoint::connectnow takesArc<dyn OutputEndpointCallbacks>; the hiddenAsyncErrorCallbacktype is removed. External implementations adapt the signature and callcallbacks.error(...).