← 输入框|github.com|刷新
Skip to content

[connectors] Give transport output endpoints a health and metrics channel - #7639

Merged
bestouff merged 2 commits into
mainfrom
issue-7578
Oct 29, 2026
Merged

bestouff merged 2 commits into
mainfrom
issue-7578

Conversation

@bestouff

@bestouff bestouff commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Linked Issue

First step for #7578. The Kafka sinks need to report queue depth and health, but a transport output endpoint only got an error closure from OutputEndpoint::connect. This PR replaces it with the trait object OutputEndpointCallbacks, the output counterpart of InputConsumer:

Method Controller side
error output_transport_error, as before
update_connector_health update_output_connector_health
set_custom_metrics ControllerStatus::set_output_custom_metrics

No connector changes behavior. Next PRs: #7653, #7682, #7683.

Describe Manual Test Plan

Engine test output_endpoint_callbacks_reach_status_and_metrics pushes a gauge, health and an error through the channel and checks the endpoint status and the Prometheus text. It then removes the endpoint and stops the controller, and checks that late reports through the same callbacks do nothing. Clippy -D warnings, cargo fmt and cargo check --all-targets pass. The broker tests were not run; their changes only rename an argument.

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated (no user-visible change)
  • Changelog updated (no user-visible change)

Breaking Changes?

  • OpenAPI / REST HTTP API / feldera-types / manager
  • Feldera SQL
  • feldera-sqllib
  • Python SDK
  • fda
  • Adapters (including configuration)
  • Storage Format / Checkpoints
  • Others (specify): feldera-adapterlib crate API

Describe Incompatible Changes

OutputEndpoint::connect now takes Arc<dyn OutputEndpointCallbacks>; the hidden AsyncErrorCallback type is removed. External implementations adapt the signature and call callbacks.error(...).

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM. This is a refactor that changes no connector's behaviour. The new channel to the controller is wired correctly: status.add_output runs before connect, so health and metrics reported at connect time are kept. Holding a Weak reference makes any call after shutdown a no-op.

What I ran

Check Result
cargo test --locked -p dbsp_adapters --lib output_endpoint_callbacks (debug) pass
Same test run 30 more times, to look for flakiness 30/30 pass. It waits on a condition with no fixed sleeps. Errors and health are written in a fixed order, so the final assertions don't race.
cargo test -p dbsp_adapters --lib output_connector 11/11 pass
pre-commit run --files <changed files> (clippy, fmt, machete, docs, OpenAPI) all pass

I couldn't run the Kafka and NATS endpoint tests, because they need brokers. Their changes only rename things.

Pitfall (medium): feldera-adapterlib is published (publish = true). The signature of OutputEndpoint::connect changes and AsyncErrorCallback is removed. Out-of-tree OutputEndpoint implementations will stop compiling. The type was #[doc(hidden)], so this is probably acceptable, but the PR description should say it is a breaking change to the crate's API.

Missing test (low, follow-up is fine): No test calls the callbacks after the endpoint has been removed. That path should be a silent no-op, because update_output_connector_health and set_output_custom_metrics skip unknown IDs. The Kafka follow-up PRs will hit it from librdkafka threads during disconnect.

@bestouff
bestouff marked this pull request as draft October 26, 2026 18:46
@bestouff bestouff changed the title [adapters] Give transport output endpoints a health and metrics channel [connectors] Give transport output endpoints a health and metrics channel Oct 27, 2026
bestouff added a commit that referenced this pull request Oct 27, 2026
The non-fault-tolerant Kafka sink dropped undelivered messages after
librdkafka's five-minute default timeout while the pipeline kept running.
Default to unbounded delivery retries so a sustained outage fills the
producer queue and applies backpressure. Preserve either explicit timeout
alias so deployments can retain a finite deadline.

Report health from pending delivery age and delivery results. An enqueue
can succeed because a failed delivery freed space, so it cannot establish
recovery. A periodic task on the shared runtime detects quiet stalled
sinks even with statistics disabled. Track delivery identities so progress
on one partition cannot hide an older pending record on another.

Expose queue, request, delivery, and wait metrics through the output
callback API. Keep broker counters monotonic across resets and include
active and interrupted queue waits. Log temporary disconnections without
reporting lost output. Document the changed default and its limits.

Validate timeout aliases, outage recovery and payloads, quiet sinks,
expiry followed by enqueue, out-of-order delivery, fatal health, counter
resets, and cancellation. Check controller callback forwarding and
backpressure with the static engine. The alias and enqueue-health tests
fail when their respective fixes are removed.

Second step for #7578, stacked on #7639.
@bestouff
bestouff marked this pull request as ready for review October 27, 2026 18:23
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM. This refactor keeps behaviour the same: the error path still goes through output_transport_error, and health and metrics reuse the existing ControllerStatus entry points. add_output registers the endpoint's status before connect, so reports made at connect time land.

What I ran

Check Result
cargo test -p static-connectors --test connector_suites -- output_endpoint_callbacks ✅ pass
Same test run 20 more times (flakiness check) ✅ 20/20. It polls with wait, uses no sleeps, and the error is reported before the health on the same thread, so the test has no race
cargo clippy -D warnings on feldera-adapterlib and dbsp_connectors (kafka, nats) with --tests ✅ clean
pre-commit run on the changed files ✅ all passed

Gates: tests ✅, flaky ✅, no unsafe blocks, no new dependencies. Docs aren't needed because users see no change. The feldera-adapterlib API break is declared in the PR description.

Gaps (medium):

  • The Weak upgrade path has no test. Nothing checks that a callback fired after remove_output_endpoint or controller.stop() is a silent no-op. This matters for the follow-up Kafka PRs, where rdkafka delivery threads can outlive the endpoint.
  • The Kafka and NATS endpoint tests weren't run, by the PR author or by me, because they need brokers. The change there is a mechanical rename, so the risk is low.

@bestouff
bestouff added this pull request to stack #7686 October 28, 2026 19:05
@bestouff
bestouff requested a review from blp October 28, 2026 19:07

@blp blp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks as I'd expect. I read the changes except that I just skimmed the test changes.

@bestouff
bestouff force-pushed the issue-7578 branch 2 times, most recently from 6a5b699 to 886f05b Compare October 28, 2026 21:20
…nnel

A transport output endpoint got only an error closure from
`OutputEndpoint::connect`. It had no way to report its health or to
export its own metrics, which only integrated sinks could do because
they hold the controller.

Replace the closure with the trait object `OutputEndpointCallbacks`,
the output counterpart of the reporting methods of `InputConsumer`:

| Method                    | Controller side                       |
|---------------------------|---------------------------------------|
| `error`                   | `output_transport_error`, as before   |
| `update_connector_health` | `update_output_connector_health`      |
| `set_custom_metrics`      | `ControllerStatus::set_output_custom_metrics` |

The two Kafka sinks keep the trait object where they kept the closure.
The other endpoints ignore it as they ignored the closure. Tests that
drive an endpoint without a controller use `MockOutputCallbacks`.

The new controller test connects an endpoint that reports a gauge, its
health and an error through the channel, and checks the endpoint status
and the Prometheus text. With the health and metrics forwarding stubbed
out, it fails on the health assertion.

First step for #7578: the Kafka sinks need this channel to show queue
depth and to report themselves unhealthy while they retry.

Signed-off-by: Xavier Bestel <xavier.bestel@feldera.com>
The non-fault-tolerant Kafka sink dropped undelivered messages after
librdkafka's five-minute default timeout while the pipeline kept running.
Default to unbounded delivery retries so a sustained outage fills the
producer queue and applies backpressure. Preserve either explicit timeout
alias so deployments can retain a finite deadline.

Report health from pending delivery age and delivery results. An enqueue
can succeed because a failed delivery freed space, so it cannot establish
recovery. A periodic task on the shared runtime detects quiet stalled
sinks even with statistics disabled. Track delivery identities so progress
on one partition cannot hide an older pending record on another.

Expose queue, request, delivery, and wait metrics through the output
callback API. Keep broker counters monotonic across resets and include
active and interrupted queue waits. Log temporary disconnections without
reporting lost output. Document the changed default and its limits.

Validate timeout aliases, outage recovery and payloads, quiet sinks,
expiry followed by enqueue, out-of-order delivery, fatal health, counter
resets, and cancellation. Check controller callback forwarding and
backpressure with the static engine. The alias and enqueue-health tests
fail when their respective fixes are removed.

Second step for #7578, stacked on #7639.

Signed-off-by: Xavier Bestel <xavier.bestel@feldera.com>
@bestouff
bestouff added this pull request to the merge queue Oct 29, 2026
Merged via the queue into main with commit 6d6aa60 Oct 29, 2026
1 check passed
@bestouff
bestouff deleted the issue-7578 branch October 29, 2026 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants