← 输入框|github.com|刷新
Skip to content

[python] Make wait_for_completion() wait through a transient UNAVAILABLE status - #7798

Merged
mihaibudiu merged 2 commits into
mainfrom
sdk-wait-for-completion-unavailable
Nov 4, 2026
Merged

mihaibudiu merged 2 commits into
mainfrom
sdk-wait-for-completion-unavailable

Conversation

@blp

@blp blp commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Linked Issue

Fixes #6986. Analysis: #6986 (comment)

The manager sets UNAVAILABLE after one failed status probe (5 s timeout). On overloaded CI nodes this occurs, and wait_for_completion() failed immediately with "Pipeline must be running to wait for completion".

Two commits:

  1. wait_for_completion() accepts UNAVAILABLE and continues to poll for up to unavailable_timeout_s (new parameter, 60 s by default). This includes client timeouts and connection errors from stats(). The "not running" error now includes the status and the runtime status details.
  2. wait_for_completion() records the incarnation UUID from the first statistics and fails if it changes. Without this, a pod that Kubernetes replaces (spot preemption, eviction) can report completion with input lost, and the wait would succeed. A pipeline with exactly_once fault tolerance recovers its input (HTTP input too, from the journal), so for it the wait continues with the new incarnation.

Not included: the 410 PipelineRestarted failure from the same issue. It comes from the completion-token wait in input_json(), after a real pipeline restart, and the SDK correctly fails there.

Describe Manual Test Plan

New unit tests in python/tests/unit/test_wait_for_completion_unavailable.py (15 tests). I checked that they catch the bugs they target: with the old pipeline.py, catching only FelderaAPIError, no reset of the UNAVAILABLE timer, no incarnation check, or a wrong fault-tolerance check, the related tests fail. pytest tests/unit: 382 passed. ruff format and ruff check are clean.

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

Mark if you think the answer is yes for any of these components:

Describe Incompatible Changes

None. The new parameter is optional. The "not running" error message keeps its old text as a prefix.

@blp
blp requested a review from mihaibudiu November 3, 2026 01:05
@blp blp self-assigned this Nov 3, 2026
@blp blp added bug Something isn't working QA Testing and quality assurance python Pull requests that update python code enterprise Issue related to Feldera Enterprise features. labels Nov 3, 2026
Comment thread python/feldera/pipeline.py Outdated
latest_stats = self.stats()
try:
latest_stats = self.stats()
except FelderaAPIError as err:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug (medium): this only catches FelderaAPIError. send_request turns a client-side read timeout into FelderaTimeoutError and a dropped connection into FelderaCommunicationError. Neither one subclasses FelderaAPIError, so they still escape on the first failure. That can happen on the overloaded node this PR targets when the client has a timeout set, or when the manager restarts. I checked it with a mocked test: [RUNNING, UNAVAILABLE] statuses and stats side effects [_stats(False), FelderaTimeoutError(...), _stats(True)] raise instead of completing. Consider catching (FelderaAPIError, FelderaTimeoutError, FelderaCommunicationError) here. It would also help to add that test case.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. The wait now also catches FelderaTimeoutError and FelderaCommunicationError from stats(), with a parametrized test for both. I also added tests for timeout_s during an UNAVAILABLE stretch, the streak reset after a successful stats(), and a restart that lands in INITIALIZING or REPLAYING.

f"pipeline '{self.name}' restarted while waiting for"
f" completion (incarnation {first_incarnation} became"
f" {incarnation}); the input it had is lost"
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): "the input it had is lost" isn't always true. With fault tolerance on (fault_tolerance.model: exactly_once), a replaced pod recovers from its checkpoint and replays, and its connectors resume. A completion reported after the restart is then valid, but this check makes it a hard failure. Before this PR the wait failed anyway, so it isn't a regression. Still, the message is misleading, and FT users get no way to tolerate the restart. Consider wording such as "input pushed since the last checkpoint may be lost". Alternatively, skip the check when the pipeline has fault tolerance enabled, or make it opt-out.

@blp blp Nov 3, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction: my first reply here said that HTTP input after the last checkpoint is lost even with fault tolerance on. That is wrong. With exactly_once, the HTTP input endpoint journals each request and replays it after a restart (crates/connectors/src/transport/http/input.rs:283), so a completion after the restart can be valid, as you said. For now the message says "input received since the last checkpoint may be lost". The right fix is probably to skip the check when the pipeline uses exactly_once, and I will follow up on that.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in b2c5899. When the incarnation changes, wait_for_completion() now reads the pipeline's fault-tolerance model. With exactly_once, it logs the restart and continues to wait on the new incarnation. With at_least_once or none, it still fails, and the message says that the pipeline has no exactly-once fault tolerance. Tests cover all three models.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
uv run --locked pytest tests/unit ✅ 374 passed
pre-commit run --files <changed> ✅ ruff and ruff-format pass
Extra QA test: FelderaTimeoutError / FelderaCommunicationError from stats() while UNAVAILABLE ❌ the error escapes (see the inline comment)
Extra QA test: timeout_s still applies while UNAVAILABLE ✅

Missing tests (medium)

  1. Non-API errors (timeout or connection) from stats() while UNAVAILABLE.
  2. timeout_s firing during an UNAVAILABLE streak, before unavailable_timeout_s.
  3. The UNAVAILABLE streak resetting after a successful stats(), so two short outages don't add up to one timeout.
  4. A restart that lands in INITIALIZING or REPLAYING, where stats() fails, rather than going straight back to RUNNING.

Gates: the tests, manual test plan, and docstring gates all pass, and there is no breaking change, since the new parameter is optional. The first commit's subject is over 72 chars. That's a soft limit, not a blocker. Apart from the two inline comments, this LGTM.

Comment thread python/feldera/pipeline.py Outdated
raise RuntimeError(
f"pipeline '{self.name}' restarted while waiting for"
f" completion (incarnation {first_incarnation} became"
f" {incarnation}); the input it had is lost"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the input it had?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, that was unclear and not always true. The message is now "... input received since the last checkpoint may be lost".

The manager reports UNAVAILABLE after one failed status probe (5 s
timeout), for example when the node is overloaded. wait_for_completion()
failed immediately on this status, which caused flaky CI failures with
"Pipeline must be running to wait for completion".

Now wait_for_completion() accepts UNAVAILABLE and continues to poll for
up to `unavailable_timeout_s` (60 s by default). This includes client
timeouts and connection errors from the statistics request. The error
for a pipeline that is not running now includes the status and the
runtime status details.

Fixes #6986.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp force-pushed the sdk-wait-for-completion-unavailable branch from e5bccef to 3dad1cd Compare November 3, 2026 03:01
A pipeline pod that Kubernetes replaces (spot preemption, eviction) is
UNAVAILABLE for a short time, then starts again. Without exactly-once
fault tolerance, it can lose input that it received since its last
checkpoint (all of its input, without fault tolerance). Now that
wait_for_completion() waits through UNAVAILABLE, the new incarnation
could report completion and the wait would succeed with input lost.

wait_for_completion() now records the incarnation UUID from the first
statistics it receives and fails if it changes. A pipeline with
exactly-once fault tolerance journals its input, including HTTP input,
and recovers it after a restart, so for such a pipeline the wait
continues with the new incarnation.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp force-pushed the sdk-wait-for-completion-unavailable branch from 3dad1cd to b2c5899 Compare November 3, 2026 17:08
@blp
blp enabled auto-merge November 3, 2026 17:11
@blp
blp disabled auto-merge November 3, 2026 17:17
@blp
blp added this pull request to the merge queue Nov 3, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 3, 2026
@mythical-fred-oss

mythical-fred-oss Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Verdict: flaky test, pre-existing on main — not caused by this PR. High confidence.

The merge queue run failed in the Adapter Tests job. The PR only touches python/feldera/pipeline.py and its test file; the failure is in Rust connector tests the PR never touched.

Job Test Adapters
Failing test integrated::postgres::cdc_input::tests::a_panic_on_the_worker_task_still_shuts_the_runtime_down_in_time
Log worker task panics deliberately at cdc_input.rs:3974, then cdc_input.rs:3983 fails with the wrapper thread hung on the runtime after a panic: Timeout — the shutdown-timing assertion the test exists to check

This is a known flake: tracked in #7716, now its 5th occurrence against unrelated PRs (docs bumps, SQL compiler changes, Kafka output changes). The test waits a fixed 6s margin for shutdown_timeout to return after a panic; under CI host contention that margin is apparently too tight.

Next step: no action needed on this PR. Re-queueing should pass — none of the prior occurrences repeated on retry.

@blp
blp added this pull request to the merge queue Nov 3, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 3, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Verdict: flaky test, pre-existing on main — not caused by this PR. High confidence.

Different test than the last fred-observe comment on this PR (that one was cdc_input / #7716). This run failed in a NATS connector test instead.

Job Test Adapters
Failing test transport::nats::input::test::hardening_tests::library_recreated_consumer_is_deleted_on_pause
Log panics at hardening_tests.rs:548 (log lines ~2466-2612): "Connector's own count reached 1, but the server disagrees: found 0 consumers" after a burst of client error: nats: IO error reconnects

PR #7798 only touches python/feldera/pipeline.py and its test file — never crates/connectors. The test last changed in 9638852 ("NATS: use async-nats 0.51.0", 2026-10-29), the likely source of the new timing sensitivity.

Filed as #7820 (assigned to the author of that commit).

Next step: no action needed on this PR. Re-queueing should pass.

@mihaibudiu
mihaibudiu added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit c8adf3f Nov 4, 2026
1 check passed
@mihaibudiu
mihaibudiu deleted the sdk-wait-for-completion-unavailable branch November 4, 2026 05:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enterprise Issue related to Feldera Enterprise features. python Pull requests that update python code QA Testing and quality assurance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Runtime Integration Tests (amd64): pipeline stopped mid-test, RuntimeExecutionError "Pipeline must be running to wait for completion"

2 participants