Repository navigation
[python] Make wait_for_completion() wait through a transient UNAVAILABLE status - #7798
Conversation
| latest_stats = self.stats() | ||
| try: | ||
| latest_stats = self.stats() | ||
| except FelderaAPIError as err: |
There was a problem hiding this comment.
bug (medium): this only catches FelderaAPIError. send_request turns a client-side read timeout into FelderaTimeoutError and a dropped connection into FelderaCommunicationError. Neither one subclasses FelderaAPIError, so they still escape on the first failure. That can happen on the overloaded node this PR targets when the client has a timeout set, or when the manager restarts. I checked it with a mocked test: [RUNNING, UNAVAILABLE] statuses and stats side effects [_stats(False), FelderaTimeoutError(...), _stats(True)] raise instead of completing. Consider catching (FelderaAPIError, FelderaTimeoutError, FelderaCommunicationError) here. It would also help to add that test case.
There was a problem hiding this comment.
Fixed. The wait now also catches FelderaTimeoutError and FelderaCommunicationError from stats(), with a parametrized test for both. I also added tests for timeout_s during an UNAVAILABLE stretch, the streak reset after a successful stats(), and a restart that lands in INITIALIZING or REPLAYING.
| f"pipeline '{self.name}' restarted while waiting for" | ||
| f" completion (incarnation {first_incarnation} became" | ||
| f" {incarnation}); the input it had is lost" | ||
| ) |
There was a problem hiding this comment.
pitfall (medium): "the input it had is lost" isn't always true. With fault tolerance on (fault_tolerance.model: exactly_once), a replaced pod recovers from its checkpoint and replays, and its connectors resume. A completion reported after the restart is then valid, but this check makes it a hard failure. Before this PR the wait failed anyway, so it isn't a regression. Still, the message is misleading, and FT users get no way to tolerate the restart. Consider wording such as "input pushed since the last checkpoint may be lost". Alternatively, skip the check when the pipeline has fault tolerance enabled, or make it opt-out.
There was a problem hiding this comment.
Correction: my first reply here said that HTTP input after the last checkpoint is lost even with fault tolerance on. That is wrong. With exactly_once, the HTTP input endpoint journals each request and replays it after a restart (crates/connectors/src/transport/http/input.rs:283), so a completion after the restart can be valid, as you said. For now the message says "input received since the last checkpoint may be lost". The right fix is probably to skip the check when the pipeline uses exactly_once, and I will follow up on that.
There was a problem hiding this comment.
Done in b2c5899. When the incarnation changes, wait_for_completion() now reads the pipeline's fault-tolerance model. With exactly_once, it logs the restart and continues to wait on the new incarnation. With at_least_once or none, it still fails, and the message says that the pipeline has no exactly-once fault tolerance. Tests cover all three models.
|
What I ran
Missing tests (medium)
Gates: the tests, manual test plan, and docstring gates all pass, and there is no breaking change, since the new parameter is optional. The first commit's subject is over 72 chars. That's a soft limit, not a blocker. Apart from the two inline comments, this LGTM. |
| raise RuntimeError( | ||
| f"pipeline '{self.name}' restarted while waiting for" | ||
| f" completion (incarnation {first_incarnation} became" | ||
| f" {incarnation}); the input it had is lost" |
There was a problem hiding this comment.
Agreed, that was unclear and not always true. The message is now "... input received since the last checkpoint may be lost".
The manager reports UNAVAILABLE after one failed status probe (5 s timeout), for example when the node is overloaded. wait_for_completion() failed immediately on this status, which caused flaky CI failures with "Pipeline must be running to wait for completion". Now wait_for_completion() accepts UNAVAILABLE and continues to poll for up to `unavailable_timeout_s` (60 s by default). This includes client timeouts and connection errors from the statistics request. The error for a pipeline that is not running now includes the status and the runtime status details. Fixes #6986. Signed-off-by: Ben Pfaff <blp@feldera.com>
e5bccef to
3dad1cd
Compare
A pipeline pod that Kubernetes replaces (spot preemption, eviction) is UNAVAILABLE for a short time, then starts again. Without exactly-once fault tolerance, it can lose input that it received since its last checkpoint (all of its input, without fault tolerance). Now that wait_for_completion() waits through UNAVAILABLE, the new incarnation could report completion and the wait would succeed with input lost. wait_for_completion() now records the incarnation UUID from the first statistics it receives and fails if it changes. A pipeline with exactly-once fault tolerance journals its input, including HTTP input, and recovers it after a restart, so for such a pipeline the wait continues with the new incarnation. Signed-off-by: Ben Pfaff <blp@feldera.com>
3dad1cd to
b2c5899
Compare
|
Verdict: flaky test, pre-existing on The merge queue run failed in the
This is a known flake: tracked in #7716, now its 5th occurrence against unrelated PRs (docs bumps, SQL compiler changes, Kafka output changes). The test waits a fixed 6s margin for Next step: no action needed on this PR. Re-queueing should pass — none of the prior occurrences repeated on retry. |
|
Verdict: flaky test, pre-existing on Different test than the last fred-observe comment on this PR (that one was
PR #7798 only touches Filed as #7820 (assigned to the author of that commit). Next step: no action needed on this PR. Re-queueing should pass. |
Linked Issue
Fixes #6986. Analysis: #6986 (comment)
The manager sets UNAVAILABLE after one failed status probe (5 s timeout). On overloaded CI nodes this occurs, and
wait_for_completion()failed immediately with "Pipeline must be running to wait for completion".Two commits:
wait_for_completion()accepts UNAVAILABLE and continues to poll for up tounavailable_timeout_s(new parameter, 60 s by default). This includes client timeouts and connection errors fromstats(). The "not running" error now includes the status and the runtime status details.wait_for_completion()records the incarnation UUID from the first statistics and fails if it changes. Without this, a pod that Kubernetes replaces (spot preemption, eviction) can report completion with input lost, and the wait would succeed. A pipeline withexactly_oncefault tolerance recovers its input (HTTP input too, from the journal), so for it the wait continues with the new incarnation.Not included: the 410
PipelineRestartedfailure from the same issue. It comes from the completion-token wait ininput_json(), after a real pipeline restart, and the SDK correctly fails there.Describe Manual Test Plan
New unit tests in
python/tests/unit/test_wait_for_completion_unavailable.py(15 tests). I checked that they catch the bugs they target: with the oldpipeline.py, catching onlyFelderaAPIError, no reset of the UNAVAILABLE timer, no incarnation check, or a wrong fault-tolerance check, the related tests fail.pytest tests/unit: 382 passed.ruff formatandruff checkare clean.Checklist
Breaking Changes?
Mark if you think the answer is yes for any of these components:
Describe Incompatible Changes
None. The new parameter is optional. The "not running" error message keeps its old text as a prefix.