Repository navigation
[python] Fix program version wait and retries of non-idempotent requests - #7795
Conversation
| return exc.status_code in self.config.retry_config.retryable_status_codes | ||
| return ( | ||
| exc.status_code in self.config.retry_config.retryable_status_codes | ||
| and (idempotent or exc.status_code in _NOT_APPLIED_STATUS_CODES) |
There was a problem hiding this comment.
bug (medium): Some manager 503s say the request was not applied, and they no longer retry for non-idempotent calls:
DeadlockDetectedandLockTookTooLongcome fromdb/error.rs:483-485after Postgres rolls the transaction back.PipelineInteractionNotDeployedandCompilerUnavailableare rejected before any side effect.
As a result, create_pipeline, patch_pipeline, ingress and checkpoint now fail when the manager is under DB contention, and these calls used to recover. I confirmed this with a unit test: a POST that gets a 503 with error_code: DeadlockDetected makes only 1 call. Suggestion: also retry when exc.error_code is in a "not applied" allow-list.
_NOT_APPLIED_ERROR_CODES = frozenset({"DeadlockDetected", "LockTookTooLong", "PipelineInteractionNotDeployed", "CompilerUnavailable"})
...
and (idempotent or exc.status_code in _NOT_APPLIED_STATUS_CODES
or getattr(exc, "error_code", None) in _NOT_APPLIED_ERROR_CODES)There was a problem hiding this comment.
Agreed, thanks. I checked each code in the manager. DeadlockDetected and LockTookTooLong come after Postgres rolls back the transaction (db/error.rs:480-486). PipelineInteractionNotDeployed (runner/interaction.rs:104) and CompilerUnavailable (pipeline_management.rs:1554-1567) come before any side effect. A non-idempotent request now retries a 503 with one of these codes. Your test is in, with all four codes, and it fails without the fix.
One exception: pause, resume, activate and approve forward to the pipeline first and then call increment_notify_counter (pipeline_interaction.rs:1803, 1868, 1958, 2060). So a DB error code from those handlers can follow an applied action. A repeat asks for the same state, so the retry is harmless. The code comment at _NOT_APPLIED_ERROR_CODES says so.
Claude (AI assistant), replying on behalf of @blp.
| # from its own (e.g. during a rolling upgrade). | ||
| current_version = p.program_version or 0 | ||
| if current_version == expected_program_version: | ||
| if current_version >= expected_program_version: |
There was a problem hiding this comment.
pitfall (medium): >= also accepts a version bumped by a concurrent PATCH from another client. The wait then reports success for a program this caller did not submit. The old != check caught that case. If the only legitimate cause is a platform-version adoption, consider checking it: accept > expected only when platform_version changed, or at least log a warning when current_version > expected.
There was a problem hiding this comment.
Fair point. A check on platform_version would need the platform version that this caller's PATCH set, and _wait_for_compilation does not have it. Also, the old == check caught a concurrent PATCH only if it landed before the poll saw success. So I took the second option: the wait now logs a warning when the compiled version is newer than expected, and a unit test checks the warning.
Claude (AI assistant), replying on behalf of @blp.
|
What I ran
Missing tests (medium): no unit test covers a 503 with a "not applied" @pytest.mark.parametrize("code", ["DeadlockDetected", "LockTookTooLong", "PipelineInteractionNotDeployed"])
def test_post_503_not_applied_is_retried(code):
client = _make_client()
body = json.dumps({"message": "x", "error_code": code, "details": {}}).encode()
with patch_requests("post", [_make_response(503, body, headers={"Retry-After": "0"}),
_make_response(200, b"{}")]) as m:
client.post("/pipelines")
assert m.call_count == 2Breaking change (medium): the Python SDK retry behaviour changes in a way users can see, and the "Changelog updated" box is unchecked. Please add a changelog entry. The gates for commit messages, manual test description and unit tests pass. |
_wait_for_compilation required the program version to equal the expected value. One edit can bump the version twice: the PATCH bumps it, then the compiler bumps it again when it adopts a pending pipeline whose platform version differs from its own (for example during a rolling upgrade). The wait then failed with "program version (3) != expected program version (2)" although the newer program compiled successfully. Accept any version at or above the expected one, and log a warning for a newer version, because a concurrent edit by another client also gives one. An older version still raises. Fixes #7748. Signed-off-by: Ben Pfaff <blp@feldera.com>
…ied. The client retried 502, 503, 504 and read timeouts for every request. After each of these the server may already have applied the request: a gateway sends 502/504 when the backend fails late, and the manager sends 503 when a request it proxies to a pipeline fails mid-way. A retried POST could then be applied twice, for example HTTP ingress inserting the same rows twice. Non-idempotent requests now retry only connect timeouts (the request was never sent), 408/429 (the server rejected it), and a 503 whose manager error code shows the request was not applied (DeadlockDetected, LockTookTooLong, PipelineInteractionNotDeployed, CompilerUnavailable). GET and requests marked idempotent=True keep the full policy. The read-only POSTs /diff, /validate_program and egress are now marked idempotent, so they keep their retries. Fixes #7793. Signed-off-by: Ben Pfaff <blp@feldera.com>
3391c7b to
9b0ff60
Compare
|
Thanks for the review. All three points are addressed, squashed into the two commits:
Claude (AI assistant), replying on behalf of @blp. |
Linked Issue
Fixes #7748. Fixes #7793.
Two independent fixes to the Python client, one commit each.
1.
_wait_for_compilationaccepts a newer program version (#7748)test_adaptive_joins_balancer_policiesfailed withprogram version (3) != expected program version (2). A retried PATCH cannot cause this: the manager bumpsprogram_versiononly when the program changes, and a replay sends the same program. The compiler bumps it a second time when it adopts apendingpipeline whose platform version is different from its own (crates/pipeline-manager/src/db/operations/pipeline.rs:1952-1957). That can happen during a rolling upgrade. The wait now accepts>= expectedand logs a warning for a newer version, because a concurrent edit by another client also gives one. An older version still raises. See the analysis on #7748.2. Requests that are not idempotent do not retry after failures that may come after the request was applied (#7793)
ConnectTimeoutReadTimeoutDeadlockDetected,LockTookTooLong,PipelineInteractionNotDeployed,CompilerUnavailable)GET requests and requests marked
idempotent=Truekeep the full policy. The manager also returns 503 after a request it forwards to a pipeline may have been applied (crates/pipeline-manager/src/runner/interaction.rs:598-610), so before this change a retried HTTP ingress POST could insert the same rows twice. The read-only POSTs/diff,/validate_programand egress are now marked idempotent, so they keep their retries.Cost:
patch_pipeline, ingress, checkpoint and other non-idempotent calls no longer retry through a 502/503 during an upgrade. Possible follow-ups: markresume,clearandstart_compactionidempotent if their API contracts allow it, and add anIdempotency-Keyon the server so ingress can retry safely.Describe Manual Test Plan
uv run pytest tests/unit: 387 passed.test_wait_for_compilation.py, 5 new tests intest_httprequests_retry.py; after review, also the 4 cases oftest_post_503_not_applied_is_retriedand the warning check). With the fix, all pass.Checklist
send_request,_is_retryableandRetryConfig)docs.feldera.com/docs/changelog.md)Breaking Changes?
Describe Incompatible Changes
Non-idempotent requests no longer retry on 502, 504, read timeouts, or a 503 that may follow an applied request. Code that relied on those retries now gets the error instead.
_wait_for_compilation(internal) no longer raises when the program version is newer than expected.PR opened by Claude (AI assistant) on behalf of @blp.