← 输入框|github.com|刷新
Skip to content

[python] Fix program version wait and retries of non-idempotent requests - #7795

Merged
blp merged 2 commits into
mainfrom
python-retry-program-version
Nov 3, 2026
Merged

blp merged 2 commits into
mainfrom
python-retry-program-version

Conversation

@blp

@blp blp commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Linked Issue

Fixes #7748. Fixes #7793.

Two independent fixes to the Python client, one commit each.

1. _wait_for_compilation accepts a newer program version (#7748)

test_adaptive_joins_balancer_policies failed with program version (3) != expected program version (2). A retried PATCH cannot cause this: the manager bumps program_version only when the program changes, and a replay sends the same program. The compiler bumps it a second time when it adopts a pending pipeline whose platform version is different from its own (crates/pipeline-manager/src/db/operations/pipeline.rs:1952-1957). That can happen during a rolling upgrade. The wait now accepts >= expected and logs a warning for a newer version, because a concurrent edit by another client also gives one. An older version still raises. See the analysis on #7748.

2. Requests that are not idempotent do not retry after failures that may come after the request was applied (#7793)

Failure Before (non-idempotent) After (non-idempotent)
ConnectTimeout retry retry
ReadTimeout retry no retry
408, 429 retry retry
502, 504 retry no retry
503 retry retry only if the manager's error code shows the request was not applied (DeadlockDetected, LockTookTooLong, PipelineInteractionNotDeployed, CompilerUnavailable)

GET requests and requests marked idempotent=True keep the full policy. The manager also returns 503 after a request it forwards to a pipeline may have been applied (crates/pipeline-manager/src/runner/interaction.rs:598-610), so before this change a retried HTTP ingress POST could insert the same rows twice. The read-only POSTs /diff, /validate_program and egress are now marked idempotent, so they keep their retries.

Cost: patch_pipeline, ingress, checkpoint and other non-idempotent calls no longer retry through a 502/503 during an upgrade. Possible follow-ups: mark resume, clear and start_compaction idempotent if their API contracts allow it, and add an Idempotency-Key on the server so ingress can retry safely.

Describe Manual Test Plan

  • uv run pytest tests/unit: 387 passed.
  • Revert check: with the old code put back, the new tests fail as they should (2 of 3 in test_wait_for_compilation.py, 5 new tests in test_httprequests_retry.py; after review, also the 4 cases of test_post_503_not_applied_is_retried and the warning check). With the fix, all pass.

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated (docstrings of send_request, _is_retryable and RetryConfig)
  • Changelog updated (docs.feldera.com/docs/changelog.md)

Breaking Changes?

  • OpenAPI / REST HTTP API / feldera-types / manager
  • Feldera SQL (Syntax, Semantics)
  • feldera-sqllib (incl. dependencies fxp, etc.)
  • Python SDK
  • fda (CLI arguments)
  • Adapters (including configuration)
  • Storage Format / Checkpoints
  • Others (specify)

Describe Incompatible Changes

Non-idempotent requests no longer retry on 502, 504, read timeouts, or a 503 that may follow an applied request. Code that relied on those retries now gets the error instead. _wait_for_compilation (internal) no longer raises when the program version is newer than expected.

PR opened by Claude (AI assistant) on behalf of @blp.

Comment thread python/feldera/rest/_httprequests.py Outdated
return exc.status_code in self.config.retry_config.retryable_status_codes
return (
exc.status_code in self.config.retry_config.retryable_status_codes
and (idempotent or exc.status_code in _NOT_APPLIED_STATUS_CODES)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug (medium): Some manager 503s say the request was not applied, and they no longer retry for non-idempotent calls:

  • DeadlockDetected and LockTookTooLong come from db/error.rs:483-485 after Postgres rolls the transaction back.
  • PipelineInteractionNotDeployed and CompilerUnavailable are rejected before any side effect.

As a result, create_pipeline, patch_pipeline, ingress and checkpoint now fail when the manager is under DB contention, and these calls used to recover. I confirmed this with a unit test: a POST that gets a 503 with error_code: DeadlockDetected makes only 1 call. Suggestion: also retry when exc.error_code is in a "not applied" allow-list.

_NOT_APPLIED_ERROR_CODES = frozenset({"DeadlockDetected", "LockTookTooLong", "PipelineInteractionNotDeployed", "CompilerUnavailable"})
...
and (idempotent or exc.status_code in _NOT_APPLIED_STATUS_CODES
     or getattr(exc, "error_code", None) in _NOT_APPLIED_ERROR_CODES)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, thanks. I checked each code in the manager. DeadlockDetected and LockTookTooLong come after Postgres rolls back the transaction (db/error.rs:480-486). PipelineInteractionNotDeployed (runner/interaction.rs:104) and CompilerUnavailable (pipeline_management.rs:1554-1567) come before any side effect. A non-idempotent request now retries a 503 with one of these codes. Your test is in, with all four codes, and it fails without the fix.

One exception: pause, resume, activate and approve forward to the pipeline first and then call increment_notify_counter (pipeline_interaction.rs:1803, 1868, 1958, 2060). So a DB error code from those handlers can follow an applied action. A repeat asks for the same state, so the retry is harmless. The code comment at _NOT_APPLIED_ERROR_CODES says so.

Claude (AI assistant), replying on behalf of @blp.

# from its own (e.g. during a rolling upgrade).
current_version = p.program_version or 0
if current_version == expected_program_version:
if current_version >= expected_program_version:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): >= also accepts a version bumped by a concurrent PATCH from another client. The wait then reports success for a program this caller did not submit. The old != check caught that case. If the only legitimate cause is a platform-version adoption, consider checking it: accept > expected only when platform_version changed, or at least log a warning when current_version > expected.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point. A check on platform_version would need the platform version that this caller's PATCH set, and _wait_for_compilation does not have it. Also, the old == check caught a concurrent PATCH only if it landed before the poll saw success. So I took the second option: the wait now logs a warning when the compiled version is newer than expected, and a unit test checks the warning.

Claude (AI assistant), replying on behalf of @blp.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
uv run --locked pytest tests/unit ✅ 382 passed
pre-commit run on changed files ✅ ruff / ruff-format pass
QA test: POST gets 503 with error_code DeadlockDetected / LockTookTooLong / PipelineInteractionNotDeployed ❌ not retried (1 call); see inline comment
QA test: POST gets ConnectionError ✅ not retried (same as before)

Missing tests (medium): no unit test covers a 503 with a "not applied" error_code body on a non-idempotent request. The suggested test is below; it fails on this PR as it stands.

@pytest.mark.parametrize("code", ["DeadlockDetected", "LockTookTooLong", "PipelineInteractionNotDeployed"])
def test_post_503_not_applied_is_retried(code):
    client = _make_client()
    body = json.dumps({"message": "x", "error_code": code, "details": {}}).encode()
    with patch_requests("post", [_make_response(503, body, headers={"Retry-After": "0"}),
                                 _make_response(200, b"{}")]) as m:
        client.post("/pipelines")
    assert m.call_count == 2

Breaking change (medium): the Python SDK retry behaviour changes in a way users can see, and the "Changelog updated" box is unchecked. Please add a changelog entry. The gates for commit messages, manual test description and unit tests pass.

@blp blp self-assigned this Nov 2, 2026
@blp blp added bug Something isn't working QA Testing and quality assurance python Pull requests that update python code labels Nov 2, 2026
blp added 2 commits November 2, 2026 16:00
_wait_for_compilation required the program version to equal the expected
value. One edit can bump the version twice: the PATCH bumps it, then the
compiler bumps it again when it adopts a pending pipeline whose platform
version differs from its own (for example during a rolling upgrade). The
wait then failed with "program version (3) != expected program version
(2)" although the newer program compiled successfully.

Accept any version at or above the expected one, and log a warning for a
newer version, because a concurrent edit by another client also gives
one. An older version still raises.

Fixes #7748.

Signed-off-by: Ben Pfaff <blp@feldera.com>
…ied.

The client retried 502, 503, 504 and read timeouts for every request.
After each of these the server may already have applied the request: a
gateway sends 502/504 when the backend fails late, and the manager sends
503 when a request it proxies to a pipeline fails mid-way. A retried
POST could then be applied twice, for example HTTP ingress inserting the
same rows twice.

Non-idempotent requests now retry only connect timeouts (the request was
never sent), 408/429 (the server rejected it), and a 503 whose manager
error code shows the request was not applied (DeadlockDetected,
LockTookTooLong, PipelineInteractionNotDeployed, CompilerUnavailable).
GET and requests marked idempotent=True keep the full policy. The
read-only POSTs /diff, /validate_program and egress are now marked
idempotent, so they keep their retries.

Fixes #7793.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp force-pushed the python-retry-program-version branch from 3391c7b to 9b0ff60 Compare November 3, 2026 00:00
@blp

blp commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

Thanks for the review. All three points are addressed, squashed into the two commits:

  • 503 "not applied" error codes: they retry again (see the inline thread).
  • Concurrent PATCH: the wait logs a warning for a newer version (see the inline thread).
  • Changelog: entry added under "Unreleased" in docs.feldera.com/docs/changelog.md.

uv run pytest tests/unit: 387 passed. Each new test fails with its fix reverted.

Claude (AI assistant), replying on behalf of @blp.

@blp
blp added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit 7e1d8fd Nov 3, 2026
1 check passed
@blp
blp deleted the python-retry-program-version branch November 3, 2026 05:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working python Pull requests that update python code QA Testing and quality assurance

Projects

None yet

2 participants