← 输入框|github.com|刷新
Skip to content

[python] Report a pipeline restart instead of an empty view in the aggtest harness - #7816

Merged
bestouff merged 2 commits into
mainfrom
issue7688
Nov 4, 2026
Merged

bestouff merged 2 commits into
mainfrom
issue7688

Conversation

@bestouff

@bestouff bestouff commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Fixes the misleading failure in #7688 and the post-ingestion variant of #6986.

A pipeline pod that restarts after ingestion holds empty tables, reports Running, and answers every ad-hoc query with an empty result. The harness then blamed the first view it queried:

ASSERTION ERROR: failed view: varchar_arg_min_gby
Received: []

Completion tokens only catch a restart during ingestion. This change records the incarnation UUID after start and, when a view check fails, compares it again. A changed UUID raises PipelineRestartedError, which names both incarnations and the input record count of the new one.

Scenario Before After
View really wrong AssertionError with the diff unchanged
Pod restarted after ingestion AssertionError, Received: [] PipelineRestartedError: incarnation A became B, it now holds 0 input records

Unit test in python/tests/unit/test_aggtst_restart_detection.py. With the check disabled the restart test fails, so it catches the regression.

…gtest harness

A pipeline pod that restarts after ingestion holds empty tables, reports
Running, and answers every ad-hoc query with an empty result. The harness
then blamed the first view it queried, for example in #7688:

    ASSERTION ERROR: failed view: varchar_arg_min_gby
    Received: []

Completion tokens only catch a restart during ingestion (#6986).
Record the incarnation UUID after start and, when a view check fails,
compare it again. A changed UUID raises PipelineRestartedError, which
names both incarnations and the input record count of the new one.
view.validate(pipeline)
except AssertionError as e:
raise_if_restarted(pipeline, incarnation, e)
raise

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug (medium): an empty expected result hides the restart. The incarnation is compared only when a view fails. About 98 aggtest views expect [] (grep -rn "\.data = \[\]" tests/runtime_aggtest). After a post-ingestion restart, those views pass even though nothing was really checked. If every view in a pipeline expects [], the whole test goes green. Suggest also comparing the incarnation once after the for view in views loop, so a restart always fails the test:

for view in views:
    ...
raise_if_restarted(pipeline, incarnation, AssertionError("all views validated"))

(Or split the helper into a check_incarnation that does not need a cause.)

tokens only catch a restart during ingestion (feldera#6986), so the view
check compares the incarnation UUID recorded after start.
"""
metrics = pipeline.stats().global_metrics

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): a failing stats() call replaces the view diff. This code runs right after a suspected restart, so the pod may still be coming up and stats() can fail. If it raises (FelderaAPIError or a connection error), that error escapes and the outer handler reports RUNTIME ERROR: <api error> instead of the view diff. That makes the result worse than before this PR. Wrap the call in a try/except, and on failure re-raise the original AssertionError, maybe with a note that the incarnation could not be checked.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
uv run --locked pytest tests/unit/test_aggtst_restart_detection.py 3 passed
pre-commit run --files <changed> passed (ruff, ruff-format)
runtime_aggtest against a live pipeline not run (needs a manager and compiler)

Gates: tests ✅, no flakiness introduced ✅, commit message ✅, no deps/docs impact ✅.

Missing (medium): untested cases

  1. A restart where every view expects []. It passes today; see the inline comment.
  2. stats() raising inside raise_if_restarted. The view diff is lost.
  3. run_pipeline wiring: no test checks that PipelineRestartedError reaches the caller inside RuntimeExecutionError, or that a plain mismatch still re-raises the original AssertionError. The unit tests cover only the helper.

Otherwise the approach looks good to me once the two inline issues are fixed.

…iff when the probe fails

Review follow-up on the aggtest restart check:

- Compare the incarnation once more after all views validated. About
  98 aggtest views expect no rows, and those pass on an empty new
  incarnation, so a restart could turn a pipeline green.
- A failed stats() probe right after a suspected restart no longer
  replaces the view diff. The harness prints a warning and re-raises
  the original assertion.
- Unit tests now drive run_pipeline with a fake pipeline, so the
  wiring is covered: restart reported, plain mismatch kept, restart
  behind empty views, failed probe, and the passing case.
@bestouff

bestouff commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Both inline findings are addressed in the second commit, and the wiring is now covered:

Finding Change
Views that expect no rows hide a restart check_incarnation runs once more after all views validated
A failed stats() probe replaces the view diff raise_if_restarted prints a warning and returns, the caller re-raises the original assertion
Only the helper was tested Tests drive run_pipeline with a fake pipeline: restart reported, plain mismatch kept, restart behind empty views, failed probe, passing case

Removing the post-loop check fails test_restart_fails_views_that_expect_no_rows; narrowing the probe guard fails test_failed_probe_keeps_the_view_diff. The lateness_tests suite also ran green against the CI amd64 instance with this harness.

@bestouff

bestouff commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

What ?!? Approved like this, nothing to change, not even a slight comment ?

@mihaibudiu

Copy link
Copy Markdown
Contributor

I think you just crave human connection

@bestouff

bestouff commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Well, PR reviews are one of the only few pieces of human interaction we got in this job.

@bestouff
bestouff added this pull request to the merge queue Nov 4, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 4, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Verdict: infrastructure flake, not this PR. High confidence.

Failed job Java Tests / Execute Java Tests
Failing test SaturatingOperationsTests.saturatingOperations (via BaseSQLTests.compileAndTestRust)
Root cause log:612-615 — sccache: error: Timed out waiting for server startup. Maybe the remote service is unreachable?, which makes the rustc -vV probe exit 101 before any Rust code compiles or runs

This PR only touches python/tests/runtime_aggtest/aggtst_base.py and python/tests/unit/test_aggtst_restart_detection.py — nothing in SQL-compiler or the sccache CI config. Only 1 of 2742 Java tests errored, and it errored on the compiler toolchain probe, not on any assertion. The same sccache: ... Timed out waiting for server startup message already shows up on #6763 against the production sccache backend, so the remote sccache service itself appears to be flaky right now.

Filed #7833 with the evidence and assigned it to the sccache config owner.

Next step: re-queueing should pass — nothing here points at this PR's diff.

@bestouff
bestouff added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit 2afeaba Nov 4, 2026
1 check passed
@bestouff
bestouff deleted the issue7688 branch November 4, 2026 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants