Repository navigation
Conversation
…gtest harness A pipeline pod that restarts after ingestion holds empty tables, reports Running, and answers every ad-hoc query with an empty result. The harness then blamed the first view it queried, for example in #7688: ASSERTION ERROR: failed view: varchar_arg_min_gby Received: [] Completion tokens only catch a restart during ingestion (#6986). Record the incarnation UUID after start and, when a view check fails, compare it again. A changed UUID raises PipelineRestartedError, which names both incarnations and the input record count of the new one.
| view.validate(pipeline) | ||
| except AssertionError as e: | ||
| raise_if_restarted(pipeline, incarnation, e) | ||
| raise |
There was a problem hiding this comment.
bug (medium): an empty expected result hides the restart. The incarnation is compared only when a view fails. About 98 aggtest views expect [] (grep -rn "\.data = \[\]" tests/runtime_aggtest). After a post-ingestion restart, those views pass even though nothing was really checked. If every view in a pipeline expects [], the whole test goes green. Suggest also comparing the incarnation once after the for view in views loop, so a restart always fails the test:
for view in views:
...
raise_if_restarted(pipeline, incarnation, AssertionError("all views validated"))(Or split the helper into a check_incarnation that does not need a cause.)
| tokens only catch a restart during ingestion (feldera#6986), so the view | ||
| check compares the incarnation UUID recorded after start. | ||
| """ | ||
| metrics = pipeline.stats().global_metrics |
There was a problem hiding this comment.
pitfall (medium): a failing stats() call replaces the view diff. This code runs right after a suspected restart, so the pod may still be coming up and stats() can fail. If it raises (FelderaAPIError or a connection error), that error escapes and the outer handler reports RUNTIME ERROR: <api error> instead of the view diff. That makes the result worse than before this PR. Wrap the call in a try/except, and on failure re-raise the original AssertionError, maybe with a note that the incarnation could not be checked.
|
What I ran
Gates: tests ✅, no flakiness introduced ✅, commit message ✅, no deps/docs impact ✅. Missing (medium): untested cases
Otherwise the approach looks good to me once the two inline issues are fixed. |
…iff when the probe fails Review follow-up on the aggtest restart check: - Compare the incarnation once more after all views validated. About 98 aggtest views expect no rows, and those pass on an empty new incarnation, so a restart could turn a pipeline green. - A failed stats() probe right after a suspected restart no longer replaces the view diff. The harness prints a warning and re-raises the original assertion. - Unit tests now drive run_pipeline with a fake pipeline, so the wiring is covered: restart reported, plain mismatch kept, restart behind empty views, failed probe, and the passing case.
|
Both inline findings are addressed in the second commit, and the wiring is now covered:
Removing the post-loop check fails |
|
What ?!? Approved like this, nothing to change, not even a slight comment ? |
|
I think you just crave human connection |
|
Well, PR reviews are one of the only few pieces of human interaction we got in this job. |
|
Verdict: infrastructure flake, not this PR. High confidence.
This PR only touches Filed #7833 with the evidence and assigned it to the sccache config owner. Next step: re-queueing should pass — nothing here points at this PR's diff. |
Fixes the misleading failure in #7688 and the post-ingestion variant of #6986.
A pipeline pod that restarts after ingestion holds empty tables, reports Running, and answers every ad-hoc query with an empty result. The harness then blamed the first view it queried:
Completion tokens only catch a restart during ingestion. This change records the incarnation UUID after start and, when a view check fails, compares it again. A changed UUID raises
PipelineRestartedError, which names both incarnations and the input record count of the new one.Received: []PipelineRestartedError: incarnation A became B, it now holds 0 input recordsUnit test in
python/tests/unit/test_aggtst_restart_detection.py. With the check disabled the restart test fails, so it catches the regression.