← 输入框|github.com|刷新
Skip to content

[python] Check a paused output connector's record counter on one host only - #7582

Merged
blp merged 2 commits into
mainfrom
orchestration-multihost-records
Oct 22, 2026
Merged

blp merged 2 commits into
mainfrom
orchestration-multihost-records

Conversation

@blp

@blp blp commented Oct 22, 2026

Copy link
Copy Markdown
Member

test_output_connector_orchestration timed out with 2 hosts.

An output connector's total_processed_input_records counts the input of its own host, but the pipeline's total_processed_records is the sum over all hosts. With more than one host, the connector's counter need not pass the pipeline's count. If its host has no input, the counter stays at zero.

Now the test does the records check only when FELDERA_TEST_NUM_HOSTS == 1. With more than one host, the coordinator tracks completion by steps, and the test still checks steps.

Tested: tests/platform/test_orchestration.py passes with 2 hosts (together with the coordinator change that adds the output connector command route).

… only

An output connector's `total_processed_input_records` counts the input of
its own host, but the pipeline's `total_processed_records` is the sum over
all hosts.  With more than one host, the connector's counter need not pass
the pipeline's count, and if its host has no input, it stays at zero, so
`test_output_connector_orchestration` timed out.

Now the test does the records check only with one host.  With more than
one host, the coordinator tracks completion by steps, which the test still
checks.
@blp blp self-assigned this Oct 22, 2026
@blp blp added QA Testing and quality assurance python Pull requests that update python code labels Oct 22, 2026
@blp
blp requested a review from ryzhyk October 22, 2026 00:20
timeout_s=30.0,
poll_interval_s=0.5,
)
metrics = output_connector_stats(pipeline_name, "v", "sink")["metrics"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing (medium): With >1 host, the transmitted_records == paused_at assert now runs right after paused_at is read, with no wait in between. So the "stays quiet however much the pipeline produces" claim is not checked on multi-host: a paused connector that leaks writes would still pass. Add an else: branch that waits for the pipeline to make progress, e.g.

    else:
        processed_at_pause = pipeline_stats(pipeline_name)["global_metrics"]["total_processed_records"]
        wait_for_condition(
            "pipeline keeps processing while the connector is paused",
            lambda: pipeline_stats(pipeline_name)["global_metrics"]["total_processed_records"] > processed_at_pause,
            timeout_s=30.0,
            poll_interval_s=0.5,
        )

After that wait, the following asserts check quietness on every host count.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Review summary: This is a test-only fix. It skips the per-host record-counter check when there is more than one host, which is correct: each connector counts only its own host's input, while total_processed_records is the total across all hosts.

What I ran Result
ruff check / ruff format --check on the changed file pass
tests/platform/test_orchestration.py not run: needs a running pipeline-manager
  • missing (medium): With more than one host, nothing waits between reading paused_at and asserting the connector is quiet, so that run no longer tests that a paused sink stays quiet. Inline comment has a suggested else branch.
  • The feldera-pr-checks gates pass: the PR changes only a test, the commit message explains why, and the PR description says what was tested.

With more than one host, the test skipped the records wait, so it checked
that the paused connector had not written anything right after it read
the connector's count.  A connector that kept writing while paused would
still pass.

Now, with more than one host, the test waits for the pipeline to process
more input, and then for the connector to process a step that started
after that, before it checks that the connector wrote nothing.
@blp

blp commented Oct 22, 2026

Copy link
Copy Markdown
Member Author

Thanks, fixed in c2a67b9. I went one step further than the suggestion: a wait on the pipeline's count alone does not make sure that the new records reached the sink. So after that wait, the test also waits for the connector to process a step that started after the input. With 2 hosts, both output connector tests pass. To check that it catches a leak, I started the connector again before the waits: the test then fails with assert 500 == 400.

@blp
blp added this pull request to the merge queue Oct 22, 2026
Merged via the queue into main with commit a2cc16c Oct 22, 2026
@blp
blp deleted the orchestration-multihost-records branch October 22, 2026 06:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Pull requests that update python code QA Testing and quality assurance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants