Repository navigation
[python] Check a paused output connector's record counter on one host only - #7582
Conversation
… only An output connector's `total_processed_input_records` counts the input of its own host, but the pipeline's `total_processed_records` is the sum over all hosts. With more than one host, the connector's counter need not pass the pipeline's count, and if its host has no input, it stays at zero, so `test_output_connector_orchestration` timed out. Now the test does the records check only with one host. With more than one host, the coordinator tracks completion by steps, which the test still checks.
| timeout_s=30.0, | ||
| poll_interval_s=0.5, | ||
| ) | ||
| metrics = output_connector_stats(pipeline_name, "v", "sink")["metrics"] |
There was a problem hiding this comment.
missing (medium): With >1 host, the transmitted_records == paused_at assert now runs right after paused_at is read, with no wait in between. So the "stays quiet however much the pipeline produces" claim is not checked on multi-host: a paused connector that leaks writes would still pass. Add an else: branch that waits for the pipeline to make progress, e.g.
else:
processed_at_pause = pipeline_stats(pipeline_name)["global_metrics"]["total_processed_records"]
wait_for_condition(
"pipeline keeps processing while the connector is paused",
lambda: pipeline_stats(pipeline_name)["global_metrics"]["total_processed_records"] > processed_at_pause,
timeout_s=30.0,
poll_interval_s=0.5,
)After that wait, the following asserts check quietness on every host count.
|
Review summary: This is a test-only fix. It skips the per-host record-counter check when there is more than one host, which is correct: each connector counts only its own host's input, while
|
With more than one host, the test skipped the records wait, so it checked that the paused connector had not written anything right after it read the connector's count. A connector that kept writing while paused would still pass. Now, with more than one host, the test waits for the pipeline to process more input, and then for the connector to process a step that started after that, before it checks that the connector wrote nothing.
|
Thanks, fixed in c2a67b9. I went one step further than the suggestion: a wait on the pipeline's count alone does not make sure that the new records reached the sink. So after that wait, the test also waits for the connector to process a step that started after the input. With 2 hosts, both output connector tests pass. To check that it catches a leak, I started the connector again before the waits: the test then fails with |
test_output_connector_orchestrationtimed out with 2 hosts.An output connector's
total_processed_input_recordscounts the input of its own host, but the pipeline'stotal_processed_recordsis the sum over all hosts. With more than one host, the connector's counter need not pass the pipeline's count. If its host has no input, the counter stays at zero.Now the test does the records check only when
FELDERA_TEST_NUM_HOSTS == 1. With more than one host, the coordinator tracks completion by steps, and the test still checks steps.Tested:
tests/platform/test_orchestration.pypasses with 2 hosts (together with the coordinator change that adds the output connectorcommandroute).