Repository navigation
[python] Check a paused output connector's record counter on every host - #7608
Conversation
This reverts bbc9569 and a2cc16c. They limited the check that a paused output connector reports input processed after the pause to one host, because the connector's `total_processed_input_records` counted only the input of its own host, and they added waits on steps for more than one host in its place. Now the coordinator reports that counter for the input of all of the hosts, so the check works with any number of hosts again, and it gives the connector time to show that it stays quiet, so the waits on steps are not needed. This needs a coordinator and pipelines that have the multihost committed records change.
|
Review summary. This PR changes only a test. It reverts the single-host guard and the multihost step waits in
Otherwise LGTM once the 2-host run described in the PR has passed. |
|
Review of #7608 (test-only revert: the paused-output-connector record-counter check runs on every host count again) What I ran
pitfall (medium): merge order. The PR is no longer marked draft, but the description still says "Draft: merge only after … deployed where the multihost Python tests run". The single-host path is unchanged. On multihost, Gates: this is test-only, so there is no behaviour change, no docs, and no deps to check. The commit message is good, and it explains the why. Flakiness looks fine: the input is an unbounded 100 rec/s datagen, so the condition is guaranteed to become true without a sleep, and the 30 s timeout leaves plenty of headroom. LGTM once the deployment precondition is met. |
Draft: merge only after #7607 and feldera/cloud#2123 are merged and deployed where the multihost Python tests run. Before that, this test hangs with more than one host.
This reverts bbc9569 and a2cc16c (from #7582). They limited the check that a paused output connector reports input processed after the pause to one host, because the connector's
total_processed_input_recordscounted only the input of its own host, and they added waits on steps for more than one host in its place.With #7607 and feldera/cloud#2123, the coordinator reports that counter for the input of all of the hosts. So the check works with any number of hosts again, and it gives the connector time to show that it stays quiet, so the waits on steps are not needed.
test_orchestration.pyis now identical to the version before bbc9569. Tested:test_output_connector_orchestrationpasses (2 passed) with 2 hosts on a local k3d cluster built from feldera/cloud#2123, which includes #7607. Still to do before merging: both changes deployed where the CIpython-multihostsuite runs.