← 输入框|github.com|刷新
Skip to content

[pipeline-manager] Collect support data for due pipelines concurrently - #8007

Draft
pavanmanishd wants to merge 1 commit into
feldera:mainfrom
pavanmanishd:fix/support-collector-concurrent
Draft

pavanmanishd wants to merge 1 commit into
feldera:mainfrom
pavanmanishd:fix/support-collector-concurrent

Conversation

@pavanmanishd

Copy link
Copy Markdown

Fixes #7933. Design: #7933 (comment)

This is a draft. I will mark it ready for review after a maintainer accepts the design in #7933.

The background support data collector collected the due pipelines one at a time. Thus a slow pipeline delayed the collection of all other due pipelines.

Changes:

  • collect_due_pipelines collects up to 4 due pipelines at the same time with buffer_unordered.
  • Each pipeline gets its next collection time when its own collection completes.
  • The run loop does not change.

Each collection keeps its bundle in memory until it is stored. Thus MAX_CONCURRENT_COLLECTIONS also limits the memory of the manager. I chose 4. Tell me if you prefer a different value.

Test: slow_pipeline_does_not_delay_other_collections. One pipeline answers each request after 2 s. The other pipeline answers at once. The slow pipeline is first in the schedule. The test checks that the data of the fast pipeline is stored in less than 2 s. Before the change, the data was stored after 6.04 s. After the change, it is stored after about 60 ms.

The background support data collector collected the due pipelines one
at a time. Thus a slow pipeline delayed the collection of all other due
pipelines.

Collect up to 4 due pipelines at the same time. Schedule the next
collection of each pipeline from the time its own collection completes.

Fixes feldera#7933.

Signed-off-by: pavanmanishd <pavanmanishd@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[pipeline-manager] Background support data collector collects one pipeline at a time, so a slow pipeline delays all others

1 participant