← 输入框|github.com|刷新
Skip to content

All items in a queue assigned to one worker. #84

Closed

Description

@rlandingham

When we are creating a RedisStreamBroker, we are seeing that if there are a ton of messages getting sent into the queue what ends up happening is that each worker first gets a few messages, then one worker ends up with all the remainder of the tasks. We lose all parralelization in this.

To reproduce.

Create a TaskIQ broker using RedisStreamBroker. Create a worker that has 20 workers. Send 1000 tasks into the queue you have the broker configured.

What you should see is at first the tasks are all going spread, but eventually one worker will process the bulk majority with no other workers getting tasks until new messages are pushed into the queue.

I think this could be solved by implemeting the "count" on the XREADGROUP as a parameter to pass.

Activity

  1. rlandingham commented on Apr 23, 2025

    @rlandingham
    Author

    You cn see here, that one worker is getting all of the requests for the most part.

    Image
  2. s3rius commented on Apr 23, 2025

    @s3rius
    Member

    Thanks for noticing it. I've made a PR that fixes this behavior.

    Should appear in an upcoming release very soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions