You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When we are creating a RedisStreamBroker, we are seeing that if there are a ton of messages getting sent into the queue what ends up happening is that each worker first gets a few messages, then one worker ends up with all the remainder of the tasks. We lose all parralelization in this.
To reproduce.
Create a TaskIQ broker using RedisStreamBroker. Create a worker that has 20 workers. Send 1000 tasks into the queue you have the broker configured.
What you should see is at first the tasks are all going spread, but eventually one worker will process the bulk majority with no other workers getting tasks until new messages are pushed into the queue.
I think this could be solved by implemeting the "count" on the XREADGROUP as a parameter to pass.
Activity
rlandingham commentedon Apr 23, 2025
You cn see here, that one worker is getting all of the requests for the most part.
s3rius commentedon Apr 23, 2025
Thanks for noticing it. I've made a PR that fixes this behavior.
Should appear in an upcoming release very soon.