updateDelaySet may cause very high redis cpu usage

[amit-gshe]

We have a very large queue which has about several millions of delay jobs. And our queue has tens of workers(pods) in k8s cluster and even double or triple more pods when the business peak time. We have encountered an issue that bull can lead high redis cpu usage and the redis gets slow then stop prosessing jobs. It's a pain last for several months. Finally we find that the updateDelaySet get triggered so frequently on each worker and thus caused the high redis cpu usage and then consumer get so slow even stop processing waiting jobs and the waiting queue get stacked and finally the entire bull queue stop working.

I found that the updateDelaySet was expensive when the queue was huge so we should stop doing updateDelaySet when the waiting queue has too many jobs to process. We can add an option say maxWaitingJobs which default set Number.MAX. The user can set this value according their needs.

[manast]

Before I dig more into this issue, could you tell me more about your queue setup? for instance why do you have so many delayed jobs, and what is the average delay for these jobs? are you using rate limiting?

[manast]

also, are you using priorities or just normal jobs?

[amit-gshe]

My queue config: new Queue('FOOD_DIGEST_JOB', { redis: { host, port }, limiter: { max: 1000, duration: 1000 } }). We have millions of users and they can feed fish and every fish has a delay job in our setting. we use normal jobs. Each of our jobs can delay tens of seconds to days.

[manast]

ok, so you have both delayed jobs and rate limited to 1000 jobs per second right?

[amit-gshe]

yes exactly

[manast]

and where are you running your redis instances?

[amit-gshe]

we run our redis instance on aws elastic cache redis.

[manast]

did you consider upgrading the instance type to a faster one? I know in aws is wasteful since larger instances also has more cores but redis will only use one.

[manast]

I assume you are using last bull version 3.18.1 right?

[amit-gshe]

yes we tried upgrading our instance but nothing difference. The queue can run a period of time but this issue will occur occasionally. we use 3.18.

[manast]

yes. Moving the delayed jobs back to wait is not very efficient, specially in this case when your queues are always full of jobs. I think I can implement a solution for this, but it may take a couple of days before it is ready (if everything goes well).

[manast]

I implemented some improvements that should reduce the CPU usage for a case like this, I wonder if you have any chance to verify it before we make a new release? #1897

[amit-gshe]

I tried #1897 with option new Queue('FOOD_DIGEST_JOB', { redis: { host, port }, limiter: { max: 1000, duration: 1000 } }) this morning and I found the CPU usage is lower but the consumimg rate get slower at the same time which result in the delayed jobs get stacked in delay queue and not consumed in time. The rate limiter setting is 1000/s, but the consume rate was just 140+/s. The adding rate was 300+/s (There are many jobs adding and consuming at the same time). BTW I found the redis slow log is clean now. The redis slow log increase quickly in the before version.