Question about lock_ttl and until_and_while_executing

[ghost]

I inherited a job set up as:

 sidekiq_options retry: 3,
                  lock: :until_and_while_executing,
                  lock_timeout: 10.minutes,
                  lock_ttl: 20.minutes,
                  on_conflict: {
                    client: :log,
                    server: :reject
                  }

These jobs take a few seconds to run, and when a large number of them queue up the time spent on the queue can exceed the lock_ttl. When that happens, the jobs appear to no-op as soon as they are picked up for processing. I'm not seeing errors, just jobs instantly completing without doing any work.

The lock_ttl was added as part of an effort to address orphaned locks, so I think the solution here is straightforward -- remove the lock_ttl and let the reaper clean up any orphaned locks.

What I'm unclear on is what the intended behavior is for this use case -- while it makes sense to me that these jobs might not process after the lock_ttl expires, I might have expected something like an error.

Can anyone shed some insight on this?

[mhenrixon]

First, lock_timeout: 10.minutes might mean that one of your workers will be waiting for 10 minutes to push a job to Sidekiq.

To be honest, the timeout should likely only be used for the server process (when your perform action is called).

You want this number to be as low as possible. Try a second or two!

Hard to say what is going wrong, but in #730, it was discovered that redis-namespace is not working with keyword arguments, and there is an old PR from 2015 that potentially fixes the issue: resque/redis-namespace#104.

[pboling]

Separately it does follow from the current documentation of lock_ttl, but it is easy to get distracted by the behavior at queue-time...

The same lock_ttl value also still has a runtime behavior (i.e. it is sort-of overloaded in the scenario of until_and_while_executing), which is to kill the lock after the ttl expires.

So if set for 5.minutes, and I have a job that takes more than 5 minutes to run, the (uniqueness) lock expires after 5 minutes, allowing an identical job to begin executing. Because in an until_and_while_executing configuration the point is to be able to have one executing and one queued at all times, if you use lock_ttl it means that your jobs must both:

1. begin executing (i.e. be removed from the queue) faster than the lock_ttl or they will be vanished (not even dead, just disappeared).
2. finish executing faster than the lock_ttl or they will allow the same job in the queue to start up alongside them.

It's the logical conclusion of expiring a lock with a TTL, but it is still hard to think about, and it might be worth mentioning that when using lock: :until_and_while_executing you almost certainly do not want a lock_ttl, unless you set it to nil.

[mhenrixon]

lock_ttl and lock_timeout should likely not be used on the first lock since I added the reaper. The reaper should make sure that no leftovers are kept in redis.

For lock_ttl, it should kick in as execution starts, but I have been too chicken to break this for people, perhaps in the next major release.

[pboling]

lock_ttl and lock_timeout should likely not be used on the first lock since I added the reaper

when you say "first lock" are you referring to the queue-time lock?

For lock_ttl, it should kick in as execution starts

It seems to currently do this for the runtime lock (we're still on v7.1), so your phrasing is confusing.

In other words, I have a job with a lock_ttl of 5 minutes, and I had two identical jobs running, and they started 5 minutes apart. There is always one in the queue to be picked up, so it clearly waited to pick it up until the lock_ttl had expired the run-time lock.

So what have you been too chicken to break? (I.e. what might change in the next major release?)

[ghost]

@pboling Thank you.

This has been nagging at me for ages, and I believe the distinction you drew between the two separate lock_ttls was the missing piece for me.

[pboling]

@mhenrixon

For lock_ttl, it should kick in as execution starts, but I have been too chicken to break this for people, perhaps in the next major release.

Can you elaborate on the difference between what it "should do" and what it currently does, and what might break if you change the behavior?

I am particularly confused because "For lock_ttl, it should kick in as execution starts" this is the apparent behavior we see on v7.1.