requeue recently fetched components

[jeffmcaffer]

We have cases now where the definitions are being built from reasonably old harvested data and as a result the definitions are substantially out of date. Previously we talked about having the service check the current crawler tool levels and requeue any definition based on older tools. We'd still return the old definition but soon enough the new one would be ready.

Thoughts:

• pretty soon now we are going to rescan the world so we'll be up to date
• pretty soon after that we'll do some sort of change that will make those obsolete
• There is an issue if we do this lazily where the same thing could get queued many times causing queue bloat and even duplicate work for long running tools.
• may be better to do it as a "job" where we periodically queue up "old" definitions.

Summary:

• we can defer for a bit but should put this in place relatively soon after rescanning everything

thoughts @dabutvin ?

[dabutvin]

I really like the idea of doing it lazily, even if we end up running a job as well.
If a definition is requested and it has an old version of a tool we should return the existing definition and queue it for fresh harvesting for next time.

[jeffmcaffer]

Updating with some thinking...

The webhooks from the crawler to the service tell the service about all new tool runs. In that run output we have the version of the tools that were used. The service should just keep a running max. Then when it sees a definition that has an older tool, it can requeue that component.

Take care to handle startup conditions where the service does not yet have a value for a given tool.

Also note that some tools can have different version numbers depending on the type/provider being crawled. For example, the ClearlyDefined tool for Crates may be 1.2.0 and fro NPMs may be 1.9.0. So the key for the table is type/provider/tools

[jeffmcaffer]

@AlexWebYourmind this is a good issue to have a service side dev look at.

[ignacionr]

Maybe once we determine that the fetched definition represents an update opportunity, we should also set a response header for the CDN to not cache it, and/or signal the caller (maybe an X-header, or another seemingly out-of-band indication).

[dabutvin]

@ignacionr that makes sense. good call