Modify greyed-out appearance on homepage relationship to a failed OBO Dashboard run

[allenbaron]

Currently, a single failed OBO dashboard run for an ontology results in the ontology being greyed out (use discouraged?) on the OBO Foundry homepage. Some failures are not due to errors in an ontology but in errors that occur when the dashboard is run, for example versionIRIs not resolving. Others are temporary errors in ontologies that occur in a single release and are then fixed.

There was some discussion about this on the OBO Operations meeting, where it was noted that there are resource constraints for running the Dashboard, both in frequency and getting it to work, and a desire to distinguish between ontologies that are actively maintained and those that are not and with errors that will not be fixed.

Would it be possible to update the link between the Dashboard and OBO Foundry main pages, so that more than one Dashboard run failure would need to happen before greying out an ontology on the OBO Foundry main page?

Note

Additional questions and suggestions were discussed at the meeting, particularly related to resource constraints for running the Dashboard, but I chose not included them in this issue to promote focused discussion on whether this specific suggestion is desirable and feasible without undue burden.

[jsstevenson]

Thanks for writing this up @allenbaron !

Would it be possible to update the link between the Dashboard and OBO Foundry main pages, so that more than one Dashboard run failure would need to happen before greying out an ontology on the OBO Foundry main page?

Just on this point -- from a technical perspective, this is very possible, but it would take a little bit of code wrangling to get it right. The way it currently works is that a given run of the dashboard generation scripts ultimately leads to a commit that updates this file. When you point your browser to obofoundry.org, your browser requests that file and uses it to programmatically generate the table (incl. whether a row is greyed out). We could add an additional step for the client to 1) get the commit hash for the last revision to the file before its current form and 2) get the state of the file at that revision, then require a check to have failed in consecutive months for it to "grey" out the table row. (We could also be more granular and set this policy for specific rules e.g. if we find that they tend to be noisier).

[matentzn]

My personal take: these are very distinct things:

Some failures are not due to errors in an ontology but in errors that occur when the dashboard is run, for example versionIRIs not resolving.

Others are temporary errors in ontologies that occur in a single release and are then fixed.

My personal preference would be to report every single case of the former and make the dashboard fully resilient. @allenbaron my use this issue here to collect all Dashboard issues filed that are representatives of that problem.

For the latter, my sense is that if the issue is temporary, the ontology can also live with the temporary grey flag. I understand its annoying - but the better way to deal with this IMO is to have more regular dashboard runs, say, every month.

@anitacaron - right now, the main issue on the latter side as I understand is that the dashboard consumes a huge amount of memory for the mega ontologies like NCBITaxon. Crazy thought - even if this is an approximation - why dont we just run the dashboard on the NCBI taxon obo slim (same for CHEBI)? This will catch all errors reduce memory requriements and runtime by half and we dont lose any meaningful signal (metadata same, whether 10K undefined classes or 100K does not matter).