Programmatically determine DGIdb “latest” download date and remove hardcoded UI label

[katie-perry]

Background

The DGIdb downloads page currently displays a row labeled latest, which maps to a directory of downloadable TSV files hosted externally (I believe these are stored in an AWS instance owned/managed by the Griffith lab - so we may need to leverage their help for this).

At present, there is no programmatic way to determine when the “latest” files were generated. As a temporary solution, the UI hardcodes a display label (e.g. latest (2024-Dec)), while continuing to use latest for the download paths.

This approach is not sustainable long-term and risks becoming inaccurate over time.

Assumptions

• the downloadable TSVs are stored in an external AWS environment (S3?) managed by the Griffith lab
• File access is based on stable endpoints/directory names (e.g. data/latest, data/2024-Dec)
• There is no exposed metadata endpoint indicating when files were generating

Goal

Introduce a mechanism for determining and displaying the generation date for the "latest" DGIdb download files

Proposed Approaches

File-level metadata

• We could add a generated-on date to the TSV files themselves
• For this, we would need to know how these files are generated - this may need to be done by the Griffith lab if we determine this is the best approach

Manifest/metadata file

• Provide a small metadata file alongside the downloads (e.g. latest/metadata.json
• Ex:

{
  "generated_at": "2024-12-15",
  "version_label": "2024-Dec"
}

Storage Metadata (S3 object metadata)

• Expose last-modified timestamps or current metadata

API solution

• Endpoint to provide available download versions and date?

Acceptance Criteria

• Investigate and document where DGIdb download files are stored and how they are updated
• Identify a reliable source of truth for the generation date of the "latest" files
• Implement a programmatic mechanism to retrieve this date (file header, metadata file, api, etc.)
• Update the downloads UI (Files.tsx) to display the date dynamically
• Remove hardcoded display label (latest (2024-Dec)) from Files.tsx
• Ensure existing download URLs and behavior remain unchanged
• Document chosen approach for future maintainers

[katie-perry]

Related issue is #612 - for now the PR for that adds the hardcoded label to make the site more friendly for users but this ticket is for addressing the longer-term issue that this should be able to be determined dynamically/programmatically

[jsstevenson]

👍 we should be attaching descriptive metadata to the top of the TSVs (I think we have started doing this somewhat but could do more).

One idea I have considered is setting up a GitHub repo dedicated to DGIdb data (eg https://github.com/dgidb/dgidb-data), and then including the data files as assets in the discrete releases. This doesn't require checking data into the git repo itself, although we could also do that -- instead we just manually drag and drop files into the release description when we make it. Then, we can use the GitHub API to get a list of releases (ideally named according to some date/versioning pattern) and programmatically generate the table on the DGIdb client side, without having to manually update the table/get the PR approved and merged/issue a new deployment each time we generate new data. It would also put that data in a space that all of us can manage, so we don't to rely on Adam or Suzy being in the office to get a new release posted.

As an example, I've been trying something like this for a data curation project as part of the MetaKB. Every time I make a new data release, I add a specific file as an asset. Then, the /releases endpoint gets updated to include that new release, including metadata (tag name + publish date being what we're interested in) and then a list of assets w/ download URLs.

• You could also just check the files into git, and write an action that adds the files as assets when you make a new release. Or you could write an action that checks that every release contains some specific files and reverts it to a draft if not. Lots of different ways to go here.
• I suspect Zenodo or something else could do the same thing, but GitHub was the first place I thought of. There are probably size limits to file assets, but I don't think there's any risk that we would ever even approach them.