Add metadata only stage to repodata generation

[soapy1]

Description

This introduces a new upstream stage called md. The existing stages in conda-index are:

• fs - means that the artifact is now available in the set of packages and is assumed by default to be the local filesystem
• indexed - means that the entry already exists in the database (same filename, same timestamp, same hash), and its package metadata has been extracted to the index_json etc

This pr adds md which is like fs in that it represents an aritfact that is now available. But, it is assumed to not be available on the local filesystem. md is a sibiling to the indexed stage. This is helpful for indexing artifacts that are not able to be represented on the local filesystem. For example, to represent pypi packages.

To include metadata sourced packages in repodata be use to include include_stages and package_extensions to the ChannelIndex.cache_kwargs during it's instantiation. For example:

channel_index = ChannelIndex(
        tmp_path,
        "haswheels",  # channel name if different than last segment of tmp_path
        repodata_v3=True,
        cache_kwargs={
            "package_extensions": CONDA_PACKAGE_EXTENSIONS + (".whl",),
            "include_stages": ["md"],
        },
    )

And inject the metadata using the store_md_state function:

cache = channel_index.cache_for_subdir("noarch")
. . .
cache.store_md_state(listdir_like())

See tests/test_demonstrate_wheel.py for how this changes the api for injecting wheel data into repodata.

ref: conda/conda-pypi#276 (comment)

Other noteable changes

• fixed import errors when importing from conda_index.postgres.cache import PsqlCache
  • this caused test failures in tests/test_psql.py - looks like these tests have been getting skipped
  • fixed those tests
• update conda_index.conda_index.cache.BaseCondaIndexCache.database_path to only prepend the database prefix if the prefix is not already included

Checklist - did you ...

• Add a file to the news directory (using the template) for the next release's release notes?
• Add / update necessary tests?
• Add / update outdated documentation?

[soapy1]

A different approach might make the cache layer know about what all the desired upstream stages are instead of getting a cache by subdir and stage.
By moving the responsibility of keeping track of the upstream stages to the cache layer, we can avoid the merging logic below. This could be replaced by accounting for the multiple stages in the query phase.

[dholth]

The database layer is the right place to merge stages. It could be accounted for in cache_kwargs when creating ChannelIndex, or in indexed_packages() when performing the query.

[soapy1]

Let's go for including this in the cache_kwargs. We'll define a new attribute include_stages which accepts a list of stages to consider in addition to upstream_stage.

[dholth]

We'll have to check what happens when path is in fs and md at the same time. We would get the same index_json twice, maybe, but it would also overwrite itself in the output dict.

[soapy1]

yes! Added a test in tests/test_index.py::test_index_noarch_with_wheels. This will put wheels and noarch conda packages in the same noarch subdir.

[dholth]

Do you think md is a sibling to the indexed phase, not the upstream fs phase? We wouldn't join md with indexed to find a list of packages whose metadata needs to be added; instead; md implies that the metadata has been loaded in the index_json table. Then we only consider md+indexed when pulling data out of the database.

[soapy1]

hmmm, I was thinking of md and fs as siblings. But I think you are right, it should be md and indexed. Will re-jig it!

[soapy1]

This means that when a user is creating a function like listdir_like they need to also add this repodata key. That isn't very listdir_stat like.....

For example, something like

def listdir_like():
        for path, repodata in wheels.items():
            assert "sha256" in repodata
            if "md5" not in repodata:
                repodata["md5"] = None
            yield {
                "path": cache.database_path(path),
                "size": repodata["size"],
                "mtime": repodata.get("timestamp", 1), 
                "repodata": repodata,
            }

This is maybe a bit too much of a hack. @dholth what do you think?

[soapy1]

hmmm, I expect these dependencies were not included for some reason. After fixing the psql cache import errors in the tests, I got new errors saying that these dependencies are missing. Is there a strong incentive to keep these out of the main package? An alternative is to add them as optional dependencies.

[dholth]

This program is written to work with "low dependencies", so it's necessary for sqlalchemy to be only an optional dependency for conda-index. This should help users who install conda-index alongside other programs or incidentally as a conda, conda-pypi, or conda-build dependency.