ARAX-Shepherd is failing TestCase_4 en masse

[edeutsch]

In the latest test run of the Refactor:
https://arax.ncats.io/?systest=281

We are failing a lot of Test Case 4 that BTE and Aragorn apparently pass:

And apparently we are causing the ARS results to fail as well.
Looking at the results:
https://arax.ci.transltr.io/?r=56b0f4a1-ba29-4c96-8f7f-93a20c151942

It would appear that while Aragorn and BTE are only returning 4 answers (and thus easy for them to duck this huge list of "NeverShow"s, ARAX is returning 500 answers, and thus more vulnerable to NeverShows

https://arax.ci.transltr.io/?r=b6469a57-3f6d-43ef-94a8-75b17282a2f1

For example, MMP3 is a NeverShow
It is result number 330.

It would appear that Retriever is providing this edge.
Since Retriever is the primary knowledge source, this is presumably a subclass reasoning edge. But due to bug #2662 we cannot see the support graph.

I suspect resolving this problem would result in much better Refactor test scores for ARAX-Shepherd as well as ARS (since many FAILs would be turned into PASSes).

We pass these in CI with legacy KPs:

Only 14 results in legacy CI:
https://arax.ci.transltr.io/?r=5ffc9544-4f84-4d6f-aafc-73cd59beae98

and does not include MMP3 et al.

It would seem that ARAX-Shepherd is failing many tests because it is getting NeverShow information from Retriever

This would seem like an impactful mystery to solve....

[saramsey]

I think #2662 has now been fixed. New code with the latest fix has been rolled out to /devED, /shepherd, and ITRB CI. The code has also been merged to master.

[saramsey]

OK, I wrote this simple QG to test the issue:

{
      "nodes": {
        "sn": {
          "ids": [
            "CHEBI:29103"
          ]
        },
        "on": {
          "ids": [
            "NCBIGene:4314"
          ]
        }
      },
      "edges": {
        "e00": {
          "subject": "sn",
          "object": "on",
          "predicates": [
            "biolink:affects"
          ]
        }
      }
    }

and fixed a bug in ARAX_filter_kg.py which that QG tripped (see issue #2680), and with that patch, running on arax.ncats.io/shepherd, I can now see the aux graph in the MMP3 result,

[saramsey]

So, I have merged the code for #2680 to master and rolled out that code to arax.ncats.io/shepherd.

Results of my little QG are recorded here:
https://arax.ncats.io/shepherd/?r=437389

[saramsey]

Hmm, I see a TRAPI validation error on the TRAPI message response, something about CHEBI:29103 not having a category. Hmm.

[saramsey]

OK, after researching it, the TRAPI validation failure seems like a separate sub-issue, so I've logged it as #2682

[dkoslicki]

Observation: switching to tier 0 makes the 500 results go away and a much more manageable number. The bunch of results all appear to come from:
infores:ubergraph
infores:text-mining-provider-targeted

Is text-mining-provider-targeted in tier 0? See most any edge of: https://arax.ci.transltr.io/?r=81e36326-6049-49a1-9dc0-6860d94b20ce

[bazarkua]

Confirmed both ubergraph and text-mining-provider (tmkp) are included in the tier 0 graph (translator_kg). In separate discussion the point was raised that discrepancy is likely due subclass reasoning that is in tier 1 sources but not in the tier 0

[dkoslicki]

@sierra-moxon is tracking down the source of the subclass issue. Status of that so far is: subclass is coming from ubergraph, and

Upstream, ubergraph is using ELK, to materialize the relationships between ontology terms, and we've got NN chemical conflation turned on in the ingests which rewrites some identifiers. NN'ing in the ingest pipeline then converts CHEBI:85264 to CHEBI:48947

[isbluis]

Just a heads-up comment on subclassing: I am not familiar on how it is actually implemented, but there is a possibility of it generating incorrect results, given that the Biolink specification seems to have some incorrect child/parent definitions. I filed an issue with that team a few weeks ago, but I am not sure if it has been looked at (or indeed if it is much of an issue).