Classify papers by OpenAlex field; surface per-journal on dashboard

papers.csv gains topic_field/topic_subfield from OpenAlex
primary_topic (blank for EPMC-only records until their journal covers
them). journals.csv gains a field column: the modal field across each
journal's classified works, which indirectly classifies EPMC-only
papers via journal overlap. The dashboard journals table adds a
sortable Field column and a field filter dropdown.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: jodeleeuw

README.md

@@ -26,6 +26,8 @@ One row per unique paper.
 | `publication_year` | Year of publication. |
 | `publication_date` | Publication date (`YYYY-MM-DD`) when known. |
 | `type` | Work type (e.g. `article`, `preprint`). |
+| `topic_field` | **Derived, recomputed every run** — the display name of the OpenAlex `primary_topic` field (e.g. `Psychology`). OpenAlex assigns topics **algorithmically**, so occasional misassignments are expected. Blank for all Europe PMC-only records and for OpenAlex records without a primary topic; newly added columns stay blank until the next full monthly fetch backfills them (`--relink` does not refetch and leaves these columns untouched). |
+| `topic_subfield` | **Derived** — the OpenAlex `primary_topic` subfield display name (e.g. `Cognitive Neuroscience`). Same provenance and caveats as `topic_field`. |
 | `is_preprint` | **Derived, recomputed every run** — manual edits will not stick (unlike `notes`/`exclude`). `True` only when a strong preprint signal fires: `type` is exactly `preprint`, the DOI prefix belongs to a known preprint server (PsyArXiv, OSF Preprints, bioRxiv/medRxiv, arXiv, Research Square, Preprints.org, SocArXiv, EdArXiv, SSRN), or the venue names one of those servers. `False` means "not confidently identified as a preprint", **not** "confirmed published". |
 | `duplicate_of` | **Derived, recomputed every run** — manual edits will not stick. When this row is a preprint (or earlier version) of another row in the dataset, the `id` of that canonical row; blank for canonical / unlinked rows. Chains are flattened (v1 → v2 → published becomes v1 → published), so a canonical row always has a blank `duplicate_of`. Rows are never deleted by linking — this column only annotates. See [Preprint → published linking](#preprint--published-linking). |
 | `match_method` | **Derived** — which signal produced the link: `crossref`, `doi_version`, or `title_author` (see below). Blank whenever `duplicate_of` is blank. |
@@ -170,8 +172,17 @@ only) derives two summary tables from the data CSVs:
   (distinct works), `n_authors` (distinct author keys), `first_use`,
   `last_use`.
 - **`analysis/journals.csv`** — one row per unique venue: `journal_key`,
-  `journal_name`, `n_papers` (distinct works), `n_authors` (distinct author
-  keys), `first_use`, `last_use`. Venues are keyed by normalized name
+  `journal_name`, `field`, `n_papers` (distinct works), `n_authors`
+  (distinct author keys), `first_use`, `last_use`. The `field` column is
+  the **modal `topic_field` across the journal's classified works** (a
+  work's own field is the modal value across its linked members; ties break
+  by count then alphabetically; blank when no member work is classified).
+  Because classification is per-journal, most Europe PMC-only papers are
+  covered indirectly: their journal is classified by its OpenAlex-covered
+  works. Expect the large majority of works to sit in a classified journal
+  once the monthly fetch has backfilled `topic_field`, with a residual
+  blank tail (unclassified OpenAlex records and journals seen only via
+  Europe PMC). The script prints the exact coverage each run. Venues are keyed by normalized name
   (lowercased, runs of punctuation/whitespace collapsed to single spaces —
   this merges MEDLINE-style variants like *Journal of experimental
   psychology. General*) since the sources provide no ISSNs; `journal_name`

analysis/dashboard.json

@@ -71,151 +71,176 @@
    "name": "PsyArXiv",
    "n_papers": 266,
    "first_use": "2017-03-03",
-   "last_use": "2026-07-02"
+   "last_use": "2026-07-02",
+   "field": ""
   },
   {
    "name": "Behavior Research Methods",
    "n_papers": 93,
    "first_use": "2014-03-28",
-   "last_use": "2026-07-06"
+   "last_use": "2026-07-06",
+   "field": ""
   },
   {
    "name": "Cognition",
    "n_papers": 64,
    "first_use": "2017-02-15",
-   "last_use": "2026-05-25"
+   "last_use": "2026-05-25",
+   "field": ""
   },
   {
    "name": "OSF Preprints",
    "n_papers": 64,
    "first_use": "2016-06-01",
-   "last_use": "2026-07-02"
+   "last_use": "2026-07-02",
+   "field": ""
   },
   {
    "name": "bioRxiv (Cold Spring Harbor Laboratory)",
    "n_papers": 59,
    "first_use": "2017-09-15",
-   "last_use": "2026-06-22"
+   "last_use": "2026-06-22",
+   "field": ""
   },
   {
    "name": "Scientific Reports",
    "n_papers": 56,
    "first_use": "2016-09-13",
-   "last_use": "2026-06-12"
+   "last_use": "2026-06-12",
+   "field": ""
   },
   {
    "name": "Frontiers in Psychology",
    "n_papers": 45,
    "first_use": "2014-08-06",
-   "last_use": "2026-06-04"
+   "last_use": "2026-06-04",
+   "field": ""
   },
   {
    "name": "Attention Perception & Psychophysics",
    "n_papers": 44,
    "first_use": "2016-02-04",
-   "last_use": "2026-07-13"
+   "last_use": "2026-07-13",
+   "field": ""
   },
   {
    "name": "arXiv (Cornell University)",
    "n_papers": 43,
    "first_use": "2019-05-17",
-   "last_use": "2026-06-24"
+   "last_use": "2026-06-24",
+   "field": ""
   },
   {
    "name": "Memory & Cognition",
    "n_papers": 39,
    "first_use": "2018-04-30",
-   "last_use": "2026-05-29"
+   "last_use": "2026-05-29",
+   "field": ""
   },
   {
    "name": "PLoS ONE",
    "n_papers": 38,
    "first_use": "2015-06-26",
-   "last_use": "2026-05-15"
+   "last_use": "2026-05-15",
+   "field": ""
   },
   {
    "name": "Psychonomic Bulletin & Review",
    "n_papers": 37,
    "first_use": "2016-07-06",
-   "last_use": "2026-07-13"
+   "last_use": "2026-07-13",
+   "field": ""
   },
   {
    "name": "Quarterly Journal of Experimental Psychology",
    "n_papers": 31,
    "first_use": "2020-01-14",
-   "last_use": "2026-05-03"
+   "last_use": "2026-05-03",
+   "field": ""
   },
   {
    "name": "Cognitive Science",
    "n_papers": 27,
    "first_use": "2015-01-01",
-   "last_use": "2026-06-01"
+   "last_use": "2026-06-01",
+   "field": ""
   },
   {
    "name": "Journal of Cognition",
    "n_papers": 24,
    "first_use": "2019-01-01",
-   "last_use": "2026-01-09"
+   "last_use": "2026-01-09",
+   "field": ""
   },
   {
    "name": "Open Mind",
    "n_papers": 23,
    "first_use": "2020-01-01",
-   "last_use": "2026-06-30"
+   "last_use": "2026-06-30",
+   "field": ""
   },
   {
    "name": "Communications Psychology",
    "n_papers": 22,
    "first_use": "2022-10-17",
-   "last_use": "2026-04-29"
+   "last_use": "2026-04-29",
+   "field": ""
   },
   {
    "name": "eLife",
    "n_papers": 21,
    "first_use": "2017-11-21",
-   "last_use": "2026-03-27"
+   "last_use": "2026-03-27",
+   "field": ""
   },
   {
    "name": "Journal of Memory and Language",
    "n_papers": 20,
    "first_use": "2018-06-11",
-   "last_use": "2026-06-20"
+   "last_use": "2026-06-20",
+   "field": ""
   },
   {
    "name": "Psychological Research",
    "n_papers": 20,
    "first_use": "2020-11-12",
-   "last_use": "2026-02-07"
+   "last_use": "2026-02-07",
+   "field": ""
   },
   {
    "name": "SSRN Electronic Journal",
    "n_papers": 18,
    "first_use": "2022-01-01",
-   "last_use": "2026-01-01"
+   "last_use": "2026-01-01",
+   "field": ""
   },
   {
    "name": "Consciousness and Cognition",
    "n_papers": 17,
    "first_use": "2019-09-12",
-   "last_use": "2026-03-12"
+   "last_use": "2026-03-12",
+   "field": ""
   },
   {
    "name": "eScholarship (California Digital Library)",
    "n_papers": 17,
    "first_use": "2013-01-01",
-   "last_use": "2024-01-01"
+   "last_use": "2024-01-01",
+   "field": ""
   },
   {
    "name": "Journal of Experimental Psychology General",
    "n_papers": 17,
    "first_use": "2018-11-28",
-   "last_use": "2026-01-26"
+   "last_use": "2026-01-26",
+   "field": ""
   },
   {
    "name": "Nature Communications",
    "n_papers": 17,
    "first_use": "2019-03-08",
-   "last_use": "2026-05-15"
+   "last_use": "2026-05-15",
+   "field": ""
   }
  ],
  "top_institutions": [

analysis/generate_summaries.py

@@ -64,7 +64,7 @@
     "n_papers", "n_authors", "first_use", "last_use",
 ]
 JOURNAL_OUT_COLUMNS = [
-    "journal_key", "journal_name", "n_papers", "n_authors",
+    "journal_key", "journal_name", "field", "n_papers", "n_authors",
     "first_use", "last_use",
 ]
 
@@ -101,6 +101,14 @@
 }
 
 
+def modal_value(counter):
+    """Most frequent value in a Counter; ties broken by count then
+    alphabetically (deterministic). '' for an empty counter."""
+    if not counter:
+        return ""
+    return min(counter.items(), key=lambda kv: (-kv[1], kv[0]))[0]
+
+
 def norm_name(name):
     """Normalize a name for matching: lowercase, strip periods, collapse ws."""
     name = name.lower().replace(".", " ")
@@ -169,6 +177,7 @@ def canonical(pid):
     excluded = set()
     type_filtered = 0
     group_dates = {}  # group id -> list of member date strings (non-excluded)
+    group_field_votes = {}  # group id -> Counter of non-blank topic_field
     for row in raw_papers:
         pid = row["id"]
         if (row.get("exclude") or "").strip() == "True":
@@ -178,7 +187,18 @@ def canonical(pid):
             excluded.add(pid)
             type_filtered += 1
             continue
-        group_dates.setdefault(group_of[pid], []).append(paper_date(row))
+        gid = group_of[pid]
+        group_dates.setdefault(gid, []).append(paper_date(row))
+        tfield = (row.get("topic_field") or "").strip()
+        if tfield:
+            group_field_votes.setdefault(gid, Counter())[tfield] += 1
+
+    # A work's field: modal topic_field across its non-excluded members
+    # (member versions of the same work almost always agree; using the modal
+    # value lets a classified preprint cover an unclassified published row
+    # and vice versa). '' when no member is classified.
+    work_field = {gid: modal_value(group_field_votes[gid])
+                  for gid in group_field_votes}
 
     # first/last use per active group (min/max across its non-excluded members)
     group_span = {}
@@ -239,9 +259,13 @@ def canonical(pid):
             nvenue = norm_venue(venue)
         j_key = f"name:{nvenue}"
         jentry = journals.setdefault(
-            j_key, {"names": Counter(), "papers": set(), "authors": set()})
+            j_key, {"names": Counter(), "papers": set(), "authors": set(),
+                    "fields": Counter()})
         jentry["names"][venue] += 1  # one spelling vote per work
         jentry["papers"].add(gid)
+        wf = work_field.get(gid, "")
+        if wf:
+            jentry["fields"][wf] += 1  # one field vote per classified work
         group_journal[gid] = j_key
 
     # ------------------------------------------------------------------ #
@@ -470,6 +494,9 @@ def date_range(group_ids):
         journal_rows.append({
             "journal_key": key,
             "journal_name": most_frequent(entry["names"]),
+            # Modal topic_field across the journal's classified works (ties:
+            # count then alphabetical); '' when no member work is classified.
+            "field": modal_value(entry["fields"]),
             "n_papers": len(entry["papers"]),
             "n_authors": len(entry["authors"]),
             "first_use": first,
@@ -499,13 +526,19 @@ def date_range(group_ids):
     works_by_year = [{"year": int(y), "n": year_counts[y]}
                      for y in sorted(year_counts)]
 
-    def top_entries(rows, name_field, limit=25):
-        return [{
-            "name": r[name_field],
-            "n_papers": r["n_papers"],
-            "first_use": r["first_use"],
-            "last_use": r["last_use"],
-        } for r in rows[:limit]]
+    def top_entries(rows, name_field, limit=25, extra_keys=()):
+        out = []
+        for r in rows[:limit]:
+            e = {
+                "name": r[name_field],
+                "n_papers": r["n_papers"],
+                "first_use": r["first_use"],
+                "last_use": r["last_use"],
+            }
+            for k in extra_keys:
+                e[k] = r[k]
+            out.append(e)
+        return out
 
     dashboard = {
         "generated": datetime.now(timezone.utc).strftime("%Y-%m-%d"),
@@ -518,7 +551,8 @@ def top_entries(rows, name_field, limit=25):
             "preprints_linked": preprints_linked,
         },
         "works_by_year": works_by_year,
-        "top_journals": top_entries(journal_rows, "journal_name"),
+        "top_journals": top_entries(journal_rows, "journal_name",
+                                    extra_keys=("field",)),
         "top_institutions": top_entries(inst_rows, "institution_name"),
         "top_authors": top_entries(author_rows, "author_name"),
     }
@@ -549,6 +583,24 @@ def top_entries(rows, name_field, limit=25):
     print(f"  Unique journals: {len(journal_rows)} "
           f"(venues recovered from DOI prefix: {venue_from_doi_prefix}, "
           f"works skipped for empty venue: {skipped_no_venue})")
+
+    # Field-classification coverage
+    n_works = len(group_dates)
+    journal_field = {r["journal_key"]: r["field"] for r in journal_rows}
+    direct = sum(1 for gid in group_dates if work_field.get(gid, ""))
+    covered = sum(
+        1 for gid in group_dates
+        if work_field.get(gid, "")
+        or journal_field.get(group_journal.get(gid, ""), ""))
+    classified_journals = sum(1 for r in journal_rows if r["field"])
+    def pct(a, b):
+        return f"{100.0 * a / b:.1f}%" if b else "n/a"
+    print(f"  Field coverage: {pct(direct, n_works)} of works classified "
+          f"directly ({direct}/{n_works}); "
+          f"{pct(classified_journals, len(journal_rows))} of journals "
+          f"classified ({classified_journals}/{len(journal_rows)}); "
+          f"{pct(covered, n_works)} of works in a classified journal or "
+          f"classified directly ({covered}/{n_works})")
     print("  Top 5 authors by n_papers:")
     for r in author_rows[:5]:
         print(f"    {r['n_papers']:>4}  {r['author_name']}  ({r['author_key']})")