Skip to content

Feat/yaml driven template docs#804

Merged
ypriverol merged 10 commits intobigbio:devfrom
ypriverol:feat/yaml-driven-template-docs
Mar 8, 2026
Merged

Feat/yaml driven template docs#804
ypriverol merged 10 commits intobigbio:devfrom
ypriverol:feat/yaml-driven-template-docs

Conversation

@ypriverol
Copy link
Member

@ypriverol ypriverol commented Mar 8, 2026

Summary by CodeRabbit

  • New Features

    • Added an interactive SDRF builder UI to simplify SDRF file creation with live preview and download functionality.
    • Introduced a comprehensive Templates Guide documenting template inheritance, layers, and combination rules.
  • Breaking Changes

    • Technical replicate comment field is now required in all templates.
    • Data acquisition method changed from data-dependent to data-independent in example templates.
    • Immunopeptidomics field renamed from HLA to MHC terminology.
  • Documentation

    • Restructured metadata guidance into dedicated Sample Guidelines and Templates Guide pages.
    • Reorganized template documentation with centralized inheritance and combination rule documentation.
  • Bug Fixes

    • Fixed BioSample accession regex validation to reject invalid prefixes.

ypriverol and others added 10 commits March 8, 2026 10:42
Add resolve_templates.py that loads YAML templates, walks the extends:
chain, and produces resolved dicts with own, inherited, and merged columns.
Includes comprehensive test suite (20 tests) covering all inheritance
patterns and merge semantics.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Jinja2 snippet and build script that resolves all templates,
classifies them into organism/technology/experiment/sample groups,
and injects the rendered section into site/index.html.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace dead adoc template loop with build_template_pages.py
- Add index template section auto-generation step
- Fix tool-support.adoc -> TOOLS.adoc filename mismatch
- Add pip install pyyaml jinja2 to CI
- Enable submodule checkout in CI for sdrf-templates access

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Sort columns by requirement: required > recommended > optional
- Replace flat table with expandable detail rows (click to expand)
- Fix documentation rendering for mixed paragraph/list blocks
- Merge index template section into one unified table (remove redundant
  Organism/Technology card sections)
- Remove broken metadata-guidelines section from index
- Fix Jinja2 dict.values() conflict by renaming to allowed_values

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Mar 8, 2026

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0d8e40ab-b421-46bd-bbef-5fa1fe953fa4

📥 Commits

Reviewing files that changed from the base of the PR and between e2b644b and dbfb270.

⛔ Files ignored due to path filters (24)
  • annotated-projects/PMID33086064/PMID33086064.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD000527/PXD000527.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD000793/PXD000793.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD003539/PXD003539.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD005163/PXD005163.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD005171/PXD005171.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD005174/PXD005174.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD005175/PXD005175.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD005177/PXD005177.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD008841/PXD008841.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD010429/PXD010429.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD010705/PXD010705.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD010708/PXD010708.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD010981/PXD010981.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD012203/PXD012203.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD012986/PXD012986.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD019185_PXD018883/PXD019185_PXD018883.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD023158/PXD023158.sdrf.tsv is excluded by !**/*.tsv
  • annotated-projects/PXD026474/PXD026474.sdrf.tsv is excluded by !**/*.tsv
  • examples/use-cases/crosslinking.sdrf.tsv is excluded by !**/*.tsv
  • examples/use-cases/single-cell.sdrf.tsv is excluded by !**/*.tsv
  • psi-document/sdrf-proteomics-specification-v1.1.0-dev.pdf is excluded by !**/*.pdf
  • sdrf-proteomics/TERMS.tsv is excluded by !**/*.tsv
  • sdrf-proteomics/metadata-guidelines/sdrf-terms.tsv is excluded by !**/*.tsv
📒 Files selected for processing (51)
  • .github/workflows/build-docs.yml
  • CHANGELOG.md
  • DEVELOPMENT.md
  • llms.txt
  • psi-document/v1.1.0-dev/supplementary-files/quickstart-v1.1.0-dev.adoc
  • psi-document/v1.1.0-dev/supplementary-files/template-definitions-v1.1.0-dev.adoc
  • scripts/build-asciidoc.sh
  • scripts/build-docs.sh
  • scripts/build_index_templates.py
  • scripts/build_sdrf_builder_data.py
  • scripts/build_template_pages.py
  • scripts/copy-assets.sh
  • scripts/inject-headers.py
  • scripts/resolve_templates.py
  • scripts/test_resolve_templates.py
  • scripts/transform-links.py
  • scripts/transform-sdrf-tables.py
  • sdrf-proteomics/DATA-ANALYSIS.adoc
  • sdrf-proteomics/README.adoc
  • sdrf-proteomics/SAMPLE-GUIDELINES.adoc
  • sdrf-proteomics/TEMPLATES.adoc
  • sdrf-proteomics/TOOLS.adoc
  • sdrf-proteomics/VERSIONING.adoc
  • sdrf-proteomics/metadata-guidelines/template-definitions.adoc
  • sdrf-proteomics/quickstart.adoc
  • sdrf-proteomics/sdrf-templates
  • sdrf-proteomics/templates/affinity-proteomics/README.adoc
  • sdrf-proteomics/templates/base/README.adoc
  • sdrf-proteomics/templates/cell-lines/README.adoc
  • sdrf-proteomics/templates/crosslinking/README.adoc
  • sdrf-proteomics/templates/dda-acquisition/README.adoc
  • sdrf-proteomics/templates/dia-acquisition/README.adoc
  • sdrf-proteomics/templates/human/README.adoc
  • sdrf-proteomics/templates/immunopeptidomics/README.adoc
  • sdrf-proteomics/templates/invertebrates/README.adoc
  • sdrf-proteomics/templates/metaproteomics/README.adoc
  • sdrf-proteomics/templates/ms-proteomics/README.adoc
  • sdrf-proteomics/templates/olink/README.adoc
  • sdrf-proteomics/templates/plants/README.adoc
  • sdrf-proteomics/templates/single-cell/README.adoc
  • sdrf-proteomics/templates/somascan/README.adoc
  • sdrf-proteomics/templates/vertebrates/README.adoc
  • site/build-search-index.py
  • site/css/style.css
  • site/index.html
  • site/js/sdrf-builder.js
  • site/quickstart.html
  • site/sdrf-data.json
  • site/sdrf-terms.html
  • site/templates/index-templates.html.j2
  • site/templates/template-page.html.j2

📝 Walkthrough

Walkthrough

A comprehensive refactoring consolidates the SDRF documentation build system from multiple shell scripts into Python-driven tooling, introduces a new sample-metadata template layer between base and technology/sample layers, replaces per-template README files with a centralized Templates Guide, and adds an interactive client-side SDRF Builder UI for configuring SDRF metadata.

Changes

Cohort / File(s) Summary
Build System Consolidation
.github/workflows/build-docs.yml, scripts/build-docs.sh
Unified build pipeline replacing separate AsciiDoc, asset, and post-processing steps with Python-driven template and index generation; added --dev flag handling for banner injection; added sdrf-builder.js asset copying.
Template Resolution Framework
scripts/resolve_templates.py, scripts/test_resolve_templates.py
New Python module implementing template inheritance resolution, column merging across chains, and provenance tracking; comprehensive test suite validating manifest loading, chain building, column inheritance, and resolved metadata.
Template Page Generation
scripts/build_template_pages.py
Python script that converts YAML template definitions to HTML pages with documentation processing, column sorting by requirement level, and Jinja2 rendering.
Template Index and Builder Data
scripts/build_index_templates.py, scripts/build_sdrf_builder_data.py
Index generation script injecting resolved templates into HTML; builder data compilation extracting combination rules, validators, and term definitions for client-side UI.
Build Script Removals
scripts/build-asciidoc.sh, scripts/copy-assets.sh
Removed legacy scripts now integrated into unified build-docs.sh.
Link and Table Transformations
scripts/transform-links.py, scripts/transform-sdrf-tables.py
Updated link patterns for TOOLS.adoc, TEMPLATES.adoc, SAMPLE-GUIDELINES.adoc; removed unused HTMLParser import.
Template Architecture and Hierarchy
psi-document/v1.1.0-dev/supplementary-files/template-definitions-v1.1.0-dev.adoc
Updated template inheritance to extend sample-metadata instead of base; changed acquisition method examples from dda-acquisition to dia-acquisition; added sample-metadata to layer definitions.
Central Templates Documentation
sdrf-proteomics/TEMPLATES.adoc
New 1,339-line comprehensive guide documenting template layers, inheritance, version pinning, combination rules, column validators, YAML schema, best practices, and validation tooling.
Template README Consolidation
sdrf-proteomics/templates/*/README.adoc, sdrf-proteomics/metadata-guidelines/template-definitions.adoc
Removed 13 per-template README files (base, human, vertebrates, invertebrates, plants, affinity-proteomics, cell-lines, crosslinking, dda-acquisition, dia-acquisition, immunopeptidomics, metaproteomics, ms-proteomics, olink, single-cell, somascan) and legacy template-definitions guide; content consolidated into TEMPLATES.adoc and Python-generated pages.
Documentation Updates
sdrf-proteomics/README.adoc, sdrf-proteomics/SAMPLE-GUIDELINES.adoc, sdrf-proteomics/quickstart.adoc, sdrf-proteomics/VERSIONING.adoc
Replaced template links with external GitHub URLs; updated terminology (HLA→MHC typing); expanded ontology guidance; reorganized sample metadata sections; added age/sex/developmental stage requirements; updated template inheritance descriptions.
Specification and Metadata
CHANGELOG.md, llms.txt, DEVELOPMENT.md
Documented template architecture refactoring, breaking changes (comment[technical replicate] now REQUIRED, organism templates become REQUIRED), and new build system in development guide; updated template inheritance documentation.
Interactive SDRF Builder
site/js/sdrf-builder.js
New 957-line JavaScript module implementing data-driven builder with technology/organism/experiment/affinity selection, column resolution, combination rule validation, column/factor pickers, and TSV export.
Builder UI and Pages
site/quickstart.html, site/css/style.css, site/templates/template-page.html.j2, site/templates/index-templates.html.j2
Replaced static quickstart with builder-driven workflow with live preview; added 270+ lines of CSS for template pages, builder cards, and column picker UI; added Jinja2 templates for dynamic template page and index rendering.
Search and Navigation
site/build-search-index.py, site/index.html, site/sdrf-terms.html
Extended search indexing to include YAML templates; updated homepage links from sample-metadata/human templates to templates.html and sample-guidelines.html; updated term definitions with revised ontologies (e.g., MHC, GENO, PRIDE).
Submodule Update
sdrf-proteomics/sdrf-templates
Updated submodule pointer reflecting template system changes.

Sequence Diagram

sequenceDiagram
    participant User
    participant Browser as Browser (quickstart.html)
    participant Builder as SDRF Builder (sdrf-builder.js)
    participant DataFile as sdrf-builder-data.json
    participant TemplateResolved as Resolved Templates
    
    User->>Browser: Load quickstart.html
    Browser->>Builder: Load sdrf-builder.js
    Builder->>DataFile: fetch('sdrf-builder-data.json')
    DataFile-->>Builder: templates, combination_rules, terms
    Builder->>TemplateResolved: resolveColumns(selections)
    TemplateResolved-->>Builder: ordered column list
    Builder->>Browser: Render technology options
    User->>Builder: Select technology
    Builder->>TemplateResolved: resolveColumns(+technology)
    TemplateResolved-->>Builder: updated columns
    Builder->>Browser: Render organism options
    User->>Builder: Select organism
    Builder->>TemplateResolved: resolveColumns(+organism)
    TemplateResolved-->>Builder: resolved columns with organism overrides
    Builder->>Browser: Live preview table
    User->>Builder: Download TSV / Open in Editor
    Builder->>Browser: Generate/export SDRF
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

Suggested labels

documentation, build-system, feature, templates

Suggested reviewers

  • nithujohn
  • noatgnu
  • enryH
  • timosachsenberg

Poem

🐰 The builder hops with Python's grace,
Templates resolve in layered space,
From YAML chains to HTML pages bright,
The SDRF dance now interactive delight!
No more per-file READMEs to maintain,
One guide to rule them all—a gain!

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@ypriverol ypriverol changed the base branch from master to dev March 8, 2026 19:33
@ypriverol ypriverol merged commit 921e1f5 into bigbio:dev Mar 8, 2026
1 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant