Development Guide

This guide explains how to build and maintain the SDRF-Proteomics documentation website.

Repository Structure

proteomics-metadata-standard/
├── scripts/                   # Build and utility scripts
│   ├── build-docs.sh          # Main build script (local & CI/CD)
│   ├── build_template_pages.py # Generate per-template HTML from YAML
│   ├── build_index_templates.py # Generate index.html template section
│   ├── resolve_templates.py   # Template inheritance resolver
│   ├── inject-headers.py      # Navigation header injection
│   ├── transform-links.py     # .adoc→.html link transformation
│   ├── transform-sdrf-tables.py # SDRF example table styling
│   ├── build_sdrf_builder_data.py # SDRF builder JSON data compiler
│   └── add-dev-banner.sh      # Dev version banner
├── sdrf-proteomics/           # Main specification source
│   ├── README.adoc            # Main specification (AsciiDoc)
│   ├── TOOLS.adoc             # Tool support page
│   ├── TEMPLATES.adoc         # Templates guide
│   ├── SAMPLE-GUIDELINES.adoc # Sample metadata guidelines
│   ├── metadata-guidelines/   # Detailed metadata guidelines
│   │   ├── data-analysis-metadata.adoc
│   │   └── sdrf-terms.tsv
│   ├── sdrf-templates/        # YAML template definitions (submodule)
│   └── images/                # Specification images
├── site/                      # Website assets and build scripts
│   ├── css/style.css          # Main stylesheet
│   ├── js/search.js           # Search functionality
│   ├── js/sdrf-builder.js     # Interactive SDRF builder logic
│   ├── index.html             # Homepage
│   ├── quickstart.html        # Quick start guide + interactive builder
│   ├── sdrf-explorer.html     # SDRF dataset explorer
│   ├── sdrf-terms.html        # Terms reference page
│   ├── build-sdrf-index.py    # Builds dataset index
│   └── build-search-index.py  # Builds search index
├── annotated-projects/        # Annotated SDRF files (~300 datasets)
├── demo_page/                 # Local preview output (git-ignored)
├── .github/workflows/         # CI/CD configuration
│   └── build-docs.yml         # Website build workflow
├── DEVELOPMENT.md             # This file
└── README.md                  # Project overview

Prerequisites

Install the required tools:

# Install Asciidoctor (Ruby)
gem install asciidoctor
gem install asciidoctor-diagram

# Python 3 is required for build scripts
python3 --version

Building the Website Locally

Using the Build Script (Recommended)

The easiest way to build the documentation is using the provided build script. This script replicates the exact CI/CD build process, ensuring consistency between local and deployed builds.

# Build to default directory (demo_page/)
./scripts/build-docs.sh

# Build with a clean start (removes existing output)
./scripts/build-docs.sh --clean

# Build to a specific directory
./scripts/build-docs.sh docs

# Build dev version (adds development banner)
./scripts/build-docs.sh docs/dev --dev

# Show help
./scripts/build-docs.sh --help

What the Build Script Does

Converts AsciiDoc to HTML using Asciidoctor with proper styling options
Copies static assets (CSS, JavaScript, images)
Copies static HTML pages (index.html, quickstart.html, sdrf-explorer.html, sdrf-editor.html, sdrf-terms.html)
Builds SDRF Explorer index from annotated-projects
Injects navigation headers into all generated HTML pages
Transforms SDRF links to use the SDRF Explorer viewer
Transforms SDRF example tables to add color-coded column styling
Builds search index for site-wide search functionality
Adds dev banner (when using --dev flag)

Manual Build (Advanced)

If you need to build individual files for quick iteration:

# Build the main specification only
asciidoctor \
  -D demo_page \
  -a stylesheet=css/style.css \
  -a linkcss \
  -a toc=left \
  -a toclevels=3 \
  -a sectanchors \
  -a sectlinks \
  -a source-highlighter=highlight.js \
  --backend=html5 \
  -o specification.html \
  sdrf-proteomics/README.adoc

Note: Manual builds will be missing navigation headers, search functionality, and other features. Use the build script for testing the complete site.

View Locally

After building, open the site directly in your browser:

open demo_page/index.html

The site is self-contained and doesn't require a web server.

Key Files to Modify

File	Purpose
`sdrf-proteomics/README.adoc`	Main specification document
`sdrf-proteomics/metadata-guidelines/*.adoc`	Detailed metadata guidelines
`site/css/style.css`	Website styling (including SDRF table colors)
`site/index.html`	Homepage
`site/sdrf-explorer.html`	Dataset explorer page
`site/sdrf-terms.html`	Terms reference page
`.github/workflows/build-docs.yml`	CI/CD build configuration

SDRF Example Table Styling

SDRF example tables use color-coded columns. Add these CSS classes:

Class	Color	Use For
`sample-col`	Blue	source name, characteristics[...]
`data-col`	Green	assay name, comment[...], technology type
`factor-col`	Orange	factor value[...]

Example HTML in AsciiDoc:

++++
<div class="sdrf-example-table">
<table>
<thead>
<tr>
<th class="sample-col">source name</th>
<th class="sample-col">characteristics [organism]</th>
<th class="data-col">assay name</th>
<th class="data-col">comment [data file]</th>
<th class="factor-col">factor value [disease]</th>
</tr>
</thead>
<tbody>
<tr>
<td class="sample-col">sample_1</td>
<td class="sample-col">homo sapiens</td>
<td class="data-col">run_1</td>
<td class="data-col">file.raw</td>
<td class="factor-col">normal</td>
</tr>
</tbody>
</table>
<div class="sdrf-legend">
<span class="legend-item"><span class="legend-color sample-bg"></span> Sample metadata</span>
<span class="legend-item"><span class="legend-color data-bg"></span> Data file metadata</span>
<span class="legend-item"><span class="legend-color factor-bg"></span> Factor values</span>
</div>
</div>
++++

CI/CD Deployment

The website is automatically built and deployed when pushing to:

master branch → Production site (https://sdrf.quantms.org/)
dev branch → Development site (https://sdrf.quantms.org/dev/)

The CI/CD workflow (.github/workflows/build-docs.yml) performs the same steps as the local build script, ensuring consistency.

Local vs CI/CD Comparison

Feature	Local (build-docs.sh)	CI/CD
AsciiDoc conversion	✓	✓
Navigation headers	✓	✓
SDRF link transformation	✓	✓
Search index	✓	✓
Dev banner	`--dev` flag	Auto for dev branch
Output directory	Configurable	docs/ or docs/dev/

Ensuring Consistency

To verify your local build matches the deployed version:

# Build same as production
./scripts/build-docs.sh --clean

# Build same as dev deployment
./scripts/build-docs.sh --clean --dev

Adding New Annotated Projects

Create a folder in annotated-projects/ with the PXD accession
Add the SDRF file as {PXD}.sdrf.tsv

Validate the file:

pip install sdrf-pipelines
parse_sdrf validate-sdrf --sdrf_file annotated-projects/PXD000000/PXD000000.sdrf.tsv

Rebuild the dataset index:
```
python3 site/build-sdrf-index.py
```

References in AsciiDoc

Use bibliography format for proper cross-references:

# In text:
This is supported by research <<ref1>>.

# In References section:
[bibliography]
== References

- [[[ref1,1]]] Author et al. Title. Journal (Year). https://doi.org/xxx[doi:xxx]

Useful Commands

# Validate all SDRF files in a folder
for f in annotated-projects/*/*.sdrf.tsv; do
  parse_sdrf validate-sdrf --sdrf_file "$f"
done

# Find all SDRF files with a specific column
grep -l "characteristics\[disease\]" annotated-projects/*/*.sdrf.tsv

# Count annotated projects
ls -d annotated-projects/PXD* | wc -l

Troubleshooting

Stylesheet Warnings

When building, you may see warnings like:

asciidoctor: WARNING: stylesheet does not exist or cannot be read

These are expected and can be ignored. We use -a linkcss which links the stylesheet at runtime rather than embedding it during build.

Missing Dependencies

If you see errors about missing commands:

asciidoctor: gem install asciidoctor
python3: Install Python 3 from https://python.org

Build Output Differences

If the local build looks different from the deployed version:

Ensure you're using the build script (./scripts/build-docs.sh), not manual asciidoctor commands
Clear the output directory with --clean
Check that you have the latest site assets (git pull)

Common Issues

Issue	Solution
No navigation header	Use the build script instead of manual asciidoctor
CSS not loading	Ensure `site/css/style.css` was copied to output
Dev banner missing	Use `--dev` flag when building
Search not working	Ensure `search-index.json` was generated

SDRF Editor Integration

The website includes an embedded SDRF Editor at /sdrf-editor.html. The editor is loaded from the sdrfedit repository via jsDelivr CDN.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│  sdrfedit repo (bigbio/sdrfedit)                                │
│                                                                  │
│  Developer workflow:                                             │
│  1. Make changes to editor source code                           │
│  2. npm run build -- --configuration=production                  │
│  3. git add dist/ && git commit && git push to main              │
│                                                                  │
│  The dist/ folder is committed:                                  │
│  dist/sdrf-editor/browser/                                       │
│  ├── main.js                                                     │
│  ├── polyfills.js                                                │
│  └── styles.css                                                  │
└─────────────────────────────────────────────────────────────────┘
                              ↓
              jsDelivr CDN (automatic, no deployment needed)
              https://cdn.jsdelivr.net/gh/bigbio/sdrfedit@main/...
                              ↓
┌─────────────────────────────────────────────────────────────────┐
│  proteomics-metadata-standard                                    │
│                                                                  │
│  site/sdrf-editor.html loads the editor:                         │
│  <script src="https://cdn.jsdelivr.net/gh/bigbio/sdrfedit@main/ │
│               dist/sdrf-editor/browser/main.js" type="module">   │
│                                                                  │
│  When this repo is rebuilt, it pulls the latest editor version   │
└─────────────────────────────────────────────────────────────────┘

Updating the Editor

When changes are made to the sdrfedit repository:

sdrfedit maintainer builds and commits the dist/ folder to main
jsDelivr automatically serves the new files (cached ~24 hours)
This website gets the updated editor on next rebuild/redeploy

To force an immediate update, the specification website needs to be rebuilt:

Push to dev branch → triggers rebuild → pulls latest editor
Push to master branch → triggers rebuild → pulls latest editor

Editor Files

File	Purpose
`site/sdrf-editor.html`	Editor page (loads Angular app from jsDelivr)
`scripts/inject-headers.py`	Adds Editor link to navigation

Testing Editor Changes Locally

To test editor changes before publishing:

# In sdrfedit repo
cd ../sdrfedit
npm run build -- --configuration=production

# Serve locally and update sdrf-editor.html to point to local files
# (temporarily change jsDelivr URLs to local paths for testing)

Interactive SDRF Builder

The quickstart page (/quickstart.html) includes an interactive SDRF template builder that helps users create custom SDRF templates by answering questions.

How It Works

Build time: scripts/build_sdrf_builder_data.py resolves all YAML templates and sdrf-terms.tsv into sdrf-builder-data.json
Runtime: site/js/sdrf-builder.js loads the JSON and renders a branching questionnaire
Users select technology, organism, experiment type, review columns, and download a TSV

Builder Files

File	Purpose
`scripts/build_sdrf_builder_data.py`	Compiles YAML templates + terms into JSON
`site/js/sdrf-builder.js`	Builder UI logic (vanilla JS)
`site/quickstart.html`	Page with simplified intro + builder scaffold
`site/css/style.css`	Builder-specific styles (`.builder-`, `.option-`)

Updating the Builder

When templates change (new columns, new templates):

The build script automatically recompiles sdrf-builder-data.json
No JS changes needed unless new template layers or combination rules are added
New templates appear automatically in the appropriate step

Contributing

Make changes to the appropriate AsciiDoc or HTML files
Build locally and test: ./scripts/build-docs.sh --clean
View locally: open demo_page/index.html
Create a pull request to the dev branch for review
After approval and merge to dev, changes deploy to https://sdrf.quantms.org/dev/
After merge to master, changes deploy to https://sdrf.quantms.org/

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Development Guide

Repository Structure

Prerequisites

Building the Website Locally

Using the Build Script (Recommended)

What the Build Script Does

Manual Build (Advanced)

View Locally

Key Files to Modify

SDRF Example Table Styling

CI/CD Deployment

Local vs CI/CD Comparison

Ensuring Consistency

Adding New Annotated Projects

References in AsciiDoc

Useful Commands

Troubleshooting

Stylesheet Warnings

Missing Dependencies

Build Output Differences

Common Issues

SDRF Editor Integration

Architecture

Updating the Editor

Editor Files

Testing Editor Changes Locally

Interactive SDRF Builder

How It Works

Builder Files

Updating the Builder

Contributing

FilesExpand file tree

DEVELOPMENT.md

Latest commit

History

DEVELOPMENT.md

File metadata and controls

Development Guide

Repository Structure

Prerequisites

Building the Website Locally

Using the Build Script (Recommended)

What the Build Script Does

Manual Build (Advanced)

View Locally

Key Files to Modify

SDRF Example Table Styling

CI/CD Deployment

Local vs CI/CD Comparison

Ensuring Consistency

Adding New Annotated Projects

References in AsciiDoc

Useful Commands

Troubleshooting

Stylesheet Warnings

Missing Dependencies

Build Output Differences

Common Issues

SDRF Editor Integration

Architecture

Updating the Editor

Editor Files

Testing Editor Changes Locally

Interactive SDRF Builder

How It Works

Builder Files

Updating the Builder

Contributing