Infra] Support Markdown/Notebook-Friendly Documentation Export for Downstream Integration

[zhaochenyang20]

Checklist

• If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
• Please use English. Otherwise, it will be closed.

Motivation

Currently, SGLang's documentation system uses a make compile instruction that targets HTML for immediate online display. This "HTML-only" output creates a bottleneck for downstream systems (like internal documentation platforms or LLM-based RAG pipelines) that require structured data.

We need to modify the documentation pipeline so that the make compile command preserves the intermediate Markdown or generates Jupyter Notebook (.ipynb) files as primary artifacts, rather than just treating them as transient steps toward an HTML file.

Problem Statement

The current build process couples content to style too early:

• Transient Markdown: The source Markdown is often discarded or transformed into HTML fragments that lose semantic structure.
• Downstream Incompatibility: Systems requiring GPT Notebook or raw Markdown cannot ingest the current HTML output without fragile parsing.
• Metadata Loss: Important structural markers (callouts, specific code block metadata) are often lost during the final HTML render.

Goals

1. Preserve Source Integrity: The make compile command must ensure that the output directory contains the structured Markdown or Jupyter Notebook version of the docs.
2. Notebook-First Workflow: Ensure that code blocks and metadata remain intact in a format that can be easily converted to or stored as Jupyter Notebooks.
3. Dual-Path Output: The system should still support the current web display, but the "source of truth" for downstream consumers must be the stable Markdown/Notebook artifacts.

Technical Tasks

• Pipeline Audit: Modify the existing Makefile and compilation scripts to prevent the deletion of intermediate Markdown files.
• Structured Export: Ensure the make compile process produces a version of the documentation where content (Markdown/Notebook) is decoupled from the HTML CSS/JS.
• Semantic Retention: Verify that custom components (e.g., specific SGLang callouts or specialized code blocks) are preserved in the Markdown output using standard semantic markers.
• Validation: Run a test to confirm the output artifacts can be successfully ingested by a standard Markdown-to-Notebook converter without losing heading hierarchies or code block functionality.

Implementation Note

We want to keep the developer experience consistent. The workflow should still be triggered via:

make compile

However, the result should be a directory of persistent, structured files (Markdown/Notebooks) ready for downstream redistribution, in addition to or instead of the raw HTML blobs.

Calling all contributors interested in DevX and Documentation Infra! Help us move SGLang toward a more portable, "LLM-ready" documentation architecture.

---

Suggested Labels

infrastructure, documentation, build-system

Related resources

No response

[klhhhhh]

Proposed make compile Behavior

I propose to change make compile from an in-place notebook execution step into an artifact-producing compilation step.

Instead of executing notebooks in place (--inplace), make compile should:

1. Execute source notebooks without modifying them

2. Write executed notebooks to a stable output directory, e.g.

   docs/_build/export/ipynb/<relative-path>.ipynb
   

3. Optionally convert executed notebooks into structured Markdown and write them to:

   docs/_build/export/md/<relative-path>.md
   

The existing HTML documentation pipeline (make html) remains unchanged and continues to render documentation for the website.

[zhaochenyang20]

https://github.com/sgl-project/sglang/blob/main/.github/workflows/release-docs.yml

[zhaochenyang20]

See this:

We have make compile, which exec ipynb into ipynb with results.

We further get:

          cd docs
          make html
          python3 wrap_run_llm.py

In my idea, I think we should have a make markdown command, which puts the executed ipynb into markdown files under _build/html/.markdown_results.

But, everything should be there, including the index.rst. Which should be cared about.

[zhaochenyang20]

#18131 (review)

this solved our issue. Thanks!

[zhaochenyang20]

#18131 (comment)

really unfortunate, this should be reopened.