livepeer · victorges · Nov 25, 2025 · Nov 25, 2025 · Nov 27, 2025
diff --git a/AGENTS.md b/AGENTS.md
@@ -50,14 +50,18 @@
 | Runtime overview | `docs/live-ai-runtime-overview.md` |
 | Local development | `docs/live-ai-local-dev.md` |
 | Container setup | `docs/runner-docker.md` |
+| Docker image hierarchy | This file → "Docker Image Hierarchy" section |
+| External pipeline development | `docs/external-pipelines.md` |
 | FastAPI entrypoint | `runner/app/main.py` |
 | Live pipeline wrapper | `runner/app/pipelines/live_video_to_video.py` |
 | Infer process | `runner/app/live/infer.py` |
 | Process management | `runner/app/live/process/` |
 | Streaming protocols | `runner/app/live/streamer/` |
 | Frame encoding/decoding | `runner/app/live/trickle/` |
 | Pipeline implementations | `runner/app/live/pipelines/` |
+| Pipeline plugin loader | `runner/app/live/pipelines/loader.py` |
 | HTTP API (internal) | `runner/app/live/api/api.py` |
+| Package dependencies | `runner/pyproject.toml` |
 
 ---
 
@@ -78,6 +82,76 @@
 
 ---
 
+## 🐳 Docker Image Hierarchy
+
+Docker images follow a **three-layer hierarchy** for build efficiency and separation of concerns:
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│  Dockerfile.live-app__PIPELINE__  (or Dockerfile.live-app-noop)    │
+│  - Installs ai-runner-base package (pyproject.toml)                │
+│  - Copies app code                                                  │
+│  - Registers entry points for pipeline discovery                   │
+└─────────────────────────────────┬───────────────────────────────────┘
+                                  │ FROM
+┌─────────────────────────────────▼───────────────────────────────────┐
+│  Dockerfile.live-base-{pipeline}  (streamdiffusion, comfyui, scope)│
+│  - Installs conda + Python environment                             │
+│  - Installs PyTorch with CUDA support (from pytorch.org URL)       │
+│  - Installs pipeline-specific libs (StreamDiffusion, ComfyUI, etc) │
+└─────────────────────────────────┬───────────────────────────────────┘
+                                  │ FROM
+┌─────────────────────────────────▼───────────────────────────────────┐
+│  Dockerfile.live-base                                               │
+│  - nvidia/cuda base image                                           │
+│  - System deps (build-essential, etc)                               │
+│  - pyenv for Python version management                              │
+│  - FFmpeg with NVIDIA hardware acceleration                         │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+### Image Files (`runner/docker/`)
+
+| File | Purpose |
+|------|---------|
+| `Dockerfile.live-base` | Base image: CUDA, pyenv, FFmpeg with NVENC/NVDEC |
+| `Dockerfile.live-base-streamdiffusion` | Adds conda, PyTorch (cu128), StreamDiffusion lib |
+| `Dockerfile.live-base-comfyui` | Based on comfyui-base, adds FFmpeg |
+| `Dockerfile.live-base-scope` | Adds conda, scope dependencies |
+| `Dockerfile.live-app__PIPELINE__` | App layer for streamdiffusion/comfyui/scope (uses conda) |
+| `Dockerfile.live-app-noop` | App layer for noop pipeline (uses pyenv, no CUDA) |
+
+### Key Design Decisions
+
+1. **PyTorch is NOT in pyproject.toml** - It's installed in base images with CUDA-specific builds from `download.pytorch.org/whl/cu128`
+
+2. **Dependencies in pyproject.toml** - All runtime deps (aiohttp, fastapi, etc.) are defined in `runner/pyproject.toml` with exact versions
+
+3. **Entry points for pipeline discovery** - Pipelines register via `[project.entry-points."ai_runner.pipeline"]` in pyproject.toml
+
+4. **Layer caching strategy** - App Dockerfiles copy pyproject.toml first, install deps, then copy app code to maximize cache hits
+
+5. **Conda vs pyenv**:
+   - `live-base-streamdiffusion/comfyui/scope` → Use conda (`comfystream` environment)
+   - `live-base` (for noop) → Uses pyenv
+
+### Building Images
+
+```bash
+# Build base image
+docker build -f docker/Dockerfile.live-base -t livepeer/ai-runner:live-base .
+
+# Build pipeline base (example: streamdiffusion)
+docker build -f docker/Dockerfile.live-base-streamdiffusion -t livepeer/ai-runner:live-base-streamdiffusion .
+
+# Build app image
+docker build -f docker/Dockerfile.live-app__PIPELINE__ \
+  --build-arg PIPELINE=streamdiffusion \
+  -t livepeer/ai-runner:live-app-streamdiffusion .
+```
+
+---
+
 ## 📐 Architecture Overview
 
 ### Two-Level Process Hierarchy
@@ -255,6 +329,12 @@ class Pipeline(ABC):
     def prepare_models(cls): ...
 ```
 
+**Plugin System**: Pipelines are discovered via Python entry points. Two entry point groups are used:
+- `ai_runner.pipeline` - Pipeline class (loaded when pipeline is instantiated)
+- `ai_runner.pipeline_params` - Params class (loaded for lightweight parameter parsing, optional - falls back to BaseParams if not specified)
+
+This avoids importing the pipeline class (and heavy dependencies like torch) when just parsing parameters. See `docs/external-pipelines.md` for details.
+
 ---
 
 ## 🔑 Key Concepts
@@ -496,11 +576,20 @@ threading.Thread(target=lambda: stdout.close(), daemon=True).start()
 
 ### Adding a New Pipeline
 
-1. Create new directory under `runner/app/live/pipelines/`
-2. Implement `Pipeline` ABC (initialize, put_video_frame, get_processed_video_frame, update_params, stop, prepare_models)
-3. Create `*Params` class extending `BaseParams`
-4. Add to `loader.py` (load_pipeline and parse_pipeline_params)
-5. Add Docker configuration if needed (`docker/Dockerfile.live-app-*`)
+**Built-in pipelines** (in this repo):
+1. Create new directory under `runner/app/live/pipelines/{name}/`
+2. Create `pipeline.py` with your `Pipeline` subclass
+3. Create `params.py` with your `*Params` class extending `BaseParams`
+4. Register entry point in `runner/pyproject.toml` under `[project.entry-points."ai_runner.pipeline"]`
+5. Add Docker configuration if needed (`docker/Dockerfile.live-base-*`)
+
+**External pipelines** (separate repos):
+1. Create a Python package with `pipeline.py` and `params.py` modules
+2. Register both entry points in your `pyproject.toml`:
+   - `ai_runner.pipeline` → your Pipeline class
+   - `ai_runner.pipeline_params` → your Params class (optional - falls back to BaseParams)
+3. Install via `pip install` or `uv pip install`
+4. See `docs/external-pipelines.md` for complete guide
 
 ### Modifying Process Architecture
 
@@ -532,6 +621,13 @@ threading.Thread(target=lambda: stdout.close(), daemon=True).start()
 **Last Updated**: 2025-11-25
 
 **Recent Changes**:
+- Added pipeline plugin system using Python entry points (`ai_runner.pipeline`)
+- Pipelines use two entry point groups: `ai_runner.pipeline` and `ai_runner.pipeline_params`
+- Created `runner/pyproject.toml` for package definition and dependencies
+- Removed `requirements.live-ai.txt` (deps now in pyproject.toml)
+- Updated Dockerfiles to use pyproject.toml for dependency installation
+- Added Docker Image Hierarchy section documenting the three-layer build structure
+- Created `docs/external-pipelines.md` for external pipeline development guide
 - Deprecated ZeroMQ protocol references (was only used for local development)
 - Clarified that Trickle is the only streaming protocol
 - Added "Stream Start Flow" section explaining how streams are initiated via API