Knowledge-Augmented Reinforcement Learning — Training Framework
A multi-turn conversational RL training framework built on verl, supporting the training of agents in knowledge-intensive settings such as KnowledgeGraph and DataBase.
KARL-Train uses GRPO (Group Relative Policy Optimization) for reinforcement learning training, with support for:
- Multi-turn conversational RL training
- Knowledge Graph tasks
- DataBase tasks
- Multi-task mixed training
- Asynchronous rollout (vLLM as rollout server)
- FSDP2 distributed training
- Python >= 3.10
- PyTorch >= 2.3
- CUDA >= 12.1
- Ray >= 2.10
- vLLM (for rollout server)
- Multi-node GPU cluster (H100/H800 recommended)
# Install dependencies
pip install -e .
pip install -r requirements.txt
# Install SGLang (from local copy, no-deps to avoid conflicts)
pip uninstall -y sglang
cd sglang
pip install -e "python[all]" --no-deps
cd ..
# Install aiomonitor (for async monitoring)
pip install aiomonitorTraining configuration files are located in configs/. Use configs/example.yaml as a template to create your own.
Replace the placeholders in configs/example.yaml before use:
# 1. Model path
actor_rollout_ref:
model:
path: <MODEL_PATH>/Qwen2.5-7B-Instruct # replace with actual path
# 2. Rollout server URL
actor_rollout_ref:
rollout:
base_url: http://<ROLLOUT_SERVER>:9000/api # replace with actual URLEdit scripts/start_agentbench.sh and fill in your WandB credentials:
export WANDB_BASE_URL="https://wandb.example.com" # WandB server URL
export WANDB_API_KEY="<YOUR_WANDB_API_KEY>" # WandB API key
export WANDB_ENTITY="<YOUR_WANDB_ENTITY>" # WandB user/organization# start ray
cd train
## single node
bash scripts/setup.sh
## multi-node
allrun bash scripts/setup.sh# start training
bash scripts/start_agentbench.sh exampleTraining logs are saved to ./logs/train_<timestamp>.log.
The eval/ directory contains post-training evaluation and analysis tools:
| File | Description |
|---|---|
check.py |
Result statistics. Reads evaluation output (JSONL) and computes average score, standard deviation, Best-of-N, Pass Rate, and Pass@K metrics. Supports single-file and batch-directory modes. |
server_agent_multitask.py |
API-based evaluation. Evaluates models via an OpenAI-compatible API, supporting multi-task evaluation. Compatible with vLLM, SGLang, or any OpenAI API-compatible service. |
verl2hf.py |
Model format conversion. Merges verl FSDP sharded checkpoints into standard HuggingFace format. Supports custom output paths and uploading to HuggingFace Hub. |
You need to first start the controller and workers at the environment side. See ../env/README.md for detailed setup.
Then you can start evaluation by:
# Compute evaluation statistics
python eval/check.py <output.jsonl>
python eval/check.py <output_dir>/ # batch mode for all jsonl files in a directory
# API-based evaluation
python eval/server_agent_multitask.py -m <model_name> -u <api_url> -c <controller_url> <task_name>
# Convert checkpoint to HuggingFace format
python eval/verl2hf.py --local_dir <checkpoint_dir> --output_dir <hf_output_dir>karl-train/
├── configs/ # Training configurations
│ └── example.yaml # Configuration template
├── scripts/ # Launch scripts
│ ├── start_agentbench.sh # AgentBench training launcher
│ ├── start_lbw.sh # LBW training launcher
│ └── setup.sh # Environment setup script
├── eval/ # Evaluation tools
│ ├── check.py # Result statistics (avg score / Pass@K)
│ ├── server_agent_multitask.py # API-based multi-task evaluation
│ └── verl2hf.py # FSDP checkpoint -> HuggingFace conversion
├── sglang/ # SGLang inference engine (local copy)
├── verl/ # Core verl codebase
│ ├── trainer/ # Trainers (main_ppo entry point)
│ ├── workers/ # Actor / Critic / Rollout / Reward workers
│ ├── models/ # Model definitions (LLaMA, Qwen2, etc.)
│ ├── tools/ # Tool-calling support (search, code sandbox, etc.)
│ ├── single_controller/ # Ray single-controller scheduling
│ ├── third_party/ # Third-party integrations (vLLM, SGLang)
│ └── utils/ # Utilities (reward scoring, checkpointing, etc.)
├── tests/ # Tests
├── requirements.txt # Python dependencies
├── pyproject.toml # Project metadata
├── setup.py # Install script
└── README.md # This file