Skip to content

Latest commit

 

History

History
152 lines (114 loc) · 5.24 KB

README.md

File metadata and controls

152 lines (114 loc) · 5.24 KB

KARL-Train

Knowledge-Augmented Reinforcement Learning — Training Framework

A multi-turn conversational RL training framework built on verl, supporting the training of agents in knowledge-intensive settings such as KnowledgeGraph and DataBase.

Overview

KARL-Train uses GRPO (Group Relative Policy Optimization) for reinforcement learning training, with support for:

  • Multi-turn conversational RL training
  • Knowledge Graph tasks
  • DataBase tasks
  • Multi-task mixed training
  • Asynchronous rollout (vLLM as rollout server)
  • FSDP2 distributed training

Requirements

  • Python >= 3.10
  • PyTorch >= 2.3
  • CUDA >= 12.1
  • Ray >= 2.10
  • vLLM (for rollout server)
  • Multi-node GPU cluster (H100/H800 recommended)

Installation

# Install dependencies
pip install -e .
pip install -r requirements.txt

# Install SGLang (from local copy, no-deps to avoid conflicts)
pip uninstall -y sglang
cd sglang
pip install -e "python[all]" --no-deps
cd ..

# Install aiomonitor (for async monitoring)
pip install aiomonitor

Configuration

Training configuration files are located in configs/. Use configs/example.yaml as a template to create your own.

Editing the Configuration

Replace the placeholders in configs/example.yaml before use:

# 1. Model path
actor_rollout_ref:
  model:
    path: <MODEL_PATH>/Qwen2.5-7B-Instruct  # replace with actual path

# 2. Rollout server URL
actor_rollout_ref:
  rollout:
    base_url: http://<ROLLOUT_SERVER>:9000/api  # replace with actual URL

Training

Environment Variables

Edit scripts/start_agentbench.sh and fill in your WandB credentials:

export WANDB_BASE_URL="https://wandb.example.com"   # WandB server URL
export WANDB_API_KEY="<YOUR_WANDB_API_KEY>"          # WandB API key
export WANDB_ENTITY="<YOUR_WANDB_ENTITY>"            # WandB user/organization

Launch Training

# start ray 
cd train
## single node
bash scripts/setup.sh

## multi-node
allrun bash scripts/setup.sh
# start training
bash scripts/start_agentbench.sh example

Training logs are saved to ./logs/train_<timestamp>.log.

Evaluation Tools (eval/)

The eval/ directory contains post-training evaluation and analysis tools:

File Description
check.py Result statistics. Reads evaluation output (JSONL) and computes average score, standard deviation, Best-of-N, Pass Rate, and Pass@K metrics. Supports single-file and batch-directory modes.
server_agent_multitask.py API-based evaluation. Evaluates models via an OpenAI-compatible API, supporting multi-task evaluation. Compatible with vLLM, SGLang, or any OpenAI API-compatible service.
verl2hf.py Model format conversion. Merges verl FSDP sharded checkpoints into standard HuggingFace format. Supports custom output paths and uploading to HuggingFace Hub.

Usage Examples

You need to first start the controller and workers at the environment side. See ../env/README.md for detailed setup.

Then you can start evaluation by:

# Compute evaluation statistics
python eval/check.py <output.jsonl>
python eval/check.py <output_dir>/          # batch mode for all jsonl files in a directory

# API-based evaluation
python eval/server_agent_multitask.py   -m <model_name>   -u <api_url>   -c <controller_url>   <task_name>

# Convert checkpoint to HuggingFace format
python eval/verl2hf.py --local_dir <checkpoint_dir> --output_dir <hf_output_dir>

Project Structure

karl-train/
├── configs/                         # Training configurations
│   └── example.yaml                # Configuration template
├── scripts/                         # Launch scripts
│   ├── start_agentbench.sh         # AgentBench training launcher
│   ├── start_lbw.sh                # LBW training launcher
│   └── setup.sh                    # Environment setup script
├── eval/                            # Evaluation tools
│   ├── check.py                    # Result statistics (avg score / Pass@K)
│   ├── server_agent_multitask.py   # API-based multi-task evaluation
│   └── verl2hf.py                  # FSDP checkpoint -> HuggingFace conversion
├── sglang/                          # SGLang inference engine (local copy)
├── verl/                            # Core verl codebase
│   ├── trainer/                    # Trainers (main_ppo entry point)
│   ├── workers/                    # Actor / Critic / Rollout / Reward workers
│   ├── models/                     # Model definitions (LLaMA, Qwen2, etc.)
│   ├── tools/                      # Tool-calling support (search, code sandbox, etc.)
│   ├── single_controller/          # Ray single-controller scheduling
│   ├── third_party/                # Third-party integrations (vLLM, SGLang)
│   └── utils/                      # Utilities (reward scoring, checkpointing, etc.)
├── tests/                           # Tests
├── requirements.txt                 # Python dependencies
├── pyproject.toml                   # Project metadata
├── setup.py                         # Install script
└── README.md                        # This file