REPRO-Bench

📄 Official codebase for the ACL 2025 paper: REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Papers?

📦 Dataset

The REPRO-Bench dataset is hosted on Hugging Face:

git clone https://huggingface.co/datasets/chuxuan/REPRO-Bench
cd REPRO-Bench
git lfs pull

It includes:

• 112 task instances (PDFs + code + data)
• Gold-standard reproducibility annotations
• Public reproduction reports

---

🚀 Getting Started

Run SWE-Agent

bash SWE-Agent/run_all.sh

Run AutoGPT

bash AutoGPT/classic/original_autogpt/run_all.sh

Run CORE-Agent

bash CORE-Agent/classic/original_autogpt/run_all.sh