MindPilot

Closed-loop Visual Stimulation Optimization for Brain Modulation with EEG-guided Diffusion

---

📰 News

• 🧠 2026.06.19: We add the encoding directory for training proxy models.
• 🚀 2026.03.08: We update the codebase and Hugging Face resources.
• 📄 2026.02.11: We upload the arXiv paper and release the code repository.
• 🎉 2026.01.26: Our paper is accepted to ICLR 2026.

📖 Overview

We propose MindPilot, which employs a simple black-box optimization approach to achieve EEG-guided closed-loop visual stimulation optimization (supporting offline learning data and experience replay) for regulating the brain activity of subjects. This paper addresses three problems: high-noise non-invasive brain signals; the non-differentiability of brain input and output; and the variability of brain activity states.

🎯 Conceptualization

The conceptualization of MindPilot: Closed-loop optimization of visual stimuli using EEG feedback

---

🏗️ Architecture

Overall architecture of MindPilot framework

Installation

Option 1: Quick Setup (Recommended)

cd MindPilot
chmod +x setup.sh
./setup.sh
conda activate MindPilot

Option 2: Manual Setup

conda env create -f environment.yml
conda activate MindPilot

---

📦 Pretrained Weights & Datasets

We provide pretrained model weights and preprocessed datasets on Hugging Face:

🤗 https://huggingface.co/datasets/LidongYang/Mindpilot

You can download using the Hugging Face CLI:

# Install huggingface_hub if not already installed
pip install huggingface_hub

# Download all files
huggingface-cli download LidongYang/Mindpilot --repo-type dataset --local-dir ./data

External Datasets

Download the additional datasets from the following sources:

Dataset	Description	Download Link
THINGS-EEG2	Natural images with EEG responses	OSF
ArtPhoto	Artistic photographs with emotion ratings	ImageEmotion
GAPED	Geneva Affective Picture Database	UNIGE
EmoSet	Large-scale emotion dataset	VCC Tech

---

🎓 Usage

0. EEG Encoding Pipeline

The reproducible EEG encoding implementation lives in encoding/. It includes DNN feature extraction, PCA, linearizing encoding training, and evaluation scripts. Large datasets, feature arrays, pretrained weights, and generated results are intentionally excluded from Git and must be provided via the documented environment variables.

1. Train EEG Readout Model

Train a neural network to predict EEG responses from visual features:

bash encoding/synthesizing_eeg_data/train.sh \
    --sub 1 \
    --dnn alexnet \
    --subjects within \
    --pretrained true \
    --layers all \
    --brain_regions occipital_parietal \
    --n_components 1000

2. Interactive Search

Perform evolutionary search in the latent space for optimal stimuli:

jupyter notebook experiments/exp-interactive_search.ipynb

3. Heuristic Generation

Generate optimized visual stimuli using gradient-based optimization:

python experiments/exp-heuristic_generation_with_guidance_anyfeature.py

4. Benchmark Evaluation

Offline Generation Benchmark

bash experiments/run_benchmark_offline_generation.sh

Heuristic Generation Benchmark

bash experiments/run_benchmark_heuristic_generation.sh

Complete Benchmark Suite

bash experiments/run_benchmark_total.sh

5. Real-time Human Experiments

Server side (runs optimization):

python server/improved_experiment.py --port 5000

Client side (presents stimuli and records EEG):

python client/client.py --server_ip 192.168.1.100 --port 5000

📝 Citation

If you find this work useful, please cite our paper:

@article{li2024visual,
  title={Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion},
  author={Li, Dongyang and Wei, Chen and Li, Shiying and Zou, Jiachen and Liu, Quanying},
  journal={Advances in Neural Information Processing Systems},
  volume={37},
  pages={102822--102864},
  year={2024}
}
@inproceedings{2026mindpilot,
  title={MindPilot: Closed-loop Visual Stimulation Optimization for Brain Modulation with {EEG}-guided Diffusion},
  author={Dongyang Li, Kunpeng Xie, Mingyang Wu, Yiwei Kong, Jiahua Tang, Haoyang Qin, Chen Wei, Quanying Liu },
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
  url={https://openreview.net/forum?id=7jdmXx869Q}
}

😺 Acknowledgement

We sincerely thank the following works and their authors for providing valuable resources that supported this project:

• The THINGS-EEG2 dataset — "A large and rich EEG dataset for modeling human visual object recognition" by Alessandro T. Gifford, Kshitij Dwivedi, Gemma Roig, and Radoslaw M. Cichy.
• FAST-DiReCT — "Query-Efficient Online Black-box Guidance for Diffusion-Model Target Generation" by Kim Yong Tan, Yueming Lyu, Ivor Tsang, and Yew-Soon Ong.
• D3PO — "Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model" by Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Qimai Li, Weihan Shen, Xiaolong Zhu, Xiu Li.
• DDPO — "Training Diffusion Models with Reinforcement Learning" by Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, Sergey Levine.
• DPOK — "Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models" by Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, Kimin Lee.

📬 Contact Dongyang Li if you have any questions or suggestions.

📜 License

This project is licensed under the MIT License.