|
1 | | - |
| 1 | +# Gameplay Companion — Self-Learning AI Coach |
2 | 2 |
|
3 | | -<h1 align="center">Open-LLM-VTuber</h1> |
4 | | -<h3 align="center"> |
| 3 | +A real-time gameplay coaching companion built on top of Open-LLM-VTuber. |
| 4 | +Watches your screen, identifies the game, gives specific tactical advice via a Live2D avatar, |
| 5 | +and remembers what it has told you across sessions. |
5 | 6 |
|
6 | | -[](https://github.com/Open-LLM-VTuber/Open-LLM-VTuber/releases) |
7 | | -[](https://github.com/Open-LLM-VTuber/Open-LLM-VTuber/blob/master/LICENSE) |
8 | | -[](https://github.com/Open-LLM-VTuber/Open-LLM-VTuber/actions/workflows/codeql.yml) |
9 | | -[](https://github.com/Open-LLM-VTuber/Open-LLM-VTuber/actions/workflows/ruff.yml) |
10 | | -[](https://hub.docker.com/r/Open-LLM-VTuber/open-llm-vtuber) |
11 | | -[](https://qm.qq.com/q/ngvNUQpuKI) |
12 | | -[&color=blue&link=https%3A%2F%2Folv.zulipchat.com)](https://olv.zulipchat.com) |
| 7 | +--- |
13 | 8 |
|
14 | | -> **📢 v2.0 Development**: We are focusing on Open-LLM-VTuber v2.0 — a complete rewrite of the codebase. v2.0 is currently in its early discussion and planning phase. We kindly ask you to refrain from opening new issues or pull requests for feature requests on v1. To participate in the v2 discussions or contribute, join our developer community on [Zulip](https://olv.zulipchat.com). Weekly meeting schedules will be announced on Zulip. We will continue fixing bugs for v1 and work through existing pull requests. |
| 9 | +## What it does |
15 | 10 |
|
16 | | -[](https://www.buymeacoffee.com/yi.ting) |
17 | | -[](https://discord.gg/3UDA8YFDXx) |
| 11 | +1. **Watches your screen** — takes a screenshot every 15–25 s via PowerShell (WSL2 compatible) |
| 12 | +2. **Identifies the game** — sends the screenshot to `claude -p` with `@/path/to/screenshot.png`; Claude visually identifies the game, boss, current state, and returns a tactical JSON |
| 13 | +3. **Caches knowledge in Qdrant** — ingested tactic + best-practices vectors are reused on the next capture (no redundant research calls) |
| 14 | +4. **Delivers advice through a Live2D avatar** — proactive `ai-speak-signal` fires if screen changed; also responds to direct voice/text questions |
| 15 | +5. **Remembers the user** — Supabase stores every piece of advice given (`user_history`), long-term preferences (`long_term_profile`), and full session chat logs (`chat_logs`). Profile is injected into the system prompt at session start. |
18 | 16 |
|
19 | | -[](https://deepwiki.com/Open-LLM-VTuber/Open-LLM-VTuber) |
| 17 | +--- |
20 | 18 |
|
21 | | -ENGLISH README | [中文 README](./README.CN.md) | [한국어 README](./README.KR.md) | [日本語 README](./README.JP.md) |
| 19 | +## Architecture |
22 | 20 |
|
23 | | -[Documentation](https://open-llm-vtuber.github.io/docs/quick-start) | [](https://github.com/orgs/Open-LLM-VTuber/projects/2) |
| 21 | +``` |
| 22 | +Screen (Windows display) |
| 23 | + ↓ PowerShell CopyFromScreen every 15–25 s |
| 24 | +ScreenWatcher (daemon thread) ← pHash Hamming < 5 → skip AFK |
| 25 | + ↓ screenshot path |
| 26 | +KnowledgeBase.get_game_context(@path) |
| 27 | + ↓ game_name == "unknown" ↓ known game |
| 28 | + claude -p @screenshot Qdrant vector search (score ≥ 0.8) |
| 29 | + → JSON: game, state, tactic → HIT: return cached tactic |
| 30 | + → Qdrant ingest (tactic + tips) → MISS: claude research + ingest |
| 31 | + → current_game_name updated |
| 32 | + ↓ tactic text |
| 33 | +ai-speak-signal → WebSocketHandler → ConversationHandler |
| 34 | + → BasicMemoryAgent → WSLClaudeLLM (claude -p subprocess) |
| 35 | + → TTS → Live2D avatar speaks |
| 36 | + ↓ |
| 37 | +MemoryManager.save_advice(game_name, advice) → Supabase user_history |
| 38 | +``` |
24 | 39 |
|
25 | | -<a href="https://trendshift.io/repositories/12358" target="_blank"><img src="https://trendshift.io/api/badge/repositories/12358" alt="Open-LLM-VTuber%2FOpen-LLM-VTuber | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a> |
| 40 | +### LLM |
26 | 41 |
|
27 | | -</h3> |
| 42 | +All inference uses `claude -p` (WSL subprocess, no API keys, no remote LLM calls). |
| 43 | +Sole provider: `WSLClaudeLLM` in `src/open_llm_vtuber/agent/stateless_llm/wsl_claude.py`. |
28 | 44 |
|
| 45 | +### RAG |
29 | 46 |
|
30 | | -> 常见问题 Common Issues doc (Written in Chinese): https://docs.qq.com/pdf/DTFZGQXdTUXhIYWRq |
31 | | -> |
32 | | -> User Survey: https://forms.gle/w6Y6PiHTZr1nzbtWA |
33 | | -> |
34 | | -> 调查问卷(中文): https://wj.qq.com/s2/16150415/f50a/ |
| 47 | +Qdrant cloud collection `game_knowledge` (384-dim Cosine, `sentence-transformers/all-minilm-l6-v2`). |
| 48 | +Game-specific tactics and best-practices are ingested on first encounter and reused thereafter. |
35 | 49 |
|
| 50 | +### Memory |
36 | 51 |
|
| 52 | +Supabase (PostgreSQL) tables: |
| 53 | +- `user_history` — every piece of advice given with game name + timestamp |
| 54 | +- `chat_logs` — full session history saved on disconnect |
| 55 | +- `long_term_profile` — persistent user preferences injected into system prompt |
37 | 56 |
|
38 | | -> :warning: This project is in its early stages and is currently under **active development**. |
| 57 | +--- |
39 | 58 |
|
40 | | -> :warning: If you want to run the server remotely and access it on a different machine, such as running the server on your computer and access it on your phone, you will need to configure `https`, because the microphone on the front end will only launch in a secure context (a.k.a. https or localhost). See [MDN Web Doc](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia). Therefore, you should configure https with a reverse proxy to access the page on a remote machine (non-localhost). |
| 59 | +## Setup |
41 | 60 |
|
| 61 | +### Prerequisites |
| 62 | +- WSL2 with Ubuntu |
| 63 | +- `claude` CLI installed and authenticated (`~/.local/bin/claude`) |
| 64 | +- Qdrant cloud account |
| 65 | +- Supabase project |
42 | 66 |
|
| 67 | +### Install |
43 | 68 |
|
44 | | -## ⭐️ What is this project? |
| 69 | +```bash |
| 70 | +uv sync |
| 71 | +``` |
45 | 72 |
|
| 73 | +### Configure |
46 | 74 |
|
47 | | -**Open-LLM-VTuber** is a unique **voice-interactive AI companion** that not only supports **real-time voice conversations** and **visual perception** but also features a lively **Live2D avatar**. All functionalities can run completely offline on your computer! |
| 75 | +Copy `.env.example` to `.env` and fill in: |
48 | 76 |
|
49 | | -You can treat it as your personal AI companion — whether you want a `virtual girlfriend`, `boyfriend`, `cute pet`, or any other character, it can meet your expectations. The project fully supports `Windows`, `macOS`, and `Linux`, and offers two usage modes: web version and desktop client (with special support for **transparent background desktop pet mode**, allowing the AI companion to accompany you anywhere on your screen). |
| 77 | +``` |
| 78 | +QDRANT_API_KEY=... |
| 79 | +QDRANT_CLUSTER_ENDPOINT=https://<id>.qdrant.io |
| 80 | +SUPABASE_URL=https://<id>.supabase.co |
| 81 | +SUPABASE_SECRET_KEY=sb_secret_... |
| 82 | +``` |
50 | 83 |
|
51 | | -Although the long-term memory feature is temporarily removed (coming back soon), thanks to the persistent storage of chat logs, you can always continue your previous unfinished conversations without losing any precious interactive moments. |
| 84 | +Create the Supabase tables using `migrations/001_create_tables.sql` in the Supabase SQL editor. |
52 | 85 |
|
53 | | -In terms of backend support, we have integrated a rich variety of LLM inference, text-to-speech, and speech recognition solutions. If you want to customize your AI companion, you can refer to the [Character Customization Guide](https://open-llm-vtuber.github.io/docs/user-guide/live2d) to customize your AI companion's appearance and persona. |
| 86 | +### Run |
54 | 87 |
|
55 | | -The reason it's called `Open-LLM-Vtuber` instead of `Open-LLM-Companion` or `Open-LLM-Waifu` is because the project's initial development goal was to use open-source solutions that can run offline on platforms other than Windows to recreate the closed-source AI Vtuber `neuro-sama`. |
| 88 | +```bash |
| 89 | +uv run run_server.py |
| 90 | +``` |
56 | 91 |
|
57 | | -### 👀 Demo |
58 | | -|  |  | |
59 | | -|:---:|:---:| |
60 | | -|  |  | |
| 92 | +Open `http://localhost:12393` in a browser. Start playing a game — the companion will begin commenting within 15–25 seconds. |
61 | 93 |
|
| 94 | +--- |
62 | 95 |
|
63 | | -## ✨ Features & Highlights |
| 96 | +## Key files |
64 | 97 |
|
65 | | -- 🖥️ **Cross-platform support**: Perfect compatibility with macOS, Linux, and Windows. We support NVIDIA and non-NVIDIA GPUs, with options to run on CPU or use cloud APIs for resource-intensive tasks. Some components support GPU acceleration on macOS. |
| 98 | +| File | Role | |
| 99 | +|------|------| |
| 100 | +| `src/.../agent/stateless_llm/wsl_claude.py` | Claude subprocess LLM (only provider) | |
| 101 | +| `src/.../modules/knowledge_base.py` | Qdrant RAG + Claude visual research | |
| 102 | +| `src/.../modules/vision_loop.py` | Screen watcher + proactive triggers | |
| 103 | +| `src/.../integrations/memory_manager.py` | Supabase CRUD | |
| 104 | +| `src/.../service_context.py` | DI container + system prompt assembly | |
| 105 | +| `src/.../conversations/single_conversation.py` | Pipeline + advice saving | |
| 106 | +| `src/.../websocket_handler.py` | WS routing, session init, vision loop | |
| 107 | +| `migrations/001_create_tables.sql` | Supabase DDL | |
| 108 | +| `docs/PHASE_*.md` | Per-phase development logs with live test results | |
66 | 109 |
|
67 | | -- 🔒 **Offline mode support**: Run completely offline using local models - no internet required. Your conversations stay on your device, ensuring privacy and security. |
68 | | - |
69 | | -- 💻 **Attractive and powerful web and desktop clients**: Offers both web version and desktop client usage modes, supporting rich interactive features and personalization settings. The desktop client can switch freely between window mode and desktop pet mode, allowing the AI companion to be by your side at all times. |
70 | | - |
71 | | -- 🎯 **Advanced interaction features**: |
72 | | - - 👁️ Visual perception, supporting camera, screen recording and screenshots, allowing your AI companion to see you and your screen |
73 | | - - 🎤 Voice interruption without headphones (AI won't hear its own voice) |
74 | | - - 🫱 Touch feedback, interact with your AI companion through clicks or drags |
75 | | - - 😊 Live2D expressions, set emotion mapping to control model expressions from the backend |
76 | | - - 🐱 Pet mode, supporting transparent background, global top-most, and mouse click-through - drag your AI companion anywhere on the screen |
77 | | - - 💭 Display AI's inner thoughts, allowing you to see AI's expressions, thoughts and actions without them being spoken |
78 | | - - 🗣️ AI proactive speaking feature |
79 | | - - 💾 Chat log persistence, switch to previous conversations anytime |
80 | | - - 🌍 TTS translation support (e.g., chat in Chinese while AI uses Japanese voice) |
81 | | - |
82 | | -- 🧠 **Extensive model support**: |
83 | | - - 🤖 Large Language Models (LLM): Ollama, OpenAI (and any OpenAI-compatible API), Gemini, Claude, Mistral, DeepSeek, Zhipu AI, GGUF, LM Studio, vLLM, etc. |
84 | | - - 🎙️ Automatic Speech Recognition (ASR): sherpa-onnx, FunASR, Faster-Whisper, Whisper.cpp, Whisper, Groq Whisper, Azure ASR, etc. |
85 | | - - 🔊 Text-to-Speech (TTS): sherpa-onnx, pyttsx3, MeloTTS, Coqui-TTS, GPTSoVITS, Bark, CosyVoice, Edge TTS, Fish Audio, Azure TTS, etc. |
86 | | - |
87 | | -- 🔧 **Highly customizable**: |
88 | | - - ⚙️ **Simple module configuration**: Switch various functional modules through simple configuration file modifications, without delving into the code |
89 | | - - 🎨 **Character customization**: Import custom Live2D models to give your AI companion a unique appearance. Shape your AI companion's persona by modifying the Prompt. Perform voice cloning to give your AI companion the voice you desire |
90 | | - - 🧩 **Flexible Agent implementation**: Inherit and implement the Agent interface to integrate any Agent architecture, such as HumeAI EVI, OpenAI Her, Mem0, etc. |
91 | | - - 🔌 **Good extensibility**: Modular design allows you to easily add your own LLM, ASR, TTS, and other module implementations, extending new features at any time |
92 | | - |
93 | | - |
94 | | -## 👥 User Reviews |
95 | | -> Thanks to the developer for open-sourcing and sharing the girlfriend for everyone to use |
96 | | -> |
97 | | -> This girlfriend has been used over 100,000 times |
98 | | -
|
99 | | - |
100 | | -## 🚀 Quick Start |
101 | | - |
102 | | -Please refer to the [Quick Start](https://open-llm-vtuber.github.io/docs/quick-start) section in our documentation for installation. |
103 | | - |
104 | | - |
105 | | - |
106 | | -## ☝ Update |
107 | | -> :warning: `v1.0.0` has breaking changes and requires re-deployment. You *may* still update via the method below, but the `conf.yaml` file is incompatible and most of the dependencies needs to be reinstalled with `uv`. For those who came from versions before `v1.0.0`, I recommend deploy this project again with the [latest deployment guide](https://open-llm-vtuber.github.io/docs/quick-start). |
108 | | -
|
109 | | -Please use `uv run update.py` to update if you installed any versions later than `v1.0.0`. |
110 | | - |
111 | | -## 😢 Uninstall |
112 | | -Most files, including Python dependencies and models, are stored in the project folder. |
113 | | - |
114 | | -However, models downloaded via ModelScope or Hugging Face may also be in `MODELSCOPE_CACHE` or `HF_HOME`. While we aim to keep them in the project's `models` directory, it's good to double-check. |
115 | | - |
116 | | -Review the installation guide for any extra tools you no longer need, such as `uv`, `ffmpeg`, or `deeplx`. |
117 | | - |
118 | | -## 🤗 Want to contribute? |
119 | | -Checkout the [development guide](https://docs.llmvtuber.com/docs/development-guide/overview). |
120 | | - |
121 | | - |
122 | | -# 🎉🎉🎉 Related Projects |
123 | | - |
124 | | -[ylxmf2005/LLM-Live2D-Desktop-Assitant](https://github.com/ylxmf2005/LLM-Live2D-Desktop-Assitant) |
125 | | -- Your Live2D desktop assistant powered by LLM! Available for both Windows and MacOS, it senses your screen, retrieves clipboard content, and responds to voice commands with a unique voice. Featuring voice wake-up, singing capabilities, and full computer control for seamless interaction with your favorite character. |
126 | | - |
127 | | - |
128 | | - |
129 | | - |
130 | | - |
131 | | - |
132 | | -## 📜 Third-Party Licenses |
133 | | - |
134 | | -### Live2D Sample Models Notice |
135 | | - |
136 | | -This project includes Live2D sample models provided by Live2D Inc. These assets are licensed separately under the Live2D Free Material License Agreement and the Terms of Use for Live2D Cubism Sample Data. They are not covered by the MIT license of this project. |
137 | | - |
138 | | -This content uses sample data owned and copyrighted by Live2D Inc. The sample data are utilized in accordance with the terms and conditions set by Live2D Inc. (See [Live2D Free Material License Agreement](https://www.live2d.jp/en/terms/live2d-free-material-license-agreement/) and [Terms of Use](https://www.live2d.com/eula/live2d-sample-model-terms_en.html)). |
139 | | - |
140 | | -Note: For commercial use, especially by medium or large-scale enterprises, the use of these Live2D sample models may be subject to additional licensing requirements. If you plan to use this project commercially, please ensure that you have the appropriate permissions from Live2D Inc., or use versions of the project without these models. |
141 | | - |
142 | | - |
143 | | -## Contributors |
144 | | -Thanks our contributors and maintainers for making this project possible. |
145 | | - |
146 | | -<a href="https://github.com/Open-LLM-VTuber/Open-LLM-VTuber/graphs/contributors"> |
147 | | - <img src="https://contrib.rocks/image?repo=Open-LLM-VTuber/Open-LLM-VTuber" /> |
148 | | -</a> |
149 | | - |
150 | | - |
151 | | -## Star History |
152 | | - |
153 | | -[](https://star-history.com/#Open-LLM-VTuber/open-llm-vtuber&Date) |
| 110 | +--- |
154 | 111 |
|
| 112 | +## Development |
155 | 113 |
|
| 114 | +```bash |
| 115 | +# Lint |
| 116 | +ruff check . |
156 | 117 |
|
| 118 | +# Format |
| 119 | +ruff format . |
157 | 120 |
|
| 121 | +# Run simulation test |
| 122 | +uv run tests/simulation.py |
| 123 | +``` |
158 | 124 |
|
| 125 | +See `CLAUDE.md` for full architecture notes and `PLAN.md` for the transformation roadmap. |
0 commit comments