Memory proxy as baseUrl: transparent long-term memory via OpenAI-compatible middleware

[flobo3]

Idea: Transparent memory layer via OpenAI-compatible proxy

I've been experimenting with adding semantic long-term memory to nanobot without touching the core — by inserting an OpenAI-compatible memory proxy between nanobot and the LLM provider.

Architecture

nanobot (baseUrl: localhost:8100/v1)
    |
    v
Memory Proxy (e.g. agent-cli memory proxy)
    |  1. Intercepts /v1/chat/completions
    |  2. Embeds user message -> ChromaDB vector search
    |  3. Injects relevant memories into system prompt
    |  4. Forwards augmented request to real LLM
    |  5. Extracts new facts from response -> stores as markdown
    v
LLM Provider (OpenRouter, OpenAI, etc.)

The proxy is agent-cli (pip install "agent-cli[memory]"). It stores memories as human-readable markdown files + ChromaDB for vector search, with built-in git versioning. Embeddings can run locally via Ollama (nomic-embed-text) so no extra API cost.

Setup

agent-cli memory proxy \
  --memory-path ~/.nanobot/workspace/memory-proxy \
  --openai-base-url https://openrouter.ai/api/v1 \
  --embedding-base-url http://localhost:11434/v1 \
  --embedding-model nomic-embed-text \
  --port 8100

Then in nanobot config:

{
  "baseUrl": "http://localhost:8100/v1",
  "model": "google/gemini-2.5-flash-preview"
}

What works

• Transparent: nanobot doesn't know it's there, no core changes needed
• Works with any OpenAI-compatible provider
• Memories are markdown files on disk (readable, editable)
• Semantic search (ChromaDB) + recency weighting + MMR diversity
• Fact reconciliation (updates old facts instead of just appending)
• Local embeddings via Ollama = zero extra cost

Concerns

• Two memory layers running in parallel (built-in MEMORY.md + proxy's ChromaDB) - some duplication and extra token usage
• The proxy doesn't see nanobot's tool calls / tool results, only the raw chat messages - so it can't extract facts from tool outputs
• Streaming support - need to verify it handles nanobot's streaming mode correctly
• No awareness of nanobot's session boundaries (/new, /reset) - memories persist across sessions by design, but there's no way to scope them per-session

Questions

1. Has anyone tried this approach (memory proxy as middleware) with nanobot?
2. Are there any known issues with putting a proxy in front of nanobot's LLM calls (streaming, tool calls, error handling)?
3. Is there a better way to achieve transparent semantic memory while waiting for the hooks/plugin system (Implement hooks feature (like Claude Code/GitHub Copilot CLI) #2182, Feature request: Plugin system for agent extensibility (like Copilot CLI or Claude Code) #2231)?
4. Would it make sense to build something like this as a native nanobot feature (Dream PR feat(memory): two-stage memory system with Dream consolidation #2717 looks related but different)?

Would love to hear if anyone has experimented with similar setups.