The source-available pipeline for your personal data: 20+ apps synced into one place you can graph, chat with and learn from.
Quick start · Features · Integrations · Storage · CLI, API & MCP · Deploy · License
There are thousands of products helping companies use their knowledge bases and analytics. Our most important data stays neglected - every person has millions of data points recorded, tracking on purpose or not.
agentqs is the pipeline for it: connect 20+ apps, scrape the ones that lock your data in. It structures, indexes and embeds everything into one searchable record - ask it anything, or learn from it with graphs, AI with skills, or voice sessions. Works out of the box in the web app, CLI, Slack or Telegram.
docker run -d --name agentqs -p 3000:3000 -v agentqs-data:/data flowenginecloud/agentqsOpen http://localhost:3000.
Everything lives in the agentqs-data volume, session key included, so restarts keep you signed in. Set -e SESSION_SECRET=... only if you want to pin that key yourself.
agentqs is CLI-first, and this setup needs no API keys and no environment variables.
git clone https://github.com/Ami3466/agentqs.git && cd agentqs
npm install
npm run dev # → http://localhost:3000Point an AI coding agent at it and it can import the data already on your machine, connect your accounts and set up syncs.
Contributing? npm run verify:tree type-checks the tree git would push (not your working copy), which catches the one thing that has ever broken this repo's CI: a new module imported but never committed. Each feature also has a deterministic script - npm run capture:test covers the capture path, npm run channels:test the Slack/Telegram bots, npm run rebuild:verify the derived cache.
Better on the cloud: it needs to be live for the Slack and Telegram integrations, and OAuth apps need a public, stable callback URL. Either way you can back your data up, encrypted, to your own Google Drive - on the schedule you set.
![]() |
![]() |
![]() |
![]() |
| Data Pipelines: integrations, auto scraping, manual import, and Chrome extension for Google data. |
Journal View your structured data. Edit and organize. |
Graphs Lines and correlations across any data points. Save the views. |
Chat Add skills, chat with AI, or have a voice session. |
One daily record of everything: sleep, steps, mood, focus, screen time, workouts, commits - plain CSVs you own, as a table, a timeline, or the Coverage map (the same source × year heatmap Pipeline shows - click a cell to filter the table to that source). Scan data keeps it clean: duplicate columns merge, dead columns drop, messy values get fixed. One click each, all undoable. agentqs audit hands your AI agent the evidence for a deeper review - impossible dates, coverage holes, sources gone quiet, outlier values.
Connect a source, or drop any file and hit Structure - it lands in your record. Clean CSVs map instantly, no LLM. Every capture is logged: structured, pending or rejected, and everything is revertible. Click any source's name to jump to the Journal filtered to just its data (/journal?source=<id>). Under the source list, a coverage heatmap grades what you have: every stream by year, how far back each goes, and where the holes are - click a cell to open that source in the Journal (the Journal's own Coverage view shows the same map). Same map over the API (GET /api/coverage), the CLI (agentqs coverage), and the coverage MCP tool.
Correlate anything - sleep vs focus, screen time vs mood - and save the views.
Answers come from your actual data: SQL over metrics, text search over memos and sessions, semantic search over everything. // logs a memo, / runs a command.
Give the AI a persona - mentor, coach, therapist, or your own prompt. Skills run in chat, grounded in your record, and a chat session saves its insights and commitments back to it.
Text or voice, from the app, CLI or a channel. They land in your inbox - structure them into daily rows or keep them as searchable notes. A dropped file is keyed by its contents, so dropping the same one again says it is already there instead of failing - and if you had discarded that file, dropping it again is taken as asking for it back and returns it to the queue. A filename your alphabet doesn't survive (Hebrew, Arabic, Chinese) falls back to a real source name rather than becoming a daily column called 2. A dropped photo lands as a capture too - recognised by its name as well as its type, so an iPhone .HEIC, an UPPERCASE.JPG or anything your browser declines to label still arrives as a picture instead of being read as text and quietly skipped. Lists describe it (beach.jpg · image/jpeg · 3.0 MB) instead of shipping the whole file back to your browser, and a photo too big for a raw capture is told so in a photo's own terms, pointing at agentqs photos import. Nothing is ever skipped without a reason you can act on.
Slack and Telegram: log memos and ask your record questions from where you already are. They are capture channels, not scrapers - what you send the bot lands in your inbox, and your Slack history is never read. The Pipeline lists them among your connections, in the same row as everything else, with the count of what they captured and a link to their setup, so a bot you haven't linked yet never looks like a broken data source and a linked one is never asked to connect again. Your message is written to the record and acknowledged before the reply is composed, because Slack drops a delivery it hasn't heard back from in 3 seconds and eventually disables the subscription - a bot that answers slowly must never become a bot that stops receiving. Every inbound delivery is written down, refusals included, and the row says which failure you have: a bot nothing ever reaches (the platform stopped calling - check its Event Subscriptions) reads differently from one whose every message the app refuses (a rotated signing secret). Both used to look identical: an empty inbox.
Polling, not just push. Name a conversation under Settings → Channels → Also poll this channel (a channel name, a comma-separated list, or * for every conversation the bot is in, DMs included — because "my week of logs vanished" almost always means they were written somewhere other than the one channel the poll was aimed at) and agentqs checks it on its own schedule (hourly by default) as well as receiving webhooks - so the messages you sent while the webhook was down are still collected on the next sweep instead of being lost. Both paths key the capture on the platform's own message identity (Slack's channel + timestamp, Telegram's chat + message id), so a message that arrives twice lands once, and the bot's own replies are never captured as entries. The two directions are also told apart on the row: a poll that could not reach Slack says exactly that, instead of accusing Slack of sending a message the app refused, and a poll quietly collecting everything never certifies a webhook that has not fired once - that combination gets its own warning pointing at the app's Event Subscriptions. It is an ordinary due-source swept by the in-app scheduler on the same host as your record: no external CI minutes to run out, nothing to keep in sync with a second repo. agentqs sync --id slack runs it by hand. Email is the fourth channel, and the only one with no webhook: every email agentqs sends you ends with a short aqs# footer, your reply quotes it, and one Gmail search on that tag finds every reply to every notification - no labels, no inbound host, no tunnel. Replies are read only on the Gmail transport with Capture replies ticked (SMTP is send-only, by design); a reply counts only if it carries your instance's token, comes from an address agentqs has mailed and is not in Spam, and it lands under Gmail's own message id with the quoted original cut off (a heuristic: good for Gmail, Apple Mail and Outlook, not exact for every client). agentqs sync --id email runs it by hand; npm run email:test proves it.
Agent rules (Settings → Agent) message you the other way: when X, ping me. X is a time (an 8pm evening brief) or a data line crossing (resting HR over 55, social over 60 min today) - the threshold is a plain compare over your record, no AI. The message is a fixed line, or an AI brief the agent writes from your day. The in-app scheduler checks every 15 minutes, so an alert is as fresh as the source behind its metric. Set them in the app, from the CLI (agentqs rules add), or let the agent add its own via MCP. A plain daily notification (Settings → Channels, agentqs notifications add, or the notification_upsert MCP tool) is the same send with no trigger to think about: a fixed line like an 8pm "How was your day?", or --recap, a recap written from your record when it sends - to Slack, Telegram or your email.
Email (Settings → Channels → Email) gives agentqs a way to mail you: plain SMTP (send-only - Gmail app passwords, Fastmail, SES, Postmark), or your Google account through its own send-only grant that reuses the Google key you already saved, with read access added only if you tick Capture replies. For SMTP, port 587 is STARTTLS and 465 is TLS from the first byte; a server that offers neither is refused by design, not downgraded, because your password would cross in the clear - and for Gmail the password is an app password (2-Step Verification has to be on to create one), not your account password. Prove it with the Test button, agentqs mail test --to you@example.com, or npm run mail:live, which sends through your own server and reads every message back over IMAP (it skips cleanly when no credentials are set). Setup, replies, notifications and reset in full: docs/email-and-notifications.md. The same transport powers Forgot password? on the sign-in page: a single-use link, good for 30 minutes, that also signs out every existing session. No email set up? agentqs password --set on the machine resets it at a hidden prompt.
Connect by API or OAuth - GitHub · Whoop · Oura · Fitbit · Withings · Strava · RescueTime · Toggl Track · Todoist · WakaTime · YNAB · Readwise · Google (Calendar, Gmail) · Spotify · Last.fm · Deezer · Trakt · Notion · Swarm · Mastodon · Granola*
Google is one connection. One account, one key, and you tick what it imports - Calendar, and Gmail down to Inbox and Sent separately. The scope Google is asked for is only what you ticked, so if you never want Gmail you are never asked for your mail; tick it later and it re-authorizes the same key. Gmail counts, it never reads: message IDs only, landing emails_received and emails_sent per day. The products Google gives no API for - Search, Maps, YouTube, Gemini - come in through the Chrome extension under Automated imports instead. (Google Drive backup is not a source: backups send data out, and never appear in the pipeline.)
OAuth sources (Google, Spotify, Fitbit, Strava, Whoop, Withings, Trakt) connect with Authorize - register the redirect URI the form shows, paste client id + secret, approve. Every sync mints a fresh token from there on. Syncing lands only the rows it fetched: an import never re-reads your whole record, so a hosted instance stays responsive no matter how big the record gets.
Every source is read to the end. Each importer follows its API's pagination to the last page rather than taking the first one and stopping, asks about every year back to 2000 (a quiet stretch in your life is not the end of it), and refuses to write a partial answer as if it were the whole one. A source that lands nothing on its first import fails loudly instead of reporting success.
Connecting a source imports its history, not the last 90 days. The first sync of a source takes everything the API will give (years, not a window); every sync after it resumes from the last day you already have, so nothing is re-fetched and nothing is left behind. Slow is not a ceiling: Gmail is counted one day at a time, so its first import walks your whole account year by year and can run for minutes - it runs in the background and saves each year as it lands. Where an API has a hard ceiling the row says so instead of looking broken - Spotify only ever returns your last ~50 plays, so a sync covers a few days and your listening history comes from their account export. Drop that export in (or agentqs source file spotify --path my_spotify_data.zip) and it fills the same Spotify source, so one row shows the lifetime while the sync keeps the last few days fresh.
Every connection shows its receipts, and clicking one opens its data. A connected row carries a Connected badge, what actually landed (days, events, date range) and when it last synced - all counted from the record, so a connected-but-empty source reads "no data yet" instead of passing for a healthy one. Whoop rows also name which athlete they are authorized as, so two straps are never twins. Click the row and the Journal opens filtered to that source alone: "what did this give me?" is one click, never a guess.
"Connected" means a key is stored - nothing else. A CSV you dropped, a Takeout archive you unpacked, a history file read off your disk: those are Imported and Local file, and they say so. They are still yours to filter and remove - they just have no account behind them and nothing syncs them. Only a row with a real credential gets the Connected badge, so the list answers "what is actually live?" at a glance instead of showing you thirty green lights for files you dragged in once.
A source can start over without you signing in again. agentqs source reset <id> throws away everything a source landed - its daily file, its events, Whoop's per-minute heart rate - and keeps the connection: the key, the login, the schedule. The next sync then re-walks the source's whole history into a clean file. This is how you undo a bad import, and it is the only way: a sync merges, so it can correct a value it writes again but it can never remove a day it now knows better than to write at all. Re-syncing after a fix leaves the bad days exactly where they were; resetting first is what clears them. Removing the source instead (source remove) also takes the key, which is why cleaning up used to mean signing back in to everything.
* Granola has no API key - the credential is the desktop app's refresh token (workos_tokens.refresh_token in its supabase.json). Running agentqs on the same machine? It finds the login itself. Whoop connects two ways, and both ship: the official API (Authorize) for daily summaries, and the unofficial app login (email + password) - the only source of per-minute heart rate. Connecting Whoop imports its entire history, not a recent window - it finds where your data starts on its own, and later syncs resume from the last day you have (agentqs sync whoop --all-time re-runs a full backfill any time).
Scrape with the Chrome extension - the agentqs extension exports your entire Google MyActivity from a signed-in tab. Checkpointed: survives restarts and resumes on its own.
Schedule scraping with Playwright - for anything without an API: record the click-path to your data once, store the login, set an interval - it replays headless and lands like any other source. A Google MyActivity scraper ships out of the box.
Import files - drop anything: Google Takeout archives (calendar event titles land searchable) · Apple Health export.zip (steps, sleep, HR, workouts - lifetime, device-deduped) · Chrome and Safari browser history · Google Timeline · iPhone backups · Notion exports · Spotify data export · photos · PDFs (statements, invoices, papers - the text layer is extracted server-side and lands like any memo; a scanned PDF is refused by name with "no text layer", never skipped in silence, and there is no OCR) · any CSV, TSV, Markdown or text file. Point agentqs import at a whole folder and every file is accounted for - structured, landed raw, routed to its importer, or reported as residue in a saved receipt; nothing is silently skipped
Semantic search runs on local embeddings - no API key needed, and the model is multilingual, so a journal written in any language is searchable from a question asked in any other. Long documents are chunked, so a memo is found by any passage in it, not just its opening; a structured CSV is left to its daily cells rather than indexed twice. Chat and structuring use whatever provider you add: Anthropic, OpenAI, Gemini or any compatible endpoint.
Everything lives in one data directory you own, in three layers:
- The record - plain text, the source of truth.
record/daily/*.csvholds one row per day per source, numbers first - that's what Graphs and the Journal read.record/events.jsonlholds items - a meeting, a page visit, a track - one line each with title, text and link. Memos ininbox.jsonl, AI sessions insessions.jsonl. Per-minute streams (WHOOP heart rate) are one small CSV per day, rolled up into daily columns. - Derived indexes - rebuildable, never committed. A SQLite cache for SQL, a full-text index, on-device embeddings for semantic search, and a photo index.
agentqs rebuildrecreates all of them from the record, byte-identical. Nothing you click can trigger that rebuild: it re-reads the whole record, SQLite is synchronous, and one of those on a web request is not a slow page - it is every page in the app hanging at once. Every action instead patches only the rows it changed, and the app refuses (loudly, namingagentqs rebuild) rather than re-deriving your record behind a button. - The detail store (
detail.db) - every point behind the rollups. Streams too dense for one row per day - per-minute heart rate, every browser visit - live in a local SQLite as normal numeric tables, so chat andagentqs querycorrelate at full grain (detail.heart_rate).dailykeeps one value per day;detailkeeps them all.
Every new source follows the same rule: numbers go to daily columns; items with text (meetings, emails, messages) go to events, searchable by keyword and meaning; dense streams go to per-day files in the record and are indexed into detail, with a daily rollup; photos are indexed on-device. The same explainer lives in the app, behind the ? next to the Journal, Graphs and Pipeline titles.
Every action, no API key. Everything works from the terminal or any CLI agent:
agentqs chat "what changed this week?"
agentqs import ./export.csv --name mood
agentqs query "select date, value_num from daily where metric='mood'"No AI key? Use a CLI agent as the AI. Everything the app does is reachable key-free: capture, import, sync, query, semantic recall (local embeddings), rebuild - and for the two flows that normally need a model, the agent does the reasoning itself:
agentqs inbox --json # pending captures, full text
agentqs structure --id <id> --csv "date,mood,steps
2026-01-05,8,14200" # the agent supplies the extracted CSV
agentqs recall "days that felt burned out" # meaning-search, fully on-deviceExpose it over MCP - the agent reads, writes and queries through local tools. Register the server with any MCP client:
{ "command": "agentqs", "args": ["serve", "--mcp"] }Over HTTP, the same core. A running instance exposes every capability as a REST endpoint behind one bearer key (mint it in Settings → API). Nothing is CLI-only that an agent driving the API can't reach - the faces are thin, the brain is shared.
curl -s "$URL/api" # discovery manifest: every endpoint + its CLI/MCP twin, and the routing rules
curl -s "$URL/api/query" -H "authorization: Bearer $KEY" # schema + LIVE metric catalog (every metric, its source + span) + query recipes
curl -s "$URL/api/query" -H "authorization: Bearer $KEY" \
-d '{"sql":"select metric, count(*) from daily group by metric","limit":500}' # POST body = read-only SQL, the analytical doorGET /api is a machine-readable map an agent reads first, so it never guesses a route. POST /api/query is the analytical door - arbitrary read-only SQL over the record, run on a worker thread with a timeout so a heavy scan never freezes the instance; GET /api/query self-describes it (the metric names live in a column, not as columns, and it tells you so). The key-free reasoning flows work here too: POST /api/structure takes the CSV your agent extracted, POST /api/sessions records the insights and commitments it reasoned - the instance needs no model key of its own. POST /api/chat and POST /api/search (semantic recall) round out the natural-language side.
Keep your record outside the repo:
export AGENTQS_DATA_DIR="$HOME/agentqs-data"
npm run build
npm run devYour record is plain text (record/daily/*.csv, inbox.jsonl, sessions.jsonl) and can live in a private git repo. Databases and embeddings are derived - agentqs rebuild recreates them. By default the store lives in your platform's app-data folder, out of iCloud/Dropbox/OneDrive's reach - sync engines corrupt live stores. agentqs doctor checks yours; agentqs migrate-store moves an exposed one to safety.
The image is rebuilt and published on every push to main, and it is public, so there is no docker login and no build step. It supports linux/amd64 and linux/arm64; use the Quick start to run it directly. Already pinned to GHCR? ghcr.io/ami3466/agentqs:latest is the same image, mirrored digest-for-digest.
Version tags (:0.3.1, :0.3) and per-commit tags (:sha-abc1234) are there too, so a rollback is one tag away. The repo's docker-compose.yml uses the same image and passes provider keys through:
export SESSION_SECRET=$(openssl rand -hex 32)
docker compose pull && docker compose up -dAdd --build to up to build from your working tree instead of pulling the published image.
Optional env: ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_GENERATIVE_AI_API_KEY for chat and structuring. Local file sources (Chrome history, iPhone backups) only work when the container runs on the machine that holds the files - otherwise run the daemon there (npm run daemon -- run --push) and let the instance pull the record via git.
While the app is running it schedules its own syncs - set a source's interval and it happens, no setup. For times the app isn't open, one crontab line covers it: 0 * * * * agentqs sync --due runs every source whose interval says it's due, browser scrapes included (copy it from Settings → API → Connect). agentqs pipeline shows the whole truth in one table: where each source's data comes from, credential provenance, what's scheduled, and whether the last run actually worked.
Deploy on FlowEngine - up 24/7, persistent storage at /data, syncs run around the clock.
Any other host: mount storage at /data, set AGENTQS_DATA_DIR=/data, keep a stable SESSION_SECRET. Cloud can't read your laptop - for browser history, iPhone backups or photos, run the CLI on the machine that has the files.
- Private by default. Data stays in your data directory. Model calls only get what you ask to send.
- Off-site backups built in.
agentqs backup github --remote <url>pushes a snapshot of your plain-text record to a private repo (files past GitHub's size limit are excluded and named, never silently dropped), andagentqs backup driveuploads the whole store as one AES-256-GCM-encrypted archive to your Google Drive on a schedule (--schedule daily|off, or the switches in Settings → Data) -agentqs backup restorebrings either back. Backups are where your data is kept, not where it comes from: Google Drive here is a destination, so it never shows up among your data sources. - Raw files on request, from Drive. For raw content the record deliberately never stores (emails, whole message threads, PDFs), keep it in a Google Drive folder and point agentqs at it in Settings → Data → Drive import.
agentqs drive listshows what's there andagentqs drive pull <file>reads one file's text on demand (a PDF comes back as its extracted text, not a stub) - nothing is synced into your record, and the chat agent pulls a file only when a question needs it. Read-only access, a separate grant from the backup one. - Token use is explicit. Capture and search are local. AI runs only when you chat, structure prose, or run a channel - CLI-agent workflows skip it entirely. The local models those searches use (the text embedder, photo search and captioning, built-in Whisper) load on demand and are unloaded again after five minutes idle, so a single search or captioned photo doesn't leave hundreds of MB parked for the life of the process -
AGENTQS_MODEL_IDLE_MStunes the window. - Nothing is scheduled outside the app. There is no crontab, no launchd line and no CI job driving your data: the running server sweeps every 15 minutes for due syncs, scheduled notifications and agent rules (
AGENTQS_NO_SCHEDULER=1turns it off;agentqs sync --dueis the headless equivalent). GitHub Actions only builds the container image. - Tabs remember what they loaded. Switching between Journal, Graphs and Pipeline paints the last answer immediately and refreshes behind it, and each tab shows its own shape while a first load runs instead of a bare "Loading…". A capture patches the derived cache in place rather than rebuilding it, so logging a memo costs milliseconds on a record of any size.
- No action can hang the app. Structuring, keeping and discarding all cost work proportional to what you changed, not to the size of your record, and they run as background jobs on one serial queue - so a slow merge answers with a job to follow instead of holding the connection, and closing the tab no longer kills it.
npm run freeze:testproves it against a seeded 200,000-event record: structuring an item takes about 50ms where re-deriving that record takes seconds.
MIT with the Commons Clause - free to use, modify and self-host, including inside a company; not for resale.



