Skip to content

backup-export.md

Latest commit

 

History

History
139 lines (107 loc) · 4.63 KB

backup-export.md

File metadata and controls

139 lines (107 loc) · 4.63 KB

Backup export: memex → Obsidian vault

Per DESIGN.md §7.4, the Obsidian vault is the canonical long-term store and memex is the capture-side surface. scripts/memex-export.sh is a one-way mirror: it pulls every capture from the Worker and writes one markdown file per doc into ${MEMEX_VAULT_DIR}/captures/. Vault edits never flow back; the next export run is authoritative.

This is a local-side job (the target is your local filesystem), so it runs as a systemd user timer — not a Cloudflare Cron Trigger.

What it produces

$MEMEX_VAULT_DIR/
└── captures/
    ├── 1a2b3c4d-….md
    └── …

Each file:

---
id: 1a2b3c4d-…
created_at: 2026-04-12T19:03:11Z
updated_at: 2026-05-08T22:14:02Z
updated_at_ms: 1746742442000
content_hash: 9f86d081…
source: "voice"
summary: "Why captures-only is the right v1 stance for the second brain."
tags:
  - "second-brain"
  - "design"
---

Raw markdown body of the capture goes here…

Filename is the doc UUID: stable, collision-free, decoupled from titles.

Idempotency

Re-running is cheap and safe.

  1. The script paginates GET /thoughts?limit=100&before=<cursor> and gets back (id, created_at, updated_at) for every doc.
  2. For each doc, it reads updated_at_ms from the existing file's frontmatter.
    • If the file's updated_at_ms is ≥ upstream, skip (no HTTP GET).
    • Otherwise fetch GET /thought/:id, recompute the SHA-256 of content, and compare to the stored content_hash.
      • Hash match → only the updated_at_ms line is touched.
      • Hash differs → rewrite the file.

Net effect: nightly runs after the corpus is caught up are O(list), one cheap paginated read of doc summaries. New / changed docs are the only ones that incur a per-doc fetch.

Required env

Variable Purpose
MEMEX_URL e.g. https://serverless-memex.<account>.workers.dev
MEMEX_CLIENT_ID Cloudflare Access service token client ID
MEMEX_CLIENT_SECRET Cloudflare Access service token client secret
MEMEX_VAULT_DIR Absolute path to your Obsidian vault root (the captures/ subdir is created).

The first three are already in ~/.secrets. Add MEMEX_VAULT_DIR there too:

export MEMEX_VAULT_DIR="$HOME/Documents/ObsidianVault"

Manual run

# one-time: mark the script executable after first checkout
chmod +x scripts/memex-export.sh

# from a shell that has ~/.secrets sourced
./scripts/memex-export.sh             # full sync
./scripts/memex-export.sh --dry-run   # report-only; no writes

Verify:

ls "$MEMEX_VAULT_DIR/captures" | wc -l
# Spot-check a file's frontmatter:
head -15 "$MEMEX_VAULT_DIR/captures/"*.md | less

Install as a daily systemd user timer

systemd (EnvironmentFile=) wants plain KEY=VALUE lines without export, but ~/.secrets uses export. Easiest path: keep a stripped copy at ~/.config/memex/env.

mkdir -p ~/.config/memex
# Generate the env file from ~/.secrets (drop `export`, keep only memex vars):
grep -E '^export MEMEX_' ~/.secrets | sed 's/^export //' > ~/.config/memex/env
chmod 600 ~/.config/memex/env

mkdir -p ~/.config/systemd/user
cp scripts/systemd/memex-export.service ~/.config/systemd/user/
cp scripts/systemd/memex-export.timer   ~/.config/systemd/user/

systemctl --user daemon-reload
systemctl --user enable --now memex-export.timer

Confirm it's armed:

systemctl --user list-timers --all | grep memex
systemctl --user status memex-export.timer

Trigger an on-demand run (useful for the first sync):

systemctl --user start memex-export.service
journalctl --user -u memex-export.service -n 200 --no-pager

Operational notes

  • The script is read-only against the API. It does not touch the AI / embedding path and adds no new Cloudflare bindings.
  • The before query param on GET /thoughts is an additive cursor for this job; for normal "recent" reads, omit it and use limit alone.
  • The vault side is treated as a mirror, not a working tree. If you delete a capture in memex, the corresponding captures/<id>.md will not be removed on next run (intentional — keeps deletions reversible by hand). Prune manually if/when that becomes a problem.
  • Conflict with vault edits: this script overwrites captures/<id>.md when upstream changes. Don't hand-edit files under captures/ — promote to a sibling folder first.