Skip to content

Commit aea69dc

Browse files
committed
Merge branch 'feat/semantic-vault'
2 parents 32e8e7f + 78fc535 commit aea69dc

63 files changed

Lines changed: 7200 additions & 77 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/cerbo-cli/SKILL.md‎

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
name: cerbo-cli
3-
description: "Use when the user wants to work with a Cerbo vault from the terminal: create or register a vault, create/read/write/delete pages, link and annotate pages, find backlinks, query the knowledge graph with SPARQL, search pages by meaning, rebuild the index, import a URL or an ontology, build the symlink tree, or run a vocabulary migration. Examples: \"Create a page about X\", \"Which pages link here?\", \"Find notes about Berlin\", \"Reindex the vault\""
3+
description: "Use when the user wants to work with a Cerbo vault from the terminal: create or register a vault, create/read/write/delete pages, link and annotate pages, find backlinks, query the knowledge graph with SPARQL, search pages by meaning, rebuild the index, import a URL, an ontology or an Obsidian/Tolaria vault, build the symlink tree, or run a vocabulary migration. Examples: \"Create a page about X\", \"Import my Obsidian vault\", \"Which pages link here?\", \"Find notes about Berlin\", \"Reindex the vault\""
44
---
55

66
# Cerbo CLI
@@ -55,6 +55,7 @@ cwd is auto-registered silently and shows as `(auto)` in `cerbo vault list`.
5555
| `cerbo info` | Config dir, cache dir, registered vaults |
5656
| `cerbo import <URL>` | A read-only Source object (`schema:WebPage`) for the URL |
5757
| `cerbo import-ontology <URL> [--prefix p --namespace IRI]` | An Ontology object; with both flags, registers prefix `p` for annotations |
58+
| `cerbo import-vault <DIR> [--from obsidian\|tolaria] [--vault ID \| --new PATH [--name N]] [--force] [--dry-run]` | Import an Obsidian or Tolaria vault as pages; the source folder is only read |
5859
| `cerbo index [--vault ID\|PATH] [--page UUID] [--embeddings]` | Rebuild `annotations.ttl` and the semantic index, report broken links; `--embeddings` also computes vectors for search |
5960
| `cerbo sparql '<QUERY>'` / `--file q.rq` / `-` | Read-only SPARQL 1.1 against the vault's graph |
6061
| `cerbo search "<TEXT>"` | Pages closest in meaning to the text, any language |
@@ -160,6 +161,36 @@ Built-in prefixes (`schema`, `foaf`, `dcterms`, `skos`, `rdf`, `rdfs`, `owl`,
160161
`xsd`) need no registration. An unknown prefix is kept as
161162
`cerbo:unresolvedType` with a warning, not an error.
162163

164+
### Import an Obsidian or Tolaria vault
165+
166+
```bash
167+
cerbo import-vault ~/Notes --dry-run # report only: counts, unresolved links, skipped files
168+
cerbo import-vault ~/Notes --vault <ID> --json # into a registered vault
169+
cerbo import-vault ~/Sync/Tolaria/Braindump --new ~/cerbo/braindump --name Braindump
170+
```
171+
172+
- **Always pass `--vault <ID>` or `--new <PATH>`.** Without them the command
173+
asks on a terminal and fails otherwise ("no target vault: pass --vault <ID>
174+
or --new <PATH>"); `--vault` and `--new` are exclusive. `--new` needs a
175+
missing or empty folder (`.git` alone is fine) and is created only once the
176+
source has been read.
177+
- Format: `.obsidian/` means Obsidian; `views/*.yml` or `type:` frontmatter
178+
means Tolaria; otherwise pass `--from obsidian|tolaria`.
179+
- Notes become pages (folder → virtual path, file stem → slug, title from the
180+
first H1 for Tolaria or the file name for Obsidian). `[[Note|text]]`,
181+
`![[file]]` and relative Markdown links become `cerbo://<uuid>` links;
182+
referenced files become attachments; links inside code are untouched;
183+
unresolved links stay as written and are reported.
184+
- Frontmatter becomes RDF in the import's namespace `cerbo://vocab/<uuid>/`
185+
(`type: X` → extra `rdf:type`, `[[wikilink]]` values → relations, others →
186+
typed literals, `_`-keys dropped). Its prefix is the source folder name,
187+
slugified (`braindump`, `braindump-2`), registered in the vault's prefix map.
188+
- One import per source folder and vault: a repeat is refused, `--force` makes
189+
a second copy. The `--json` report has `vault`, `format`, `pages`,
190+
`attachments`, `relations`, `skipped`, `unreferenced`, `unresolved`,
191+
`ambiguous`, `renamedSlugs`, `frontmatterErrors`, `warnings`.
192+
- Backlinks and SPARQL see the imported pages right away; no `cerbo index`.
193+
163194
### Maintenance
164195

165196
```bash
@@ -184,6 +215,9 @@ A `cerbo index` run that exits 0 has fully rebuilt the vault's metadata.
184215
| Broken link warning on `page write` | The target UUID does not exist; the page was still saved |
185216
| `cerbo symlink` skips pages | They have no `cerbo:slug`; run `cerbo index` first |
186217
| `Cannot specify both --vault and --page` | Pick one for `cerbo index` |
218+
| `no target vault: pass --vault <ID> or --new <PATH>` | `import-vault` without a terminal; pass a target |
219+
| `cannot detect vault format` | Pass `--from obsidian` or `--from tolaria` to `import-vault` |
220+
| `… was already imported into this vault on <date>` | Expected; `--force` imports a second copy |
187221

188222
Changing the CLI? Update `cli/man/cerbo.md` in the same change (`cerbo.1` is
189223
generated at build time and not committed).

‎Cargo.lock‎

Lines changed: 117 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎cli/Cargo.toml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,4 +16,8 @@ tokio = { version = "1", features = ["full"] }
1616
cerbo-core = { path = "../core", features = ["test-utils"] }
1717
tempfile = "3"
1818

19+
[target.'cfg(unix)'.dev-dependencies]
20+
# Opens a pseudo-terminal for the interactive import-vault test.
21+
libc = "0.2"
22+
1923
[build-dependencies]

‎cli/man/cerbo.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,72 @@ Options:
148148
- `--namespace <IRI>` - Namespace IRI the prefix expands to, e.g. `https://example.org/vocab#` (requires `--prefix`)
149149
- `--json` - Output result as JSON with uuid, url, prefix and namespace fields
150150

151+
## import-vault
152+
153+
Import an Obsidian or Tolaria vault (a folder of Markdown notes with YAML frontmatter and `[[wikilinks]]`) into a Cerbo vault. Every note becomes a page, links become `cerbo://` links, referenced files become Attachment objects, and frontmatter becomes RDF in `meta.ttl`. The source folder is only read: no file under it is created, changed, moved or deleted.
154+
155+
Usage: `cerbo import-vault <DIR> [--from obsidian|tolaria] [--vault <ID> | --new <PATH> [--name <NAME>]] [--force] [--dry-run] [--json]`
156+
157+
Arguments:
158+
- `DIR` - The folder to import (the Obsidian or Tolaria vault)
159+
160+
Options:
161+
- `--from <FORMAT>` - Source format, `obsidian` or `tolaria`; overrides detection
162+
- `--vault <ID>` - Import into this registered vault (an ID from `cerbo vault list`). An unknown ID fails with "Vault not found" and writes nothing.
163+
- `--new <PATH>` - Create a vault in `PATH` (missing, or empty but for `.git`), initialise it as `cerbo init` does, register it, and import into it. A folder holding other files is refused. The vault is created only after the source has been read and planned, so a source that cannot be imported leaves no empty vault behind.
164+
- `--name <NAME>` - Name of the vault created by `--new` (default: the source folder name). Needs `--new`.
165+
- `--force` - Import a folder that was already imported into this vault: makes a second, separate copy with its own ontology and prefix
166+
- `--dry-run` - Read the source and print the report without asking for a vault or writing anything (no vault, no registry file)
167+
- `--json` - Print only the report, as one JSON object (see below). An error prints `{"error": true, "message": ...}`.
168+
169+
`--vault` and `--new` cannot be combined: that is a usage error (exit status 2).
170+
171+
**Target vault:** without `--vault` or `--new`, on a terminal, the command lists the registered vaults by number (name and path) plus "Create a new vault" on stderr and reads the answer from stdin; "Create a new vault" then asks for a folder (`~/` is expanded) and a name. When stdin is not a terminal the command fails with "no target vault: pass --vault <ID> or --new <PATH>" and writes nothing.
172+
173+
**Format detection:** a folder with `.obsidian/` is Obsidian. A folder without it that has a `views/` folder of `.yml` files, or a Markdown note with a `type:` frontmatter field, is Tolaria. Anything else fails with "cannot detect vault format, pass --from obsidian|tolaria". A folder holding `.cerbo/` is a Cerbo vault and is refused, also with `--from`. The report says which format was used and why.
174+
175+
**Mapping:**
176+
- Note (`*.md`) - a page (`rdf:type schema:NoteDigitalDocument`); `page.md` holds the body without its frontmatter
177+
- Title - Tolaria: the first H1, else the file stem; Obsidian: the file stem. `# <title>` is prepended when the body does not start with it.
178+
- Folder - `cerbo:virtualPath` (the page's folder in the sidebar and symlink tree)
179+
- File stem - `cerbo:slug`, transliterated to ASCII; a second note in one folder with the same slug gets `-2`, then `-3`, and is listed under renamed slugs
180+
- File times - `schema:dateCreated` (birth time, else modification time) and `schema:dateModified`
181+
- `[[Note]]`, `[[Note|text]]`, `[[Note#Heading]]`, `![[Note]]` - `[text](cerbo://<uuid>)`, where text is the `|text` part, else the target's title; the `#Heading` is dropped and counted. A target matches a note by relative path, then file stem, then title, then Obsidian alias, ignoring case; with several matches the one nearest the linking note's folder wins and the link is listed as ambiguous.
182+
- `[text](other.md)`, `[text](../other%20note.md)` - `[text](cerbo://<uuid>)`; external URLs are left alone
183+
- `![[photo.png]]`, `![alt](path)`, `[text](path)` to a local file - one Attachment object per file, however many notes use it, linked as `![alt](cerbo://<uuid>)` or `[text](cerbo://<uuid>)`. Files are matched by relative path, then by file name anywhere in the vault. Files no note references are listed, not imported.
184+
- Links inside code blocks and inline code - left unchanged
185+
- Unresolved links - kept as written and listed in the report
186+
- Frontmatter `type: X` (Tolaria) - a second `rdf:type <ns>X`; a note with `type: Type` declares the class `X` in the import's ontology, labelled with its title
187+
- Frontmatter values holding `[[wikilinks]]` - one `<ns><key> <cerbo://objects/<uuid>>` relation per resolved link; an unresolved one is kept as a string and listed
188+
- Other frontmatter values - literals `<ns><key>`: strings, booleans, integers, ISO dates and date-times with their XSD type, one triple per list item, nested maps as a YAML string. Keys are percent-encoded where they are not IRI-safe (`Belongs to` - `Belongs%20to`).
189+
- Keys starting with `_` (Tolaria app state such as `_favorite`) - dropped
190+
- Obsidian `tags` / `aliases` - `schema:keywords` (without `#`) / `schema:alternateName`; aliases also resolve wikilinks. Inline `#tags` stay text.
191+
- Frontmatter that cannot be parsed - kept at the top of `page.md` in a `yaml` code block and listed in the report
192+
- Skipped and listed: Obsidian `.canvas` and `.excalidraw.md` files, Tolaria `views/`, symbolic links. Left out silently: `.obsidian/`, `.git/`, `.trash/` and every other hidden file or folder.
193+
194+
`<ns>` is the import's own namespace `cerbo://vocab/<uuid>/`, owned by one Ontology object titled `<Format>: <source folder name>`. That object records the source as a `file://` URL (`schema:url`), the format (`cerbo:importFormat`) and the import time, and its `meta.ttl` declares every class and property the import used. Its prefix, the source folder name slugified (`braindump`, or `braindump-2` when taken), is registered in `.cerbo/ontology-map.json`, so annotations can write `[Text]{braindump:Journal}`.
195+
196+
**One import per source:** importing a folder into a vault that already holds an import of the same folder (same canonical path, in any format) fails before anything is written, naming the earlier import's date and format. `--force` imports a second copy. Updating an earlier import is not supported.
197+
198+
**Target outside the source:** the target vault (from `--vault`, `--new` or the prompt) may not be the source folder or lie inside it, since the source is never written; such a target is refused before anything is created. Only one import runs into a vault at a time; a second one started meanwhile is refused.
199+
200+
**All or nothing:** objects are written to `.cerbo/import-staging/` first and then moved into `.cerbo/objects/`; the prefix map is updated last. A failed import leaves the vault's pages and prefix map as they were; should some moved objects not go back to the trash, the next import into that vault finishes the cleanup. If an import was interrupted while moving, the next `cerbo import-vault` into that vault moves its objects to the trash first and reports how many. After the import the semantic index is brought up to date, so `cerbo page backlinks` and `cerbo sparql` see the new pages without `cerbo index`; should that fail, the import still succeeds and the report carries a warning.
201+
202+
**Report:** the human report says `Imported N pages and M attachments (size) into "<vault>" (<id>)` (`Dry run: would import ...` with `--dry-run`), the source and format, the ontology and prefix, the counts of rewritten links, relations and dropped heading fragments, then the lists that need attention: skipped files, unreferenced files, unresolved links (with the note and, for frontmatter, the key), ambiguous links (with the chosen target and the candidates), renamed slugs and frontmatter errors. Warnings go to stderr. With `--json` the report is one JSON object with the fields `vault` (`{id, name, path}`; `null` on a dry run), `source`, `format`, `formatReason`, `dryRun`, `pages`, `attachments`, `attachmentBytes`, `relations`, `links`, `droppedFragments`, `ontology` (`{uuid, title, namespace, prefix}`), `skipped`, `unreferenced`, `unresolved` (`[{note, link, property}]`), `ambiguous` (`[{note, link, chosen, candidates}]`), `renamedSlugs` (`[{note, from, to}]`), `frontmatterErrors` (`[{note, error}]`), `recovered` and `warnings`.
203+
204+
Examples:
205+
206+
```
207+
# See what an Obsidian vault would become, without writing anything
208+
cerbo import-vault ~/Notes --dry-run
209+
210+
# Import it into a registered vault
211+
cerbo import-vault ~/Notes --vault 3f2a
212+
213+
# Import a Tolaria vault into a new vault named Braindump
214+
cerbo import-vault ~/Sync/Tolaria/Braindump --from tolaria --new ~/cerbo/braindump --name Braindump --json
215+
```
216+
151217
## index
152218

153219
Rebuild metadata (annotations.ttl) from page content, report broken links, and bring the semantic index up to date. Used for recovery after manual edits or bulk imports. Backlinks are not files; see `cerbo page backlinks` and SEMANTIC INDEX.
@@ -409,6 +475,17 @@ Import an ontology and register its annotation prefix:
409475
cerbo import-ontology https://example.org/vocab.ttl --prefix ex --namespace https://example.org/vocab#
410476
```
411477

478+
Preview, then import an Obsidian vault into a registered vault:
479+
```
480+
cerbo import-vault ~/Notes --dry-run
481+
cerbo import-vault ~/Notes --vault 3f2a
482+
```
483+
484+
Import a Tolaria vault into a new vault:
485+
```
486+
cerbo import-vault ~/Sync/Tolaria/Braindump --new ~/cerbo/braindump --name Braindump
487+
```
488+
412489
Preview, then run, the vocabulary migration of one vault:
413490
```
414491
cerbo migrate vocabulary --vault /path/to/my/vault --dry-run

0 commit comments

Comments
 (0)