Grimmory
Annotation & Note Sidecar Files (KOReader-compatible) #1237
Replies: 6 comments
-
🔗 Related PRs#16 - fix(api): Use fixed-layout epub format in Kobo sync when necessary (Booklore PR Port) [merged] 📝 Issue PlannerCheck the box below or use the
🧪 Issue enrichment is currently in open beta.You can configure auto-planning by selecting labels in the issue_enrichment configuration. To disable automatic issue enrichment, add the following to your issue_enrichment:
auto_enrich:
enabled: false💬 Have feedback or questions? Drop into our discord! |
Beta Was this translation helpful? Give feedback.
-
|
Just a few days ago, I've started investigating the possibility of adding annotation sync with Kobo to Grimmory. How would that fit in here? I think your idea is pretty good, but I'd like to add the possibility for two-way sync of annotations between Kobo and Grimmory, with the native firmware supplied by Kobo. |
Beta Was this translation helpful? Give feedback.
-
|
@JVT038 I investigated that as well. We could create a native Kobo sync endpoint in Grimmory as well as the KOReader one. Calibre-web did it in python, Komga has it in Kotlin, Kavita has it in C#. It would likely need to be re-implemented in Java unless I'm missing something, but the structure is known well. I use KOReader and the Kobo, so I could definitely help test this part or do the development as well. It's just a bit bigger than what I'm proposing, but I definitely think that it would be a worthwhile addition. I would use it. |
Beta Was this translation helpful? Give feedback.
-
|
I have logged the network traffic on my Kobo ereader, and extracted this relevant logs from it: Network traffic for annotations
----------------------
--- ADD ANNOTATIONS---
----------------------
As you can see, the annotation location is using Kobo's span notation, so I'm hoping that #253 can provide us some code to make it easier to convert the location formats between each other (e.g. koreader to kobo, to Grimmory, to whatever). |
Beta Was this translation helpful? Give feedback.
-
|
@chernesk Thanks for consolidating this info into the issue- I think a spike it a great idea to help us identify what the scope of this would look like, and so we can agree on some specific technical details prior to implementation. As you mentioned, your goal is to reduce lock in, and to better surface annotations/highlights/notes/bookmarks across platforms. Question- you mentioned having the database effectively be a "queryable cache"- in your proposal, we would still have the DB act as the source of truth, correct? Is it correct to assume in this proposal that Sidecar annotation data then becomes an optional form of data resilience users can enable (disabled by default), and like book sidecar data, imports/exports can be run at any given time? Curious to see a spike, and looking forward to hearing your thoughts on:
This is very exciting work, and we appreciate you putting your time and thought towards helping improve this project for the community :) |
Beta Was this translation helpful? Give feedback.
-
|
"DB as source of truth / sidecar as optional resilience" Yes, exactly our model. Mirrors how the existing metadata sidecar works: disabled by default, user-enabled, can be imported/exported on demand. The DB is the queryable cache the app reads from at runtime. The sidecar is what survives a container wipe. "User identifiers — filename or file content?" Both, for different reasons:
I'd propose a top-level booklore_meta header table in the file: -- KOReader bookmark file
-- version: 1
-- booklore:schema_version=1
{
["booklore_meta"] = {
["username"] = "alice",
["user_id"] = 7,
["book_id"] = 42,
["exported_at"] = "2024-01-15 10:30:00",
["schema_version"] = 1,
},
["highlights"] = {
[1] = {
-- KOReader fields (used by KOReader, ignored otherwise)
["pos0"] = "...",
["pos1"] = "...",
["drawer"] = "lighten",
["notes"] = "the highlighted text",
-- Grimmory extension (ignored by KOReader)
["booklore"] = {
["id"] = 42,
["color"] = "#FFFF00",
["style"] = "highlight",
["note"] = "My personal comment",
["version"] = 3,
},
},
},
["bookmarks"] = {},
}KOReader ignores both booklore_meta and per-entry booklore keys. The schema version in the comment line (--booklore:schema_version=1) lets a future parser fast-fail on a format it doesn't understand without parsing the Lua. "What structure — any open specs to draw from?" The base format is KOReader's own .sdr/metadata.epub.lua — it's the closest thing to an open community standard for EPUB annotation sidecars, used by Calibre plugins, Obsidian importers, and a handful of other tools. Building on it means maximum transferability outside of Grimmory. For the booklore extension namespace, there isn't a formal spec to draw from, but the pattern of adding vendor namespaces to an extensible format is well-established (same pattern as EPUB's custom metadata, OPDS extensions, etc.). The schema_version field means we can evolve it without breaking imports. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
What's Your Idea?
Store highlights and notes as KOReader-compatible sidecar files (
.sdr/metadata.*.lua) alongside each book in the library, in addition to the database. The database becomes a queryable cache that can be rebuilt from the sidecar files, rather than the sole copy of annotation data.Why Would This Be Helpful?
Right now, highlights and notes only exist in the database. If the database is lost or corrupted, all annotations are permanently gone with no way to recover them from the book files themselves.
This also makes cross-device reading painful. A user reading across the web reader, a Kobo running KOReader or just a user using KOReader, and a mobile browser has no way to get highlights from one surface to another — reading progress syncs via the existing KOReader endpoint, but highlights and notes do not travel at all.
There are two complementary improvements that together solve this:
1. Sidecar files alongside books
Write annotations to KOReader's
.sdr/metadata.*.luaformat alongside each EPUB in the library. This gives:.sdrfolder anywhere and highlights come with it2. Extend KOReader Sync endpoints to include annotations
This path applies to Kobo users who have KOReader installed on their device. The existing
/api/koreaderendpoints handle reading progress. AddingGETandPUTendpoints for annotations at the same base path would give users who already have KOReader Sync pointed at Grimmory full bidirectional highlight sync with no new configuration on their end — the same credentials and server URL they already use.This is the only path for KOReader → Grimmory sync in the typical setup where the Kobo connects to a remote Grimmory server. OPDS is download-only and cannot push highlights back.
Users running the native Kobo firmware (without KOReader) would only benefit from the sidecar path above — highlights made in the native Kobo reader would not sync back to Grimmory.
Anything Else? (Optional)
The main design decision would be conflict resolution when annotations diverge between the sidecar and the database (e.g. a highlight made on the Kobo while offline, and a different highlight made in the web reader on the same passage). Last-write-wins is probably the pragmatic starting point.
Want to Help Out?
Yes! I'd love to help implement this
Have You Considered Any Alternatives? (Optional)
Calibre-web was the obvious comparison point but it stores annotations in its own SQLite database with the same fragility. KOReader's sidecar approach is the only self-hosted solution that actually keeps annotations with the book file.
Before Submitting
Beta Was this translation helpful? Give feedback.
All reactions