Add window.nostrdb NIP

[hzrd149]

Adds the NIP-DB from nostr-bucket and window.nostrdb.js to this repo

[vitorpamplona]

Nice!

Should it specify more behaviors to make sure every DB implementation delivers the exact same results? For instance:

• Does it expire events automatically? Does it send updates on subscriptions when things expire?
• Does it delete on kind 5 events? Does it update subscriptions when it deletes events?
• Does it delete on kind 62 events? If so, devs should use a unique "relay name" for each instance of the db?
• Is it expected to support NIP-17 kind 5 and 62 deletions by p-tags?
• What is the order of the return for filters and subscriptions?
• Does the add return false when inserting an old replaceable the db already has a newer version for?
• Does the add return false if the event has already expired?
• Does the add return false if there is a kind 5 or kind 62 event for that event?
• Are ephemeral events stored if sent to the DB?
• Is a streamhandler complete equal to an eose?
• What happens when the DB is out of space?

A few extra API calls that I use quite a lot on Amethyst:

/** Delete events by filters */
delete(filters: Filter[]): Promise<boolean>;

/** Count events by filters in a subscription */
count(filters: Filter[], handlers: StreamHandlers): Subscription;

/** Deletes expired events if the DB doesn't have a process to run this periodically */
deleteExpiredEvents(): Promise<boolean>;

On subscriptions, this could complicate things a bit, but it simplifies so much for the user of the lib that is may make sense to be considered here too. We also use an all-in-one update, like

type StreamHandlers = {
  events?: (events: NostrEvent[]) => void;
  ...
};

This makes it easier to maintain a "feed" without having to re-code by-address updates (find the old, remove the old, and insert the new in the order it needs to be) when they arrive later in the stream, which is quite a common occurrence.

In fact, we do two types of "bundled" returns, a regular one with simply Nostr Events directly and a Wrapped version that allows the UI to listen to changes to specific events and only update that line on the feed, without redrawing the entire list.

type StreamHandlers = {
  events?: (events: Observable<NostrEvent>[]) => void;
  ...
};

In this case, the stream handler is only called if there are additions or deletions to the list, ignoring addressable updates in each item. In that way, the UI doesn't need to figure out what changed in the list to only update one line. It can just listen to updates in that line specifically.

Finally, these two calls are largely unnecessary because they can be done with filters directly:

/** Get a single event by ID */
event(id: string): Promise<NostrEvent | undefined>;

/** Get the latest version of a replaceable event */
replaceable(
  kind: number,
  author: string,
  identifier?: string,
): Promise<NostrEvent | undefined>;

[alexgleason]

The interface used here is like a weird bastardization of the Nostrify store interface. Why not use the Nostrify one?

/** Nostr event store. */
export interface NStore {
  /** Add an event to the store (equivalent of `EVENT` verb). */
  event(event: NostrEvent): Promise<void>;
  /** Get an array of events matching filters. */
  query(filters: NostrFilter[]): Promise<NostrEvent[]>;
  /** Get the number of events matching filters (equivalent of `COUNT` verb). */
  count(filters: NostrFilter[]): Promise<NostrRelayCOUNT[2]>;
  /** Remove events from the store. This action is temporary, unless a kind `5` deletion is issued. */
  remove(filters: NostrFilter[]): Promise<void>;
}

[hzrd149]

Should it specify more behaviors to make sure every DB implementation delivers the exact same results? For instance:

The NIP may need more clarification but the general idea is for a browser extension ( or browser ) to provide access to the local event store on the system. so something similar to nostr-relay-tray but without the web app needing to test ws://localhost:4869

So the window.nostrdb API doesn't need to be and probably shouldn't be too extensive because it should be compatible with many types of backends.

• Does the add return false when inserting an old replaceable the db already has a newer version for?
• Does the add return false if the event has already expired?
• Does the add return false if there is a kind 5 or kind 62 event for that event?

These things should be clarified, since it would help to know exactly what the return value of the add meant

Is a streamhandler complete equal to an eose?

That's the idea, but I'm not sure how useful that would be vs complete meaning "closed from database side"

A few extra API calls that I use quite a lot on Amethyst:

A count and delete methods should probably be added, although its important to point out that the app shouldn't expect to have full control over the database. i.e. apps should not try to clear the whole event store via delete({})

I kind of like the idea of adding a events callback to the subscription, I think it would work well with most database implementations and as you said could help the app avoid having to de-duplicate replaceable events

Finally, these two calls are largely unnecessary because they can be done with filters directly:

The event and replaceable methods are there for performance and convenience. they can be done with filters but most database implementations have optimizations for fetching by id, or by replaceable address. also very convenient for the client to have await window.nostrdb.replaceable(0, <pubkey>)

The interface used here is like a weird bastardization of the Nostrify store interface. Why not use the Nostrify one?

I like the name query vs filters and remove vs delete. but the event method should retrieve an event and not add one to the store

@alexgleason @vitorpamplona Finally Id like to get your thoughts on the supports() method and signaling support for things like NIP-50 search and the live subscription method. I extended the interface a little in window.nostrdb.js with a lookup method that allows clients to resolve user searches to pubkeys https://github.com/hzrd149/window.nostrdb.js/blob/master/src/interface.ts#L80 but I'm not sure if that would really belong here

[staab]

This is great, one thing that is worth considering is soft delete. In flotilla, in order to avoid reflow I like to show deleted events with a "deleted" badge until the page is reloaded to avoid things just disappearing for no reason. Maybe that's not an issue if the client buffers events it has already retrieved and detects deletes that way, but something to consider.

Same thing for subscribe, it would be nice to subscribe to deletions as well.

Welshman's repository does both of these things — deleted events are retained and queryable via an opt-in flag, and updates are subscribable with both added and removed data.

[fiatjaf]

Should subscribe() return only events received after the subscription started?

[vitorpamplona]

@alexgleason @vitorpamplona Finally Id like to get your thoughts on the supports() method

I hate optional things in specs. I'd be fine with either forcing or removing search/subscriptions from the nip entirely.

Though "dbs" and "reactive dbs" are VERY different concepts. We cannot build a DB that receives reactivity (subscribe) from something that listens to it because there is nothing to listen to. That plugin has to recode many behaviors that are already coded in the DB itself (when to replace events, when to delete/expire, and what happens when those things happen) and thus run the risk of being out of sync with each other.

So, to me, I think all dbs MUST be reactive dbs by definition.

[hzrd149]

Should subscribe() return only events received after the subscription started?

The client can set a since field on the filter to only get new events

I hate optional things in specs. I'd be fine with either forcing or removing search/subscriptions from the nip entirely.

I like the idea of forcing the database to be reactive, but we will still need a way to signal support for the NIP-50 search field

[alexgleason]

the event method should retrieve an event and not add one to the store

The verb is ["EVENT", event] not ["ADD", event]

thoughts on the supports() method

How about:

async info(): RelayInfoDocument // NIP-11 document

[alexgleason]

This method is not needed. Just call .query

[vitorpamplona]

Agree, the .query can check if the filter is only made by one address and send it here if this is more performant.

[alexgleason]

This method is not needed. Just call .query

[vitorpamplona]

Agree, the .query can check if the filter is only made by one ID and send it here if this is more performant.

[alexgleason]

Yes, it's important this interface is minimal for extension implementers. Client code can always wrap it with convenience methods.

[alexgleason]

Rename .req and return an AsyncIterable or a simple callback. I don't understand this interface

[vitorpamplona]

lol, I was going to say, I thought JavaScript had a native async streaming scheme...

[hzrd149]

I had a version that used the AsyncIterable interface but it was difficult to work with. it is a native api but it always requires apps to wrap it in some way in order to work with it

[vitorpamplona]

we will still need a way to signal support for the NIP-50 search field

I am not sure how useful this will be since you probably want to know what exactly the DB indexes instead. If your app needs search in specific parts, you will choose a DB that offers search for those parts. If you don't need search, you will choose a DB without search. To me there is no point in asking if the feature exists because the dev is already choosing when selecting a DB.

[DeepDoge]

I don't know if my opinion matters here, but I’m strongly against injecting the API directly into the window.

It causes extensions to compete with each other during injection, and it makes it harder for users to run multiple extensions at the same time. NIP-07 runs into the same issue.

Instead, I’d suggest an event-based approach.

For example, extensions could add an event listener to the window for something like nip123:requestAPI.

When the web app loads, it would start listening for nip123:announceAPI and then dispatch nip123:requestAPI.

Any extension listening for nip123:requestAPI would respond by dispatching nip123:announceAPI with its name, description, logo, and API object. (and maybe more)

From there, the app can choose one automatically, let the user pick one, or let them change it later in the settings, or some apps may even support multiple extensions.

This avoids fighting over window and makes multi-extension support much cleaner.

[hzrd149]

@alexgleason With regards to the add(), and the replaceable() and event() methods. my idea is that this interface isn't supposed to mirror the NIP-01 relays API. even though that api does have everything an app needs to talk to an event store. apps can already do that by opening a websocket to a local relay port.

This interface is primary designed for convenience and to allow apps to plug into whatever local event store the user has configured. which is why I moved away from the AsyncIterable interface in favor of "simple" callback methods

@vitorpamplona I don't think the apps need to know the exact details of how the event store is implemented like what tags or content is indexed because this interface isn't supposed to be a custom database provided to the app. the app shouldn't rely on this interface in the same why they would with their own custom database.

The way I've been thinking of this is a convenient interface that apps can use almost as a cache, to load events more quickly and attempt to store new events locally for later. if the app needs a database or highly specific feature then they should be using a WASM sqlite or indexeddb database

@DeepDoge I tend to agree, injecting interface into the window context feels a lot like the old days of extending the proto object to add methods... always ended poorly. I'm open to event or message based API if you can find a way to make it simple without the need for a library (less than ~10 lines to send a message). then might also be cool to look into supporting multiple stores, at which point it could make sense to use the NIP-01 message format... 🤔

[vitorpamplona]

I don't think the apps need to know the exact details of how the event store is implemented like what tags or content is indexed

If you offer "search" you NEED to specify what can be searched so that app devs can figure out if they can use this DB or if they need a custom DB. We can specify in the NIP (as opposed to a dynamic function like "supports") that all dbs MUST index all .content and all tags of all kinds. Or some other type of expected behavior. Devs just need to know what will be there, regardless of who the DB provider is, to figure out if they can use this or not.

I am in favor of getting rid of the "supports" and requiring a specific search indexing behavior by NIP.

On the AsyncIterable

You have to decide how high-level or how low-level you want this spec to be. If you offer replaceable and event as niceties, then you might as well add a bunch of other niceties to the spec, like query with a single Filter, not an array.

I think those niceties are not needed at this level. These methods will most likely be wrapped by each client anyway. So users of this spec can add only the niceties they need to their own wrapping mechanism.

AsyncIterable has close and error handling, but it doesn't have EOSEs. Managing EOSE's is also lower level than anything else in here. That's why I mentioned I switched over to having an array of events in the stream and the lib managing the EOSE of that list.

Also, managing EOSE as a callback is much harder than as a stream of events|EOSEs because the order of execution is important. You definitely don't want to process events and eose callbacks in parallel. So, I suggest doing an AsyncIterable<Event|EOSE> thing.

[staab]

my idea is that this interface isn't supposed to mirror the NIP-01 relays API

What if it did though? nostrdb could just implement the relay interface. It would be fully reactive too, in the same way relays are.

[vitorpamplona]

This nostrdb object can be provided by an extension, right?

If so, then it must be a relay-like api.

If not, and nostrdb is something created by each web app, then it should have a different interface.

Local DBs should offer custom projections, for instance, because the client doesn't need to re-verify events that are already in that DB, and picking just the properties you need can easily 10x a local query.

But if the nostrdb object is out of the dev's control, the dev MUST even verify events coming from it too... because who knows what's there?

Maybe it should be a .localrelay property instead.

[DeepDoge]

@hzrd149

I'm open to event or message based API if you can find a way to make it simple without the need for a library (less than ~10 lines to send a message)

Sure, what I’m talking about is very simple and doesn’t require a library at all.

Instead of doing this on the extension side:

if (!window.nostrdb) {
   window.nostrdb = { /* nostrdb impl */ }
}

You’d do something like:

window.addEventListener("nip123:requestAPI", () => {
   window.dispatchEvent(new CustomEvent("nip123:announceAPI", { 
      detail: {  name: "My Extension", icon: "data uri here", api: { /* nostrdb impl */  } }
   }))
})

And on the app side:

// Basic example
const nostrdbExtensions = []
window.addEventListener("nip123:announceAPI", (event) => nostrdbExtensions.push(event.detail))
window.dispatchEvent(new CustomEvent("nip123:requestAPI"))

Apps can make this reactive, update their selection list whenever they receive nip123:announceAPI, auto-select one, let users choose, or even support multiple simultaneously. These are application side choices.

[hzrd149]

@staab @vitorpamplona Your right that simply exposing a local NIP-01 ( websocket like ) interface would cover everything in this NIP and it would allow apps to access a common local event store ( like nostr-relay-tray at ws://localhost:4869 ).
However when I implemented it that way initially I discovered simple JavaScript apps don't natively talk NIP-01, they need another npm package or library to handle the connection and subscriptions. which means I needed to bundle a full nostr SDK just to access local events.

My point is that while NIP-01 is the core of nostr, its designed for communication with a remote relay over a websocket connection and so its a messaging protocol. This NIP is intended to be a JavaScript interface so making it conform to the messaging format of NIP-01 would just add extra badge that the apps would need to parse and handle.

@alexgleason I updated the subscribe method to return an AsyncGenerator and it is cleaner 👍

@DeepDoge using custom events seems heavy, it could be cleaner to use something like the window.postMessage API to communicate with the parent window or extension. still not sure if this would be better though since it would require apps to implement a lot of boilerplate

[vitorpamplona]

Why not just use objects directly?

if (supportedFeatures.search) {
}

Then the UI doesn't need to loop through the array of features and compare strings every time it needs to check.

[fiatjaf]

Accessing fields in a hashmap also involves string comparisons, and some hashing too.

Who knows what the JIT compiler does with these "objects", but in theory a single-item array is faster than a single-item hashmap.

[vitorpamplona]

Accessing fields in a hashmap also involves string comparisons

Only if there are collisions in each hashcode, otherwise, it's an int comparison. The hashcode is cached after the first usage.

single-item array is faster than a single-item hashmap

Only if you are not accounting for the string comparison.

[vitorpamplona]

We still need to clarify what must be implemented by the DB:

• What is the order of the return for filters and subscriptions?
  -- NIP-50 Searches must be sorted by score, not by created at.

• Does the add return false when inserting an old replaceable the db already has a newer version for?

• Does the add return false if the new event is expired?

• Does the add return false if there is a kind 5 or kind 62 event for that event?

• Does the add return false if the dev is trying to store an ephemeral event?

• Does it expire events automatically? Does it send updates on subscriptions when things expire while being subscribed?

• Does it delete on kind 5 events? Does it update subscriptions containing deleted events?

• Does it delete on kind 62 events? If so, devs should use a unique "relay name" for each instance of the db?

• Is it expected to support GiftWrap kind 5 and 62 deletions by p-tags?

[DeepDoge]

@hzrd149

using custom events seems heavy

It’s honestly pretty lightweight. It’s just a one-time handshake when the app loads.

App:
listen for nip123:announceAPI
dispatch nip123:requestAPI

Extension:
listen for nip123:requestAPI
respond with nip123:announceAPI

That’s basically it. No channels, no origin checks, no extra abstraction.

it could be cleaner to use window.postMessage

postMessage is more for cross-window / iframe communication. Here we’re already in the same page context.

It also serializes data, so passing live API objects or functions gets awkward. With CustomEvent, you can just pass the API object directly since it’s all in the same JS context.

So it’s actually simpler in practice.

My main concern with window.nostrdb is just that it turns into a race condition. Whoever injects first wins. That works if we never care about multiple extensions.

But if we ever want, user choice, switching providers, supporting multiple stores, better ecosystem competition.

Then globals don’t scale very well.

Also If an app only wants the first one, the event version of it is still like 3 lines literally:

let nostrdb;
window.addEventListener("nip123:announceAPI", (event) => (nostrdb = event.detail.api), { once: true })
window.dispatchEvent(new CustomEvent("nip123:requestAPI"))

It just now apps have option to support multiple extensions.

This pattern is also very similar to EIP-6963 for multi-provider discovery, so I didn't made it up.