diff --git a/DESIGN.md b/DESIGN.md index 0b4659a..e9c0095 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -7,6 +7,33 @@ library. If you are embedding it, [README.md](README.md) is what you want, and the public `NearestNeighborSearchIndex` API is what you should use rather than managing any of the structures below directly. +## Rows and Records + +Four names sit close enough together to look like duplicates. They are four +levels of one thing. + +A **row** is the unit a searchable structure stores: a record together with +its metadata, the extra data a filter matches on. A row is identified by a +`rowNum`, which is the handle the internals pass around in place of the row +itself. + +A **record** is the feature data of a row. It is a concept, not a type, and +it can take the shape of a vector, a time series, a histogram, a sparse +(multi-)set, or a sequence of terms. `RecordType` names which shape a given +record is in; a comparator declares the shapes it reads and a structure +declares the one it stores, so the two pair up by agreeing on one. + +`TermsAndValues` is the public data structure a record arrives in: two +parallel arrays, `String[] terms` and `float[] values`, which between them +express every shape above depending on how they are populated. Terms alone +is a sequence, values alone is dense, both together is sparse. + +`LongTermsAndValues` is the internal form of the same pair, with terms +hashed to `long` and the comparator-derived `uniValue` cached alongside them. + +So a record is the idea and a `TermsAndValues` is the array pair implementing +it, which is why the enum is `RecordType` and not `TermsAndValuesType`. + ## Structure Lifecycle `NearestNeighborSearchIndex` owns one active cache, zero or more graduating @@ -100,6 +127,23 @@ Enums a config names implement `ConfigVocabulary`, which holds the one case-insensitive lookup and the one unsupported-value message. A config never names a record type, so `RecordType` is not among them. +## Two Senses Of "Index" + +The word covers two levels, and both are correct in their place. + +The library is an index: `NearestNeighborSearchIndex` is the whole searchable +thing, which is the sense the project's own name carries in Uber Similarity +Search Index. It stores rows, answers queries, and is the only type an +embedder needs. + +An `Index` is one searchable structure inside it, a sibling of a `Cache` and +a `SearchableStructure` like it. The facade owns a list of these and a list +of caches, and merges their results. + +So the facade holding indexes rather than being one is not a contradiction. +A reader who expects `NearestNeighborSearchIndex` to extend `Index` has the +narrow sense in mind; nothing does, and nothing should. + ## Package Layout ### Searchable Structures diff --git a/README.md b/README.md index ca214e6..fd53ec5 100644 --- a/README.md +++ b/README.md @@ -88,7 +88,7 @@ that cannot work together all fail at creation. ## Records -A record is a `TermsAndValues`, built from parallel arrays: +A record reaches USSI as a `TermsAndValues`, built from parallel arrays: ```java new TermsAndValues(String[] terms, float[] values)