Skip to content

Document the row / record / TermsAndValues levels and the two senses of "index" - #24

Merged
drametwally merged 2 commits into
mainfrom
document-record-vocabulary
Sep 15, 2026
Merged

drametwally merged 2 commits into
mainfrom
document-record-vocabulary

Conversation

@drametwally

@drametwally drametwally commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Docs only, two vocabulary levels that keep getting mistaken for duplication.

Rows and records

Four names sit close enough together to read as duplicates, and the record/TermsAndValues pairing has now been flagged twice as if one of them were redundant. They are four levels of one thing:

  • A row is what a searchable structure stores: a record plus its metadata, the extra data a filter matches on. It is identified by a rowNum, the handle the internals pass around in place of the row.
  • A record is the feature data of a row — a concept, not a type. It can be a vector, a time series, a histogram, a sparse (multi-)set, or a sequence of terms. RecordType names which shape, and a comparator and a structure pair up by agreeing on one.
  • TermsAndValues is the public structure a record arrives in: String[] terms and float[] values, which between them express every shape depending on population. Terms alone is a sequence, values alone is dense, both is sparse.
  • LongTermsAndValues is the internal form of the same pair, terms hashed to long with the comparator-derived uniValue cached alongside.

Stating it also answers why the enum is RecordType and not TermsAndValuesType.

README.md read "A record is a TermsAndValues", which flattens the two levels into each other. It now reads "A record reaches USSI as a TermsAndValues" — the same instruction for someone embedding the library, without the conflation.

Two senses of "index"

This one is a correction to my own audit. I had flagged NearestNeighborSearchIndex as misnamed because it holds List<Index> and List<Cache>, making it the engine rather than an index, and proposed renaming it to SimilaritySearchEngine.

That reasoning missed that the word covers two levels, and both are right in their place. The library as a whole is an index — that is the sense the project's own name carries in Uber Similarity Search Index — while an Index is one searchable structure inside it, a sibling of Cache. Renaming the facade would have left a library called an index with nothing named one at the top, and broken every embedder's entry point to do it.

So the name stays and the real confusion, one word covering two levels, is written down instead:

So the facade holding indexes rather than being one is not a contradiction. A reader who expects NearestNeighborSearchIndex to extend Index has the narrow sense in mind; nothing does, and nothing should.

Checked, not guessed

The population rules were read back off TermsAndValues.isDenseFeature, isSparseFeature, and isSequence. The facade's composition was read off its field declarations: implements AutoCloseable only, with List<Index> indexes and List<Cache> graduatingCaches.

One thing deliberately left out: why we say "active cache" rather than the SPARS paper's "live cache" — the engine owns one active cache plus zero or more graduating caches, and graduating caches are still searched, so "live" would imply the others are dead. That note only makes sense once the paper is cited here, so it rides with the terminology-alignment PR.

Four names sit close enough together to read as duplicates of each other,
and the pairing of record with TermsAndValues has now been flagged twice as
though one of them were redundant. They are four levels: a row is a record
plus metadata, identified by a rowNum; a record is the feature data as a
concept, whose shape RecordType names; TermsAndValues is the public array
pair a record arrives in; LongTermsAndValues is the internal form with
terms hashed to long and the uniValue cached alongside.

Stating it also explains why the enum is RecordType rather than
TermsAndValuesType.

README said "A record is a TermsAndValues", which flattens the two levels
into each other. It now says a record reaches USSI as one, which is the
same instruction for an embedder without the conflation.

Docs only.
The audit flagged NearestNeighborSearchIndex as misnamed on the grounds
that it holds List<Index> and List<Cache>, so it is the engine rather than
an index. That reasoning missed that the word covers two levels and both
are right in their place: the library as a whole is an index, which is the
sense Uber Similarity Search Index carries, while an Index is one
searchable structure inside it.

So the facade keeps its name, and the actual confusion - one word, two
levels - is stated where it belongs rather than fixed by a rename that
would leave a library called an index with nothing named one at the top.
@drametwally drametwally changed the title Document the row / record / TermsAndValues levels Document the row / record / TermsAndValues levels and the two senses of "index" Sep 15, 2026
@drametwally
drametwally merged commit 26aa788 into main Sep 15, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant