Docs: add lancedb integration page #9594
Open
+87
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ozkatz
added
docs
|
📚 Documentation preview at https://pr-9594.docs-lakefs-preview.io/ |
nopcoder
approved these changes
Oct 23, 2025
Comment on lines
+18
to
+20
| ```python | ||
|
|
||
| db = lancedb.connect( |
There was a problem hiding this comment.
Adding the requirements for installation and import will help
|
|
||
| ### Multimodal data storage | ||
|
|
||
| In many cases, lanceDB stores embeddings of data that exists elsewhere: documents, images, text files etc. These "raw" data files are processed to extract embeddings, which are then stored in lanceDB - but they are also stored in their raw form for retrieval, and in some cases metadata about them is stored in other formats for use in a data warehouse or data lake. |
There was a problem hiding this comment.
Suggested change
| In many cases, lanceDB stores embeddings of data that exists elsewhere: documents, images, text files etc. These "raw" data files are processed to extract embeddings, which are then stored in lanceDB - but they are also stored in their raw form for retrieval, and in some cases metadata about them is stored in other formats for use in a data warehouse or data lake. | |
| In many cases, LanceDB stores embeddings of data that exists elsewhere: documents, images, text files etc. These "raw" data files are processed to extract embeddings, which are then stored in LanceDB - but they are also stored in their raw form for retrieval, and in some cases metadata about them is stored in other formats for use in a data warehouse or data lake. |
|
|
||
| lakeFS provides a highly performant and scalable way understand how data changes over time. For example, say we store raw image data in `images/` - we can update the raw data by adding, removing or updating images in the `images/` directory - and lakeFS will capture the changes as a commit. | ||
|
|
||
| This allows you to perform differential processing of new data: If we have an `images` table in lanceDB we can keep track of the latest commit represented in that table. As new data arrives, we can update our embeddings with the latest commit by diffing the previous commit and the new one, resulting in a minimal set of embeddings to add, remove or update: |
There was a problem hiding this comment.
Suggested change
| This allows you to perform differential processing of new data: If we have an `images` table in lanceDB we can keep track of the latest commit represented in that table. As new data arrives, we can update our embeddings with the latest commit by diffing the previous commit and the new one, resulting in a minimal set of embeddings to add, remove or update: | |
| This allows you to perform differential processing of new data: If we have an `images` table in LanceDB we can keep track of the latest commit represented in that table. As new data arrives, we can update our embeddings with the latest commit by diffing the previous commit and the new one, resulting in a minimal set of embeddings to add, remove or update: |
Comment on lines
+21
to
+23
| uri="s3://example-repo/example-branch/path/to/lancedb", | ||
| storage_options={ | ||
| "endpoint": "https://example.lakefs.io", |
There was a problem hiding this comment.
Comment these line explain about the s3 endpoint to lakeFS repository and the lakefs endpoint.
|
|
||
| # Using lakeFS with LanceDB | ||
|
|
||
| [LanceDB](https://lancedb.com/) is a vector database that allows you to store and query vector data. |
There was a problem hiding this comment.
consider adding another line about the use-case of using LanceDB.
Used for storing and querying embeddings often used in AI/ML pipelines... something like that.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a documentation page describing the use cases and how-to guide on using LanceDB on top of lakeFS