[WIP][POC][DeltaLake] Prototype PR to add support reading tables using Delta Kernel library #23119
+4,206
−58
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
findinpath
suggested changes
Aug 26, 2024
There was a problem hiding this comment.
I skimmed the io.trino.plugin.deltalake.kernel.clients code and the usage of hadoop's Configuration class is a no-go for trinodb/trino. Please see into #15921 for details.
There was a problem hiding this comment.
Removing unrelated property files is likely unintended.
| private final TypeManager typeManager; | ||
|
|
||
| public KernelTableClient( | ||
| Configuration configuration, |
There was a problem hiding this comment.
trinodb/trino has migrated off from hadoop related API in favor of using natively the file system clients.
We'd likely not accept reintroducing this compile dependency in trino-delta-lake
| } | ||
|
|
||
| Utils.closeCloseables(currentFileReader); | ||
| if (!fileIter.hasNext()) { |
There was a problem hiding this comment.
nit: fileIter -> fileIterator
Trino code conventions state that variable name shortcuts are disincouraged.
vkorukanti
force-pushed
the
kernel-poc
branch
from
8afcabb
to
53e5df1
Compare
vkorukanti
force-pushed
the
kernel-poc
branch
from
53e5df1
to
d388182
Compare
add config to enable kernel, create KernelDeltaLakeMetadata override get table handle using Kernel APIs, stubs for TableClient custom impls, build changes delegators, split manager, page source provider (not yet tested) end-2-end working! support for partition column
vkorukanti
force-pushed
the
kernel-poc
branch
from
d388182
to
e267a4d
Compare
mosabua
added
stale-ignore
|
Switched to stale-ignore label since this is an ongoing effort. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The Delta Kernel project is a set of Java libraries for building Delta connectors that can read from and write into Delta tables using a narrow set of APIs without understanding the Delta protocol details.
There are two sets of public APIs to build connectors.
More Information about Delta Kernel can be found:
Example Java programs that illustrate how to read and write Delta tables using the Kernel APIs.
Migration guide
Currently, the Trino Delta connector has its own implementation of the Delta Log. We want to see if there is a way we can use Delta Kernel in Trino Delta connector so that Trino-Delta connector doesn't need to reimplement the same protocol updates.
This PR is just attempt to use the Kernel for read path with a session option. The prototype is in a very early stage, and a lot of details need to be implemented. Currently, it allows reading Delta tables, including the one with deletion vectors. It uses Trino's own Parquet reader. Looking for some early feedback on how Kernel can help reduce the development burden on the Trino-Delta connector to keep up with the Delta protocol updates.