[feat](iceberg)Supports using `rest` type catalog to read tables in unity catalog #43525

wuwenchi · 2024-11-08T11:15:54Z

What problem does this PR solve?

We now support using the rest type catalog to read tables in the unity catalog (https://github.com/unitycatalog/unitycatalog).
When reading the parquet file on the be side, we find the corresponding column name based on the column id, which naturally supports the column rename function.

example：

CREATE CATALOG `uc3`
PROPERTIES (
"type"  =  "iceberg",
"iceberg.catalog.type"  =  "rest",
"uri"  =  "http://127.0.0.1:8080/api/2.1/unity-catalog/iceberg",
"external_catalog.name" = "unity"  --- catalog name in unity catalog
);

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-08T11:16:00Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

github-actions

clang-tidy made some suggestions

github-actions · 2024-11-08T11:20:48Z

be/src/vec/exec/format/parquet/vparquet_reader.cpp

@@ -259,6 +259,10 @@ std::vector<tparquet::KeyValue> ParquetReader::get_metadata_key_values() {
    return _t_metadata->key_value_metadata;
 }

+const FieldDescriptor ParquetReader::get_file_metadata_schema() {


warning: return type 'const std::doris::vectorized::FieldDescriptor' is 'const'-qualified at the top level, which may reduce code readability without improving const correctness [readability-const-return-type]

Suggested change

const FieldDescriptor ParquetReader::get_file_metadata_schema() {

FieldDescriptor ParquetReader::get_file_metadata_schema() {

be/src/vec/exec/format/parquet/vparquet_reader.h:151:

- const FieldDescriptor get_file_metadata_schema(); + FieldDescriptor get_file_metadata_schema();

wuwenchi · 2024-11-08T11:27:42Z

run buildall

doris-robot · 2024-11-08T11:52:54Z

TeamCity be ut coverage result:
Function Coverage: 37.91% (9867/26030)
Line Coverage: 29.09% (82270/282777)
Region Coverage: 28.26% (42368/149947)
Branch Coverage: 24.80% (21452/86500)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0b472c1d54bb438937c943b2672ca3caa904e331_0b472c1d54bb438937c943b2672ca3caa904e331/report/index.html

wuwenchi · 2024-11-08T12:11:37Z

run buildall

doris-robot · 2024-11-08T12:36:44Z

TeamCity be ut coverage result:
Function Coverage: 37.91% (9868/26030)
Line Coverage: 29.10% (82278/282777)
Region Coverage: 28.25% (42367/149947)
Branch Coverage: 24.80% (21451/86500)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b2105ca203b130292bf414c0203cbd4d5b5bf1bf_b2105ca203b130292bf414c0203cbd4d5b5bf1bf/report/index.html

morningman · 2024-11-11T07:47:24Z

fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataCache.java

-            icebergCatalog = ((HMSExternalCatalog) key.catalog).getIcebergHiveCatalog();
+            Catalog icebergCatalog = ((HMSExternalCatalog) key.catalog).getIcebergHiveCatalog();
+            icebergTable = HiveMetaStoreClientHelper.ugiDoAs(((ExternalCatalog) key.catalog).getConfiguration(),
+                () -> icebergCatalog.loadTable(TableIdentifier.of(key.dbName, key.tableName)));


We should unify the interface, both using catalog.loadTable() or using metadataOps.loadTable()

morningman · 2024-11-11T07:52:16Z

fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalCatalog.java

@@ -39,6 +39,7 @@ public abstract class IcebergExternalCatalog extends ExternalCatalog {
    public static final String ICEBERG_HADOOP = "hadoop";
    public static final String ICEBERG_GLUE = "glue";
    public static final String ICEBERG_DLF = "dlf";
+    public static final String EXTERNAL_SERVER_CATALOG_NAME = "external_server_catalog_name";


Suggested change

public static final String EXTERNAL_SERVER_CATALOG_NAME = "external_server_catalog_name";

public static final String EXTERNAL_SERVER_CATALOG_NAME = "external_catalog.name";

morningman · 2024-11-11T07:52:52Z

be/src/vec/exec/format/table/iceberg_reader.cpp

-                            _has_schema_change = true;
-                        }
-                    }
+Status IcebergParquetReader::_gen_col_name_maps(FieldDescriptor field_desc) {


Suggested change

Status IcebergParquetReader::_gen_col_name_maps(FieldDescriptor field_desc) {

Status IcebergParquetReader::_gen_col_name_maps(const FieldDescriptor& field_desc) {

morningman · 2024-11-11T07:55:31Z

be/src/vec/exec/format/parquet/vparquet_reader.h

@@ -149,6 +149,7 @@ class ParquetReader : public GenericReader {
            const std::unordered_map<std::string, VExprContextSPtr>& missing_columns) override;

    std::vector<tparquet::KeyValue> get_metadata_key_values();


This method can be removed?

morningman · 2024-11-11T07:57:26Z

be/src/vec/exec/format/table/iceberg_reader.h

@@ -218,7 +218,7 @@ class IcebergParquetReader final : public IcebergTableReader {
        parquet_reader->set_delete_rows(&_iceberg_delete_rows);
    }

-    Status _gen_col_name_maps(std::vector<tparquet::KeyValue> parquet_meta_kv);
+    Status _gen_col_name_maps(FieldDescriptor field_desc);


Need also modify this method in IcebergOrcReader

The orc format is originally associated according to the id, so there is no need to modify it.

wuwenchi · 2024-11-11T13:28:13Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-11-11T13:32:38Z

be/src/vec/exec/format/parquet/schema_desc.cpp

@@ -147,6 +150,14 @@ Status FieldDescriptor::parse_from_thrift(const std::vector<tparquet::SchemaElem
    return Status::OK();
 }

+const doris::Slice FieldDescriptor::get_column_name_from_field_id(int32_t id) const {


warning: return type 'const doris::Slice' is 'const'-qualified at the top level, which may reduce code readability without improving const correctness [readability-const-return-type]

Suggested change

const doris::Slice FieldDescriptor::get_column_name_from_field_id(int32_t id) const {

doris::Slice FieldDescriptor::get_column_name_from_field_id(int32_t id) const {

be/src/vec/exec/format/parquet/schema_desc.h:137:

- const doris::Slice get_column_name_from_field_id(int32_t id) const; + doris::Slice get_column_name_from_field_id(int32_t id) const;

github-actions · 2024-11-11T13:32:39Z

be/src/vec/exec/format/parquet/vparquet_reader.cpp

-// This is for iceberg schema evolution.
-std::vector<tparquet::KeyValue> ParquetReader::get_metadata_key_values() {
-    return _t_metadata->key_value_metadata;
+const FieldDescriptor ParquetReader::get_file_metadata_schema() {


warning: return type 'const std::doris::vectorized::FieldDescriptor' is 'const'-qualified at the top level, which may reduce code readability without improving const correctness [readability-const-return-type]

Suggested change

const FieldDescriptor ParquetReader::get_file_metadata_schema() {

FieldDescriptor ParquetReader::get_file_metadata_schema() {

be/src/vec/exec/format/parquet/vparquet_reader.h:150:

- const FieldDescriptor get_file_metadata_schema(); + FieldDescriptor get_file_metadata_schema();

wuwenchi · 2024-11-12T12:14:46Z

run buildall

wuwenchi · 2024-11-13T08:30:48Z

run buildall

fix

c45ada8

wuwenchi requested review from platoneko, gavinchou and dataroaring as code owners November 8, 2024 11:15

github-actions bot reviewed Nov 8, 2024

View reviewed changes

format

0b472c1

format

b2105ca

morningman reviewed Nov 11, 2024

View reviewed changes

morningman added dev/2.1.x dev/3.0.x labels Nov 11, 2024

wuwenchi added 2 commits November 11, 2024 19:35

fix

924cfa8

fix

df0243c

github-actions bot reviewed Nov 11, 2024

View reviewed changes

add test

101e272

morningman changed the title ~~[bugfix](iceberg)Supports using rest type catalog to read tables in unity catalog~~ [feat](iceberg)Supports using rest type catalog to read tables in unity catalog Nov 12, 2024

wuwenchi added 2 commits November 13, 2024 15:07

test

270850c

test

e9f2152

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat](iceberg)Supports using `rest` type catalog to read tables in unity catalog #43525

[feat](iceberg)Supports using `rest` type catalog to read tables in unity catalog #43525

wuwenchi commented Nov 8, 2024 •

edited

Loading

doris-robot commented Nov 8, 2024

github-actions bot left a comment

github-actions bot Nov 8, 2024

wuwenchi commented Nov 8, 2024

doris-robot commented Nov 8, 2024

wuwenchi commented Nov 8, 2024

doris-robot commented Nov 8, 2024

morningman Nov 11, 2024

morningman Nov 11, 2024

morningman Nov 11, 2024

morningman Nov 11, 2024

morningman Nov 11, 2024

wuwenchi Nov 11, 2024

wuwenchi commented Nov 11, 2024

github-actions bot left a comment

github-actions bot Nov 11, 2024

github-actions bot Nov 11, 2024

wuwenchi commented Nov 12, 2024

wuwenchi commented Nov 13, 2024

	const FieldDescriptor ParquetReader::get_file_metadata_schema() {
	FieldDescriptor ParquetReader::get_file_metadata_schema() {

	public static final String EXTERNAL_SERVER_CATALOG_NAME = "external_server_catalog_name";
	public static final String EXTERNAL_SERVER_CATALOG_NAME = "external_catalog.name";

	Status IcebergParquetReader::_gen_col_name_maps(FieldDescriptor field_desc) {
	Status IcebergParquetReader::_gen_col_name_maps(const FieldDescriptor& field_desc) {

		@@ -149,6 +149,7 @@ class ParquetReader : public GenericReader {
		const std::unordered_map<std::string, VExprContextSPtr>& missing_columns) override;

		std::vector<tparquet::KeyValue> get_metadata_key_values();

	const doris::Slice FieldDescriptor::get_column_name_from_field_id(int32_t id) const {
	doris::Slice FieldDescriptor::get_column_name_from_field_id(int32_t id) const {

[feat](iceberg)Supports using rest type catalog to read tables in unity catalog #43525

Are you sure you want to change the base?

[feat](iceberg)Supports using rest type catalog to read tables in unity catalog #43525

Conversation

wuwenchi commented Nov 8, 2024 • edited Loading

What problem does this PR solve?

Check List (For Author)

Check List (For Reviewer who merge this PR)

doris-robot commented Nov 8, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Nov 8, 2024

Choose a reason for hiding this comment

wuwenchi commented Nov 8, 2024

doris-robot commented Nov 8, 2024

wuwenchi commented Nov 8, 2024

doris-robot commented Nov 8, 2024

morningman Nov 11, 2024

Choose a reason for hiding this comment

morningman Nov 11, 2024

Choose a reason for hiding this comment

morningman Nov 11, 2024

Choose a reason for hiding this comment

morningman Nov 11, 2024

Choose a reason for hiding this comment

morningman Nov 11, 2024

Choose a reason for hiding this comment

wuwenchi Nov 11, 2024

Choose a reason for hiding this comment

wuwenchi commented Nov 11, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Nov 11, 2024

Choose a reason for hiding this comment

github-actions bot Nov 11, 2024

Choose a reason for hiding this comment

wuwenchi commented Nov 12, 2024

wuwenchi commented Nov 13, 2024

[feat](iceberg)Supports using `rest` type catalog to read tables in unity catalog #43525

[feat](iceberg)Supports using `rest` type catalog to read tables in unity catalog #43525

wuwenchi commented Nov 8, 2024 •

edited

Loading