GH-34785: [C++][Parquet] Parquet Bloom Filter Writer Implementation #37400

mapleFU · 2023-08-26T14:26:49Z

Rationale for this change

Currently we allow reading bloom filter for specific column and rowgroup, now this patch allow it writing BF.

This patch is just a skeleton. If reviewer thinks interface would be OK, I'll go on and add testing.

What changes are included in this PR?

Allow writing bf:

Add WriterProperties config for writing bloom filter, including bf and (per-rowgroup) ndv estimation.
Add BloomFilterBuilder for parquet
From FileSerializer to ColumnWriter, adding bloomfilter
Ensure Bloom Filter info is written to the file
Testing logic for BloomFilterBuilder
Testing logic for BloomFilter and ColumnWriter
Testing whole roundtrip like ParquetPageIndexRoundTripTest

Are these changes tested?

Yes

Are there any user-facing changes?

User can create Bloom Filter in parquet with C++ api

Closes: [C++][Parquet] Allow writing BloomFilter for specific column #34785
GitHub Issue: [C++][Parquet] Allow writing BloomFilter for specific column #34785

mapleFU · 2023-08-26T14:28:40Z

This is port of #35691 . I'm busy previous days and now I've time on it now.

The previous comment are solved. cc @pitrou @wgtmac @emkornfield

cpp/src/parquet/bloom_filter.h

cpp/src/parquet/bloom_filter_parquet_test.cc

cpp/src/parquet/column_writer.cc

cpp/src/parquet/bloom_filter_parquet_test.cc

cpp/src/parquet/metadata.h

cpp/src/parquet/file_writer.cc

wgtmac

Thanks for adding this! I just did an initial review except the test.

cpp/src/parquet/bloom_filter.h

cpp/src/parquet/properties.h

cpp/src/parquet/CMakeLists.txt

cpp/src/parquet/bloom_filter_builder.cc

cpp/src/parquet/arrow/arrow_reader_writer_test.cc

cpp/src/parquet/bloom_filter.h

cpp/src/parquet/bloom_filter_builder.h

cpp/src/parquet/bloom_filter_builder.cc

# Conflicts: # cpp/src/parquet/column_writer_test.cc

mapleFU · 2024-11-12T10:45:56Z

Two need fix:

/arrow/cpp/src/parquet/bloom_filter.h:118: error: The following parameter of parquet::BloomFilter::Hash(const FLBA &value, uint32_t type_len) const is not documented:
  parameter 'type_len' (warning treated as error, aborting now)

D:/a/arrow/arrow/build/cpp/src/parquet/CMakeFiles/parquet_shared.dir/Unity/unity_3_cxx.cxx
In file included from D:/a/arrow/arrow/build/cpp/src/parquet/CMakeFiles/parquet_shared.dir/Unity/unity_3_cxx.cxx:16:
D:/a/arrow/arrow/cpp/src/parquet/schema.cc: In function 'void parquet::schema::PrintRepLevel(parquet::Repetition::type, std::ostream&)':
D:/a/arrow/arrow/cpp/src/parquet/schema.cc:630:30: error: expected unqualified-id before ':' token
  630 |     case Repetition::OPTIONAL:
      |                              ^

mapleFU · 2024-11-19T09:28:09Z

@pitrou @wgtmac @emkornfield Sorry for late reply, I believe all comments are replyed or fixed now. Now the bloom filter becoming map in all use cases, since it would be more sparse then page-indices.

amoeba · 2024-12-19T17:56:08Z

The feature freeze for Arrow 19 is planned for January 6, 2025 and I'm curious if there might be capacity to get this fully reviewed and merged by then (or soon after). If not, feel free to comment with what you'd need (more reviewers, more time, etc). cc @mapleFU @pitrou @wgtmac @emkornfield

wgtmac · 2024-12-20T01:29:16Z

Sorry for missing this! I will take a look. Meanwhile, I don't think we need to hurry for the code freeze.

# Conflicts: # cpp/src/parquet/column_writer.cc # cpp/src/parquet/type_fwd.h

mapleFU · 2024-12-20T02:43:54Z

Rebased, ready for review now

wgtmac

I have reviewed for another pass and generally LGTM.

It would be good if @pitrou @emkornfield can take a look after the holiday season.

cpp/src/parquet/CMakeLists.txt

cpp/src/parquet/arrow/arrow_reader_writer_test.cc

wgtmac · 2024-12-23T05:25:01Z

cpp/src/parquet/arrow/arrow_reader_writer_test.cc

+          ASSERT_EQ(nullptr, bloom_filter);
+        } else {
+          ASSERT_NE(nullptr, bloom_filter);
+          bloom_filters_.push_back(std::move(bloom_filter));


What about changing bloom_filters_ to be an output parameter to function ReadBloomFilters instead of a class member variable?

wgtmac · 2024-12-23T05:27:12Z

cpp/src/parquet/arrow/arrow_reader_writer_test.cc

+  std::vector<std::unique_ptr<BloomFilter>> bloom_filters_;
+};
+
+TEST_F(ParquetBloomFilterRoundTripTest, SimpleRoundTrip) {


The three test cases below share a lot of common logic (with exactly same data). Should we refactor them to eliminate the duplicate?

cpp/src/parquet/bloom_filter.h

wgtmac · 2024-12-23T08:30:30Z

cpp/src/parquet/metadata.h

+struct BloomFilterLocation {
+  /// Row group bloom filter index locations which uses row group ordinal as the key.
+  ///
+  /// Note: Before Parquet 2.10, the bloom filter index only have "offset". But here


Suggested change

/// Note: Before Parquet 2.10, the bloom filter index only have "offset". But here

/// Note: Before Parquet Format v2.10, the bloom filter index only have "offset". But here

wgtmac · 2024-12-23T08:32:39Z

cpp/src/parquet/metadata.h

+///
+/// Number of columns with a bloom filter to be relatively small compared to
+/// the number of overall columns, so map is used.
+using RowGroupBloomFilterLocation = std::map<int32_t, IndexLocation>;


What about defining RowGroupBloomFilterLocation and FileBloomFilterLocation in the BloomFilterLocation?

I think PageIndexLocation doesn't defined them here. what about keeping it consistent?

cpp/src/parquet/metadata.cc

cpp/src/parquet/column_writer_test.cc

# Conflicts: # cpp/src/parquet/file_writer.cc # cpp/src/parquet/properties.h

wgtmac · 2025-02-06T16:20:44Z

Could you please fix the CI failure?

mapleFU · 2025-03-26T07:49:21Z

I've resolve the comments and fix the ci, would you mind re-check ? @wgtmac

Parquet: Implement skeleton for BloomFilter

f1c6dc0

mapleFU requested a review from wgtmac as a code owner August 26, 2023 14:26

github-actions bot added Component: Parquet Component: C++ awaiting review labels Aug 26, 2023

tiny fixing

6ebd6da

mapleFU commented Aug 26, 2023

View reviewed changes

github-actions bot added awaiting committer review and removed awaiting review labels Aug 26, 2023

mapleFU added 2 commits August 26, 2023 23:32

tiny update test

70c9267

trying to fix ci

48350d8

mapleFU commented Aug 26, 2023

View reviewed changes

cpp/src/parquet/bloom_filter_parquet_test.cc Outdated Show resolved Hide resolved

fix lint

d2a659e

mapleFU commented Aug 26, 2023

View reviewed changes

cpp/src/parquet/metadata.h Show resolved Hide resolved

mapleFU added 4 commits August 27, 2023 00:44

fix some style problem

41236d8

add file roundtrip test

8afba81

add file roundtrip test

96c6691

fix document and ci

c131341

mapleFU commented Aug 26, 2023

View reviewed changes

cpp/src/parquet/file_writer.cc Outdated Show resolved Hide resolved

mapleFU requested review from pitrou and emkornfield August 26, 2023 18:43

Update: tiny style fix

220b58e

mapleFU changed the title ~~GH-34785: [C++][Parquet] Parquet Bloom Filter Implement~~ GH-34785: [C++][Parquet] Parquet Bloom Filter Write Implement Aug 27, 2023

wgtmac reviewed Aug 30, 2023

View reviewed changes

wgtmac changed the title ~~GH-34785: [C++][Parquet] Parquet Bloom Filter Write Implement~~ GH-34785: [C++][Parquet] Parquet Bloom Filter Writer Implementation Aug 30, 2023

pitrou requested changes Aug 31, 2023

View reviewed changes

github-actions bot added awaiting changes and removed awaiting committer review labels Sep 1, 2023

Merge branch 'main' into parquet/support-write-bloom-filter

ad96c48

mapleFU added 3 commits November 11, 2024 12:22

Merge branch 'main' into parquet/support-write-bloom-filter

2223423

# Conflicts: # cpp/src/parquet/column_writer_test.cc

change the bloom filter from vector to map

Loading
Loading status checks…

22030db

fix lint

Loading
Loading status checks…

e9c550a

mapleFU force-pushed the parquet/support-write-bloom-filter branch from 505e23b to e9c550a Compare November 12, 2024 05:48

fix lint

Loading
Loading status checks…

23fb3fa

mapleFU force-pushed the parquet/support-write-bloom-filter branch from fd3856d to 79bdaeb Compare November 15, 2024 04:49

fix comment

Loading
Loading status checks…

d892819

mapleFU force-pushed the parquet/support-write-bloom-filter branch from 79bdaeb to d892819 Compare November 15, 2024 06:57

mapleFU requested review from emkornfield and wgtmac November 19, 2024 09:27

Merge branch 'main' into parquet/support-write-bloom-filter

Loading
Loading status checks…

ef3291d

# Conflicts: # cpp/src/parquet/column_writer.cc # cpp/src/parquet/type_fwd.h

wgtmac reviewed Dec 23, 2024

View reviewed changes

mapleFU added 2 commits January 13, 2025 14:33

Merge branch 'main' into parquet/support-write-bloom-filter

7aee7dd

Resolve comments

Loading
Loading status checks…

c5b1fb1

Fokko mentioned this pull request Feb 5, 2025

support bloom-filter writing apache/iceberg-python#850

Open

Merge branch 'main' into parquet/support-write-bloom-filter

Loading
Loading status checks…

0898466

# Conflicts: # cpp/src/parquet/file_writer.cc # cpp/src/parquet/properties.h

mapleFU added 7 commits February 7, 2025 14:38

Merge branch 'main' into parquet/support-write-bloom-filter

71f5906

minor fix

Loading
Loading status checks…

d57ceea

address some comments

Loading
Loading status checks…

26c2d07

Merge branch 'main' into parquet/support-write-bloom-filter

d422ffa

Minor fix

Loading
Loading status checks…

e6bc6e1

try to fix lint

Loading
Loading status checks…

dfaf0e8

Merge branch 'main' into parquet/support-write-bloom-filter

Loading
Loading status checks…

0bafe78

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-34785: [C++][Parquet] Parquet Bloom Filter Writer Implementation #37400

GH-34785: [C++][Parquet] Parquet Bloom Filter Writer Implementation #37400

mapleFU commented Aug 26, 2023 •

edited by github-actions bot

Loading

mapleFU commented Aug 26, 2023 •

edited

Loading

wgtmac left a comment

mapleFU commented Nov 12, 2024

mapleFU commented Nov 19, 2024

amoeba commented Dec 19, 2024

wgtmac commented Dec 20, 2024

mapleFU commented Dec 20, 2024

wgtmac left a comment

wgtmac Dec 23, 2024

wgtmac Dec 23, 2024

wgtmac Dec 23, 2024

wgtmac Dec 23, 2024

mapleFU Feb 7, 2025

wgtmac commented Feb 6, 2025

mapleFU commented Mar 26, 2025

	/// Note: Before Parquet 2.10, the bloom filter index only have "offset". But here
	/// Note: Before Parquet Format v2.10, the bloom filter index only have "offset". But here

GH-34785: [C++][Parquet] Parquet Bloom Filter Writer Implementation #37400

Are you sure you want to change the base?

GH-34785: [C++][Parquet] Parquet Bloom Filter Writer Implementation #37400

Conversation

mapleFU commented Aug 26, 2023 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

mapleFU commented Aug 26, 2023 • edited Loading

wgtmac left a comment

Choose a reason for hiding this comment

mapleFU commented Nov 12, 2024

mapleFU commented Nov 19, 2024

amoeba commented Dec 19, 2024

wgtmac commented Dec 20, 2024

mapleFU commented Dec 20, 2024

wgtmac left a comment

Choose a reason for hiding this comment

wgtmac Dec 23, 2024

Choose a reason for hiding this comment

wgtmac Dec 23, 2024

Choose a reason for hiding this comment

wgtmac Dec 23, 2024

Choose a reason for hiding this comment

wgtmac Dec 23, 2024

Choose a reason for hiding this comment

mapleFU Feb 7, 2025

Choose a reason for hiding this comment

wgtmac commented Feb 6, 2025

mapleFU commented Mar 26, 2025

mapleFU commented Aug 26, 2023 •

edited by github-actions bot

Loading

mapleFU commented Aug 26, 2023 •

edited

Loading