GH-34785: [C++][Parquet] Parquet Bloom Filter Writer Implementation #37400

mapleFU · 2023-08-26T14:26:49Z

Rationale for this change

Currently we allow reading bloom filter for specific column and rowgroup, now this patch allow it writing BF.

This patch is just a skeleton. If reviewer thinks interface would be OK, I'll go on and add testing.

What changes are included in this PR?

Allow writing bf:

Add WriterProperties config for writing bloom filter, including bf and (per-rowgroup) ndv estimation.
Add BloomFilterBuilder for parquet
From FileSerializer to ColumnWriter, adding bloomfilter
Ensure Bloom Filter info is written to the file
Testing logic for BloomFilterBuilder
Testing logic for BloomFilter and ColumnWriter
Testing whole roundtrip like ParquetPageIndexRoundTripTest

Are these changes tested?

Yes

Are there any user-facing changes?

User can create Bloom Filter in parquet with C++ api

Closes: [C++][Parquet] Allow writing BloomFilter for specific column #34785
GitHub Issue: [C++][Parquet] Allow writing BloomFilter for specific column #34785

mapleFU · 2023-08-26T14:28:40Z

This is port of #35691 . I'm busy previous days and now I've time on it now.

The previous comment are solved. cc @pitrou @wgtmac @emkornfield

cpp/src/parquet/bloom_filter.h

cpp/src/parquet/bloom_filter_parquet_test.cc

cpp/src/parquet/column_writer.cc

cpp/src/parquet/bloom_filter_parquet_test.cc

cpp/src/parquet/metadata.h

cpp/src/parquet/file_writer.cc

wgtmac

Thanks for adding this! I just did an initial review except the test.

cpp/src/parquet/bloom_filter.h

cpp/src/parquet/properties.h

cpp/src/parquet/CMakeLists.txt

cpp/src/parquet/bloom_filter_builder.cc

cpp/src/parquet/arrow/arrow_reader_writer_test.cc

cpp/src/parquet/bloom_filter.h

cpp/src/parquet/bloom_filter_builder.h

cpp/src/parquet/bloom_filter_builder.cc

mapleFU · 2024-11-12T10:45:56Z

Two need fix:

/arrow/cpp/src/parquet/bloom_filter.h:118: error: The following parameter of parquet::BloomFilter::Hash(const FLBA &value, uint32_t type_len) const is not documented:
  parameter 'type_len' (warning treated as error, aborting now)

D:/a/arrow/arrow/build/cpp/src/parquet/CMakeFiles/parquet_shared.dir/Unity/unity_3_cxx.cxx
In file included from D:/a/arrow/arrow/build/cpp/src/parquet/CMakeFiles/parquet_shared.dir/Unity/unity_3_cxx.cxx:16:
D:/a/arrow/arrow/cpp/src/parquet/schema.cc: In function 'void parquet::schema::PrintRepLevel(parquet::Repetition::type, std::ostream&)':
D:/a/arrow/arrow/cpp/src/parquet/schema.cc:630:30: error: expected unqualified-id before ':' token
  630 |     case Repetition::OPTIONAL:
      |                              ^

mapleFU · 2024-11-19T09:28:09Z

@pitrou @wgtmac @emkornfield Sorry for late reply, I believe all comments are replyed or fixed now. Now the bloom filter becoming map in all use cases, since it would be more sparse then page-indices.

amoeba · 2024-12-19T17:56:08Z

The feature freeze for Arrow 19 is planned for January 6, 2025 and I'm curious if there might be capacity to get this fully reviewed and merged by then (or soon after). If not, feel free to comment with what you'd need (more reviewers, more time, etc). cc @mapleFU @pitrou @wgtmac @emkornfield

wgtmac · 2024-12-20T01:29:16Z

Sorry for missing this! I will take a look. Meanwhile, I don't think we need to hurry for the code freeze.

# Conflicts: # cpp/src/parquet/column_writer.cc # cpp/src/parquet/type_fwd.h

mapleFU · 2024-12-20T02:43:54Z

Rebased, ready for review now

wgtmac

I have reviewed for another pass and generally LGTM.

It would be good if @pitrou @emkornfield can take a look after the holiday season.

cpp/src/parquet/CMakeLists.txt

cpp/src/parquet/arrow/arrow_reader_writer_test.cc

wgtmac · 2024-12-23T05:25:01Z

cpp/src/parquet/arrow/arrow_reader_writer_test.cc

+          ASSERT_EQ(nullptr, bloom_filter);
+        } else {
+          ASSERT_NE(nullptr, bloom_filter);
+          bloom_filters_.push_back(std::move(bloom_filter));


What about changing bloom_filters_ to be an output parameter to function ReadBloomFilters instead of a class member variable?

wgtmac · 2024-12-23T05:27:12Z

cpp/src/parquet/arrow/arrow_reader_writer_test.cc

+  std::vector<std::unique_ptr<BloomFilter>> bloom_filters_;
+};
+
+TEST_F(ParquetBloomFilterRoundTripTest, SimpleRoundTrip) {


The three test cases below share a lot of common logic (with exactly same data). Should we refactor them to eliminate the duplicate?

cpp/src/parquet/bloom_filter.h

wgtmac · 2024-12-23T08:30:30Z

cpp/src/parquet/metadata.h

+struct BloomFilterLocation {
+  /// Row group bloom filter index locations which uses row group ordinal as the key.
+  ///
+  /// Note: Before Parquet 2.10, the bloom filter index only have "offset". But here


Suggested change

/// Note: Before Parquet 2.10, the bloom filter index only have "offset". But here

/// Note: Before Parquet Format v2.10, the bloom filter index only have "offset". But here

wgtmac · 2024-12-23T08:32:39Z

cpp/src/parquet/metadata.h

+///
+/// Number of columns with a bloom filter to be relatively small compared to
+/// the number of overall columns, so map is used.
+using RowGroupBloomFilterLocation = std::map<int32_t, IndexLocation>;


What about defining RowGroupBloomFilterLocation and FileBloomFilterLocation in the BloomFilterLocation?

I think PageIndexLocation doesn't defined them here. what about keeping it consistent?

cpp/src/parquet/metadata.cc

cpp/src/parquet/column_writer_test.cc

# Conflicts: # cpp/src/parquet/file_writer.cc # cpp/src/parquet/properties.h

wgtmac · 2025-02-06T16:20:44Z

Could you please fix the CI failure?

mapleFU · 2025-03-26T07:49:21Z

I've resolve the comments and fix the ci, would you mind re-check ? @wgtmac

wgtmac · 2025-03-27T07:32:01Z

cpp/src/parquet/bloom_filter_builder.h

+#include "arrow/io/type_fwd.h"
+#include "parquet/types.h"
+
+namespace parquet::internal {


If this is an internal namespace, how about renaming these files to bloom_filter_internal.h/cc ?

wgtmac · 2025-03-27T07:34:17Z

cpp/src/parquet/bloom_filter_builder.h

+  ///
+  /// The bloom filter cannot be modified after this method is called.
+  ///
+  /// \param[out] sink The output stream to write the bloom filter.


sink should be param[in/out] I guess?

wgtmac · 2025-03-27T07:35:03Z

cpp/src/parquet/bloom_filter_builder.h

+/// ```
+class PARQUET_EXPORT BloomFilterBuilder {
+ public:
+  /// \brief API to create a BloomFilterBuilder.


nit: explain the expecting lifetime of schema and properties.

wgtmac · 2025-03-27T07:35:44Z

cpp/src/parquet/bloom_filter_builder.h

+
+  /// \brief Get the BloomFilter from column ordinal.
+  ///
+  /// \param column_ordinal Column ordinal in schema, which is only for leaf columns.


Why some comments use \param while others use @param?

wgtmac · 2025-03-27T07:47:49Z

cpp/src/parquet/column_writer.cc

+
+template <typename ArrayType>
+void UpdateBinaryBloomFilter(BloomFilter* bloom_filter, const ArrayType& array) {
+  // Using a smaller size because an extra `byte_arrays` are used.


Suggested change

// Using a smaller size because an extra `byte_arrays` are used.

// Using a smaller size because an extra `byte_arrays` is used.

wgtmac · 2025-04-02T08:56:33Z

cpp/src/parquet/bloom_filter_builder.cc

+  bool finished_ = false;
+
+  using RowGroupBloomFilters = std::map<int32_t, std::unique_ptr<BloomFilter>>;
+  // Using unique_ptr because the `std::unique_ptr<BloomFilter>` is not copyable.


Suggested change

// Using unique_ptr because the `std::unique_ptr<BloomFilter>` is not copyable.

// Using `std::map` because `std::unique_ptr<BloomFilter>` is not copyable.

wgtmac · 2025-04-02T09:06:15Z

cpp/src/parquet/bloom_filter_builder.cc

+    return nullptr;
+  }
+  const BloomFilterOptions& bloom_filter_options = *bloom_filter_options_opt;
+  // CheckState() should have checked that file_bloom_filters_ is not empty.


I still think that we don't need to be fool-proof here especially when CheckState() has done duplicate work. Removing these DCHECK improves readability.

wgtmac · 2025-04-02T09:14:54Z

cpp/src/parquet/column_writer.cc

    }
    if (page_statistics_ != nullptr) {
      page_statistics_->UpdateSpaced(values, valid_bits, valid_bits_offset,
                                     num_spaced_values, num_values, num_nulls);
    }
    UpdateUnencodedDataBytes();
  }
+
+  void UpdateBloomFilter(const T* values, int64_t num_values);


Does it make sense to move all these UpdateXXX to the BloomFilter class? These functions look more like the detail of bloom filter instead of a column writer.

wgtmac · 2025-04-02T09:17:50Z

cpp/src/parquet/column_writer.cc

+                                                              int64_t valid_bits_offset) {
+  if (bloom_filter_) {
+    std::array<uint64_t, kHashBatchSize> hashes;
+    ::arrow::internal::VisitSetBitRunsVoid(


Same as my comment above, moving all UpdateBloomFilterXXX functions to the BloomFilter class can significantly reduce the changes in this file.

wgtmac · 2025-04-02T09:20:32Z

cpp/src/parquet/column_writer.cc

+    std::array<uint64_t, kHashBatchSize> hashes;
+    for (int64_t i = 0; i < num_values; i += kHashBatchSize) {
+      int64_t current_hash_batch_size = std::min(kHashBatchSize, num_values - i);
+      bloom_filter_->Hashes(values, static_cast<int>(current_hash_batch_size),


I just found that parquet-cpp has a lot of int num_values used in the function signature. Is it worth changing all of them to int32_t? cc @emkornfield @pitrou

Parquet: Implement skeleton for BloomFilter

f1c6dc0

mapleFU requested a review from wgtmac as a code owner August 26, 2023 14:26

github-actions bot added Component: Parquet Component: C++ awaiting review Awaiting review labels Aug 26, 2023

tiny fixing

6ebd6da

mapleFU commented Aug 26, 2023

View reviewed changes

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Aug 26, 2023

mapleFU added 2 commits August 26, 2023 23:32

tiny update test

70c9267

trying to fix ci

48350d8

mapleFU commented Aug 26, 2023

View reviewed changes

cpp/src/parquet/bloom_filter_parquet_test.cc Outdated Show resolved Hide resolved

fix lint

d2a659e

mapleFU commented Aug 26, 2023

View reviewed changes

cpp/src/parquet/metadata.h Show resolved Hide resolved

mapleFU added 4 commits August 27, 2023 00:44

fix some style problem

41236d8

add file roundtrip test

8afba81

add file roundtrip test

96c6691

fix document and ci

c131341

mapleFU commented Aug 26, 2023

View reviewed changes

cpp/src/parquet/file_writer.cc Outdated Show resolved Hide resolved

mapleFU requested review from pitrou and emkornfield August 26, 2023 18:43

Update: tiny style fix

220b58e

mapleFU changed the title ~~GH-34785: [C++][Parquet] Parquet Bloom Filter Implement~~ GH-34785: [C++][Parquet] Parquet Bloom Filter Write Implement Aug 27, 2023

wgtmac reviewed Aug 30, 2023

View reviewed changes

wgtmac changed the title ~~GH-34785: [C++][Parquet] Parquet Bloom Filter Write Implement~~ GH-34785: [C++][Parquet] Parquet Bloom Filter Writer Implementation Aug 30, 2023

pitrou requested changes Aug 31, 2023

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Sep 1, 2023

Merge branch 'main' into parquet/support-write-bloom-filter

ad96c48

mapleFU added 2 commits November 11, 2024 12:22

change the bloom filter from vector to map

22030db

fix lint

e9c550a

mapleFU force-pushed the parquet/support-write-bloom-filter branch from 505e23b to e9c550a Compare November 12, 2024 05:48

fix lint

23fb3fa

mapleFU force-pushed the parquet/support-write-bloom-filter branch from fd3856d to 79bdaeb Compare November 15, 2024 04:49

fix comment

d892819

mapleFU force-pushed the parquet/support-write-bloom-filter branch from 79bdaeb to d892819 Compare November 15, 2024 06:57

mapleFU requested review from emkornfield and wgtmac November 19, 2024 09:27

Merge branch 'main' into parquet/support-write-bloom-filter

ef3291d

# Conflicts: # cpp/src/parquet/column_writer.cc # cpp/src/parquet/type_fwd.h

wgtmac reviewed Dec 23, 2024

View reviewed changes

mapleFU added 2 commits January 13, 2025 14:33

Merge branch 'main' into parquet/support-write-bloom-filter

7aee7dd

Resolve comments

c5b1fb1

Fokko mentioned this pull request Feb 5, 2025

support bloom-filter writing apache/iceberg-python#850

Open

Merge branch 'main' into parquet/support-write-bloom-filter

0898466

# Conflicts: # cpp/src/parquet/file_writer.cc # cpp/src/parquet/properties.h

mapleFU added 7 commits February 7, 2025 14:38

Merge branch 'main' into parquet/support-write-bloom-filter

71f5906

minor fix

d57ceea

address some comments

26c2d07

Merge branch 'main' into parquet/support-write-bloom-filter

d422ffa

Minor fix

e6bc6e1

try to fix lint

dfaf0e8

Merge branch 'main' into parquet/support-write-bloom-filter

0bafe78

wgtmac requested changes Apr 2, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-34785: [C++][Parquet] Parquet Bloom Filter Writer Implementation #37400

GH-34785: [C++][Parquet] Parquet Bloom Filter Writer Implementation #37400

mapleFU commented Aug 26, 2023 •

edited by github-actions bot

Loading

mapleFU commented Aug 26, 2023 •

edited

Loading

wgtmac left a comment

mapleFU commented Nov 12, 2024

mapleFU commented Nov 19, 2024

amoeba commented Dec 19, 2024

wgtmac commented Dec 20, 2024

mapleFU commented Dec 20, 2024

wgtmac left a comment

wgtmac Dec 23, 2024

wgtmac Dec 23, 2024

wgtmac Dec 23, 2024

wgtmac Dec 23, 2024

mapleFU Feb 7, 2025

wgtmac commented Feb 6, 2025

mapleFU commented Mar 26, 2025

wgtmac Mar 27, 2025

wgtmac Mar 27, 2025

wgtmac Mar 27, 2025

wgtmac Mar 27, 2025

wgtmac Mar 27, 2025

wgtmac Apr 2, 2025

wgtmac Apr 2, 2025

wgtmac Apr 2, 2025

wgtmac Apr 2, 2025

wgtmac Apr 2, 2025

	/// Note: Before Parquet 2.10, the bloom filter index only have "offset". But here
	/// Note: Before Parquet Format v2.10, the bloom filter index only have "offset". But here

	// Using a smaller size because an extra `byte_arrays` are used.
	// Using a smaller size because an extra `byte_arrays` is used.

	// Using unique_ptr because the `std::unique_ptr<BloomFilter>` is not copyable.
	// Using `std::map` because `std::unique_ptr<BloomFilter>` is not copyable.

GH-34785: [C++][Parquet] Parquet Bloom Filter Writer Implementation #37400

Are you sure you want to change the base?

GH-34785: [C++][Parquet] Parquet Bloom Filter Writer Implementation #37400

Conversation

mapleFU commented Aug 26, 2023 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

mapleFU commented Aug 26, 2023 • edited Loading

wgtmac left a comment

Choose a reason for hiding this comment

mapleFU commented Nov 12, 2024

mapleFU commented Nov 19, 2024

amoeba commented Dec 19, 2024

wgtmac commented Dec 20, 2024

mapleFU commented Dec 20, 2024

wgtmac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wgtmac commented Feb 6, 2025

mapleFU commented Mar 26, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mapleFU commented Aug 26, 2023 •

edited by github-actions bot

Loading

mapleFU commented Aug 26, 2023 •

edited

Loading