Skip to content

Conversation

@SkyFan2002
Copy link
Member

@SkyFan2002 SkyFan2002 commented Nov 12, 2025

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

After #18893, TPC-H SF1000 performance degraded, with the main bottleneck being the slow construction of runtime filters. This PR makes the following optimizations:

  1. In the runtime filter implementation, the XOR filter was replaced with a blocked Bloom filter to improve build performance.
  2. Support concurrent construction of Bloom runtime filters.

The overhead of runtime filters has been greatly reduced, but they are still far from fast enough; follow-up PRs will continue to improve this.

╭──────────────────────────────────────────────────────────────────────────╮
│ query_id │        main       │         pr        │          diff         │
│   UInt8  │ Nullable(Float32) │ Nullable(Float32) │   Nullable(Float64)   │
├──────────┼───────────────────┼───────────────────┼───────────────────────┤
│        114.29114.6160.3249998092651367 │
│        26.6526.7790.1269998550415039 │
│        391.26453.25-38.013999938964844 │
│        458.28242.258-16.024002075195313 │
│        547.31637.601-9.71500015258789 │
│        610.93710.9720.034999847412109375 │
│        775.86746.802-29.064998626708984 │
│        849.21740.354-8.862998962402344 │
│        9386.424132.262-254.16201782226563 │
│       1085.13560.513-24.62200164794922 │
│       115.1145.3230.2090001106262207 │
│       1252.66547.336-5.329002380371094 │
│       1352.58650.265-2.3209991455078125 │
│       1448.73331.97-16.763002395629883 │
│       1525.56526.1330.5679988861083984 │
│       167.1946.877-0.3169999122619629 │
│       1738.35938.301-0.058002471923828125 │
│       18132.159135.3473.1880035400390625 │
│       1936.05136.8520.8010025024414063 │
│       2027.72127.316-0.4050006866455078 │
│       21263.919147.043-116.87600708007813 │
│       2210.36410.223-0.14100074768066406 │
╰──────────────────────────────────────────────────────────────────────────╯
22 rows read in 0.987 sec. Processed 46 rows, 42.99 KiB (46.61 rows/s, 43.56 KiB/s)

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-chore this PR only has small changes that no need to record, like coding styles. label Nov 12, 2025
@SkyFan2002 SkyFan2002 added the ci-cloud Build docker image for cloud test label Nov 12, 2025
@github-actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-18955-cd05e00-1762921234

note: this image tag is only available for internal use.

@SkyFan2002 SkyFan2002 closed this Nov 15, 2025
@SkyFan2002 SkyFan2002 force-pushed the improve_runtime_filter_2 branch from a670615 to e9cc22c Compare November 15, 2025 08:26
@SkyFan2002 SkyFan2002 reopened this Nov 15, 2025
@SkyFan2002 SkyFan2002 changed the title chore: replace HashSet with Vec for bloom filter construction feat: improve runtime filter [Part 2] Nov 15, 2025
@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Nov 15, 2025
@SkyFan2002 SkyFan2002 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Nov 15, 2025
@github-actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-18955-5b31dd6-1763224263

note: this image tag is only available for internal use.

@SkyFan2002 SkyFan2002 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Nov 17, 2025
@github-actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-18955-bbf2025-1763402915

note: this image tag is only available for internal use.

@SkyFan2002 SkyFan2002 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Nov 18, 2025
@SkyFan2002 SkyFan2002 force-pushed the improve_runtime_filter_2 branch from 27a02c0 to e2bddfb Compare November 18, 2025 01:20
@SkyFan2002 SkyFan2002 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Nov 18, 2025
@github-actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-18955-20895c0-1763430626

note: this image tag is only available for internal use.

@github-actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-18955-dfdb6a4-1763431711

note: this image tag is only available for internal use.

@SkyFan2002 SkyFan2002 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Nov 18, 2025
@github-actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-18955-1f06bbb-1763495247

note: this image tag is only available for internal use.

@SkyFan2002 SkyFan2002 marked this pull request as ready for review November 19, 2025 09:21
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@zhang2014
Copy link
Member

before:
image
after:
image

@zhang2014 zhang2014 merged commit 82f7c9e into databendlabs:main Nov 20, 2025
96 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-cloud Build docker image for cloud test pr-chore this PR only has small changes that no need to record, like coding styles. pr-feature this PR introduces a new feature to the codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants