Skip to content

Optimize Prometheus storage and reduce Docker volume usage #88

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Optimize Prometheus storage and reduce Docker volume usage #88

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
36a630c
Optimize Prometheus storage and reduce Docker volume usage
xyephy Aug 19, 2025
d9be00d
Revert to original scrape interval times
xyephy Aug 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion docker-compose-config-a.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -418,13 +418,16 @@ services:
- "NTM_FILTERS="

prometheus:
image: prom/prometheus:v2.36.2
image: prom/prometheus:v2.47.2
volumes:
- ./prometheus/:/etc/prometheus/
- prometheus_data_config_a:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--storage.tsdb.retention.time=7d"
- "--storage.tsdb.retention.size=2GB"
Comment on lines +428 to +429
Copy link
Member

@GitGab19 GitGab19 Aug 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it mean we are going to have benchmarks data only for the latest 7 days until we reach 2GB?

Copy link
Author

@xyephy xyephy Aug 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, what it means is oldest data is cleaned up either after 7days or if data goes above 2GB, this allows for data stored to always be below 2GB of storage.

Copy link
Member

@GitGab19 GitGab19 Aug 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we cannot have data which are older than 7days with this change.

It's not what we want, since we run benchmarks for more than that.

Copy link
Author

@xyephy xyephy Aug 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for highlighting that. I got a few questions to guide in how I address this: what's the typical maximum benchmark duration? also what approach do you recommend I look into so as to make the solution more robust in addressing the concerns you've raised?

Copy link
Member

@GitGab19 GitGab19 Aug 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's not a typical duration, but we did run benchmarks for months in the past..

Tbh, I don't even know if the issue this PR wants to solve is still valid or not (it has been opened an year ago) and I feel it lacks proper context: how much time was the system running?

Did you run some tests to see how much prometheus volume grows in time?

Copy link
Author

@xyephy xyephy Aug 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the questions! Here are the test results I got when I was working on the issue last year:

Q1: How much time was the system running?

21 days continuous (October 29 - November 19, 2024)

Q2: Did you run tests to see Prometheus volume growth over time?

Yes, here are the measured results:

Day Date Prometheus Volume Daily Growth
0 Oct 29 512 MB Baseline
7 Nov 5 5,321 MB +687 MB/day
14 Nov 12 10,130 MB +687 MB/day
21 Nov 19 14,939 MB +687 MB/day

Summary:

  • Growth rate: 687 MB per day (unconstrained growth)
  • Total growth: 14.4 GB over 21 days

Based on these findings, implementing configurable retention limits instead of removing them entirely would prevent unbounded growth while giving users flexibility. Enabling WAL compression provides additional storage optimization.

Proposed implementation:

command:
  - '--config.file=/etc/prometheus/prometheus.yml'
  - '--storage.tsdb.path=/prometheus'
  - '--storage.tsdb.retention.time=${PROMETHEUS_RETENTION_TIME:-30d}'
  - '--storage.tsdb.retention.size=${PROMETHEUS_RETENTION_SIZE:-10GB}'
  - '--storage.tsdb.wal-compression'
  - '--web.enable-lifecycle'  # Allows runtime config reload

This approach would:

  • Prevent unbounded growth (default 30d/10GB limits)
  • Allow extended test periods via environment variables
  • Reduce storage overhead through WAL compression
  • Address issue Large Prometheus Docker Volume #52 concern by @average-gary about long-running tests

Would this configurable approach work better than removing all limits? Happy to update the PR with this implementation if you think it's a good middle ground.

Copy link
Member

@GitGab19 GitGab19 Aug 25, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How many miners were you running back then? Which config were you testing?

Would you run the tool for a couple of days again to see if the situation improved now?

I feel that imposing some "too strict" hard-coded boundaries could be a bad idea if the user wants to run benchmarks for more days and doesn't have problems with storage..

Copy link
Author

@xyephy xyephy Aug 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me run it over the next 7 days then give you my findings, I have some bitaxes and Apollo this time around.

- "--storage.tsdb.wal-compression"
- "--web.console.libraries=/usr/share/prometheus/console_libraries"
- "--web.console.templates=/usr/share/prometheus/consoles"
- "--web.enable-lifecycle"
Expand Down
5 changes: 4 additions & 1 deletion docker-compose-config-c.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -343,13 +343,16 @@ services:
- "NTM_FILTERS="

prometheus:
image: prom/prometheus:v2.36.2
image: prom/prometheus:v2.47.2
volumes:
- ./prometheus/:/etc/prometheus/
- prometheus_data_config_c:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--storage.tsdb.retention.time=7d"
- "--storage.tsdb.retention.size=2GB"
- "--storage.tsdb.wal-compression"
- "--web.console.libraries=/usr/share/prometheus/console_libraries"
- "--web.console.templates=/usr/share/prometheus/consoles"
- "--web.enable-lifecycle"
Expand Down
Loading