feat: Use S3 node store with seaweedfs #3498

BYK · 2024-12-31T13:18:43Z

Enables S3 node store using SeaweedFS and sentry-nodestore-s3 by @stayallive

This should alleviate all the issues stemming from (ab)using PostgreSQL as the node store.

codecov · 2024-12-31T13:54:45Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.49%. Comparing base (2be8c79) to head (27dce85).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3498   +/-   ##
=======================================
  Coverage   99.49%   99.49%           
=======================================
  Files           3        3           
  Lines         197      197           
=======================================
  Hits          196      196           
  Misses          1        1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

aldy505 · 2024-12-31T14:47:08Z

Any reason why you didn't use SeaweedFS per what you said yesterday?

garage.toml

install/bootstrap-garage.sh

docker-compose.yml

sentry/sentry.conf.example.py

install/bootstrap-garage.sh

Co-authored-by: Reinaldy Rafli <[email protected]>

docker-compose.yml

BYK · 2024-12-31T21:08:59Z

@aldy505

Any reason why you didn't use SeaweedFS per what you said yesterday?

Well I started with that and realized 3 things:

It really is not geared towards single-node setups and have nodes with different roles. This makes is more challenging to scale up or set up in our setup
It has this paid admin interface. Not a deal breaker but it is clear that it is geared towards more "professional" setups
Its S3 API interface support is not really great

Garage fits the bill much better as it is explicitly created for smaller setups like this, easy to expand without specialized roles, doesn't have any paid thing in it, and has much more decent and familiar S3 interface support.

doc-sheet · 2025-05-25T09:29:51Z

It really is not geared towards single-node setups and have nodes with different roles. This makes is more challenging to scale up or set up in our setup

when I tried seaweedfs last time (and I still use it for sourcemap/profile storage tbh) it had single node ability via weed server command.
Like

weed server -filter=true -s3=true -master=true -volume=true

Some of them enabled by default.

doc-sheet · 2025-05-25T09:33:02Z

I think garage/minio simpler for small setups, seaweedfs looks necessary for mid to high setups because all other services I know keep files as is.

And thousands of thousands small files like profiles not ideal to store on most popular filesystems i guess.

aldy505 · 2025-06-04T01:24:30Z

I think garage/minio simpler for small setups, seaweedfs looks necessary for mid to high setups because all other services I know keep files as is.

@doc-sheet Hey, I'm going to work on this PR. I'd think seaweed is better for self-hosted Sentry. One thing I don't like about Garage is that we need to specify the storage allocation beforehand, if we set it to 100GB, there might be some people that have more data than 100GB, I don't want that to cause any issues.

That said, since you said you've used seaweed before: How was your experience? How does it compare to MinIO or Ceph?

And thousands of thousands small files like profiles not ideal to store on most popular filesystems i guess.

Yeah if we set up an object storage, we might as well move filestore & profiles there too. But let's focus on nodestore first.

doc-sheet · 2025-06-04T07:23:55Z

How was your experience? How does it compare to MinIO or Ceph?

It is a bit strange sometimes. But it is fine.

It has multiple options for filer store.
I didn't try leveldb storage aiming to fault tolerance.

At first I tried redis it worked for several months and then... I just lost all data.
It was there physically but wasn't available from API (s3 or web) - each list call returned different results.

I don't know if issue was in redis or weed itself. i suspect bug with ttl could be the reason too.

But after that incident I wiped cluster and started new one with scylla as a filer backend and it works fine for almost a year already despite that ttl bug.

Seaweedfs have multiple versions like

3.89
3.89_full
3.89_large_disk
3.89_large_disk_full

I suggest to use large_disk always. Documentation is not clear but it is easy to reach that limit
https://github.com/seaweedfs/seaweedfs/wiki/FAQ#how-to-configure-volumes-larger-than-30gb

I don't know difference between full and normal and just use _large_disk_full builds :)

Also I don't use s3 auth - I was too lazy to set it up.

Other than all that I have no problems and barely touched it after initial setup. It just works.
I added some volumes but not removed any yet.

As for minio and ceph.
I never used ceph.

But minio was the reason to look for alternatives.

Tons of profiles from js-sdk stored as different files started to affect my monitoring script and soon it might start to affect minio performance too.

And it is not that easy to scale minio. And probably impossible to optimize for small-files storage. At least in my low-cost setup.

doc-sheet · 2025-06-04T07:31:32Z

let's focus on nodestore first.

If seaweedfs would control ttl then there is another catch.
I'm not sure if it is possible to control ttl with s3-api already.

weed have it's own settings for collections and it creates collection for each s3-bucket.
https://github.com/seaweedfs/seaweedfs/wiki/S3-API-FAQ#setting-ttl

But if sentry itself would cleanup old data I guess there is no difference.

aldy505 · 2025-06-05T05:49:36Z

How was your experience? How does it compare to MinIO or Ceph?

It is a bit strange sometimes. But it is fine.

It has multiple options for filer store. I didn't try leveldb storage aiming to fault tolerance.

At first I tried redis it worked for several months and then... I just lost all data. It was there physically but wasn't available from API (s3 or web) - each list call returned different results.

I don't know if issue was in redis or weed itself. i suspect bug with ttl could be the reason too.

But after that incident I wiped cluster and started new one with scylla as a filer backend and it works fine for almost a year already despite that ttl bug.

Seaweedfs have multiple versions like

3.89

3.89_full

3.89_large_disk

3.89_large_disk_full

I suggest to use large_disk always. Documentation is not clear but it is easy to reach that limit https://github.com/seaweedfs/seaweedfs/wiki/FAQ#how-to-configure-volumes-larger-than-30gb

I don't know difference between full and normal and just use _large_disk_full builds :)

Also I don't use s3 auth - I was too lazy to set it up.

Other than all that I have no problems and barely touched it after initial setup. It just works. I added some volumes but not removed any yet.

Good to know about Seaweed

As for minio and ceph. I never used ceph.

But minio was the reason to look for alternatives.

Tons of profiles from js-sdk stored as different files started to affect my monitoring script and soon it might start to affect minio performance too.

And it is not that easy to scale minio. And probably impossible to optimize for small-files storage. At least in my low-cost setup.

Ah so everyone has the same experience with minio.

let's focus on nodestore first.

If seaweedfs would control ttl then there is another catch. I'm not sure if it is possible to control ttl with s3-api already.

weed have it's own settings for collections and it creates collection for each s3-bucket. https://github.com/seaweedfs/seaweedfs/wiki/S3-API-FAQ#setting-ttl

But if sentry itself would cleanup old data I guess there is no difference.

The sentry cleanup job only cleans up the one on filesystem. If we're using S3, it won't clean up anything. We need to configure S3 data cleanup on our own.

doc-sheet · 2025-06-06T20:24:16Z

Looks like i missed that seaweedfs now have an ability to control ttl with s3 api. And I even linked to correct section of FAQ. :)

I'd like to look into new integraton with seaweedfs.

Nad by the way I like the idea of expanding sentry images.

I am myself install some packages and modules.

Like maybe an extra step in install to build user provided Dockerfiles.

aldy505 · 2025-08-06T13:33:57Z

Integration tests isn't passing. I think we should hold this off for a bit.

hubertdeng123 · 2025-08-06T18:20:26Z

@aldy505 I also noticed that recently we've added objectstore. Perhaps related here?
getsentry/sentry#97271

aldy505 · 2025-08-07T03:51:51Z

@aldy505 I also noticed that recently we've added objectstore. Perhaps related here? getsentry/sentry#97271

@hubertdeng123 I asked Jan last week, it's not being used on SaaS yet. Quoting him:

Right, we're planning to make this an intermediary layer to some backend - we do not have a strong story for self hosted yet. We wouldn't be using postgres for sure, instead offer two alternatives: Any S3-compatible backend or raw disk.
Our first use case is event attachments followed, by release files and debug files. We do consider replacing nodestore, but it's not on our roadmap yet. Will likely take months to get to the point where we can plan that .

Turns out the arm64 runners already have 3GB of swap

aldy505 · 2025-08-17T09:12:42Z

Yay it's green.

One bad thing is that we need to tell people to have at least another 16GB set aside for swapfile.

aldy505 · 2025-08-17T09:14:50Z

Ah right, we're missing the lifecycle thing.

Previously, I've seen folks that increase the number of retention days. So if we want to keep that behaviour, it wouldn't be possible by setting S3 retention days (and modifying it every once in a while during installation). I can think of setting 1 cron container, but would it be a sensible thing to do?

BYK · 2025-08-18T22:39:42Z

@aldy505

We should have a proper migration path for existing installs

Don't think we have this either?

aldy505 · 2025-09-06T06:44:06Z

Config migration is done and is behind a prompt/flag. Next up is to think how to manage the retention. Should we do it with a cron/scheduled job (cons: probably heavy process, pros: can be configured dynamically), or a S3 lifecycle (cons: can't be configured dynamically, pros: won't be a heavy process).

aldy505 · 2025-09-09T14:20:43Z

I'm just gonna go forward with this https://github.com/seaweedfs/seaweedfs/wiki/S3-API-FAQ#setting-ttl

This reverts commit 2f1575d.

aldy505 · 2025-09-09T15:22:04Z

Great, I believe that's all.

aldy505 · 2025-09-09T15:22:47Z

@aminvakil @doc-sheet Hi, would you mind reviewing this PR?

aldy505

Unblocking myself.

BYK

Arguing with myself 😝 (comments for @aldy505 )

BYK · 2025-09-11T11:07:41Z

install/bootstrap-s3-nodestore.sh

@@ -0,0 +1,90 @@
+echo "${_group}Bootstrapping seaweedfs (node store)..."
+
+$dc up --wait seaweedfs postgres


Does the --wait thing work for podman?

Good point. Will check later.

BYK · 2025-09-11T11:09:14Z

install/bootstrap-s3-nodestore.sh

+
+bucket_list=$($s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' ls)
+
+if [[ $($bucket_list | tail -1 | awk '{print $3}') != 's3://nodestore' ]]; then


This if condition really needs some explanation about what it is checking and how -- like what are the assumptions here?

BYK · 2025-09-11T11:10:29Z

sentry/sentry.conf.example.py

+# Other backend implementations for node storage developed by the community
+# are available in public GitHub repositories.
+
+SENTRY_NODESTORE = "sentry_nodestore_s3.S3PassthroughDjangoNodeStorage"


Do we need Passthrough for fresh installs?

This is your original code, so I assume, yes.

I would not assume anything about my original code 😅

BYK · 2025-09-11T11:11:46Z

install/bootstrap-s3-nodestore.sh

+        read -p "y or n? " yn
+        case $yn in
+        y | yes | 1)
+          export APPLY_AUTOMATIC_CONFIG_UPDATES=1


This feels dangerous as once you export it like this wouldn't it also enable auto update for pgbouncer? I think that should be a separate questions and decision?

Ah good point.

BYK · 2025-09-11T11:12:07Z

install/bootstrap-s3-nodestore.sh

+        n | no | 0)
+          export APPLY_AUTOMATIC_CONFIG_UPDATES=0
+          echo
+          echo -n "Alright, you will need to update your sentry.conf.py file manually before running 'docker compose up'."


Again, provide details and link to this PR (or some doc). Also skip setting up seaweed at this point or at least mention that the service will be running for no good reason?

BYK · 2025-09-11T11:13:32Z

install/bootstrap-s3-nodestore.sh

+
+    if [[ "$APPLY_AUTOMATIC_CONFIG_UPDATES" == 1 ]]; then
+      nodestore_config=$(sed -n '/SENTRY_NODESTORE/,/[}]/{p}' sentry/sentry.conf.example.py)
+      if [[ $($dc exec postgres psql -qAt -U postgres -c "select exists (select * from nodestore_node limit 1)") = "f" ]]; then


Up until this point you don't need the postgres service so maybe instead of bringing it up at the beginning, just use $dcr postgres psql ... here?

BYK · 2025-09-11T11:14:49Z

install/bootstrap-s3-nodestore.sh

+  $dc exec seaweedfs mkdir -p /data/idx/
+  $s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' mb s3://nodestore


Why are these commands repeated (from lines 6 and 7 above)?

BYK · 2025-09-11T11:15:39Z

install/bootstrap-s3-nodestore.sh

+  # XXX(aldy505): Should we refactor this?
+  lifecycle_policy=$(
+    cat <<EOF
+<?xml version="1.0" encoding="UTF-8"?>
+<LifecycleConfiguration>
+    <Rule>
+        <ID>Sentry-Nodestore-Rule</ID>
+        <Status>Enabled</Status>
+        <Filter></Filter>
+        <Expiration>
+            <Days>$SENTRY_EVENT_RETENTION_DAYS</Days>
+        </Expiration>
+    </Rule>
+</LifecycleConfiguration>
+EOF
+  )
+  $dc exec seaweedfs sh -c "printf '%s' '$lifecycle_policy' > /tmp/nodestore-lifecycle-policy.xml"


Yes, make it a config file and mount it? Is this because of the $SENTRY_EVENT_RETENTION_DAYS variable? If yes, we may wanna construct a custom entry point script for seaweed so when people change this value and restart, it is updated without running ./install.sh again?

Restarting seaweed wouldn't take anything into effect. You will need to execute s3cmd setlifecycle for that

And I'm saying we should put whatever is necessary into a custom entrypoint script for seaweed so it picks the changes up?

I don't like that approach, it complicates stuff.

Okay, until someone compalins let's go with the simpler approach :)

BYK · 2025-09-11T11:19:06Z

install/bootstrap-s3-nodestore.sh

+  echo "Making sure the bucket lifecycle policy is all set up correctly..."
+  $s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' getlifecycle s3://nodestore


This seems it belongs to tests instead of here? Is this because of the unknown env variable value?

aminvakil

Can we make it opt-in for a release? And get some feedback regarding this change?

BYK added 6 commits December 31, 2024 16:18

feat: Use S3 node store with garage

364455b

lol, fix bash

803bef3

moar bash

58b301c

lol

bd414d5

hate bash

a2ac10d

fix moar bash

7b28dfb

aldy505 reviewed Dec 31, 2024

View reviewed changes

garage.toml Outdated Show resolved Hide resolved

garage.toml Outdated Show resolved Hide resolved

install/bootstrap-garage.sh Outdated Show resolved Hide resolved

docker-compose.yml Outdated Show resolved Hide resolved

sentry/sentry.conf.example.py Outdated Show resolved Hide resolved

aldy505 reviewed Dec 31, 2024

View reviewed changes

install/bootstrap-garage.sh Outdated Show resolved Hide resolved

BYK and others added 2 commits December 31, 2024 15:23

Add healthcheck to garage service

4a6d337

Co-authored-by: Reinaldy Rafli <[email protected]>

revert +x

595ed68

hubertdeng123 reviewed Dec 31, 2024

View reviewed changes

docker-compose.yml Outdated Show resolved Hide resolved

docker-compose.yml Outdated Show resolved Hide resolved

BYK added 2 commits December 31, 2024 23:43

fix healthcheck, fix config

99576bf

add env var for garage size

2a55a40

BYK added 4 commits January 1, 2025 00:22

use better compression level

a29cf0b

simpler garage config

a0ae9b1

Merge branch 'master' into byk/feat/s3-nodestore

20e608b

add migration support

8d7c1ff

aldy505 mentioned this pull request Jan 5, 2025

Cleaning nodestore_node table #1808

Open

This was referenced Mar 18, 2025

Ongoing Work & Feedback #3625

Open

Redundancy for a self-host installation running on a linux VM on Azure #3626

Closed

aldy505 added 5 commits August 13, 2025 10:19

Merge branch 'master' into byk/feat/s3-nodestore

7af258a

Merge branch 'master' into byk/feat/s3-nodestore

9c1c324

chore: add swap for arm64 runners

61e0684

ci: debug memory issues for arm64 runners

8db8b5c

ci: turn off swapfile first

ca22d67

Turns out the arm64 runners already have 3GB of swap

aldy505 requested review from hubertdeng123 and aminvakil August 17, 2025 09:12

aldy505 added 3 commits September 3, 2025 07:26

Merge branch 'master' into byk/feat/s3-nodestore

3a95a77

Merge remote-tracking branch 'origin/master' into byk/feat/s3-nodestore

75a9ca5

feat: nodestore config update behind a prompt/flag

81cde28

aldy505 added 7 commits September 9, 2025 21:20

Merge branch 'master' into byk/feat/s3-nodestore

5aa35db

feat: set s3 lifecycle policy

7eb223c

fix: seaweed is a busybox

fa3fc92

fix: try xml policy

ace9191

fix: go back to simplified json

2f1575d

Revert "fix: go back to simplified json"

5753667

This reverts commit 2f1575d.

chore: reword debug lifecycle policy

27dce85

aldy505 approved these changes Sep 11, 2025

View reviewed changes

BYK commented Sep 11, 2025

View reviewed changes

aminvakil reviewed Sep 11, 2025

View reviewed changes

		@@ -0,0 +1,90 @@
		echo "${_group}Bootstrapping seaweedfs (node store)..."

		$dc up --wait seaweedfs postgres


		bucket_list=$($s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' ls)

		if [[ $($bucket_list \| tail -1 \| awk '{print $3}') != 's3://nodestore' ]]; then

		$dc exec seaweedfs mkdir -p /data/idx/
		$s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' mb s3://nodestore

		echo "Making sure the bucket lifecycle policy is all set up correctly..."
		$s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' getlifecycle s3://nodestore

Uh oh!

feat: Use S3 node store with seaweedfs #3498

Are you sure you want to change the base?

feat: Use S3 node store with seaweedfs #3498

Uh oh!

Conversation

BYK commented Dec 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

aldy505 commented Dec 31, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BYK commented Dec 31, 2024

Uh oh!

doc-sheet commented May 25, 2025

Uh oh!

doc-sheet commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aldy505 commented Jun 4, 2025

Uh oh!

doc-sheet commented Jun 4, 2025

Uh oh!

doc-sheet commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aldy505 commented Jun 5, 2025

Uh oh!

doc-sheet commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aldy505 commented Aug 6, 2025

Uh oh!

hubertdeng123 commented Aug 6, 2025

Uh oh!

aldy505 commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aldy505 commented Aug 17, 2025

Uh oh!

aldy505 commented Aug 17, 2025

Uh oh!

BYK commented Aug 18, 2025

Uh oh!

aldy505 commented Sep 6, 2025

Uh oh!

aldy505 commented Sep 9, 2025

Uh oh!

aldy505 commented Sep 9, 2025

Uh oh!

aldy505 commented Sep 9, 2025

Uh oh!

aldy505 left a comment

Choose a reason for hiding this comment

Uh oh!

BYK left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

BYK commented Dec 31, 2024 •

edited

Loading

codecov bot commented Dec 31, 2024 •

edited

Loading

doc-sheet commented May 25, 2025 •

edited

Loading

doc-sheet commented Jun 4, 2025 •

edited

Loading

doc-sheet commented Jun 6, 2025 •

edited

Loading

aldy505 commented Aug 7, 2025 •

edited

Loading