Describe the bug
If the s3 storage sends http 429 with "SlowDown" and "Reduced your rewuest rate" the thanos Client continues to Hammer the s3 leading to more 429 SlowDown which makes the Situation worse than better.
To Reproduce
Steps to reproduce the behavior:
- Started Loki 3.7.3 monolithic
- Created many requests to s3 and Run into rate limit
Expected behavior
The thanos Object Store Client should use a backoff mechanism to Slow down and wir If Errors Like 429 SlowDown are received to give the s3 the Chance to unblock the Client.
Further our local Nutanix s3 May send Errors Like http 503 "StorageFull" and s3 still Hammers the s3. It should wait and slow down to Reduce Error rate.
Environment:
- Infrastructure: binary, baremetal
- Deployment tool: binary, monolithic
Screenshots, Alloy config, or terminal output
N/A
Describe the bug
If the s3 storage sends http 429 with "SlowDown" and "Reduced your rewuest rate" the thanos Client continues to Hammer the s3 leading to more 429 SlowDown which makes the Situation worse than better.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The thanos Object Store Client should use a backoff mechanism to Slow down and wir If Errors Like 429 SlowDown are received to give the s3 the Chance to unblock the Client.
Further our local Nutanix s3 May send Errors Like http 503 "StorageFull" and s3 still Hammers the s3. It should wait and slow down to Reduce Error rate.
Environment:
Screenshots, Alloy config, or terminal output
N/A