[tempo-distributed] More autoscaling configurations #3908
Conversation
jordan-simonovski
requested review from
a team,
BitProcessor,
Sheikh-Abubaker,
faustodavid and
swartz-k
as code owners
…mponents Signed-off-by: Jordan Simonovski <[email protected]>
jordan-simonovski
force-pushed
the
autoscaling-improvements
branch
from
8b5bcfc to
0168401
Compare
Signed-off-by: Jordan Simonovski <[email protected]>
… enabled Signed-off-by: Jordan Simonovski <[email protected]>
|
I'm currently optimising our Tempo setup and have just added KEDA to the Compactor. We are also searching for a solution to automatically scale the metrics generator. Therefore, the PR would be a great help. I do have one remark, though: do you think it would make sense to allow the advanced configuration to be configured in the ScaledObject? This would also allow HPA behaviours to be set. |
I'm definitely open to this if the maintainers are happy with it as an additional config. |
| # -- Init containers for the metrics generator pod | ||
| initContainers: [] | ||
| autoscaling: | ||
| # -- Enable autoscaling for the metrics-generator. WARNING: Autoscaling metrics-generators can result in lost data. Only do this if you know what you're doing. |
There was a problem hiding this comment.
WARNING: Autoscaling metrics-generators can result in lost data.
Where can I find more info about this? I haven't been able to find any documentation about autoscaling the metrics-generator.
3 participants
Added some more autoscaling configurations to Tempo components:
Happy to discuss further. We've been running the metrics generator HPA in production without any issues.
The context behind adding KEDA support to the ingester and distributor is to allow for more flexible scaling on external metrics such as pipeline queue backlogs, spikes in ingestion volume, etc.