Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(lambda-promtail): allow setting Lambda reserved concurrency #16783

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

sapslaj
Copy link
Contributor

@sapslaj sapslaj commented Mar 17, 2025

What this PR does / why we need it:

TLDR: We were running into issues with lambda-promtail eating up all of our Lambda function concurrency and adding reserved concurrency will mitigate that.

For the uninitiated, from https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html

Reserved concurrency – This represents the maximum number of
concurrent instances allocated to your function. When a function has
reserved concurrency, no other function can use that concurrency.
Reserved concurrency is useful for ensuring that your most critical
functions always have enough concurrency to handle incoming requests.
Configuring reserved concurrency for a function incurs no additional
charges.

Given this, being able to configure reserved concurrency will help in AWS accounts that utilize Lambda heavily (not just for lambda-promtail) in a few ways. Since reserved concurrency is the max concurrency a function can run, it means that a deluge of events being sent to lambda-promtail won't exhaust all of the Lambda concurrency for the entire AWS account's region. On the other hand, since setting reserved concurrency means that function will always have that amount of concurrency reserved just for it, it can mitigate issues with delayed log messages during periods of heavily Lambda usage since the lambda-promtail function will be guaranteed to be able to handle N number of events (N being the reserved concurrency) number of events at any given time. My company was mainly having issues with the former where lambda-promtail would eat up all of our Lambda function concurrency and causing Lambda invokes to fail causing error rates for our entire platform to spike (not good!).

The default behavior in the Terraform AWS provider is to not set any reserved concurrency. It uses the -1 value to denote this. To preserve that behavior, the default value for the new input variable is -1.

For the keen-eyed, provisioned concurrency is a different thing to reserved concurrency. Reserved concurrency is configured on the aws_lambda_function resource, but provisioned concurrency is configured via a separate resource
aws_lambda_provisioned_concurrency_config. That distinction is the main reason this PR exists, since the Terraform module in this repo manages the aws_lambda_function resource and currently does not expose the reserved_concurrent_executions attribute.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

For the uninitiated, from https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html

> Reserved concurrency – This represents the maximum number of
> concurrent instances allocated to your function. When a function has
> reserved concurrency, no other function can use that concurrency.
> Reserved concurrency is useful for ensuring that your most critical
> functions always have enough concurrency to handle incoming requests.
> Configuring reserved concurrency for a function incurs no additional
> charges.

Given this, being able to configure reserved concurrency will help in
AWS accounts that utilize Lambda heavily (not just for lambda-promtail)
in a few ways. Since reserved concurrency is the max concurrency a
function can run, it means that a deluge of events being sent to
lambda-promtail won't exhaust all of the Lambda concurrency for the
entire AWS account's region. On the other hand, since setting reserved
concurrency means that function will always have that amount of
concurrency reserved just for it, it can mitigate issues with delayed
log messages during periods of heavily Lambda usage since the
lambda-promtail function will be guaranteed to be able to handle N
number of events (N being the reserved concurrency) number of events at
any given time.

The default behavior in the Terraform AWS provider is to _not_ set any
reserved concurrency. It uses the `-1` value to denote this. To preserve
that behavior, the default value for the new input variable is `-1`.

For the keen-eyed, _provisioned_ concurrency is a different thing to
_reserved_ concurrency. _Reserved_ concurrency is configured on the
`aws_lambda_function` resource, but _provisioned_ concurrency is
configured via a separate resource
`aws_lambda_provisioned_concurrency_config`.
@sapslaj sapslaj requested a review from a team as a code owner March 17, 2025 13:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant