Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Environment and Versions
aws-sdk-go-base: v2.0.0-beta.53
Expected Behavior
Logging of HTTP requests or responses shouldn't lead to excessive memory usage.
Actual Behavior
When creating or updating Lambdas with the code as a Zip file, the base64 encoded archive gets included in the request.
The request body gets copied and converted to a string no matter what log level is set (https://github.com/hashicorp/aws-sdk-go-base/blob/main/logger.go#L136).
Under the hood this calls out to the stdlib's httputil.DumpRequestOut https://github.com/hashicorp/aws-sdk-go-base/blob/main/logging/http.go#L119.
httputil.DumpRequestOut allocates memory for the whole body, just so it's truncated to maxRequestBodyLen (1024 chars) later (see: https://github.com/hashicorp/aws-sdk-go-base/blob/main/logging/http.go#L134).
When creating a lot of Lambda functions with inline code archives, this can drastically increase the memory usage.
What makes this worse is that the higher memory pressure can lead to request failures and retries on machines with lower amounts of memory. When this happens more requests (because of the retries) get logged and more memory is allocated. I believe this might be the root cause of: hashicorp/terraform#9364.
I am investigating the removal of the limitation introduced in hashicorp/terraform#9364, that forces Lambdas to be created in serial.
While testing with 50 Lambdas a 25MB I was observing the memory usage of the provider using pprof. I found that more memory was used on logging the Lambda requests in (*defaultRequestBodyLogger).Log (~3.8GB) then to serialize the bodies in awsRestjson1_serializeDocumentFunctionCode.
S3 PutObject & UploadPart have a similar problem and for this the sdk wrapper has a special request logger that redacts the body: https://github.com/hashicorp/aws-sdk-go-base/blob/main/logging/http.go#L148. I think doing the same thing for Lambda CreateFunction & UpdateFunctionCode would improve the memory usage of the provider and provide a path to reverting on the decision in hashicorp/terraform#9364 and ultimately speeding up Lambda function creation/updates.
Steps to Reproduce
- Create a lambda function with a big inline code archive (e.g. 25MB).
- Run
terraform apply and attach pprof to provider
curl http://localhost:6060/debug/pprof/heap > heap.out to take a heap dumpgo tool pprof heap.out to analyze the heap dump (e.g. web)
Debug Output
n/a
References
Community Note
Environment and Versions
aws-sdk-go-base:v2.0.0-beta.53Expected Behavior
Logging of HTTP requests or responses shouldn't lead to excessive memory usage.
Actual Behavior
When creating or updating Lambdas with the code as a Zip file, the base64 encoded archive gets included in the request.
The request body gets copied and converted to a string no matter what log level is set (https://github.com/hashicorp/aws-sdk-go-base/blob/main/logger.go#L136).
Under the hood this calls out to the stdlib's
httputil.DumpRequestOuthttps://github.com/hashicorp/aws-sdk-go-base/blob/main/logging/http.go#L119.httputil.DumpRequestOutallocates memory for the whole body, just so it's truncated tomaxRequestBodyLen(1024 chars) later (see: https://github.com/hashicorp/aws-sdk-go-base/blob/main/logging/http.go#L134).When creating a lot of Lambda functions with inline code archives, this can drastically increase the memory usage.
What makes this worse is that the higher memory pressure can lead to request failures and retries on machines with lower amounts of memory. When this happens more requests (because of the retries) get logged and more memory is allocated. I believe this might be the root cause of: hashicorp/terraform#9364.
I am investigating the removal of the limitation introduced in hashicorp/terraform#9364, that forces Lambdas to be created in serial.
While testing with 50 Lambdas a 25MB I was observing the memory usage of the provider using
pprof. I found that more memory was used on logging the Lambda requests in(*defaultRequestBodyLogger).Log(~3.8GB) then to serialize the bodies inawsRestjson1_serializeDocumentFunctionCode.S3 PutObject & UploadPart have a similar problem and for this the sdk wrapper has a special request logger that redacts the body: https://github.com/hashicorp/aws-sdk-go-base/blob/main/logging/http.go#L148. I think doing the same thing for Lambda CreateFunction & UpdateFunctionCode would improve the memory usage of the provider and provide a path to reverting on the decision in hashicorp/terraform#9364 and ultimately speeding up Lambda function creation/updates.
Steps to Reproduce
terraform applyand attach pprof to providercurl http://localhost:6060/debug/pprof/heap > heap.outto take a heap dumpgo tool pprof heap.outto analyze the heap dump (e.g.web)Debug Output
n/a
References