amplify push failed with Too Many Requests error

[kouz75]

How did you install the Amplify CLI?

yarn

If applicable, what version of Node.js are you using?

v20.12.2

Amplify CLI Version

12.14.3

What operating system are you using?

mac

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

Describe the bug

Hi,

We are not able to deploy our project anymore.
We have update our GraphQL schema (add some tables).
During "amplify push" CloudFormation failed to update the stack with following error:
Too Many Requests (Service: AppSync, Status Code: 429, Request ID: ff83079c-06e2-4015-9beb-39641616f185)" (RequestToken: 50759b9e-1c38-ad42-8951-b7f1c3e02962, HandlerErrorCode: GeneralServiceException)

Is there a way to reduce the number of requests send to AppSync during Stack update ?

thanks for your help

Expected behavior

Reproduction steps

Project Identifier

Project Identifier: ed57bb8a46476511162afa8497d78147

Log output

# Put your logs below this line

Additional information

No response

Before submitting, please confirm:

• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I have removed any sensitive information from my code snippets and submission.

[kouz75]

After several attempts, I was able to apply the change

[ykethan]

Hey, thanks for raising this! I'm going to transfer this over to our API repository for better assistance.

[sundersc]

There are two control-plane APIs (create-function and update-function) that contribute to these throttling errors during stack update. Both these APIs have throttle limits of 70 tps each per accountId. Each user gets 100 tps across all operations (gets, create, update).

The resolvers and functions are spread across multiple child stacks and each of these child stacks have dependencies on the underlying datasource (DynamoDb tables in this case). So they naturally provide a gate to limit the number of simultaneous API calls.

We will continue to monitor and determine if any adjustments are required to the limits. Please feel free to reopen the issue if you encounter the issue again.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

[AnilMaktala]

The issue has been reopened as it is still unresolved.

[bobbyu99]

@kouz75 How often does this issue occur/not occur? How many tables does your schema have? Is it possible to share your schema? It could be helpful for reproducing this issue.

[kouz75]

Hi,

Today, I was able to push the schema once, but it failed many times...

The schema contains 37 tables.
->There are 37 CloudFormation stacks managing these tables.
Each CloudFormation stack has approximately 40 AppSync resources (Resolver/Function configurations).
->The project contains around 1,500 AppSync resources that need to be updated when I push the project.
All these updates are performed in parallel (during a sort periode), which may explain the throttling issue ?

[kouz75]

As a temporary solution, I removed subscriptions on most tables (added subscriptions: null on each table).
This helps reduce the number of API calls to AppSync.

[AnilMaktala]

@kouz75, Glad to hear that disabling subscriptions resolved the issue. We are actively investigating the problem on our end and will keep you updated once we have more information.

[sundersc]

@kouz75 As mentioned earlier, AppSync throttling limits are account-specific, which can sometimes impact deployment performance. If multiple developers are working in the same AWS account and running concurrent sandbox or branch deployments, this might contribute to throttling issues.

To help optimize your deployment experience, we recommend:

• Sequencing deployments: Try running one deployment at a time per account
• Streamlining your GraphQL API: Consider removing unused models and disabling unnecessary operations (mutations, queries, and subscriptions)

These strategies can:
✓ Reduce the number of resources Amplify generates
✓ Minimize API calls from CloudFormation to AppSync during deployment
✓ Improve overall deployment efficiency

[sundersc]

Related Note: If you are using Gen2 sandboxes, we have observed an issue with AppSync functions hotswap that is making redundant API calls. We have a PR open with CDK team to fix it.
aws/aws-cdk-cli#99

[kouz75]

Hi,

I understand all the limitations, but we have one project with a single GraphQL schema, and we are already running only one deployment at a time. We don't use Gen2 sandboxes.

We can remove all unnecessary operations (subscriptions), but removing queries/mutations doesn’t make sense.
I'm not sure that a schema with 35 tables and one or two extra indexes per table should cause this kind of issue.

I believe the amplify push command should handle the throttling issue.