backend-deployer 2.2.0 pins @aws-cdk/toolkit-lib 1.32.0, which forces every sandbox hotswap-fallback deploy into CloudFormation EXPRESS mode with rollback disabled (reverted upstream in toolkit-lib 1.38.1) #3327

Environment information

System:
  OS: Windows 11 10.0.26200
Binaries:
  Node: 24.15.0
  npm: 11.12.1
NPM Packages:
  @aws-amplify/backend: 1.24.0
  @aws-amplify/backend-cli: 1.9.0
  @aws-amplify/backend-deployer: 2.2.0 (nested under backend-cli and sandbox)
  @aws-amplify/data-schema: 1.26.0
  @aws-amplify/plugin-types: 1.12.2
  @aws-amplify/sandbox: 2.3.0
  @aws-cdk/toolkit-lib: 1.32.0 (nested under backend-cli / sandbox / plugin-types; the root hoists 1.19.0, which is what `ampx info` reports)
  aws-amplify: 6.20.0
  aws-cdk-lib: 2.259.0
  typescript: 6.0.3

Describe the bug

Since @aws-amplify/backend-cli 1.9.0 / @aws-amplify/backend-deployer 2.2.0, every ampx sandbox deploy that cannot be hotswapped (any change other than Lambda code) is sent to CloudFormation as DeploymentConfig.Mode: EXPRESS with rollback disabled, even though --express was never passed. A single transient resource failure then parks the root stack in UPDATE_FAILED with no rollback, and a re-run of ampx sandbox reports Deployment completed without touching the stack.

Where it comes from. backend-deployer requests sandbox deploys as { method: 'hotswap', fallback: { method: 'direct' } } (lib/cdk_deployer.js:118). backend-deployer@2.2.0 pins "@aws-cdk/toolkit-lib": "1.32.0" exactly. In toolkit-lib 1.32.0 the hotswap fallback path hardcodes express (lib/api/deployments/deploy-stack.js:119):

if (deploymentMethod.fallback) {
  await ioHelper.defaults.info('Falling back to doing a full deployment');
  deploymentMethod = deploymentMethod.fallback;
  options = { ...options, express: true };
}

and deployConfig() only re-enables rollback when the caller passes rollback: true, which backend-deployer does not. The Amplify-side opt-in added for #3261 (deployProps.express, cdk_deployer.js:122) is not involved; the toolkit forces it regardless of that flag.

CDK considers this a bug and reverted it in aws/aws-cdk-cli#1801 ("hotswap fallback deployments were updated to use express mode. This change was technically breaking and should not have been made"), released in @aws-cdk/toolkit-lib 1.38.1 on 2026-08-07. I unpacked 1.38.1 and confirmed the express: true line is gone. backend-deployer@2.2.0 (still the latest as of 2026-09-04) pins 1.32.0, so Amplify users are stuck inside the broken window. toolkit-lib 1.19.0, which backend-cli 1.8.x used, has no EXPRESS code at all.

Why it hurts. Our data stack re-puts ~140 AWS::IAM::Policy (GraphqlAccessPolicy*) resources on every deploy. IAM throttled one (Rate exceeded, HandlerErrorCode: Throttling, 5 SDK attempts) on two consecutive days. In STANDARD mode that is a rollback and a retry. In EXPRESS mode with rollback disabled:

• root and data stacks stay UPDATE_FAILED;
• AWS::IAM::Policy update is delete-then-create, so the affected Lambda role is left with no AppSync policy and the function fails with an authorization error;
• the cleanup phase never runs, so resources the update removed remain orphaned in their nested stacks;
• re-running ampx sandbox hotswaps against the stack's recorded template (already the failed one), finds no diff, prints Deployment completed in 84s, writes a fresh amplify_outputs.json, and changes nothing;
• aws cloudformation create-change-set --use-previous-template is refused (Follow-up operations must use the same DeploymentConfig (mode=EXPRESS)), and the AWS CLI has no EXPRESS option.

The only recovery we found is calling the bundled toolkit directly with { deploymentMethod: { method: 'direct' }, express: true, forceDeployment: true } against the already-synthesized .amplify/artifacts/cdk.out, which took about four minutes and repaired the stack. That is not something most Amplify users can reach.

CloudTrail on our sandbox stack shows the switch exactly at the dependency bump: STANDARD on 2026-08-25 (toolkit-lib 1.19.0, user-agent cdk-hotswap/fallback), EXPRESS on all 15 UpdateStack calls since 2026-08-26 (toolkit-lib 1.32.0, same fallback marker, no --express anywhere).

Ask. Bump @aws-cdk/toolkit-lib in backend-deployer to >= 1.38.1 (1.40.0 is current), so that sandbox fallback deploys return to STANDARD mode with rollback, and EXPRESS stays behind the explicit --express flag from #3261 as intended. Related precedent: #3140 (a stale toolkit-lib pin causing a different failure).

Reproduction steps

1. Any Gen 2 project on @aws-amplify/backend-cli@1.9.0. Confirm node_modules/@aws-amplify/backend-cli/node_modules/@aws-cdk/toolkit-lib/package.json is 1.32.0.
2. Make a non-hotswappable change (add an IAM grant, add a function, change a schema) and run npx ampx sandbox --once. Do not pass --express.
3. In CloudTrail, look at the UpdateStack event for the root stack: requestParameters.deploymentConfig is { "mode": "EXPRESS" } and disableRollback is absent; the user agent contains cdk-hotswap/fallback.
4. Make one resource fail during that update (an AWS::IAM::Policy with an invalid principal is enough). The stack goes UPDATE_FAILED and does not roll back.
5. Run npx ampx sandbox --once again unchanged: it prints Deployment completed, the stack is still UPDATE_FAILED.
6. Repeat steps 2–3 with toolkit-lib 1.38.1 or later substituted under backend-deployer: deploymentConfig is absent (STANDARD) and a failed update rolls back.