SignatureDoesNotMatch when sending several S3 commands

[lotas]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

We noticed our tests started to fail with this dependabot upgrade: taskcluster/taskcluster#7365 (comment) (please note that this PR contains multiple packages, it was tested separately and confirmed with just @aws-sdk/[email protected] to be broken for us)

Tests were failing with SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.

Upon further investigation it was determined that breaking change was introduced around v3.645.0...v3.649.0 (with 3.645.0 being the last version working, and 3.649.0 first one being broken)

After adding debug middleware I've discovered that request objects might be sharing some properties from the previous request, namely path. I suspect this might be a reason, but I'm not too sure.

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/[email protected]

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v22.11.0, v20.16.0

Reproduction Steps

Code:

import {
  ListObjectsCommand,
  PutObjectCommand,
  S3Client,
} from "@aws-sdk/client-s3";


const secret = {
  accessKeyId: "xx",
  secretAccessKey: "yy",
  testBucket: "test-bucket-for-any-garbage",
}

async function run(name, data, prefix) {
  const s3 = new S3Client({
    credentials: {
      accessKeyId: secret.accessKeyId,
      secretAccessKey: secret.secretAccessKey,
    },
    region: "us-west-2",
    followRegionRedirects: true,
  });
  s3.middlewareStack.add(
    (next) => async (args) => {
      console.log("a1>", args.request);
      return next(args);
    }, { step: "finalizeRequest" },
  );

  // 1st request
  await s3.send(
    new PutObjectCommand({
      Bucket: secret.testBucket,
      Key: name,
      Body: data,
    }),
  );

  // 2nd request fails here with SignatgureDoesNotMatch
  const objects = await s3.send(
    new ListObjectsCommand({
      Bucket: secret.testBucket,
      Prefix: prefix,
    }),
  );

  console.log(objects);
}

run('test1/file.txt', 'data', 'test1/').catch(console.error);

Running this sequence on 3.645.0 works and produces different output.

Here is how the output of the 3.649.0 looks like:

HttpRequest {
  method: 'PUT',
  hostname: 'test-bucket-for-any-garbage.s3.us-west-2.amazonaws.com',
  port: undefined,
  query: { 'x-id': 'PutObject' },
  headers: {
    'content-type': 'application/octet-stream',
    'content-length': '256',
    Expect: '100-continue',
    'x-amz-user-agent': 'aws-sdk-js/3.649.0',
    'user-agent': 'aws-sdk-js/3.649.0 ua/2.0 os/darwin#24.1.0 lang/js md/nodejs#22.11.0 api/s3#3.649.0',
    host: 'test-bucket-for-any-garbage.s3.us-west-2.amazonaws.com',
    'amz-sdk-invocation-id': 'b9e8aa45-0e6a-45a3-9050-f234e44a5eea',
    'amz-sdk-request': 'attempt=1; max=3',
    'x-amz-date': '20241105T115452Z',
    'x-amz-content-sha256': 'zz',
    authorization: 'AWS4-HMAC-SHA256 Credential=xx/20241105/us-west-2/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent, Signature=yy
  },
  body: <Buffer 80 ...>,
  protocol: 'https:',
  path: '/QsGI3EhSQR6tXuNHPDRIUw/0/test/%26/%24/%40/%3D/%3B/%3A/%2B/%2C/%3F/%5C/%7B%7D/%5E/%25/%5B%5D/%3C%3E/%23/~/%7C/%60/',
  username: undefined,
  password: undefined,
  fragment: undefined
}

HttpRequest {
  method: 'GET',
  hostname: 'test-bucket-for-any-garbage.s3.us-west-2.amazonaws.com',
  port: undefined,
  query: { prefix: 'QsGI3EhSQR6tXuNHPDRIUw/' },
  headers: {
    'x-amz-user-agent': 'aws-sdk-js/3.649.0',
    'user-agent': 'aws-sdk-js/3.649.0 ua/2.0 os/darwin#24.1.0 lang/js md/nodejs#22.11.0 api/s3#3.649.0',
    host: 'test-bucket-for-any-garbage.s3.us-west-2.amazonaws.com',
    'amz-sdk-invocation-id': 'b6d1fb01-c844-471c-b901-83d530c6ce19',
    'amz-sdk-request': 'attempt=1; max=3',
    'x-amz-date': '20241105T115456Z',
    'x-amz-content-sha256': 'zz',
    authorization: 'AWS4-HMAC-SHA256 Credential=xx/20241105/us-west-2/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent, Signature=yy'
  },
  body: undefined,
  protocol: 'https:',
  path: '/QsGI3EhSQR6tXuNHPDRIUw%2F0%2Ftest%2F%26%2F%24%2F%40%2F%3D%2F%3B%2F%3A%2F%2B%2F%2C%2F%3F%2F%5C%2F%7B%7D%2F%5E%2F%25%2F%5B%5D%2F%3C%3E%2F%23%2F~%2F%7C%2F%60%2F/',
  username: undefined,
  password: undefined,
  fragment: undefined,
}

Second one contains path which belongs to the previous command that was sent.

With 3.645.0 this request looked like:

HttpRequest {
  method: 'GET',
  hostname: 'test-bucket-for-any-garbage.s3.us-west-2.amazonaws.com',
  port: undefined,
  query: { prefix: 'Ipj9GENTTgS3eJYkfnOoQw/' },
  headers: {
    'x-amz-user-agent': 'aws-sdk-js/3.645.0',
    'user-agent': 'aws-sdk-js/3.645.0 ua/2.0 os/darwin#24.1.0 lang/js md/nodejs#22.11.0 api/s3#3.645.0',
    host: 'test-bucket-for-any-garbage.s3.us-west-2.amazonaws.com',
    'amz-sdk-invocation-id': 'efe404aa-60f2-4533-bc99-10bb3c04322c',
    'amz-sdk-request': 'attempt=1; max=3',
    'x-amz-date': '20241105T112452Z',
    'x-amz-content-sha256': 'zz',
    authorization: 'AWS4-HMAC-SHA256 Credential=xx/20241105/us-west-2/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent, Signature=yy'
  },
  body: undefined,
  protocol: 'https:',
  path: '/',
  username: undefined,
  password: undefined,
  fragment: undefined
}

Observed Behavior

List command fails:

{
  clientName: 'S3Client',
  commandName: 'ListObjectsCommand',
  input: {
    Bucket: 'test-bucket-for-any-garbage',
    Prefix: 'ecA-5h0hTZG1J7QADK17eQ/'
  },
  error: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
      at throwDefaultError (/Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@smithy/smithy-client/dist-cjs/index.js:836:20)
      at /Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@smithy/smithy-client/dist-cjs/index.js:845:5
      at de_CommandError (/Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4751:14)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async /Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
      at async /Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:482:18
      at async /Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
      at async /Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:110:22
      at async /Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:138:14
      at async /Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@aws-sdk/client-s3/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
      at async Context.cleanup (file:///Users/ykurmyza/dev/Mozilla/taskcluster/services/object/test/backends/aws_test.js:165:21) {
    '$fault': 'client',
    '$metadata': {
      httpStatusCode: 403,
      requestId: 'P8ZVW69H68980T8M',
      extendedRequestId: 'I/A0hhql9G14dncL6DvbWNmW+DzU8Rx5tg1VVE6wqHrtGPepRFKt+vUVwikSLsmGYowSq6FFf/RMAedpyBYwtw==',
      cfId: undefined,
      attempts: 1,
      totalRetryDelay: 0
    },
    Code: 'SignatureDoesNotMatch',
    AWSAccessKeyId: 'AKIA2CNU74ZQRSDNSKMH',
    StringToSign: 'AWS4-HMAC-SHA256\n' +
      '20241105T123748Z\n' +
      '20241105/us-west-2/s3/aws4_request\n' +
      '23ada6b1782ed966cb9d7ccedd7198f50e384e04b19d5429776737a41be11df6',
    SignatureProvided: '33e7b01214be5a3bfe3e2781599fd5b1176a37a54d945da4438ad3776e81e197',
    StringToSignBytes: '41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 34 31 31 30 35 54 31 32 33 37 34 38 5a 0a 32 30 32 34 31 31 30 35 2f 75 73 2d 77 65 73 74 2d 32 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 32 33 61 64 61 36 62 31 37 38 32 65 64 39 36 36 63 62 39 64 37 63 63 65 64 64 37 31 39 38 66 35 30 65 33 38 34 65 30 34 62 31 39 64 35 34 32 39 37 37 36 37 33 37 61 34 31 62 65 31 31 64 66 36',
    CanonicalRequest: 'GET\n' +
      '/ecA-5h0hTZG1J7QADK17eQ/0/test/%26/%24/%40/%3D/%3B/%3A/%2B/%2C/%3F/%5C/%7B%7D/%5E/%25/%5B%5D/%3C%3E/%23/~/%7C/%60//\n' +
      'prefix=ecA-5h0hTZG1J7QADK17eQ%2F\n' +
      'amz-sdk-invocation-id:c830f1e3-03e0-41c8-833a-7a74a2f7d820\n' +
      'amz-sdk-request:attempt=1; max=3\n' +
      'host:test-bucket-for-any-garbage.s3.us-west-2.amazonaws.com\n' +
      'x-amz-content-sha256:yy\n' +
      'x-amz-date:20241105T123748Z\n' +
      'x-amz-user-agent:aws-sdk-js/3.649.0\n' +
      '\n' +
      'amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent\n' +
      'zz',
    CanonicalRequestBytes: '47 45 54 0a 2f 65 63 41 2d 35 68 30 68 54 5a 47 31 4a 37 51 41 44 4b 31 37 65 51 2f 30 2f 74 65 73 74 2f 25 32 36 2f 25 32 34 2f 25 34 30 2f 25 33 44 2f 25 33 42 2f 25 33 41 2f 25 32 42 2f 25 32 43 2f 25 33 46 2f 25 35 43 2f 25 37 42 25 37 44 2f 25 35 45 2f 25 32 35 2f 25 35 42 25 35 44 2f 25 33 43 25 33 45 2f 25 32 33 2f 7e 2f 25 37 43 2f 25 36 30 2f 2f 0a 70 72 65 66 69 78 3d 65 63 41 2d 35 68 30 68 54 5a 47 31 4a 37 51 41 44 4b 31 37 65 51 25 32 46 0a 61 6d 7a 2d 73 64 6b 2d 69 6e 76 6f 63 61 74 69 6f 6e 2d 69 64 3a 63 38 33 30 66 31 65 33 2d 30 33 65 30 2d 34 31 63 38 2d 38 33 33 61 2d 37 61 37 34 61 32 66 37 64 38 32 30 0a 61 6d 7a 2d 73 64 6b 2d 72 65 71 75 65 73 74 3a 61 74 74 65 6d 70 74 3d 31 3b 20 6d 61 78 3d 33 0a 68 6f 73 74 3a 74 65 73 74 2d 62 75 63 6b 65 74 2d 66 6f 72 2d 61 6e 79 2d 67 61 72 62 61 67 65 2e 73 33 2e 75 73 2d 77 65 73 74 2d 32 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 32 34 31 31 30 35 54 31 32 33 37 34 38 5a 0a 78 2d 61 6d 7a 2d 75 73 65 72 2d 61 67 65 6e 74 3a 61 77 73 2d 73 64 6b 2d 6a 73 2f 33 2e 36 34 39 2e 30 0a 0a 61 6d 7a 2d 73 64 6b 2d 69 6e 76 6f 63 61 74 69 6f 6e 2d 69 64 3b 61 6d 7a 2d 73 64 6b 2d 72 65 71 75 65 73 74 3b 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 3b 78 2d 61 6d 7a 2d 75 73 65 72 2d 61 67 65 6e 74 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35',
    RequestId: 'P8ZVW69H68980T8M',
    HostId: 'I/A0hhql9G14dncL6DvbWNmW+DzU8Rx5tg1VVE6wqHrtGPepRFKt+vUVwikSLsmGYowSq6FFf/RMAedpyBYwtw=='
  },
  metadata: {
    httpStatusCode: 403,
    requestId: 'P8ZVW69H68980T8M',
    extendedRequestId: 'I/A0hhql9G14dncL6DvbWNmW+DzU8Rx5tg1VVE6wqHrtGPepRFKt+vUVwikSLsmGYowSq6FFf/RMAedpyBYwtw==',
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }

Expected Behavior

ListCommand is being sent with the correct signature

Possible Solution

No response

Additional Information/Context

No response

[lotas]

Oh, I just found out that it is not reproducible on a standalone script, which means something inside the test suite is breaking S3Client.

I'll keep updating my findings though here.
Found out that context is getting "poisoned" in https://github.com/smithy-lang/smithy-typescript/blob/main/packages/middleware-serde/src/serializerMiddleware.ts#L28 where context.endpointV2.url already contains pathname from the previous request, so url builder thinks that basepath is not /