AmazonServiceException returning null requestIds in stackTrace

[Naman-555]

Describe the bug

We are using doesBucketExistsInV2() to check if bucket exists in s3, we are getting AmazonServiceException with error code 500 for multiple different buckets.
We have observed that the stacktrace doesn't gives the requestIds . They are marked as null.

Expected Behavior

RequestIds should be printed in the stacktrace of AmazonServiceException as it happens in AmazonS3Exception.

Current Behavior

<aws.sdk.version>1.11.828</aws.sdk.version>
<kotlin.version>1.7.22</kotlin.version>

This is our s3 client .
val builder = AmazonS3ClientBuilder.standard()
.withClientConfiguration(clientConfiguration)
.withCredentials(DefaultAWSCredentialsProviderChain())
.withRegion(awsProperties.region)

We have used below method to verify if bucket exists-
amazonS3.doesBucketExistV2(bucketName)

Exception:
message: Exception while fetching data (/consumer/upsert) : java.lang.IllegalArgumentException: Error when attempting to verify existence of s3 bucket: "apiary-044815873833-us-east-1-lz-datascience"
spanId: 16d21cb68e768f1e
stack_trace: c.a.AmazonServiceException: Internal Server Error (Service: null; Status Code: 500; Error Code: null; Request ID: null; Proxy: null)
at c.a.i.EC2ResourceFetcher.handleErrorResponse(EC2ResourceFetcher.java:149)
at c.a.i.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:94)
at c.a.i.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70)
at c.a.i.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75)
at c.a.i.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
at c.a.a.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:47)
at c.a.a.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)
at c.a.a.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)
at c.a.a.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:165)
at c.a.a.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)
at c.a.a.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:111)
at c.a.h.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1257)
at c.a.h.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:833)
at c.a.h.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:783)
at c.a.h.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
... 13 frames truncated
... 154 common frames omitted

Reproduction Steps

Can use above code and when AmazonServiceException occurs , requestIds are marked as null.

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

1.11.828

JDK version used

11.0.15.1

Operating System and version

our app is deployed in AWS.

[debora-ito]

RequestIds should be printed in the stacktrace of AmazonServiceException as it happens in AmazonS3Exception.

RequestIDs will only be returned if the request was processed by the service. 500s are internal server errors, the service experienced an unexpected error and was not able to process the request.

Moving this to Github Discussions.

[debora-ito]

As I mentioned, there is no requestId to retrieve. And yes, the service team need requestIds so they can scan their logs and determine the exact point in time when your request was received.

In this case, I would investigate why your application started to see 500 errors.

• Was there any change that could have triggered the errors, like a code change, a version upgrade or a change in the application environment?
• Can you reproduce the error reliably in a local environment (not production)? If so, can you enable the verbose wirelogs? Instructions on how to enable wirelogs can be found here: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-logging.html#sdk-net-logging-verbose
  If you share the verbose wirelogs here, please make sure to redact any sensitive information, like access keys.

[Naman-555]

To answer your questions:

1. There was no change at all in the app or so.
2. We don't get these errors always, they are intermittent.So we can't enable verbose logging always in our app,
   Whenever multiple user face such issues and like this time, we have seen a sudden rise in 500 errors with AmazonServiceException,
   so we want to capture those requestIds when such error occurs so that we can pass those with AWS support team .
   Is there any other way we can capture such requestIds .
   We used doesBucketExistsInV2() for checking if bucket exists in S3 but as suggested by one of the AWS support member, using this method we won't be able to capture requestIds in metadata,
   So they suggested to use low level API GetBucketAcl() .
   So, I want to be 100% sure, if such errors (AmazonServiceException-500) occurs again and faced by many of our customers then will this below code will capture requestIds??? or there is any other way to do so? Because verbose logging cant be enabled always in the app.
   Sample code shared by them.

import com.amazonaws.services.s3.S3ResponseMetadata

GetBucketAclRequest req = new GetBucketAclRequest(bucketName);
try {
s3.GetBucketAcl(req);
}
catch(Exception e) {
// Block of code to handle errors
}

S3ResponseMetadata md = s3.getCachedResponseMetadata(req);
System.out.println("Host ID: " + md.getHostId() + " RequestID: " + md.getRequestId());

[debora-ito]

if such errors (AmazonServiceException-500) occurs again and faced by many of our customers then will this below code will capture requestIds?

No, there's no guarantee that 5xx errors will have a requestId associated, even when s3.getCachedResponseMetadata(req) is called. The issue is not on the client side, the issue is that the service will not send request Ids at all, like the first stacktrace illustrates:

c.a.AmazonServiceException: Internal Server Error (Service: null; Status Code: 500; Error Code: null; Request ID: null; Proxy: null)
at c.a.i.EC2ResourceFetcher.handleErrorResponse(EC2ResourceFetcher.java:149)
at c.a.i.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:94)

[Naman-555]

@debora-ito So in order to capture requestIds for all the errors including 5xx , will this helps?

log4j.rootLogger=WARN, A1
log4j.appender.A1=org.apache.log4j.ConsoleAppender
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c -  %m%n
**log4j.logger.com.amazonaws.request=DEBUG`**

If yes, we will enable this always in our app in all environments so that we can easily get requestIds whenever any error occurred.

We are hesitant to put **log4j.logger.org.apache.http.wire=DEBUG** in our prod environment "always" as it will display confidential info as well.

[debora-ito]

log4j.logger.com.amazonaws.request=DEBUG
Yes, this will show a summary of all the request and responses. I would run some tests before enabling in production, because it still can generate lots of logs, depending on the volume of requests your application has.

log4j.logger.org.apache.http.wire=DEBUG
Yes, wirelogs are definitely not recommended in production.

There is another option: to use RequestHandlers in the S3 client to intercept only the errors you want to log.

In the example below, the SQS client has a custom RequestHandler that logs the status code, requestID and HTTP headers every time a AmazonServiceException error is received, using the afterError hook.

public static void main(String... args) {
    AmazonSQS sqsClient =
        AmazonSQSClient.builder()
                       .withRequestHandlers(new LoggingRequestHandler())
                       .build();
}

private static class LoggingRequestHandler extends RequestHandler2 {
    @Override
    public void afterError(Request request, Response response, Exception e) {
        if (e instanceof AmazonServiceException) {
            AmazonServiceException ase = (AmazonServiceException) e;
            System.err.println("Raw response..." +
                               "HTTP status code: " + ase.getStatusCode() +
                               "HTTP requestId: " + ase.getRequestId() +
                               "HTTP headers: " + ase.getHttpHeaders());
        }
    }
}

This is just an example, you can tweak the LoggingRequestHandler to log whatever you need.
Check the RequestHandler2 javadoc for more details: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/handlers/RequestHandler2.html