StsAssumeRoleWithWebIdentityCredentialsProvider does not call refreshRequest

[paul-gardiner]

Describe the bug

I am using StsAssumeRoleWithWebIdentityCredentialsProvider to assume an AWS role using a token created on Azure AD/Entra. When the AWS token expires, it will get a new token as long as the WebIdentityToken from Entra has not expired. Once the WebIdentityToken expires, the next refresh to get an AWS token results in an ExpiredTokenException.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

When refreshing the AWS token, I expected refreshRequest to be called, which would update the WebIdentityToken. However, it is not called.

Current Behavior

s.a.a.s.s.m.ExpiredTokenException: Token expired: current date/time 1731098245 must be before the expiration date/time 1731095240 (Service: Sts, Status Code: 400, Request ID: 22679931-6a1d-4db3-abd0-a189de181809)
at s.a.a.c.i.h.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125)
at s.a.a.c.i.h.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82)
at s.a.a.c.i.h.CombinedResponseHandler.handle(CombinedResponseHandler.java:60)
at s.a.a.c.i.h.CombinedResponseHandler.handle(CombinedResponseHandler.java:41)
at s.a.a.c.i.h.p.s.HandleResponseStage.execute(HandleResponseStage.java:50)
at s.a.a.c.i.h.p.s.HandleResponseStage.execute(HandleResponseStage.java:38)
at s.a.a.c.i.h.p.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74)
at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43)
at s.a.a.c.i.h.p.s.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:79)
at s.a.a.c.i.h.p.s.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:41)
at s.a.a.c.i.h.p.s.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55)
at s.a.a.c.i.h.p.s.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39)
at s.a.a.c.i.h.p.s.RetryableStage2.executeRequest(RetryableStage2.java:93)
at s.a.a.c.i.h.p.s.RetryableStage2.execute(RetryableStage2.java:56)
at s.a.a.c.i.h.p.s.RetryableStage2.execute(RetryableStage2.java:36)
at s.a.a.c.i.h.p.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at s.a.a.c.i.h.StreamManagingStage.execute(StreamManagingStage.java:53)
at s.a.a.c.i.h.StreamManagingStage.execute(StreamManagingStage.java:35)
at s.a.a.c.i.h.p.s.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
at s.a.a.c.i.h.p.s.Api...

Reproduction Steps

Following is the config class (springboot) that creates the StsAssumeRoleWithWebIdentityCredentialsProvider bean.

@Configuration
@Slf4j
class AwsAssumeRoleConfig {

  @Value("${application.aws-role-session-duration}")
  private int sessionDuration;
  @Value("${application.aws-assumed-role-arn}")
  private String assumedRoleArn;

  @Bean
  StsAssumeRoleWithWebIdentityCredentialsProvider assumeRoleWithWebIdentityCredentialsProvider(
      StsClient stsClient, EntraTokenProvider entraTokenProvider) {
    String sessionName = "cct-" + UUID.randomUUID().toString();
    log.info("Creating StsAssumeRole provider for role: {} with session name {} with duration {}",
        assumedRoleArn, sessionName,
        sessionDuration);

    return StsAssumeRoleWithWebIdentityCredentialsProvider.builder()
        .stsClient(stsClient)
        .asyncCredentialUpdateEnabled(true)
        .refreshRequest(assumeRoleRequest -> {
          log.info("entra token requested for assume role");
          OAuth2AccessToken token = entraTokenProvider.getAccessToken();
          if (log.isDebugEnabled()) {
            log.debug("Token received from entra: {}***********",
                token.getTokenValue().substring(0, 5));
          } else {
            log.info("Token received from entra, length {}", token.getTokenValue().length());
          }
          assumeRoleRequest.webIdentityToken(token.getTokenValue());
          assumeRoleRequest.roleArn(assumedRoleArn);
          assumeRoleRequest.roleSessionName(sessionName);
          assumeRoleRequest.durationSeconds(sessionDuration);
        })
        .build();
  }

  @SuppressWarnings("java:S6242")
  @Bean
  StsClient stsClient() {
    return StsClient.builder()
        .region(Region.CA_CENTRAL_1)

        .build();
  }
}

Possible Solution

No response

Additional Information/Context

Dependencies are:

implementation 'org.springframework.cloud:spring-cloud-stream-binder-kinesis:4.0.4'
implementation 'org.apache.httpcomponents.client5:httpclient5'
implementation 'software.amazon.awssdk:kinesis:2.29.1'
implementation 'software.amazon.awssdk:dynamodb:2.29.1'
implementation 'software.amazon.awssdk:cloudwatch:2.29.1'
implementation 'software.amazon.awssdk:sts:2.29.1'
implementation 'org.springframework.security:spring-security-oauth2-client'
implementation 'com.azure.spring:spring-cloud-azure-stream-binder-eventhubs:5.17.1'
implementation 'com.google.protobuf:protobuf-java:4.28.3'

AWS Java SDK version used

2.29.1

JDK version used

OpenJDK Runtime Environment Temurin-21.0.5+11 (build 21.0.5+11-LTS)

Operating System and version

Oracle Linux Server 8.10

[bhoradc]

Hello @paul-gardiner,

Thank you for reporting the issue.

Firstly, can you please confirm if the following best describes the issue you're experiencing?

1. When the AWS temporary credentials is about to expire, the refreshRequest lambda is invoked to obtain a new web identity token and request new AWS credentials from the AWS STS.
2. However, when the web identity token itself expires, the refreshRequest lambda is not being called, leading to an ExpiredTokenException when attempting to refresh the AWS credentials.

If the above understanding is correct, then this looks an expected behaviour with the StsAssumeRoleWithWebIdentityCredentialsProvider. As I believe the refreshRequest lambda is supposed to be called when the temporary AWS credentials need to be refreshed, regardless of the state of the web identity token.

When the refreshRequest lambda is invoked, isn't the EntraTokenProvider resulting in a fresh web identity token each time, irrespective of the previous token expiration status?

Can you confirm that the EntraTokenProvider implementation is correctly providing a fresh web identity token when called within the refreshRequest lambda, even after the previous token has expired? And if possible, enable the debug logs(redact sensitive info) and share with us?

If my understanding of this problem has any deviation, please provide any additional information or clarification that might help to better understand and troubleshoot this issue.

Regards
Chaitanya

[paul-gardiner]

Not quite. When the AWS credentials expire, if the web identity token is still valid, everything is fine, but if it has expired, it isn't refreshed. My expectation is that StsAssumeRoleWithWebIdentityCredentialsProvider will refresh the web identity token before it expires.

I have worked around the issue by wrapping the StsClient, and then in AssumeRoleWithWebIdentityResponse assumeRoleWithWebIdentity(AssumeRoleWithWebIdentityRequest), I create a new AssumeRoleWithWebIdentityRequest, with a new web identity token. This works, but is a bit messy.

It has been a couple of months, but I'll see if I can get a version going with the issue, and turn on debug.