Intermittent ongoing issue in connecting IMDS endpoint in the eks cluster

[shambhand]

Our java based services which are authenticating through the IAM role suddenly failing since Friday 25th 2022
We are seeing the aws-sdk-java calls internally GET http://169.254.169.254/latest/meta-data/iam/security-credentials/<Iam-role> and intermittently see the below failure for it. We have overridden the default timeout to 2 seconds from 1 second but the issue still persists.

java.net.SocketTimeoutException: Read timed out
  at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:283)
  at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309)
  at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
  at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
  at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
  at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
  at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
  at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:343)
  at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:825)
  at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:760)
  at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1688)
  at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
  at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529)
  at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:82)
  at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70)
  at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75)
  at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
  at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:49)
  at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:124)
  at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:80)
  at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)
  at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)
  at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1269)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:845)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:794)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
  at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:6409)
  at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:6376)

Since the IMDS call fails the SDK uses a stale credential (expired) to communicate to Dynamodb and it fails.

After restarting the k8s pod's/instance it stables for 30 minutes and we are again seeing the same error intermittently.

What could be the cause of an issue? Why SDK is using the expired credential? Is there a way to retry?

There is a below env variable available in python aws sdkboto3 but not in aws-java-sdk

AWS_METADATA_SERVICE_NUM_ATTEMPTS
When attempting to retrieve credentials on an Amazon EC2 instance that has been configured with an IAM role, Boto3 will make >only one attempt to retrieve credentials from the instance metadata service before giving up. If you know your code will be >running on an EC2 instance, you can increase this value to make Boto3 retry multiple times before giving up.

AWS Java SDK Used: v1.12.296
Java used: eclipse-temurin:17-jdk
IAM Component: jtblin/kube2iam:kube2iam-2.6.0
Kubernetes: v1.21.5

[debora-ito]

What changed in the day that you started to see the errors, what could potentially have triggered them?
Any SDK version upgrade, or change in the environment?

[shambhand]

Nothing changed for a long. The aws-java-sdk is hitting client-side timeout (1-sec default) and is not able to refresh the credential(IMDS) and falling back to using the expired credential.

For now, we have increased the timeout to 3 seconds with the help environment variables AWS_METADATA_SERVICE_TIMEOUT; but I think it's not a permanent and the most reliable solution if somehow the 3 seconds timeout hits.

How can we retry the IMDS credential refresh calls immediately for the n number of attempts for failure?