Check if user access key is visible in S3

[arseniymerkulov]

Scenario:

1. create iam user via boto
2. add access key to the user
3. use that access key to create s3 bucket

Here's my code:

class AWSIAMManager:
    def __init__(self, user: AWSUser):
        self.user = user
        self.group = 'regular-permissions'
        self.client = boto3.Session(profile_name='user-manager').client('iam')

    def create_user(self):
        try:
            response = self.client.create_user(
                UserName=self.user.name,
            )
            self.client.get_waiter('user_exists').wait(UserName=self.user.name)
      
            response = self.client.add_user_to_group(
                GroupName=self.group,
                UserName=self.user.name
            )
            
            response = self.client.create_access_key(
                UserName=self.user.name
            )
            
            access_key_id = response['AccessKey']['AccessKeyId']
            access_key_secret = response['AccessKey']['SecretAccessKey']
            self.user.access_key = (access_key_id, access_key_secret)

            print(self.client.list_access_keys(UserName=self.user.name))
            # new key already accessible

        except ClientError as e:
            logger.error(e)
            return False
        return True

        
class AWSS3Manager:
    def __init__(self, user: AWSUser, region: str):
        self.bucket = f'aibb-aws-{user.name.lower()}'
        self.region = region
        self.user = user

        key_id, key_secret = user.access_key
        session = boto3.Session(aws_access_key_id=key_id,
                                aws_secret_access_key=key_secret)
        self.client = session.client('s3', region_name=region)
        self.resource = session.resource('s3', region_name=region)

    def create_bucket(self):
        try:
            location = {'LocationConstraint': self.region}
            self.client.create_bucket(Bucket=self.bucket, CreateBucketConfiguration=location)
            
        except ClientError as e:
            logger.error(e)
            return False
        return True
        
user = AWSUser('user')
iam = AWSIAMManager(user)
iam.create_user()
# time.sleep(10)

s3 = AWSS3Manager(user, 'eu-west-1')
s3.create_bucket()

Error occures in bucket creation: "The AWS Access Key Id you provided does not exist in our records."
If i wait 10 seconds everything works fine. When i request access key from IAM service it responds immediately. Is there a way to wait until user access key is ready to authenticate to S3?

[tim-finnigan]

Hi @arseniymerkulov thanks for reaching out. I think this may be an eventual consistency issue, as described in the IAM Troubleshooting Guide section on "Changes that I make are not always immediately visible": https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_general.html

As a service that is accessed through computers in data centers around the world, IAM uses a distributed computing model called eventual consistency

Any change that you make in IAM (or other AWS services), including tags used in attribute-based access control (ABAC), takes time to become visible from all possible endpoints. Some of the delay results from the time it takes to send the data from server to server, from replication zone to replication zone, and from Region to Region around the world. IAM also uses caching to improve performance, but in some cases this can add time. The change might not be visible until the previously cached data times out.

You must design your global applications to account for these potential delays. Ensure that they work as expected, even when a change made in one location is not instantly visible at another. Such changes include creating or updating users, groups, roles, or policies. We recommend that you do not include such IAM changes in the critical, high-availability code paths of your application. Instead, make IAM changes in a separate initialization or setup routine that you run less frequently. Also, be sure to verify that the changes have been propagated before production workflows depend on them.

Due to the nature of the eventual consistency, I'm not sure there is a good solution here other than using time.sleep(). Unfortunately there isn't a waiter like AccessKeyExists, similar to the UserExists one you are using. The UserExists waiter is defined here, and polls the get_user command for updates. There is not an equivalent get_access_key command so I think that would be needed for a waiter.