Skip to content

AWS prefect work-pool create --provision-infra doesn't add cloudwatch permissions to assumed role #20237

Description

@alastair

Bug summary

I use prefect work-pool create --type ecs:push --overwrite --provision-infra ecs-push-pool to provision items in my AWS account for running an ECS Push work pool.

In prefect cloud I edited the work pool to enable CloudWatch logging.

After submitting the flow, I get an error in AWS when trying to start the task (the same error is also returned to prefect cloud)

Flow run could not be submitted to infrastructure: TaskFailedToStart - ResourceInitializationError: failed to validate logger args: create stream has been retried 1 times: failed to create Cloudwatch log group: operation error CloudWatch Logs: CreateLogGroup, https response error StatusCode: 400, RequestID: c8e093f6-7267-4fe9-b8a0-2a35dc60d982, api error AccessDeniedException: User: arn:aws:sts::xxxx:assumed-role/PrefectEcsTaskExecutionRole/7a29b7fb28d7464198e5a95d259867b5 is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:us-east-2:xxxx:log-group:prefect:log-stream: because no identity-based policy allows the logs:CreateLogGroup action : exit status 1

I see that the prefect-ecs-user user which was created does have access to some logs: permissions via the prefect-ecs-policy policy.

However the PrefectEcsTaskExecutionRole role only has AmazonECSTaskExecutionRolePolicy attached to it. If I edit the role and add logs:* (or a more limited set of permissions) then it works fine.

I'm not sure the best way to surface this in the setup script. It could be via an additional flag --enable-logs which adds the permissions to the Role and also enables the setting in the work pool. However with this option then you'd need to work out the behaviour when the user turns off this setting (remove permissions from the role?) or turns it on without having used the flag (you'd need to add them but don't have access to a key/secret with enough permissions to add it).

Perhaps since the user already has these permissions it's OK to add them directly to the Role as well.

Version info

Version:              3.6.6
API version:          0.8.4
Python version:       3.13.11
Git commit:           f7d4baf4
Built:                Thu, Dec 11, 2025 08:20 PM
OS/Arch:              darwin/arm64
Profile:              ephemeral
Server type:          cloud
Pydantic version:     2.12.5
Server:
  Database:           sqlite
  SQLite version:     3.51.1
Integrations:
  prefect-aws:        0.7.2
  prefect-slack:      0.3.1
  prefect-github:     0.3.2

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions