2.18.3

Features

Argo Events sensor namespace configuration

This release enables deploying the Argo sensor in its own namespace. The sensors are required when using Metaflow triggers when deploying to Argo Workflows.

The sensor namespace can be configured by either by the Metaflow config, or through an environment variable

export METAFLOW_ARGO_EVENTS_SENSOR_NAMESPACE=example

Compressing of large commands on AWS Step Functions

AWS Step Functions has a maximum size for the step command that can become a problem with complex flows. To get around this limitation, this release introduces a way to compress the command being executed by offloading it to S3 instead.

Setting the following environment variable or Metaflow config value to true will enable the feature.

export METAFLOW_SFN_COMPRESS_STATE_MACHINE=True

Improvements

Improve support for conditional steps with Argo Workflows

This release fixes an issue with conditional steps on Argo Workflows, where deployment would fail with a flow that had a conditional step where a case would not introduce its own steps, but instead skip ahead in the graph.

What's Changed

• Make argo events sensor namespace configurable by @dhpollack in #1463
• feat: add S3 command upload support for AWS Step Functions to handle large commands by @nlwstein in #2534
• Patch 2025-09-08-01 fix minor typos by @myatt83 in #2591
• Update metaflow client to accept both pathspec and _object together by @talsperre in #2590
• fix: argo conditional skip step parsing by @saikonen in #2587
• release: 2.18.3 by @saikonen in #2592

New Contributors

• @nlwstein made their first contribution in #2534
• @myatt83 made their first contribution in #2591

Full Changelog: 2.18.2...2.18.3

2.18.2

Improvements

Provide git-info metadata for deployed flows

Git repository information is not available during execution of a scheduled flow, such as with Argo Workflows or AWS Step Functions due to Metaflow only packaging files necessary for flow execution.

This release introduces a change where the git info is read during deployment time and retained as part of the deployment.

What's Changed

• fix: missing git info on remote flows by @saikonen in #2583
• fix: R test dependencies by @saikonen in #2586
• release: 2.18.2 by @saikonen in #2585

Full Changelog: 2.18.1...2.18.2

2.18.1

Features

Support custom AWS Batch tags

This release adds support for adding custom tags to AWS Batch.
In order to enable Batch tagging, a config flag needs to be set either in the environment, or in Metaflow config

export METAFLOW_BATCH_EMIT_TAGS=True

The supported ways for adding custom tags are

1. Through the @batch decorator

@batch(aws_batch_tags={"my tag": "my custom value"})

2. Through the CLI when deploying to AWS Step Functions

python BatchTags.py step-functions create \
--aws-batch-tag "tagging from"="CLI with value" \
--aws-batch-tag another=value2

3. Specifying default through an environment variable

export METAFLOW_BATCH_DEFAULT_TAGS='{"envtag": "value123", "another":"from env"}'

or in Metaflow config:

{
    "METAFLOW_BATCH_DEFAULT_TAGS": {"envtag": "value123", "another":"from env"}
}

What's Changed

• fix: Card dag breaking up in loops by @rsanteri in #2581
• feature: Batch tags [updated] by @RikishK, @Limess, @saikonen in #2578
• fix: shortened argo workflow name sanitising by @saikonen in #2580
• release: 2.18.1 by @saikonen in #2582

Full Changelog: 2.18.0...2.18.1

2.18.0

What's Changed

• chore: clarify argo conditionals exception by @saikonen in #2569
• Fix idiotic typo in exception handling by @romain-intel in #2567
• Fix an issue with storing sub-configs. by @romain-intel in #2566
• Fix save function to write package blob correctly by @savingoyal in #2575
• disallow conditional to join directly by @mt-ob in #2571
• Bump metaflow version to 2.18.0 by @savingoyal in #2577

Full Changelog: 2.17.5...2.18.0

2.17.5

What's Changed

• feature: support recursive conditionals on Argo Workflows by @saikonen in #2562
• Bump version to 2.17.5 by @savingoyal in #2565

Full Changelog: 2.17.4...2.17.5

2.17.4

What's Changed

• fix: DAG for default card with conditionals by @rsanteri in #2561
• Fix/allow raise in user decos by @romain-intel in #2555
• recursive conditionals with resume by @mt-ob in #2549
• Bump metaflow version to 2.17.4 by @savingoyal in #2563

New Contributors

• @rsanteri made their first contribution in #2561

Full Changelog: 2.17.3...2.17.4

2.17.3

What's Changed

• feature: conditionals for Argo Workflows by @saikonen in #2550
• fix: disallow unsupported conditional DAG structures by @saikonen in #2556
• fix: disable conditionals inside foreach on argo by @saikonen in #2559
• chore: re-enable conditionals in foreach on argo workflows by @saikonen in #2560
• patch release by @savingoyal in #2554

Full Changelog: 2.17.2...2.17.3

2.17.2

What's Changed

• Feat/conditionals by @romain-intel in #2545
• Fix an issue where user code exceptions were gobbled. by @romain-intel in #2548
• patch release by @savingoyal in #2553

Full Changelog: 2.17.1...2.17.2

2.17.1

What's Changed

• Bump tmp from 0.2.3 to 0.2.4 in /metaflow/plugins/cards/ui by @dependabot[bot] in #2536
• fix: metaflow-dev improvements by @saikonen in #2527
• [bug fix] fix subtle bug from #2463 by @valayDave in #2540
• Fix bug in deployer reload logic by @talsperre in #2533
• patch release by @savingoyal in #2541
• [bug fix] fix subtle bug from #2463 by @valayDave in #2542
• Fix bug in metaflow version when public=True by @talsperre in #2539

Full Changelog: 2.17.0...2.17.1

2.17.0

Improvements

Argo Workflows name length restriction

This release introduces a limit to the workflow name lengths that are deployed to Argo Workflows.

Why was this necessary?

Argo Workflows attaches the workflow template name (the deployed flow name) as a label to the Kubernetes resources it creates when argo itself launches a flow (events, schedule, Argo UI). The problem is that Kubernetes labels have a maximum length of 63 characters, and if a workflow name is longer than that, it will fail to launch due to not fitting in the label.

Launching flows on Argo Workflows through the Metaflow CLI has always been exempt of this issue, as this has not tried to attach such a label to the resources.

What about existing flows?

Existing flows continue to operate as previously, and these can still be operated on with the Metaflow CLI commands, e.g. argo-workflows trigger/argo-workflows delete.

An existing flow will always be replaced when running argo-workflows create. With the new version this might also result in a differently named deployment, in case the old one had a name that was too long.

What happens with long names now?

When using the @project decorator, Metaflow will automatically truncate the deployed name if it is too long. A part of each name partial (project / branch / flow_name) is kept to help with finding the deployments in Argo UI.

If the name of the deployment is too long when deploying without @project, then this is treated as an error and it is up to the user to provide a shorter name. This can be done by either renaming the flow class, or by supplying a custom name with --name

⚠️ NOTE
Deploying the same flow with an older version of Metaflow will lead to two separate deployments existing at once if the length of the name exceeded the limit.
For such flows be sure to only deploy them with Metaflow versions >=2.17 or use the --name CLI option to specify a custom deploy name.

What's Changed

• [cards] add attempt to default card. by @valayDave in #2532
• address name contraction issues in argo-workflows create by @savingoyal in #2082
• release: 2.17.0 by @saikonen in #2535

Full Changelog: 2.16.8...2.17.0