v1.77.0

Update `atmos validate stacks` command @aknysh (#611)

what

• Update atmos validate stacks command
• Improve stack validation error messages

why

• When checking for misconfiguration and duplication of components in stacks, throw errors only if the duplicate component configurations in the same stack are different (this will allow importing the base default/abstract components into many stack manifest files)

• The atmos validate stacks command check the following

  • All YAML manifest files for YAML errors and inconsistencies

  • All imports: if they are configured correctly, have valid data types, and point to existing manifest files

  • Schema: if all sections in all YAML manifest files are correctly configured and have valid data types

  • Misconfiguration and duplication of components in stacks. If the same Atmos component in the same Atmos stack is defined in more than one stack manifest file, and the component configurations are different, an error message will be displayed similar to the following:

    The Atmos component 'vpc' in the stack 'plat-ue2-dev' is defined in more than one
    top-level stack manifest file: orgs/acme/plat/dev/us-east-2-extras, orgs/acme/plat/dev/us-east-2.
    
    The component configurations in the stack manifests are different.
    
    To check and compare the component configurations in the stack manifests, run the following commands:
    - atmos describe component vpc -s orgs/acme/plat/dev/us-east-2-extras
    - atmos describe component vpc -s orgs/acme/plat/dev/us-east-2
    
    You can use the '--file' flag to write the results of the above commands to files
    (refer to https://atmos.tools/cli/commands/describe/component).
    
    You can then use the Linux 'diff' command to compare the files line by line and show the differences
    (refer to https://man7.org/linux/man-pages/man1/diff.1.html)
    
    When searching for the component 'vpc' in the stack 'plat-ue2-dev', Atmos can't decide which
    stack manifest file to use to get configuration for the component. This is a stack misconfiguration.
    
    Consider the following solutions to fix the issue:
    
      - Ensure that the same instance of the Atmos 'vpc' component in the stack 'plat-ue2-dev'
        is only defined once (in one YAML stack manifest file)
      
      - When defining multiple instances of the same component in the stack,
        ensure each has a unique name
      
      - Use multiple-inheritance to combine multiple configurations together
        (refer to https://atmos.tools/core-concepts/components/inheritance)   

notes

• This is an improvement of the previous release https://github.com/cloudposse/atmos/releases/tag/v1.75.0. The previous release introduced too strict checking and disabled the case where the same component in the same stack was just imported into two or more stack manifest files (this type of configuration is acceptable since the component config is always the same in the stack manifests since it's just imported and not modified)

v1.76.0

Add Atmos manifest lists merge strategies @aknysh (#609)

what

• Add Atmos manifest lists merge strategies
• Update docs
  • Settings section in CLI Configuration

why

• Allow using the following list merge strategies in Atmos stack manifests:

  • replace - Most recent list imported wins (the default behavior).

  • append - The sequence of lists is appended in the same order as imports.

  • merge - The items in the destination list are deep-merged with the items in the source list. The items in the source list take precedence. The items are processed starting from the first up to the length of the source list (the remaining items are not processed). If the source and destination lists have the same length, all items in the destination lists are deep-merged with all items in the source list.

The list merging strategies are configured in atmos.yaml CLI config file in the settings.list_merge_strategy section

settings:
  # `list_merge_strategy` specifies how lists are merged in Atmos stack manifests.
  # Can also be set using 'ATMOS_SETTINGS_LIST_MERGE_STRATEGY' environment variable, or '--settings-list-merge-strategy' command-line argument
  # The following strategies are supported:
  # `replace`: Most recent list imported wins (the default behavior).
  # `append`:  The sequence of lists is appended in the same order as imports.
  # `merge`:   The items in the destination list are deep-merged with the items in the source list.
  #            The items in the source list take precedence.
  #            The items are processed starting from the first up to the length of the source list (the remaining items are not processed).
  #            If the source and destination lists have the same length, all items in the destination lists are
  #            deep-merged with all items in the source list.
  list_merge_strategy: replace

v1.75.0

Improve `atmos validate stacks` and `atmos describe affected` commands @aknysh (#608)

what

• Improve atmos validate stacks and atmos describe affected commands
• Update docs
  • atmos validate stacks
  • alias: Multiple Provider Configuration in Atmos Manifests

why

• atmos validate stacks now detects if the same component in the same stack are defined in more than one Atmos stack manifest files. If such a misconfiguration is detected, the following error is shown:

  the Atmos component 'vpc' in the stack 'plat-ue2-dev' is defined in more than one top-level 
  stack manifest file: orgs/acme/plat/dev/us-east-2, orgs/acme/plat/dev/us-east-2-extras.
  Atmos can't decide which stack manifest to use to get configuration for the component in the stack.
  This is a stack misconfiguration.

• atmos describe affected now has better error messages and also executes atmos validate stacks before detecting the affected components and stacks. This prevents the issue when the same component in the same stack is defined in more than one stack manifest file, and the command used one file from the current local branch and the other file from the target branch, resulting in drift in affected components and stacks. Since this is a stack misconfiguration and is not permitted, atmos describe affected will display the error and exit

• alias: Multiple Provider Configuration in Atmos Manifests

  Atmos allows you to define multiple configurations for the same provider using a list of provider blocks and the alias meta-argument.

  The generated providers_override.tf.json file will have a list of provider configurations, and Terraform will use and override the providers as long as the aliased providers are defined in the Terraform component.

  For example:

  components: 
    terraform:
      vpc:
        providers:
          aws:
            - region: us-west-2
              assume_role:
                role_arn: "role-1"
            - region: us-west-2
              alias: "account-2"
              assume_role:
                role_arn: "role-2"

Use Darker Theme, Add File Component @osterman (#607)

what

• Add a component to render files distinct from console commands
• Darken theme

why

• Previous background felt too washed out
• It's much easier to understand files and commands separately, when they are clearly distinguished.

v1.74.0

Update Atmos logs. Update docs @aknysh (#605)

what

• Update Atmos logs
• Make the logs respect the standard file descriptors like /dev/stderr
• Update docs
  • https://atmos.tools/cli/configuration/#logs

why

Atmos logs are configured in the logs section in the atmos.yaml CLI config file:

logs:
  # Can also be set using 'ATMOS_LOGS_FILE' ENV var, or '--logs-file' command-line argument
  # File or standard file descriptor to write logs to
  # Logs can be written to any file or any standard file descriptor, including `/dev/stdout`, `/dev/stderr` and `/dev/null`
  file: "/dev/stderr"
  # Supported log levels: Trace, Debug, Info, Warning, Off
  # Can also be set using 'ATMOS_LOGS_LEVEL' ENV var, or '--logs-level' command-line argument
  level: Info

• logs.file - the file to write Atmos logs to. Logs can be written to any file or any standard file descriptor, including /dev/stdout, /dev/stderr and /dev/null. If omitted, /dev/stdout will be used. The environment variable ATMOS_LOGS_FILE can also be used to specify the log file

• logs.level - Log level. Supported log levels are Trace, Debug, Info, Warning, Off. If the log level is set to Off, Atmos will not log any messages (note that this does not prevent other tools like Terraform from logging). The environment variable ATMOS_LOGS_LEVEL can also be used to specify the log level

To prevent Atmos from logging any messages (except for the outputs of the executed commands), you can do one of the following:

• Set logs.file or the ENV variable ATMOS_LOGS_FILE to /dev/null

• Set logs.level or the ENV variable ATMOS_LOGS_LEVEL to Off

Note that when you set the log level to Debug or Trace, Atmos will log additional messages before printing the output of an executed command. For example, let's consider the atmos describe affected command:

logs:
  file: "/dev/stdout"
  level: Trace

> atmos describe affected

Checking out Git ref 'refs/remotes/origin/HEAD' ...
Checked out Git ref 'refs/remotes/origin/HEAD'

Current working repo HEAD: ffd2154e1daa32357b75460b9f45d268922b51e1 refs/heads/update-logs
Remote repo HEAD: f7aa382aa8b3d48be8f06cfdb27aad344b89aff4 HEAD

Changed files:

examples/quick-start/Dockerfile
examples/quick-start/atmos.yaml

Affected components and stacks:

[
   {
      "component": "vpc",
      "component_type": "terraform",
      "component_path": "examples/quick-start/components/terraform/vpc",
      "stack": "plat-uw2-prod",
      "stack_slug": "plat-uw2-prod-vpc",
      "affected": "stack.vars"
   },
   {
      "component": "vpc",
      "component_type": "terraform",
      "component_path": "examples/quick-start/components/terraform/vpc",
      "stack": "plat-ue2-prod",
      "stack_slug": "plat-ue2-prod-vpc",
      "affected": "stack.vars"
   }
]

With logs.level: Trace, and logs.file: "/dev/stdout", all the messages and the command's JSON output will be printed to the console to the /dev/stdout standard output.

This behavior might be undesirable when you execute the command atmos describe affected in CI/CD (e.g. GitHub Actions).

For example, you might want to log all the Atmos messages (by setting logs.level: Trace) for debugging purposes, and also want to parse the JSON output of the command (e.g. by using jq) for further processing. In this case, jq will not be able to parse the JSON output because all the other messages make the output an invalid JSON document.

To deal with that, you can set logs.file to /dev/stderr in atmos.yaml:

logs:
  file: "/dev/stderr"
  level: Trace

Now when the atmos describe affected command is executed, the additional messages are printed to /dev/stderr, but the command's JSON output is printed to /dev/stdout, allowing jq to parse it without errors.

> atmos describe affected

# NOTE: These messages are printed to `/dev/stderr`

Checking out Git ref 'refs/remotes/origin/HEAD' ...
Checked out Git ref 'refs/remotes/origin/HEAD'
Current working repo HEAD: ffd2154e1daa32357b75460b9f45d268922b51e1 refs/heads/update-logs
Remote repo HEAD: f7aa382aa8b3d48be8f06cfdb27aad344b89aff4 HEAD


# NOTE: This JSON output is printed to `/dev/stdout`

[
   {
      "component": "vpc",
      "component_type": "terraform",
      "component_path": "examples/quick-start/components/terraform/vpc",
      "stack": "plat-uw2-prod",
      "stack_slug": "plat-uw2-prod-vpc",
      "affected": "stack.vars"
   },
   {
      "component": "vpc",
      "component_type": "terraform",
      "component_path": "examples/quick-start/components/terraform/vpc",
      "stack": "plat-ue2-prod",
      "stack_slug": "plat-ue2-prod-vpc",
      "affected": "stack.vars"
   }
]

v1.73.0

Allow `Go` templates in `metadata.component` section. Add `components.terraform.command` section to `atmos.yaml`. Document OpenTofu support @aknysh (#604)

what

• Allow Go templates in metadata.component section
• Add components.terraform.command and components.helmfile.command sections to atmos.yaml
• Update docs
• Document OpenTofu support
  • https://atmos.tools/integrations/opentofu/

why

• Go templates in component and metadata.component section are useful when multiple versions of the same Terraform component are used from components/terraform section, and a particular version can be defined using templates. Use Go templates in metadata.component section as follows:

components:
  terraform:
    my-component:
      settings:
        component: my-tf-component  # Point to the Terraform component
      metadata:
        component: "{{ .settings.component }}"

• The components.terraform.command and components.helmfile.command sections in atmos.yaml specify the executable to be called by atmos when running Terraform and Helmfile commands. Allows specifying different versions of OpenTofu executables

components:
  terraform:
    # Optional `command` specifies the executable to be called by `atmos` when running Terraform commands
    # If not defined, `terraform` is used
    # Examples:
    # command: terraform
    # command: /usr/local/bin/terraform
    # command: /usr/local/bin/terraform-1.8
    # command: tofu
    # command: /usr/local/bin/tofu-1.7.1
    # Can also be set using 'ATMOS_COMPONENTS_TERRAFORM_COMMAND' ENV var, or '--terraform-command' command-line argument
    command: terraform
  helmfile:
    # Optional `command` specifies the executable to be called by `atmos` when running Helmfile commands
    # If not defined, `helmfile` is used
    # Examples:
    # command: helmfile
    # command: /usr/local/bin/helmfile
    # Can also be set using 'ATMOS_COMPONENTS_HELMFILE_COMMAND' ENV var, or '--helmfile-command' command-line argument
    command: helmfile

Related

• Closes #594

chore: Updated Tips for Creating GitHub App for Component Updater @milldr (#593)

what

• Updating the docs and tips for creating the Component Updater GitHub App

why

• A customer is going through this process now. I've added a tip for anything they've had trouble with

references

• n/a

v1.72.0

Update `gomplate` datasources. Add `env` and `evaluations ` sections to `Go` template configurations @aknysh (#599)

what

• Update gomplate datasources
• Add env section to Go template configurations
• Add evaluations config to Go template configurations
• Update/improve templating docs
  • https://atmos.tools/core-concepts/stacks/templating/

why

• Allow setting environment variables in Go templates to access cloud resources from datasources
• Allow configuring the number of template evaluations for template processing pipelines

description

Atmos supports configuring the number of evaluations/passes for template processing in atmos.yaml CLI config file. It effectively allows you to define template processing pipelines.

For example:

templates:
  settings:
    # Enable `Go` templates in Atmos stack manifests
    enabled: true
    # Number of evaluations/passes to process `Go` templates
    # If not defined, `evaluations` is automatically set to `1`
    evaluations: 2

• templates.settings.evaluations - number of evaluations/passes to process Go templates. If not defined, num_steps is automatically set to 1

Template evaluations are useful in the following scenarios:

• Combining templates from different sections in Atmos stack manifests
• Using templates in the URLs of datasources

Combining templates from different sections in Atmos stack manifests

You can define more than one step/pass of template processing to use and combine the results from each step.

For example:

templates:
  settings:
    enabled: true
    # Number of evaluations/passes to process `Go` templates
    evaluations: 3

settings:
  test: "{{ .atmos_component }}"
  test2: "{{ .settings.test }}"

components:
  terraform:
    vpc:
      vars:
        tags:
          tag1: "{{ .settings.test }}-{{ .settings.test2 }}"
          tag2: "{{\"{{`{{ .atmos_component }}`}}\"}}"

When executing an Atmos command like atmos terraform plan vpc -s <stack>, the above template will be processed in three phases:

• Evaluation 1

  • settings.test is set to vpc
  • settings.test2 is set to {{ .atmos_component }}
  • vpc.vars.tags.tag1 is set to {{ .atmos_component }}-{{ .settings.test }}
  • vpc.vars.tags.tag2 is set to {{<backtick>{{ .atmos_component }}<backtick>}}

• Evaluation 2

  • settings.test is vpc
  • settings.test2 is set to vpc
  • vpc.vars.tags.tag1 is set to vpc-vpc
  • vpc.vars.tags.tag2 is set to {{ .atmos_component }}

• Evaluation 3

  • settings.test is vpc
  • settings.test2 is vpc
  • vpc.vars.tags.tag1 is vpc-vpc
  • vpc.vars.tags.tag2 is set to vpc

WARNING:
The above example shows the supported functionality in Atmos templating. You can use it for some use-cases, but it does not mean that you should use it just for the sake of using, since it's not easy to read and understand what data we have after each evaluation step.

The following use-case describes a practical approach to using steps and pipelines in Atmos templates to work
with datasources.

Using templates in the URLs of datasources

Let's suppose that your company uses a centralized software catalog to consolidate all tags for tagging all the cloud resources. The tags can include tags per account, per team, per service, billing tags, etc.

NOTE: An example of such a centralized software catalog could be https://backstage.io

Let's also suppose that you have a service to read the tags from the centralized catalog and write them into an S3 bucket in one of your accounts. The bucket serves as a cache to not hit the external system's API with too many requests and not to trigger rate limiting.

And finally, let's say that in the bucket, you have folders per account (dev, prod, staging). Each folder has a JSON file with all the tags defined for all the cloud resources in the accounts.

We can then use the Gomplate S3 datasource to read the JSON file with the tags for each account and assign the tags to all cloud resources.

In atmos.yaml, we can figure 2 evaluation steps of template processing:

templates:
  settings:
    enabled: true
    gomplate:
      enabled: true
    # Number of evaluations to process `Go` templates
    evaluations: 2

In an Atmos stack manifest, we define the environment variables in the env section (AWS profile with permissions to access the S3 bucket), and the s3-tags Gomplate datasource.

In the terraform.vars.tags section, we define all the tags that are returned from the call to the S3 datasource.

import:
  # Import the default configuration for all VPCs in the infrastructure
  - catalog/vpc/defaults

# Global settings
settings:
  templates:
    settings:
      # Environment variables passed to datasources when evaluating templates
      # https://docs.gomplate.ca/functions/aws/#configuring-aws
      # https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html
      env:
        # AWS profile with permissions to access the S3 bucket
        AWS_PROFILE: "<AWS profile>"
      gomplate:
        # Timeout in seconds to execute the datasources
        timeout: 5
        # https://docs.gomplate.ca/datasources
        datasources:
          # `s3` datasource
          # https://docs.gomplate.ca/datasources/#using-s3-datasources
          s3-tags:
            # The `url` uses a `Go` template with the delimiters `${ }`,
            # which is processed as first step in the template processing pipeline
            url: "s3://mybucket/${ .vars.stage }/tags.json"

# Global Terraform config
terraform:
  # Global variables that are used by all Atmos components
  vars:
    tags:
      atmos_component: "{{ .atmos_component }}"
      atmos_stack: "{{ .atmos_stack }}"
      terraform_component: "{{ .component }}"
      terraform_workspace: "{{ .workspace }}"
      devops_team: '{{`{{ (datasource "s3-tags").tags.devops_team }}`}}'
      billing_team: '{{`{{ (datasource "s3-tags").tags.billing_team }}`}}'
      service: '{{`{{ (datasource "s3-tags").tags.service }}`}}'

# Atmos component configurations
components:
  terraform:
    vpc/1:
      metadata:
        component: vpc  # Point to the Terraform component in `components/terraform/vpc` folder
        inherits:
          # Inherit from the `vpc/defaults` base Atmos component, which defines the default 
          # configuration for all VPCs in the infrastructure.
          # The `vpc/defaults` base component is defined in the `catalog/vpc/defaults` 
          # manifest (which is imported above).
          # This inheritance makes the `vpc/1` Atmos component config DRY.
          - "vpc/defaults"
      vars:
        name: "vpc-1"

When executing an Atmos command like atmos terraform apply vpc/1 -s plat-ue2-dev, the above template will be processed in two steps:

• Evaluation 1:

  • datasources.s3-tags.url is set to s3://mybucket/dev/tags.json

  • the tags that use the datasource templates are set to the following:

    devops_team: '{{ (datasource "s3-tags").tags.devops_team }}'
    billing_team: '{{ (datasource "s3-tags").tags.billing_team }}'
    service: '{{ (datasource "s3-tags").tags.service }}'

• Evaluation 2:

  • all s3-tags datasources get executed, the JSON file s3://mybucket/dev/tags.json with the tags
    for the dev account is downloaded from the S3 bucket, and the tags are parsed and assigned in the
    terraform.vars.tags section

After executing the two evaluation steps, the resulting tags for the Atmos component vpc/1 in the stack plat-ue2-dev would look like this:

atmos_component: vpc/1
atmos_stack: plat-ue2-dev
terraform_component: vpc
terraform_workspace: plat-ue2-dev-vpc-1
devops_team: dev_networking
billing_team: billing_net
service: net

The tags will be added to all the AWS resources provisioned by the vpc Terraform component in the plat-ue2-dev stack.

v1.71.0

Update `atmos describe affected` command. Update docs @aknysh (#590)

what

• Update atmos describe affected command

• Add --clone-target-ref flag to the atmos describe affected command

• Update docs. Add "Remote State Backend" doc

  • https://atmos.tools/cli/commands/describe/affected/
  • https://atmos.tools/core-concepts/components/remote-state-backend/

why

• "Remote State Backend" doc describes how to override Terraform Backend Configuration to access components remote state, and how to do Brownfield development in Atmos

• Simplify the atmos describe affected command to not force it to clone the remote target reference (branch or tag), but instead just check it out (assuming the target reference is already on the local file system)

• Add --clone-target-ref flag to the atmos describe affected command for backwards compatibility. If the flag is passed, the command behaves as the old version (clones the target reference first from the remote origin)

breaking changes

If the atmos describe affected command was used in a GitHub Action (similar to https://github.com/cloudposse/github-action-atmos-affected-stacks), and the action performed a shallow Git clone (instead of a deep clone), it will break with an error that the target reference (branch) does not exist on the file system. There are a few ways to fix it:

• Use the flag --clone-target-ref=true to force the command to clone the target reference from the remote origin (this flag is addd for backwards compatibility)

  • atmos describe affected --clone-target-ref=true

• Update the GitHub Action to perform a deep-clone instead of a shallow-clone

    - uses: actions/checkout@v4
      with:
        fetch-depth: 0

• Perform a clone of the target branch into a separate directory and use the --repo-path=<dir> command line parameter to specify the path to the already cloned target repository (refer to https://atmos.tools/cli/commands/describe/affected#flags)

description

The atmos describe affected command uses two different Git commits to produce a list of affected Atmos components and stacks.

For the first commit, the command assumes that the current repo root is a Git checkout. An error will be thrown if the
current repo is not a Git repository (the .git/ folder does not exist or is configured incorrectly).

The second commit can be specified on the command line by using the --ref (Git References) or --sha (commit SHA) flags. The --sha takes precedence over the --ref flag.

How does it work?

The command performs the following:

• If the --repo-path flag is passed, the command uses it as the path to the already cloned target repo with which to compare the current working branch. I this case, the command will not clone and checkout the target reference, but instead will use the already cloned one to compare the current branch with. In this case, the --ref, --sha, --ssh-key and --ssh-key-password flags are not used, and an error will be thrown if the --repo-path flag and any of the --ref, --sha, --ssh-key or --ssh-key-password flags are provided at the same time

• Otherwise, if the --clone-target-ref=true flag is specified, the command clones (into a temp directory) the remote target with which to compare the current working branch. If the --ref flag or the commit SHA flag --sha are provided, the command uses them to clone and checkout the remote target. Otherwise, the HEAD of the remote origin is used (refs/remotes/origin/HEAD Git ref, usually the main branch)

• Otherwise, (if the --repo-path and --clone-target-ref=true flags are not passed), the command does not clone anything from the remote origin, but instead just copies the current repo into a temp directory and checks out the target reference with which to compare the current working branch. If the --ref flag or the commit SHA flag --sha are provided, the command uses them to check out. Otherwise, the HEAD of the remote origin is used (refs/remotes/origin/HEAD Git ref, usually the main branch). This requires that the target reference is already cloned by Git, and the information about it exists in the .git directory (in case of using a non-default branch as the target, Git deep clone needs to be executed instead of a shallow clone). This is the recommended way to execute the atmos describe affected command since it allows working with private repositories without providing the SSH credentials (--ssh-key and --ssh-key-password flags), since in this case Atmos does not access the remote origin and instead just checks out the target reference (which is already on the local file system)

• The command deep-merges all stack configurations from both sources: the current working branch and the target reference

• The command searches for changes in the component directories

• The command compares each stack manifest section of the stack configurations from both sources looking for differences

• And finally, the command outputs a JSON or YAML document consisting of a list of the affected components and stacks and what caused it to be affected

v1.70.0

Add `gomplate` datasources to `Go` templates in Atmos stack manifests. Update docs @aknysh (#582)

what

• Add gomplate datasources to Go templates in Atmos stack manifests
• Fix an issue with enabling/disabling Go templates in imports using the templates.settings.enabled section in atmos.yaml
• Update docs
  • https://atmos.tools/core-concepts/stacks/templating/

why

• Allow using Gomplate Datasources in Go templates in Atmos stack manifests
• The templates.settings.enabled section in atmos.yaml also affected templating in imports, but templating in imports should always be enabled regardless of enabling/disabling templating in Atmos stack manifests

description

Atmos supports Go templates in stack manifests.

Sprig Functions, Gomplate Functions and Gomplate Datasources are supported as well.

Configuration

Templating in Atmos stack manifests can be configured in the following places:

• In the templates.settings section in atmos.yaml CLI config file

• In the settings.templates.settings section in Atmos stack manifests. The settings.templates.settings section can be defined globally per organization, tenant, account, or per component. Atmos deep-merges the configurations from all scopes into the final result using inheritance.

Configuring templating in atmos.yaml CLI config file

Templating in Atmos stack manifests is configured in the atmos.yaml CLI config file in the templates.settings section:

# https://pkg.go.dev/text/template
templates:
  settings:
    enabled: true
    # https://masterminds.github.io/sprig
    sprig:
      enabled: true
    # https://docs.gomplate.ca
    # https://docs.gomplate.ca/functions
    gomplate:
      enabled: true
      # Timeout in seconds to execute the datasources
      timeout: 5
      # https://docs.gomplate.ca/datasources
      datasources:
        # 'http' datasource
        # https://docs.gomplate.ca/datasources/#using-file-datasources
        ip:
          url: "https://api.ipify.org?format=json"
          # https://docs.gomplate.ca/datasources/#sending-http-headers
          # https://docs.gomplate.ca/usage/#--datasource-header-h
          headers:
            accept:
              - "application/json"
        # 'file' datasources
        # https://docs.gomplate.ca/datasources/#using-file-datasources
        config-1:
          url: "./config1.json"
        config-2:
          url: "file:///config2.json"
        # `aws+smp` AWS Systems Manager Parameter Store datasource
        # https://docs.gomplate.ca/datasources/#using-awssmp-datasources
        secret-1:
          url: "aws+smp:///path/to/secret"
        # `aws+sm` AWS Secrets Manager datasource
        # https://docs.gomplate.ca/datasources/#using-awssm-datasource
        secret-2:
          url: "aws+sm:///path/to/secret"
        # `s3` datasource
        # https://docs.gomplate.ca/datasources/#using-s3-datasources
        s3-config:
          url: "s3://mybucket/config/config.json"

• templates.settings.enabled - a boolean flag to enable/disable the processing of Go templates in Atmos stack manifests. If set to false, Atmos will not process Go templates in stack manifests

• templates.settings.sprig.enabled - a boolean flag to enable/disable the Sprig Functions in Atmos stack manifests

• templates.settings.gomplate.enabled - a boolean flag to enable/disable the Gomplate Functions and Gomplate Datasources in Atmos stack manifests

• templates.settings.gomplate.timeout - timeout in seconds to execute Gomplate Datasources

• templates.settings.gomplate.datasources - a map of Gomplate Datasource definitions:

  • The keys of the map are the datasource names, which are used in Go templates in Atmos stack manifests.
    For example:

     terraform:
       vars:
         tags:
           provisioned_by_ip: '{{ (datasource "ip").ip }}'
           config1_tag: '{{ (datasource "config-1").tag }}'
           config2_service_name: '{{ (datasource "config-2").service.name }}'

  • The values of the map are the datasource definitions with the following schema:

    • url - the Datasource URL

    • headers - a map of HTTP request headers for the http datasource. The keys of the map are the header names. The values of the map are lists of values for the header.

      The following configuration will result in the accept: application/json HTTP header being sent with the HTTP request to the datasource:

      headers:
        accept:
          - "application/json"

Configuring templating in Atmos stack manifests

The settings.templates.settings section can be defined globally per organization, tenant, account, or per component. Atmos deep-merges the configurations from all scopes into the final result using inheritance.

For example, define Gomplate Datasources for the entire organization in the stacks/orgs/acme/_defaults.yaml stack manifest:

settings:
  templates:
    settings:
      gomplate:
        # 7 seconds timeout to execute the datasources
        timeout: 7
        # https://docs.gomplate.ca/datasources
        datasources:
          # 'file' datasources
          # https://docs.gomplate.ca/datasources/#using-file-datasources
          config-1:
            url: "./my-config1.json"
          config-3:
            url: "file:///config3.json"

Atmos deep-merges the configurations from the settings.templates.settings section in Atmos stack manifests with the templates.settings section in atmos.yaml CLI config file using inheritance.

The settings.templates.settings section in Atmos stack manifests takes precedence over the templates.settings section in atmos.yaml CLI config file, allowing you to define the global datasources in atmos.yaml and then add or override datasources in Atmos stack manifests for the entire organization, tenant, account, or per component.

For example, taking into account the configurations described above in atmos.yaml CLI config file and in the stacks/orgs/acme/_defaults.yaml stack manifest, the final datasources map will look like this:

gomplate:
  timeout: 7
  datasources:
    ip:
      url: "https://api.ipify.org?format=json"
      headers:
        accept:
          - "application/json"
    config-1:
      url: "./my-config1.json"
    config-2:
      url: "file:///config2.json"
    config-3:
      url: "file:///config3.json"

Note that the config-1 datasource from atmos.yaml was overridden with the config-1 datasource from the stacks/orgs/acme/_defaults.yaml stack manifest. The timeout attribute was overridden as well.

You can now use the datasources in Go templates in all Atmos sections that support Go templates.

Atmos sections supporting Go templates

You can use Go templates in the following Atmos sections to refer to values in the same or other sections:

• vars
• settings
• env
• metadata
• providers
• overrides
• backend
• backend_type

For example, let's say we have the following component configuration using Go templates:

component:
  terraform:
    vpc:
      settings:
        setting1: 1
        setting2: 2
        setting3: "{{ .vars.var3 }}"
        setting4: "{{ .settings.setting1 }}"
        component: vpc
        backend_type: s3
        region: "us-east-2"
        assume_role: "<role-arn>"
      backend_type: "{{ .settings.backend_type }}"
      metadata:
        component: "{{ .settings.component }}"
      providers:
        aws:
          region: "{{ .settings.region }}"
          assume_role: "{{ .settings.assume_role }}"
      env:
        ENV1: e1
        ENV2: "{{ .settings.setting1 }}-{{ .settings.setting2 }}"
      vars:
        var1: "{{ .settings.setting1 }}"
        var2: "{{ .settings.setting2 }}"
        var3: 3
        # Add the tags to all the resources provisioned by this Atmos component
        tags:
          atmos_component: "{{ .atmos_component }}"
          atmos_stack: "{{ .atmos_stack }}"
          atmos_manifest: "{{ .atmos_stack_file }}"
          region: "{{ .vars.region }}"
          terraform_workspace: "{{ .workspace }}"
          assumed_role: "{{ .providers.aws.assume_role }}"
          description: "{{ .atmos_component }} component provisioned in {{ .atmos_stack }} stack by assuming IAM role {{ .providers.aws.assume_role }}"
          # Examples of using the Sprig and Gom...

v1.69.0

In Atmos v1.55 (PR #515) we switched to using the TF_WORKSPACE environment variable for selecting Terraform workspaces when issuing Terraform commands. This had the unintended consequence that the workspace was no longer being set for use outside of Atmos as a side effect. This version reverses that change, once again setting the workspace via terraform workspace select so that it remains selected after Atmos finishes. Expand the details below for an extended explanation.

Use `terraform workspace select` to select workspace @Nuru (#580)

what

• Use terraform workspace select to select the workspace when needed
• Do not ask for Read permissions for output files
• Only attempt terraform workspace new when select returns with exit code 1; report any other kind of error

why

• Restores previous behavior, which includes fixing #574, and leaving the active workspace selected for use by terraform commands outside of Atmos
• Prevents users running under atmos terraform shell from having issues running terraform workspace commands or having the workspace remain selected after changing directories
• Atmos does not need Read permission on outputs, and redirected outputs may not be readable, causing unnecessary failures
• Avoid hiding unexpected errors

discussion

While Atmos is a powerful tool and for some there is a desire to make Atmos fully capable of replacing the terraform CLI entirely, we are not there yet. For now, there is a need to be able to use terraform alongside with atmos in some edge cases. Also, for some, there is a desire to bypass some of Atmos' overhead when running repeated tasks (especially failing tasks) by invoking terraform commands directly.

The issue of selecting a Terraform "workspace" highlights the kind of difficulty one runs into when designing a solution where Atmos and Terraform can work side by side. Terraform has the concept of, for each component, a "current workspace", which is a segregated Terraform state. Cloud Posse uses a different workspace for each deployment of a component (account and region and, where the component is deployed multiple times in the same account and region, each separate deployment).

The problem with a workspace selection is that it is stateful, modal, and mostly hidden. When you run a terraform command, the component it applies to is determined by the current working directory (one form of mode/state), and the workspace it uses is determined by either:

1. The most recent invocation of terraform workspace select in that directory
2. The value of the environment variable TF_WORKSPACE

with the environment variable taking precedence.

To further complicate matters, if you run terraform workspace select foo and the workspace foo has not been created, you get an error. Likewise, if you run TF_WORKSPACE=foo terraform init -reconfigure and the workspace foo does not exist, you get an error. But if you run TF_WORKSPACE=foo terraform plan and the workspace foo does not exist, it will silently and automatically be created for you (so beware of typos).

Whichever mechanism you choose, you are setting some kind of persistent state which affects what terraform commands affect, and it becomes easy to get lost as to what you are actually working on. Atmos handles this by always setting the state for its Terraform commands, which is admirable, and it can be more helpful to people using terraform directly if it helps those users manage this hidden state effectively.

Prior to Atmos version 1.55, its behavior was that if it ran any Terraform command that applied to a specific workspace (anything other than init), it selected the workspace, creating it if needed, and left the workspace selected. This means that if you working in the component directory, then any terraform commands like terraform state show would work on the component in the workspace of the stack you most recently used for that component. After running atmos terraform plan <component> -s <stack>, you could do everything else using plain Terraform commands if you wanted to (the only trick being that, for some commands, you would need to supply the -var-file that Atmos generated). While many users would never do things this way, in some use cases it can be particularly time saving. For example, you may want to run the same set of Terraform commands repeatedly on the same component, using different stacks, as can happen when migrating a component to a new version requires some manual steps.

As of Atmos 1.55, Atmos changed how it selects a workspace. It started using the environment variable TF_WORKSPACE instead of terraform workspace select. While this works for Atmos, it changes the behavior for running the bare terraform commands. No longer did Atmos set the workspace for you, nor would atmos terraform workspace <component> -s <stack> set the workspace for you, either. In fact, there was no longer any direct way to find out the correct workspace name to use. Your plain terraform commands worked on the default workspace, or whatever one was selected the last time you used an old version of Atmos.

So in summary, the change made using terraform harder to use, without appreciably making atmos easier to use. The change did eliminate a bug that was causing problems, and the consequences were not obvious (nor universally considered negative), so it was made. This PR fixes the bug a different way and restores the behavior of v1.54 and earlier.

In the long run, we are seeking ways to make these kinds of behaviors optional and easier to choose. For now, we are just reverting to the "tried-and-true" behavior to avoid further surprises.

references

• Reverts #515
• Fixes #574

v1.68.0

Add `gomplate` templating engine to Atmos stack manifests. Update docs @aknysh (#578)

Breaking changes

If you used Go templates in Atmos stack manifest before, they were enabled by default.
Starting with this release, you have to explicitly enable Go templating in atmos.yaml and enable either Sprig or Gomplate (or both).
See the description below.

what

• Add gomplate templating engine to Atmos stack manifests
• Update docs:
  • https://atmos.tools/core-concepts/stacks/templating/
  • https://atmos.tools/cli/configuration/

why

• Allow using the rich collection of the Gomplate Functions in Atmos stack manifest

• NOTE: The Gomplate Datasources, including configuring the datasources in Atmos stack manifests and using Inheritance for datasources, will be added in the follow up PR

Description

Configuration

Templating in Atmos stack manifests is configured in the atmos.yaml CLI config file in the templates section:

# https://pkg.go.dev/text/template
templates:
  settings:
    enabled: true
    # https://masterminds.github.io/sprig
    sprig:
      enabled: true
    # https://docs.gomplate.ca
    gomplate:
      enabled: true

• templates.settings.enabled - a boolean flag to enable/disable the processing of Go templates in Atmos stack manifests. If set
  to false, Atmos will not process Go templates in stack manifests

• templates.settings.sprig.enabled - a boolean flag to enable/disable the Sprig Functions
  in Atmos stack manifests

• templates.settings.gomplate.enabled - a boolean flag to enable/disable the Gomplate Functions
  in Atmos stack manifests

Atmos sections supporting Go templates

You can use Go templates in the following Atmos sections to refer to values in the same or other sections:

• vars
• settings
• env
• metadata
• providers
• overrides
• backend
• backend_type

For example, let's say we have the following component configuration using Go templates:

component:
  terraform:
    vpc:
      settings:
        setting1: 1
        setting2: 2
        setting3: "{{ .vars.var3 }}"
        setting4: "{{ .settings.setting1 }}"
        component: vpc
        backend_type: s3
        region: "us-east-2"
        assume_role: "<role-arn>"
      backend_type: "{{ .settings.backend_type }}"
      metadata:
        component: "{{ .settings.component }}"
      providers:
        aws:
          region: "{{ .settings.region }}"
          assume_role: "{{ .settings.assume_role }}"
      env:
        ENV1: e1
        ENV2: "{{ .settings.setting1 }}-{{ .settings.setting2 }}"
      vars:
        var1: "{{ .settings.setting1 }}"
        var2: "{{ .settings.setting2 }}"
        var3: 3
        # Add the tags to all the resources provisioned by this Atmos component
        tags:
          atmos_component: "{{ .atmos_component }}"
          atmos_stack: "{{ .atmos_stack }}"
          atmos_manifest: "{{ .atmos_stack_file }}"
          region: "{{ .vars.region }}"
          terraform_workspace: "{{ .workspace }}"
          assumed_role: "{{ .providers.aws.assume_role }}"
          description: "{{ .atmos_component }} component provisioned in {{ .atmos_stack }} stack by assuming IAM role {{ .providers.aws.assume_role }}"
          # Examples of using the Gomplate and Sprig functions
          # https://docs.gomplate.ca/functions/strings
          atmos_component_description: "{{ strings.Title .atmos_component }} component {{ .vars.name | strings.Quote }} provisioned in the stack {{ .atmos_stack | strings.Quote }}"
          # https://masterminds.github.io/sprig/os.html
          provisioned_by_user: '{{ env "USER" }}'

When executing Atmos commands like atmos describe component and atmos terraform plan/apply, Atmos processes all the template tokens in the manifest and generates the final configuration for the component in the stack:

settings:
  setting1: 1
  setting2: 2
  setting3: 3
  setting4: 1
  component: vpc
  backend_type: s3
  region: us-east-2
  assume_role: <role-arn>
backend_type: s3
metadata:
  component: vpc
providers:
  aws:
    region: us-east-2
    assume_role: <role-arn>
env:
  ENV1: e1
  ENV2: 1-2
vars:
  var1: 1
  var2: 2
  var3: 3
  tags:
    assumed_role: <role-arn>
    atmos_component: vpc
    atmos_component_description: Vpc component "common" provisioned in the stack "plat-ue2-dev"
    atmos_manifest: orgs/acme/plat/dev/us-east-2
    atmos_stack: plat-ue2-dev
    description: vpc component provisioned in plat-ue2-dev stack by assuming IAM role <role-arn>
    provisioned_by_user: <user>
    region: us-east-2
    terraform_workspace: plat-ue2-dev

Use-cases

While Go templates in Atmos stack manifests offer great flexibility for various use-cases, one of the obvious use-cases is to add a standard set of tags to all the resources in the infrastructure.

For example, by adding this configuration to the stacks/orgs/acme/_defaults.yaml Org-level stack manifest:

terraform:
  vars:
    tags:
      atmos_component: "{{ .atmos_component }}"
      atmos_stack: "{{ .atmos_stack }}"
      atmos_manifest: "{{ .atmos_stack_file }}"
      terraform_workspace: "{{ .workspace }}"
      # Examples of using the Gomplate and Sprig functions
      # https://docs.gomplate.ca/functions/strings
      atmos_component_description: "{{ strings.Title .atmos_component }} component {{ .vars.name | strings.Quote }} provisioned in the stack {{ .atmos_stack | strings.Quote }}"
      # https://masterminds.github.io/sprig/os.html
      provisioned_by_user: '{{ env "USER" }}'

The tags will be processed and automatically added to all the resources provisioned in the infrastructure.

Excluding templates from processing by Atmos

If you need to provide Go templates to external systems (e.g. ArgoCD or Datadog) verbatim and prevent Atmos from processing the templates, use double curly braces + backtick + double curly braces instead of just double curly braces:

{{`{{  instead of  {{

}}`}}  instead of  }}

For example:

components:
  terraform:

    eks/argocd:
      metadata:
        component: "eks/argocd"
      vars:
        enabled: true
        name: "argocd"
        chart_repository: "https://argoproj.github.io/argo-helm"
        chart_version: 5.46.0

        chart_values:
          template-github-commit-status:
            message: |
              Application {{`{{ .app.metadata.name }}`}} is now running new version.
            webhook:
              github-commit-status:
                method: POST
                path: "/repos/{{`{{ call .repo.FullNameByRepoURL .app.metadata.annotations.app_repository }}`}}/statuses/{{`{{ .app.metadata.annotations.app_commit }}`}}"
                body: |
                  {
                    {{`{{ if eq .app.status.operationState.phase "Running" }}`}} "state": "pending"{{`{{end}}`}}
                    {{`{{ if eq .app.status.operationState.phase "Succeeded" }}`}} "state": "success"{{`{{end}}`}}
                    {{`{{ if eq .app.status.operationState.phase "Error" }}`}} "state": "error"{{`{{end}}`}}
                    {{`{{ if eq .app.status.operationState.phase "Failed" }}`}} "state": "error"{{`{{end}}`}},
                    "description": "ArgoCD",
                    "target_url": "{{`{{ .context.argocdUrl }}`}}/applications/{{`{{ .app.metadata.name }}`}}",
                    "context": "continuous-delivery/{{`{{ .app.metadata.name }}`}}"
                  }

When Atmos processes the templates in the manifest shown above, it renders them as raw strings allowing sending the templates to the external system for processing:

chart_values:
  template-github-commit-status:
    message: |
      Application {{ .app.metadata.name }} is now running new version.
    webhook:
      github-commit-status:
        method: POST
        path: "/repos/{{ call .repo.FullNameByRepoURL .app.metadata.annotations.app_repository }}/statuses/{{ .app.metadata.annotations.app_commit }}"
        body: |
          {
            {{ if eq .app.status.operationState.phase "Running" }} "state": "pending"{{end}}
            {{ if eq .app.status.operationState.phase "Succeeded" }} "state": "success"{{end}}
            {{ if eq .app.status.operationState.phase "Error" }} "state": "error"{{end}}
            {{ if eq .app.status.operationState.phase "Failed" }} "state": "error"{{end}},
            "description": "ArgoCD",
            "target_url": "{{ .context.argocdUrl }}/applications/{{ .app.metadata.name }}",
            "context": "continuous-delivery/{{ .app.metadata.name }}"
          }

The printf template function is also supported and can be used instead of double curly braces + backtick + double curly braces.

The following examples produce the same result:

chart_values:
  template-github-commit-status:
    message: >-
      Application {{`{{ .app.metadata.name }}`}} is now running new version.

chart_values:
  template-github-commit-status:
    message: "Application {{`{{ .app.metadata.name }}`}} is now running new version."

chart_values:
  template-github-commit-status:
    message: >-
      {{ printf "Application {{ .app.metadata.name }} is now running new version." ...