atlantis merges PR when apply fails and policy checks are enabled

[aston-r]

Hi,

atlantis version: v0.28.5 (current latest version). I have simple workflow:

• users create PR in Github and post comment: atlantis apply
• atlantis merges pr only if apply is succesfull

This setup worked for years. Today I extended workflow and added the following configration:

• env vars:

ATLANTIS_ENABLE_POLICY_CHECKS=true
ATLANTIS_QUIET_POLICY_CHECKS=true

• server-side config:

custom_policy_check: true
...
workflows:
  terragrunt:
    plan:
      steps:
        - run: /tg_plan.sh
    apply:
      steps:
        - run: /tg_apply.sh
    policy_check:
      steps:
        - run: /policy_check.sh

Actual Result:

• apply fails and atlantis merges pr

time=2024-08-21T10:15:43Z level=error msg=terraform invocation failed in /home/atlantis/.atlantis/repos/... 
time=2024-08-21T10:15:43Z level=error msg=1 error occurred:
	* [/home/atlantis/.atlantis/repos/...] exit status 1

{
    "level": "error",
    "ts": "2024-08-21T10:15:43.757Z",
    "caller": "models/shell_command_runner.go:158",
    "msg": "running \"/tg_apply.sh\" in \"/home/atlantis/.atlantis/repos/...\": exit status 1",
    "json": {
        "repo": "xxxx",
        "pull": "10205",
        "duration": 6.340362671
    },
    "stacktrace": "github.com/runatlantis/atlantis/server/core/runtime/models.(*ShellCommandRunner).RunCommandAsync.func1\n\tgithub.com/runatlantis/atlantis/server/core/runtime/models/shell_command_runner.go:158"
}

Automatically merging because all plans have been successfully applied.

I reverted changes regarding policies and atlantis does not merge pr. I even tried to apply changes with polices one more time and the same issue(atlantis merges pr with failed apply)

It looks as atlantis checks that policy checks are ok and merges(