Orphaned management resource groups accumulate in the CI subscription when the core deployment never completes

[JayDoubleu]

Describe the bug

The PR validation pipeline creates a management resource group before it
creates the core resource group. If the pipeline stops before the core
deployment completes, the management resource group stays in the subscription.
No automated process can find it or delete it after that.

The validation subscription now holds 8 such resource groups. The oldest dates
from 2026-06-18.

Resource group	PR	PR state	Contents
rg-tre245d1eab-mgmt	#4853	closed 2026-07-14	empty
rg-tred81faf5c-mgmt	#4764	closed 2026-06-30	empty
rg-tred8e27863-mgmt	#4918	closed 2026-07-10	empty
rg-tree82ee0ff-mgmt	#4638	closed 2026-07-31	empty
rg-tre77179ad2-mgmt	#4904	open	empty
rg-trea20534b2-mgmt	#4932	open	empty
rg-tre4403460f-mgmt	#4927	open	storage account, 2026-06-30
rg-tre2b467883-mgmt	#4964	open	storage account and Premium ACR, 2026-07-17

Each name maps to a PR through sha512("refs/pull/<N>/merge\n")[:8], as
.github/scripts/build.js:286
defines it.

None of these resource groups ever had a core deployment. Each core deployment
creates the key vault kv-<tre_id>. Teardown leaves this key vault in the
soft-deleted state, because KV_PURGE_PROTECTION_ENABLED defaults to true
(deploy_tre_reusable.yml:411).
The subscription holds 198 soft-deleted key vaults, and many of them are core
key vaults with the name kv-tre<refid>. None of the 8 identifiers above has a
key vault, live or soft-deleted.

All file references below are permalinks to main at commit
9a1691e.

Two defects make this permanent.

Defect 1: the management resource group has no owner until the core
deployment succeeds

devops/terraform/bootstrap.sh:50
runs az group create as the first Azure command, and it adds no tags. The
deploy_management
job runs first in the pipeline. The core resource group appears much later, in
the deploy_tre
job. Any failure between these two points leaves a management resource group
with no tags.

Two runs show this directly:

• Run 28439249321
  for Fix health check query not executed #4927 failed at the Deploy management step of the Deploy Management
  job. The resource group rg-tre4403460f-mgmt has the storage account that
  bootstrap.sh creates, but no ACR. The management terraform never completed.
• Run 29574605411
  for Redesign airlock storage account architecture #4964 completed the Deploy Management job. Then
  Build Core Docker Images (build-and-push-api) failed, and the core
  deployment never started. The resource group rg-tre2b467883-mgmt has the
  storage account and the ACR, but no core resource group exists.

A failure inside bootstrap.sh gives an empty resource group. Every step after
az group create can fail and stop the script. These steps
create the storage account, wait for the role assignment to propagate, create
the containers, and run terraform init.
Six of the 8 resource groups above are empty, which matches this failure.

Defect 2: the ci_git_ref tag goes only on the core resource group

core/terraform/main.tf:62
is the only place that sets this tag. The workflow passes TF_VAR_ci_git_ref
into make deploy-core,
but not into make bootstrap mgmt-deploy.
The cleanup query is:

az group list --query "[?starts_with(name, 'rg-tre') && tags.ci_git_ref != null && starts_with(tags.ci_git_ref, 'refs')].[name, tags.ci_git_ref]" -o tsv

(devops/scripts/clean_ci_validation_envs.sh:30)

A management resource group has no such tag. clean_ci_validation_envs.sh
therefore cannot list it. Every -mgmt resource group in the subscription
reports ci_git_ref: null, and this includes the groups of healthy live
environments. The cleanup reaches a management resource group only as a side
effect, when it deletes the core resource group with the
same prefix.
If the core resource group never existed, no code path calls the destroy script
for that identifier.

#4925 corrected a related fault. Before that fix,
destroy_env_no_terraform.sh stopped early when the core resource group was
absent. /test-destroy-env now deletes these management resource groups
correctly. But rg-tre4403460f-mgmt and rg-tre2b467883-mgmt appeared after
that fix, and they are still present. The automatic cleanup never finds them.

Steps to reproduce

1. Comment /test on a PR.
2. Make the pipeline fail after the Deploy Management job starts and before
   the Deploy TRE job completes.
3. Look at the subscription. The resource group rg-tre<refid>-mgmt exists and
   has no ci_git_ref tag.
4. Close the PR, or wait for the Clean Validation Environments workflow. The
   resource group stays.

Azure TRE release version (e.g. v0.14.0 or main): main (9a1691e)

Deployed Azure TRE components - click the (i) in the UI: not applicable,
this is CI infrastructure

Expected behaviour

The cleanup must delete a management resource group on the same schedule as its
core resource group. This applies when the PR closes, when the branch is gone,
and after the inactivity limit.

Suggested fix

Add the tag when the script creates the resource group, not when terraform
applies it. The tag then survives a failure of make bootstrap. In
devops/terraform/bootstrap.sh:50:

az group create --resource-group "$TF_VAR_mgmt_resource_group_name" \
  --location "$LOCATION" \
  ${TF_VAR_ci_git_ref:+--tags "ci_git_ref=$TF_VAR_ci_git_ref"} -o table

Pass TF_VAR_ci_git_ref into the
make bootstrap mgmt-deploy
step. Add the same tag to the management resource group in devops/terraform,
so terraform does not remove it again.

This tag alone makes these resource groups visible to the current query,
because rg-tre<refid>-mgmt already matches the rg-tre prefix. One more
change is necessary in clean_ci_validation_envs.sh. The
stopEnv()
function removes the rg- prefix to get the TRE identifier. For a management
resource group it gets tre<refid>-mgmt, and control_tre.sh stop fails. Send
management resource groups direct to the destroy step, and skip the stop step.

Additional context

Two smaller items, for completeness:

• Deployments that use the rg-mgmt-<tre_id> name from config.sample.yaml
  and AzureTRE-Deployment never match the
  prefix logic
  in destroy_env_no_terraform.sh, because rg-mgmt-foo does not start with
  rg-foo. make mgmt-destroy exists, but no automated path calls it.
• Clean validation workflow failing due to az CLI change #4971 reports that the cleanup workflow failed from 2026-07-13. Runs
  succeeded again later, but the last one was on 2026-08-03. This can add to
  the problem, separate from the two defects above.