feat: Enable skip of CI deployment tests #4203

ReneHezser · 2025-01-17T14:33:16Z

Description

Allow testcases to be excluded from deplyoments by the CI deployment.

Pipeline Reference

Pipeline

(intended)

Type of Change

Update to CI Environment or utilities (Non-module affecting changes)
Azure Verified Module updates:
- Bugfix containing backwards-compatible bug fixes, and I have NOT bumped the MAJOR or MINOR version in version.json:
  - Someone has opened a bug report issue, and I have included "Closes #{bug_report_issue_number}" in the PR description.
  - The bug was found by the module author, and no one has opened an issue to report it yet.
- Feature update backwards compatible feature updates, and I have bumped the MINOR version in version.json.
- Breaking changes and I have bumped the MAJOR version in version.json.
- Update to documentation

Checklist

I'm sure there are no other open Pull Requests for the same update/change
I have run Set-AVMModule locally to generate the supporting module files.
My corresponding pipelines / checks run clean and green without any errors or warnings

.github/actions/templates/avm-validateModuleDeployment/action.yml

eriqua · 2025-01-17T22:17:47Z

.github/CODEOWNERS

@@ -180,3 +180,4 @@
 /avm/res/web/static-site/ @Azure/avm-res-web-staticsite-module-owners-bicep @Azure/avm-module-reviewers-bicep
 /avm/utl/types/avm-common-types/ @Azure/avm-utl-types-avmcommontypes-module-owners-bicep @Azure/avm-module-reviewers-bicep
 *avm.core.team.tests.ps1 @Azure/avm-core-team-technical-bicep
+*.e2eignore @Azure/avm-core-team-technical-bicep


Looking forward to discussing, great work meanwhile!
The main suggestion I'd have for now is to add tests. At least one pipeline run with one deployment test skipped and one base case pipeline with no test skipped

Thanks @eriqua.
I am not sure I understand your remark. You mean we should have a failed test before the test can be excluded?

Hi @ReneHezser exactly, suggested full test coverage for CI updates before they are introduced.

Static validation: we introduced 2 pester tests so we need:

module run with no test skipped --> should pass

module run where deployment tests are skipped as intended--> should pass

module run with one test skipped incorrectly (.e2eignore added in defaults or waf-aligned test) --> should fail pester test 1

module run with one test skipped incorrectly (.e2eignore is empty) --> should fail pester test 2

Hi @eriqua
I added testcases for the mentioned scenarios. Before merging, I need to remove the tests again.

Hey @ReneHezser, thanks. To show that the Pester test fails as intended, we need a link to the corresponding failing pipeline(s). Could you please attach a link to the failing pipeline runs?

I thought I already did, but probably forgot to save the update on the description. Sorry.

Sorry but still don't see them. Looking for specific pipeline runs, validating the four cases above, not pipeline badges.

.github/actions/templates/avm-getModuleTestFiles/action.yml

.github/actions/templates/avm-validateModuleDeployment/action.yml

.github/workflows/avm.template.module.yml

AlexanderSehr · 2025-03-01T17:25:18Z

utilities/pipelines/staticValidation/compliance/module.tests.ps1

+                [string] $moduleFolderPath
+            )
+
+            $e2eTestFolderPathList = Get-ChildItem -Directory (Join-Path -Path $moduleFolderPath 'tests' 'e2e') | Where-Object { $_.Name -in 'defaults', 'waf-aligned' }


Not sure if that's a good check. Our convention is not that the folders must be called defaults, but that there must be a test folder that ends with that in the name (same for waf).
You can find an example for this in e.g., the res/compute/virtual-machine module.
Also there the test in this file calls this out

"Module should contain a [tests/e2e/*defaults] folder."

A fix would be as easy as to thest that the name matches either defaults/waf-aligned (you could check for both in the same regex if you want).

Good catch. I understand that some other test already failed over this.

I guess we may want to also have the same logic for the waf-aligned test? I.e., -like '*waf-aligned'?

utilities/pipelines/staticValidation/compliance/module.tests.ps1

AlexanderSehr · 2025-03-01T17:30:41Z

utilities/tools/Test-ModuleLocally.ps1

+            $passCiFilePath = Join-Path $moduleTestFolderPath '.e2eignore'
+            if (Test-Path $passCiFilePath) {
+                Write-Output "File '.e2eignore' exists in the folder: $moduleTestFolderPath"
+                # end here, as the test should be bypassed
+                return


If we must have it in here, please make it conditional via a switch statement the user must enable.
Where I to run tests locally, this would prevent me from running the test case even if I wanted to and would anyways only make sense if the user did not specify a single test file. Would feel very odd if I specifically provide a test file that happens to have the .e2eignore file in the folder (which may not work in the CI/AVM subscription, but may in mine), yet I cannot test it with this script 😄

Co-authored-by: Alexander Sehr <[email protected]>

AlexanderSehr · 2025-03-04T07:14:59Z

utilities/pipelines/staticValidation/compliance/module.tests.ps1

+            foreach ($e2eTestFolderPath in $e2eTestFolderPathList) {
+                $filePath = Join-Path -Path $e2eTestFolderPath '.e2eignore'
+                if (Test-Path $filePath) {
+                    $incorrectParameters += $e2eTestFolderPath.Name


I guess it's more like 'incorrectFolders', not 'incorrectParameters'

AlexanderSehr · 2025-03-04T07:15:29Z

utilities/pipelines/staticValidation/compliance/module.tests.ps1

+                if ($pathExisting) {
+                    $fileContent = Get-Content -Path $filePath
+                    if (-not $fileContent) {
+                        $incorrectParameters += $e2eTestFolderPath.Name + '\.e2eignore'


Same as above

AlexanderSehr · 2025-03-04T07:26:16Z

utilities/tools/Test-ModuleLocally.ps1

@@ -234,8 +235,18 @@ function Test-ModuleLocally {
            # -------------------------
            if ((Get-Item -Path $ModuleTestFilePath) -is [System.IO.DirectoryInfo]) {
                $moduleTestFiles = (Get-ChildItem -Path $ModuleTestFilePath -File).FullName
+                $moduleTestFolderPath = $ModuleTestFilePath


Beware, moduleTestFiles is an array in this case and this logic should not work. Did you validate how this behaves? Your logic would need to run below in the loops across the test files.

AlexanderSehr · 2025-03-04T07:34:01Z

utilities/tools/Test-ModuleLocally.ps1

@@ -19,7 +19,7 @@ Optional. A string array that can be specified to run only Pester tests with the
 Optional. A switch parameter that triggers a Pester test for the module

 .PARAMETER ValidateOrDeployParameters


Ideally, we don't inject the new condition into this object as it represents the parameters that are passed into the Test-TemplateDeployment & New-TemplateDeployment scripts (i.e., we want to be as close to the workflow as possible). I'd instead recommend an additional parameter. For example,

[Parameter(Mandatory = $false)] [switch] $RespectE2eIgnoreFlag,

Still very much an active comment 😉

.github/actions/templates/avm-validateModuleDeployment/action.yml

ReneHezser and others added 6 commits January 14, 2025 20:22

check for a file .e2eignore in test folders

475bfce

LF

3c848bd

formats output

219f6ff

ensure the defaults and waf-aligned test for res modules

17e6b6d

require core team approval

5aec7c1

Merge branch 'Azure:main' into enable-skip-ci-deployment-tests

8a79bc3

microsoft-github-policy-service bot added Needs: Triage 🔍 Maintainers need to triage still Type: AVM 🅰️ ✌️ Ⓜ️ This is an AVM related issue labels Jan 17, 2025

cleanup

c1171ec

ReneHezser mentioned this pull request Jan 17, 2025

feat: Added documentation for skipping test deployments Azure/Azure-Verified-Modules#1796

Merged

6 tasks

ReneHezser added Type: CI 🚀 This issue is related to the AVM CI github_actions Pull requests that update GitHub Actions code and removed Needs: Triage 🔍 Maintainers need to triage still labels Jan 17, 2025

eriqua reviewed Jan 17, 2025

View reviewed changes

.github/actions/templates/avm-validateModuleDeployment/action.yml Outdated Show resolved Hide resolved

eriqua reviewed Jan 17, 2025

View reviewed changes

ReneHezser and others added 15 commits January 20, 2025 10:21

Merge branch 'Azure:main' into enable-skip-ci-deployment-tests

8e4dc0d

ensure the .e2eignore file contains text

c382179

test

b28e053

revert test

c362c71

skip in deployment task

f547f3c

test job output result

870bc34

test

e88659b

errorhandling

8f47a3d

error test

1ab729c

fixes

01a1201

testing

eb492e3

testing

47e9282

test

9e5b97c

fix for test

1630bcb

skip more steps, if the deployment test is skipped

6f08745

ReneHezser requested review from a team as code owners February 26, 2025 13:01

avm-team-linter bot added the Needs: Core Team 🧞 This item needs the AVM Core Team to review it label Feb 26, 2025

Merge branch 'Azure:main' into enable-skip-ci-deployment-tests

b7e5e43