Skip to content

Commit a098472

Browse files
MoChiliaMoChilia
authored
[CI Example Analyzer] Recover severity for errors (Azure#18894)
* recover severity * fix a bug * add some tips * fix a bug * fix some bugs for mismatchedType and change severity==1 * add tips * add tips
1 parent 222407e commit a098472

File tree

6 files changed

+49
-72
lines changed

6 files changed

+49
-72
lines changed

‎documentation/Debugging-StaticAnalysis-Errors.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -19,10 +19,10 @@ d:\workspace\powershell\build.proj(511,5): error MSB3073: The command "d:\worksp
1919
## Where to find StaticAnalysis reports
2020

2121
The StaticAnalysis reports could show up in two different places in the CI build:
22-
- On the status page in Jenkins, under the Build Artifacts: the relevant files are `BreakingChangeIssues.csv`, `SignatureIssues.csv`, and/or `HelpIssues.csv`.
23-
- On the status page in Jenkins, click Build Artifacts then navigate to artifacts. You will see `BreakingChangeIssues.csv`, `SignatureIssues.csv`, and/or `HelpIssues.csv`.
22+
- On the status page in Jenkins, under the Build Artifacts: the relevant files are `BreakingChangeIssues.csv`, `SignatureIssues.csv`, `HelpIssues.csv` and/or `ExampleIssues.csv`.
23+
- On the status page in Jenkins, click Build Artifacts then navigate to artifacts. You will see `BreakingChangeIssues.csv`, `SignatureIssues.csv`, `HelpIssues.csv` and/or `ExampleIssues.csv`.
2424

25-
Locally, the StaticAnalysis report will show up under Azure-PowerShell/artifacts. You will see `BreakingChangeIssues.csv`, `SignatureIssues.csv`, and/or `HelpIssues.csv`. You can generate these files by running
25+
Locally, the StaticAnalysis report will show up under Azure-PowerShell/artifacts. You will see `BreakingChangeIssues.csv`, `SignatureIssues.csv`, `HelpIssues.csv` and/or `ExampleIssues.csv`. You can generate these files by running
2626
```
2727
msbuild build.proj
2828
```
@@ -55,12 +55,12 @@ Signature issues occur when your cmdlets do not follow PowerShell standards. Pl
5555
Most help issues that cause StaticAnalysis to fail occur when help has not been added for a particular cmdlet. If you have not generated help for your new cmdlets, please follow the instructions [here](https://github.com/Azure/azure-powershell/blob/main/documentation/development-docs/help-generation.md). If this is not the issue, follow the steps listed under "Remediation" for each violation listed in HelpIssues.csv.
5656

5757
### Example Issues
58-
Example issues occur when your changed markdown files in the `help` folder (_e.g.,_ `src/Accounts/Accounts/help`) violate PowerShell language best practices. Please follow the suggestion displayed in "Remediation" entry for each violation listed in `ExampleIssues.csv`. If you have an issue with severity 0 or 1 that has been approved by the Azure PowerShell team, you can suppress them following these steps:
58+
Example issues occur when your changed markdown files in the `help` folder (_e.g.,_ `src/Accounts/Accounts/help`) violate PowerShell language best practices. Please follow the suggestion displayed in "Remediation" entry for each violation listed in `ExampleIssues.csv`. Issues with severity 0 or 1 must be addressed, while issues with severity 2 are advisory. To better standardize the writing of documents, please also check the warning issues with severity 2 in log or download the `ExampleIssues.csv` file. If you have an issue with severity 0 or 1 that has been approved by the Azure PowerShell team, you can suppress them following these steps:
5959

6060
- Download the `ExampleIssues.csv` file from the CI pipeline artifacts
6161
- Open the file using a text editor (such as VS Code) and copy each of the errors you'd like to suppress
6262
- Paste each of these errors into the `ExampleIssues.csv` file found in their respective [module folder](../tools/StaticAnalysis/Exceptions) (_e.g.,_ if an example issue is being suppressed for Accounts, then you would paste the corresponding line(s) in the `tools/StaticAnalysis/Exceptions/Az.Accounts/ExampleIssue.csv` file) using the same text editor
6363
- Copy each of the errors you would like to suppress directly from the ExampleIssues.csv file output in the CI pipeline artifacts
6464
- Push the changes to the .csv file and ensure the errors no longer show up in the `ExampleIssues.csv` file output from the CI pipeline artifacts.
6565

66-
To better standardize the writing of documents, please also check the warning issues with severity 2 by downloading the `ExampleIssues.csv` file.
66+
If you have unexpected errors, please check whether you have splitted outputs from codes. If outputs cannot be separated from codes, then please add the tag `<!-- Skip: Output cannot be splitted from code -->` to the next line of the example title and in front of the code block.

‎tools/StaticAnalysis/ExampleAnalyzer/AnalyzeRules/ParameterNameAndValue.psm1‎

Lines changed: 29 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,19 @@ function Get-RecoveredValueType{
9999
if ($global:AssignmentLeftAndRight.ContainsKey($CommandElementAst.Extent.Text)){
100100
$CommandElementAst = $global:AssignmentLeftAndRight.($CommandElementAst.Extent.Text)
101101
}
102+
elseif ($null -ne $CommandElementAst.Left) {
103+
if($CommandElementAst.Left -eq $VariableExpressionAst){
104+
if($CommandElementAst.Right -eq $VariableExpressionAst){
105+
$CommandElementAst = $null
106+
}
107+
else{
108+
$CommandElementAst = $CommandElementAst.Right
109+
}
110+
}
111+
else{
112+
$CommandElementAst = $CommandElementAst.Left
113+
}
114+
}
102115
elseif ($null -ne $CommandElementAst.Expression) {
103116
if($null -ne $CommandElementAst.Member){
104117
$Items += $CommandElementAst.Member
@@ -161,6 +174,9 @@ function Measure-IsTypeMatched{
161174
[System.Reflection.TypeInfo]$ExpectedType,
162175
[System.Reflection.TypeInfo]$ActualType
163176
)
177+
if($ActualType -eq $null) {
178+
return $false
179+
}
164180
if($ActualType.IsArray) {
165181
$ActualType = $ActualType.GetElementType()
166182
}
@@ -219,6 +235,9 @@ function Get-AssignedParameterExpression {
219235
}
220236
return $null
221237
}
238+
if($CommandElement_Copy -is [System.Management.Automation.Language.ConvertExpressionAst]){
239+
$CommandElement_Copy = $CommandElement_Copy.Type
240+
}
222241
while ($ExpectedType.IsArray) {
223242
$ExpectedType = $ExpectedType.GetElementType()
224243
}
@@ -283,7 +302,12 @@ function Get-AssignedParameterExpression {
283302
elseif($CommandElement_Copy -is [System.Management.Automation.Language.TypeExpressionAst] -or
284303
$CommandElement_Copy -is [System.Management.Automation.Language.TypeConstraintAst]){
285304
$ReturnType = $CommandElement_Copy.TypeName.ToString() -as [Type]
286-
$ActualType = Get-RecoveredValueType $CommandElement $ReturnType
305+
if($null -eq $ReturnType){
306+
$ActualType = $null
307+
}
308+
else{
309+
$ActualType = Get-RecoveredValueType $CommandElement $ReturnType
310+
}
287311
if (!(Measure-IsTypeMatched $ExpectedType $ActualType)) {
288312
# Mismatched_Parameter_Value_Type
289313
$ExpressionToParameter = "$($CommandElement.Extent.Text)-#-$ExpectedType. Now the type is $ActualType.(Type)"
@@ -293,6 +317,7 @@ function Get-AssignedParameterExpression {
293317
elseif($CommandElement_Copy -is [System.Management.Automation.Language.ExpressionAst]) {
294318
# Value is a constant expression
295319
$ConvertedObject = $CommandElement_Copy.Extent.text -as $ExpectedType
320+
# Value of Automatic Variable
296321
if($null -eq $ConvertedObject){
297322
if($null -ne (Get-Variable | Where-Object {$_.Name -eq $CommandElement_Copy.VariablePath})){
298323
$value = (Get-Variable | Where-Object {$_.Name -eq $CommandElement_Copy.VariablePath}).Value
@@ -682,21 +707,21 @@ function Measure-ParameterNameAndValue {
682707
$RuleName = [RuleNames]::Invalid_Parameter_Name
683708
$Severity = "Error"
684709
$RuleSuppressionID = "5011"
685-
$Remediation = "Check validity of the parameter $($CommandParameterPair[$i].ParameterName)."
710+
$Remediation = "Check validity of the parameter -$($CommandParameterPair[$i].ParameterName)."
686711
}
687712
elseif ($global:CommandParameterPair[$i].ExpressionToParameter -eq "<duplicate>") {
688713
$Message = "$($CommandParameterPair[$i].ModuleCmdletExNum)-#@#$($CommandParameterPair[$i].CommandName) -$($CommandParameterPair[$i].ParameterName) appeared more than once."
689714
$RuleName = [RuleNames]::Duplicate_Parameter_Name
690715
$Severity = "Error"
691716
$RuleSuppressionID = "5012"
692-
$Remediation = "Remove redundant parameter $($CommandParameterPair[$i].ParameterName)."
717+
$Remediation = "Remove redundant parameter -$($CommandParameterPair[$i].ParameterName)."
693718
}
694719
elseif ($null -eq $global:CommandParameterPair[$i].ExpressionToParameter) {
695720
$Message = "$($CommandParameterPair[$i].ModuleCmdletExNum)-#@#$($CommandParameterPair[$i].CommandName) -$($CommandParameterPair[$i].ParameterName) must be assigned with a value."
696721
$RuleName = [RuleNames]::Unassigned_Parameter
697722
$Severity = "Error"
698723
$RuleSuppressionID = "5013"
699-
$Remediation = "Assign value for the parameter $($CommandParameterPair[$i].ParameterName)."
724+
$Remediation = "Assign value for the parameter -$($CommandParameterPair[$i].ParameterName)."
700725
}
701726
elseif ($global:CommandParameterPair[$i].ExpressionToParameter.EndsWith(" is a null-valued parameter value.")) {
702727
$Message = "$($CommandParameterPair[$i].ModuleCmdletExNum)-#@#$($CommandParameterPair[$i].CommandName) -$($CommandParameterPair[$i].ParameterName) $($CommandParameterPair[$i].ExpressionToParameter)"

‎tools/StaticAnalysis/ExampleAnalyzer/ExampleIssue.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ public string FormatRecord()
4141
Module, Cmdlet, Example, RuleName, ProblemId, Severity, Description, Extent, Remediation);
4242
}
4343

44+
// The code that excludes exceptions is in tools/StaticAnalysis/ExampleAnalyzer/utils.ps1 Get-NonExceptionRecord.
4445
public bool Match(IReportRecord other)
4546
{
4647
var result = false;
@@ -50,7 +51,6 @@ public bool Match(IReportRecord other)
5051
result = (record.Module == Module)&&
5152
(record.Cmdlet == Cmdlet)&&
5253
(record.Example == Example)&&
53-
(record.ProblemId == ProblemId)&&
5454
(record.Description == Description);
5555
}
5656
return result;

‎tools/StaticAnalysis/ExampleAnalyzer/Measure-MarkdownOrScript.ps1‎

Lines changed: 0 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -22,23 +22,16 @@ param (
2222
[string]$OutputFolder = "$PSScriptRoot\..\..\..\artifacts\StaticAnalysisResults\ExampleAnalysis",
2323
[switch]$AnalyzeScriptsInFile,
2424
[switch]$OutputScriptsInFile,
25-
[switch]$OutputResultsByModule,
2625
[switch]$CleanScripts
2726
)
2827

2928
. $PSScriptRoot\utils.ps1
3029

31-
if ($PSCmdlet.ParameterSetName -eq "Markdown") {
32-
$scaleTable = @()
33-
$missingTable = @()
34-
$deletePromptAndSeparateOutputTable = @()
35-
}
3630
$analysisResultsTable = @()
3731

3832
# Clean caches, remove files in "output" folder
3933
if ($OutputScriptsInFile.IsPresent) {
4034
Remove-Item $OutputFolder\TempScript.ps1 -ErrorAction SilentlyContinue
41-
Remove-Item $OutputFolder\*.csv -Recurse -ErrorAction SilentlyContinue
4235
Remove-Item $PSScriptRoot\..\..\..\artifacts\StaticAnalysisResults\ExampleIssues.csv -ErrorAction SilentlyContinue
4336
Remove-Item $OutputFolder -ErrorAction SilentlyContinue
4437
}
@@ -67,25 +60,12 @@ if ($PSCmdlet.ParameterSetName -eq "Markdown") {
6760
$result = Measure-SectionMissingAndOutputScript $module $cmdlet $_.FullName `
6861
-OutputScriptsInFile:$OutputScriptsInFile.IsPresent `
6962
-OutputFolder $OutputFolder
70-
$scaleTable += $result.Scale
71-
$missingTable += $result.Missing
72-
$deletePromptAndSeparateOutputTable += $result.DeletePromptAndSeparateOutput
7363
$analysisResultsTable += $result.Errors
7464
}
7565
}
7666
if ($AnalyzeScriptsInFile.IsPresent) {
7767
$ScriptPaths = "$OutputFolder\TempScript.ps1"
7868
}
79-
# Summarize searching results
80-
if($scaleTable){
81-
$scaleTable | Where-Object {$_ -ne $null} | Export-Csv "$OutputFolder\Scale.csv" -NoTypeInformation
82-
}
83-
if($missingTable){
84-
$missingTable | Where-Object {$_ -ne $null} | Export-Csv "$OutputFolder\Missing.csv" -NoTypeInformation
85-
}
86-
if($deletePromptAndSeparateOutputTable){
87-
$deletePromptAndSeparateOutputTable | Where-Object {$_ -ne $null} | Export-Csv "$OutputFolder\DeletingSeparating.csv" -NoTypeInformation
88-
}
8969
}
9070

9171
# Analyze scripts

‎tools/StaticAnalysis/ExampleAnalyzer/utils.ps1‎

Lines changed: 7 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -149,7 +149,7 @@ function Get-ExamplesDetailsFromMd {
149149
}
150150
<#
151151
.SYNOPSIS
152-
Except the suppressed records
152+
Except the suppressed records. It is independent of ExampleIssues.cs.
153153
#>
154154
function Get-NonExceptionRecord{
155155
param(
@@ -190,7 +190,7 @@ function Measure-SectionMissingAndOutputScript {
190190
[string]$OutputFolder
191191
)
192192
$results = @()
193-
$missingSeverity = 2
193+
$missingSeverity = 1
194194

195195
$fileContent = Get-Content $MarkdownPath -Raw
196196

@@ -207,7 +207,6 @@ function Measure-SectionMissingAndOutputScript {
207207
$missingExampleOutput = 0
208208
$missingExampleDescription = 0
209209
$needDeleting = 0
210-
$needSplitting = 0
211210

212211
# If Synopsis section exists
213212
if ($indexOfSynopsis -ne -1) {
@@ -382,7 +381,7 @@ function Measure-SectionMissingAndOutputScript {
382381
# Output example codes to "TempScript.ps1"
383382
if ($OutputScriptsInFile.IsPresent) {
384383
$cmdletExamplesScriptPath = "$OutputFolder\TempScript.ps1"
385-
if($exampleCodes -ne $null){
384+
if($null -ne $exampleCodes -and $exampleCodes -ne ""){
386385
$exampleCodes = $exampleCodes.Trim()
387386
$functionHead = "function $Module-$Cmdlet-$exampleNumber{"
388387
Add-Content -Path (Get-Item $cmdletExamplesScriptPath).FullName -Value $functionHead
@@ -394,38 +393,6 @@ function Measure-SectionMissingAndOutputScript {
394393
}
395394
}
396395

397-
# ScaleTable
398-
$examples = $examplesDetails.Count
399-
$scale = [Scale]@{
400-
Module = $module
401-
Cmdlet = $cmdlet
402-
Examples = $examples
403-
}
404-
405-
# MissingTable
406-
if ($missingSynopsis -ne 0 -or $missingDescription -ne 0 -or $missingExampleTitle -ne 0 -or $missingExampleCode -ne 0 -or $missingExampleOutput -ne 0 -or $missingExampleDescription -ne 0) {
407-
$missing = [Missing]@{
408-
Module = $module
409-
Cmdlet = $cmdlet
410-
MissingSynopsis = $missingSynopsis
411-
MissingDescription = $missingDescription
412-
MissingExampleTitle = $missingExampleTitle
413-
MissingExampleCode = $missingExampleCode
414-
MissingExampleOutput = $missingExampleOutput
415-
MissingExampleDescription = $missingExampleDescription
416-
}
417-
}
418-
419-
# DeletePromptAndSeparateOutputTable
420-
if ($needDeleting -ne 0 -or $needSplitting -ne 0) {
421-
$deletePromptAndSeparateOutput = [DeletePromptAndSeparateOutput]@{
422-
Module = $module
423-
Cmdlet = $cmdlet
424-
NeedDeleting = $needDeleting
425-
NeedSplitting = $needSplitting
426-
}
427-
}
428-
429396
# Except the suppressed records
430397
$results = Get-NonExceptionRecord $results
431398

@@ -484,16 +451,16 @@ function Get-ScriptAnalyzerResult {
484451
$results = @()
485452
foreach($analysisResult in $analysisResults){
486453
if($analysisResult.Severity -eq "ParseError"){
487-
$Severity = 2
454+
$Severity = 1
488455
}
489456
elseif($analysisResult.Severity -eq "Error"){
490-
$Severity = 2
457+
$Severity = 1
491458
}
492459
elseif($analysisResult.Severity -eq "Warning"){
493-
$Severity = 3
460+
$Severity = 2
494461
}
495462
elseif($analysisResult.Severity -eq "Information"){
496-
$Severity = 4
463+
$Severity = 3
497464
}
498465
if($analysisResult.RuleSuppressionID -ge 5000 -and $analysisResult.RuleSuppressionID -le 5199){
499466
$result = [AnalysisOutput]@{

‎tools/StaticAnalysis/IssueChecker/IssueChecker.cs‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,8 @@ public void Analyze(IEnumerable<string> scopes, IEnumerable<string> modulesToAna
7979
{
8080
continue;
8181
}
82-
if (IsSingleExceptionFileHasCriticalIssue(exceptionFilePath, recordTypeName))
82+
bool outputWarning = recordTypeName.Equals(typeof(ExampleIssue).FullName);
83+
if (IsSingleExceptionFileHasCriticalIssue(exceptionFilePath, recordTypeName, outputWarning))
8384
{
8485
hasCriticalIssue = true;
8586
}
@@ -92,7 +93,7 @@ public void Analyze(IEnumerable<string> scopes, IEnumerable<string> modulesToAna
9293
}
9394
}
9495

95-
private bool IsSingleExceptionFileHasCriticalIssue(string exceptionFilePath, string reportRecordTypeName)
96+
private bool IsSingleExceptionFileHasCriticalIssue(string exceptionFilePath, string reportRecordTypeName, bool outputWarning)
9697
{
9798
bool hasError = false;
9899
using (var reader = new StreamReader(exceptionFilePath))
@@ -114,6 +115,10 @@ private bool IsSingleExceptionFileHasCriticalIssue(string exceptionFilePath, str
114115
hasError = true;
115116
errorText.AppendLine(record.FormatRecord());
116117
}
118+
else if (record.Severity == 2 && outputWarning)
119+
{
120+
errorText.AppendLine(record.FormatRecord());
121+
}
117122
}
118123
if (hasError)
119124
{

0 commit comments

Comments
 (0)