Skip to content

Performance Hotfix for 2.15.1 #261

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Mar 17, 2023
Merged

Conversation

jsinglet
Copy link
Contributor

@jsinglet jsinglet commented Mar 17, 2023

Description

Performance Hotfix for 2.15.1.

Change request type

  • Release or process automation (GitHub workflows, internal scripts)
  • Internal documentation
  • External documentation
  • Query files (.ql, .qll, .qls or unit tests)
  • External scripts (analysis report or other code shipped as part of a release)

Rules with added or modified queries

  • No rules added
  • Queries have been added for the following rules:
    • rule number here
  • Queries have been modified for the following rules:
    • rule number here

Release change checklist

A change note (development_handbook.md#change-notes) is required for any pull request which modifies:

  • The structure or layout of the release artifacts.
  • The evaluation performance (memory, execution time) of an existing query.
  • The results of an existing query in any circumstance.

If you are only adding new rule queries, a change note is not required.

Author: Is a change note required?

  • Yes
  • No

🚨🚨🚨
Reviewer: Confirm that format of shared queries (not the .qll file, the
.ql file that imports it) is valid by running them within VS Code.

  • Confirmed

Reviewer: Confirm that either a change note is not required or the change note is required and has been added.

  • Confirmed

Query development review checklist

For PRs that add new queries or modify existing queries, the following checklist should be completed by both the author and reviewer:

Author

  • Have all the relevant rule package description files been checked in?
  • Have you verified that the metadata properties of each new query is set appropriately?
  • Do all the unit tests contain both "COMPLIANT" and "NON_COMPLIANT" cases?
  • Are the alert messages properly formatted and consistent with the style guide?
  • Have you run the queries on OpenPilot and verified that the performance and results are acceptable?
    As a rule of thumb, predicates specific to the query should take no more than 1 minute, and for simple queries be under 10 seconds. If this is not the case, this should be highlighted and agreed in the code review process.
  • Does the query have an appropriate level of in-query comments/documentation?
  • Have you considered/identified possible edge cases?
  • Does the query not reinvent features in the standard library?
  • Can the query be simplified further (not golfed!)

Reviewer

  • Have all the relevant rule package description files been checked in?
  • Have you verified that the metadata properties of each new query is set appropriately?
  • Do all the unit tests contain both "COMPLIANT" and "NON_COMPLIANT" cases?
  • Are the alert messages properly formatted and consistent with the style guide?
  • Have you run the queries on OpenPilot and verified that the performance and results are acceptable?
    As a rule of thumb, predicates specific to the query should take no more than 1 minute, and for simple queries be under 10 seconds. If this is not the case, this should be highlighted and agreed in the code review process.
  • Does the query have an appropriate level of in-query comments/documentation?
  • Have you considered/identified possible edge cases?
  • Does the query not reinvent features in the standard library?
  • Can the query be simplified further (not golfed!)

lcartey added 4 commits March 17, 2023 11:21
The calculatons for signed and unsigned type of lowest rank were
expensive when there were a large number of constant expressions in the
program. This commit improves performance by:
 1. Creating classes to represent the set of integer constant
expressions and integer constants in the program.
 2. Creating a candidate table of IntegralTypes for each known constant
integer value in the program, to avoid duplicate work.
 3. Calculating the stlr and utlr for each constant value, not each
constant expression.
This improves the performance of Rule 10.5 by computing all the types of
the cast first, before determining whether they were invalid. Previously
the join orderer found the essential type category of the "from" type,
then joined that will invalid type categories, then joined that with all
expressions of that type before filtering down to the cast.
@github-actions
Copy link

🤖 Beep Boop! Matrix Testing for this PR has been initiated. Please check back later for results.

💡 If you do not hear back from me please check my status! I will report even if this PR does not contain files eligible for matrix testing.

@jsinglet
Copy link
Contributor Author

🤖 Beep Boop! gcc/cpp/X86_64 Matrix Testing for this PR has been completed but I didn't find anything to test!

@jsinglet
Copy link
Contributor Author

🤖 Beep Boop! clang/cpp/X86_64 Matrix Testing for this PR has been completed but I didn't find anything to test!

@jsinglet
Copy link
Contributor Author

🤖 Beep Boop! qcc/cpp/AARCH64LE Matrix Testing for this PR has been completed but I didn't find anything to test!

@jsinglet
Copy link
Contributor Author

🤖 Beep Boop! gcc/c/X86_64 Matrix Testing for this PR has been completed. See below for the results!


RULE                 : RULE-10-5
PACKAGE              : EssentialTypes
TEST_DIFFERENCE      : 
COMPILE_PASS         : True
QUERY                : InappropriateEssentialTypeCast
COMPILE_ERROR_OUTPUT : 
TEST_PASS            : True
SUITE                : MISRA-C-2012


@jsinglet
Copy link
Contributor Author

🤖 Beep Boop! clang/c/X86_64 Matrix Testing for this PR has been completed. See below for the results!


COMPILE_PASS         : True
COMPILE_ERROR_OUTPUT : 
PACKAGE              : EssentialTypes
RULE                 : RULE-10-5
SUITE                : MISRA-C-2012
TEST_PASS            : True
QUERY                : InappropriateEssentialTypeCast
TEST_DIFFERENCE      : 


@github-actions
Copy link

🤖 Beep Boop! Matrix Testing for this PR has been initiated. Please check back later for results.

💡 If you do not hear back from me please check my status! I will report even if this PR does not contain files eligible for matrix testing.

@jsinglet
Copy link
Contributor Author

🤖 Beep Boop! qcc/c/AARCH64LE Matrix Testing for this PR has been completed. See below for the results!


COMPILE_ERROR_OUTPUT : 
RULE                 : RULE-10-5
QUERY                : InappropriateEssentialTypeCast
COMPILE_PASS         : True
TEST_PASS            : True
PACKAGE              : EssentialTypes
TEST_DIFFERENCE      : 
SUITE                : MISRA-C-2012


@jsinglet
Copy link
Contributor Author

🤖 Beep Boop! Matrix Testing for this PR has been completed. If no reports were posted it means this PR does not contain things that need matrix testing!

@jsinglet
Copy link
Contributor Author

/test-performance

@github-actions
Copy link

🏁 Beep Boop! Performance testing for this PR has been initiated. Please check back later for results. Note that the query package generation step must complete before testing will start so it might be a minute.

💡 If you do not hear back from me please check my status! I will report even if I fail!

@jsinglet
Copy link
Contributor Author

🤖 Beep Boop! clang/cpp/X86_64 Matrix Testing for this PR has been completed but I didn't find anything to test!

@jsinglet
Copy link
Contributor Author

🤖 Beep Boop! clang/c/X86_64 Matrix Testing for this PR has been completed. See below for the results!


PACKAGE              : EssentialTypes
COMPILE_PASS         : True
SUITE                : MISRA-C-2012
TEST_DIFFERENCE      : 
COMPILE_ERROR_OUTPUT : 
TEST_PASS            : True
QUERY                : InappropriateEssentialTypeCast
RULE                 : RULE-10-5


@jsinglet
Copy link
Contributor Author

🤖 Beep Boop! gcc/c/X86_64 Matrix Testing for this PR has been completed. See below for the results!


COMPILE_PASS         : True
TEST_PASS            : True
RULE                 : RULE-10-5
QUERY                : InappropriateEssentialTypeCast
COMPILE_ERROR_OUTPUT : 
SUITE                : MISRA-C-2012
TEST_DIFFERENCE      : 
PACKAGE              : EssentialTypes


@jsinglet
Copy link
Contributor Author

🤖 Beep Boop! gcc/cpp/X86_64 Matrix Testing for this PR has been completed but I didn't find anything to test!

@jsinglet
Copy link
Contributor Author

🤖 Beep Boop! qcc/cpp/AARCH64LE Matrix Testing for this PR has been completed but I didn't find anything to test!

@jsinglet
Copy link
Contributor Author

🤖 Beep Boop! qcc/c/AARCH64LE Matrix Testing for this PR has been completed. See below for the results!


TEST_PASS            : True
RULE                 : RULE-10-5
TEST_DIFFERENCE      : 
SUITE                : MISRA-C-2012
QUERY                : InappropriateEssentialTypeCast
PACKAGE              : EssentialTypes
COMPILE_PASS         : True
COMPILE_ERROR_OUTPUT : 


@jsinglet
Copy link
Contributor Author

🤖 Beep Boop! Matrix Testing for this PR has been completed. If no reports were posted it means this PR does not contain things that need matrix testing!

@jsinglet
Copy link
Contributor Author

🏁 Beep Boop! Performance testing complete! See below for performance of the last 3 runs vs your PR. Times are based on predicate performance. You can find full graphs and stats in the PR that was created for this test in the release engineering repo.


Release                            : v2.13.0
Platform                           : x86-windows
Language                           : c
Total_Serialized_Execution_Time_Ms : 3268794
Mean_Predicate_Execution_Time_Ms   : 141.10917332182171
Median_Predicate_Execution_Time_Ms : 5.0
Standard_Deviation_Ms              : 2347.189663268768
Total_Serialized_Execution_Time_s  : 3268.794
Mean_Query_Execution_Time_s        : 0.1411091733218217
Median_Predicate_Execution_Time_s  : 0.005
Percentile95_Ms                    : 300.0
Number_of_Predicates               : 23165

Release                            : v2.13.0
Platform                           : x86-windows
Language                           : cpp
Total_Serialized_Execution_Time_Ms : 4405198
Mean_Predicate_Execution_Time_Ms   : 110.18504252126064
Median_Predicate_Execution_Time_Ms : 3.0
Standard_Deviation_Ms              : 1816.991907084716
Total_Serialized_Execution_Time_s  : 4405.198
Mean_Query_Execution_Time_s        : 0.1101850425212606
Median_Predicate_Execution_Time_s  : 0.003
Percentile95_Ms                    : 222.0
Number_of_Predicates               : 39980

Release                            : v2.13.0
Platform                           : x86-linux
Language                           : cpp
Total_Serialized_Execution_Time_Ms : 3877306
Mean_Predicate_Execution_Time_Ms   : 97.3048410168896
Median_Predicate_Execution_Time_Ms : 2.0
Standard_Deviation_Ms              : 1572.1314154912643
Total_Serialized_Execution_Time_s  : 3877.306
Mean_Query_Execution_Time_s        : 0.0973048410168896
Median_Predicate_Execution_Time_s  : 0.002
Percentile95_Ms                    : 200.0
Number_of_Predicates               : 39847

Release                            : v2.13.0
Platform                           : x86-linux
Language                           : c
Total_Serialized_Execution_Time_Ms : 2851919
Mean_Predicate_Execution_Time_Ms   : 123.07077201916024
Median_Predicate_Execution_Time_Ms : 3.0
Standard_Deviation_Ms              : 1960.041520356055
Total_Serialized_Execution_Time_s  : 2851.919
Mean_Query_Execution_Time_s        : 0.1230707720191602
Median_Predicate_Execution_Time_s  : 0.003
Percentile95_Ms                    : 266.0
Number_of_Predicates               : 23173

Release                            : v2.14.0
Platform                           : x86-windows
Language                           : c
Total_Serialized_Execution_Time_Ms : 3330223
Mean_Predicate_Execution_Time_Ms   : 125.4557543793558
Median_Predicate_Execution_Time_Ms : 4.0
Standard_Deviation_Ms              : 2125.712933594389
Total_Serialized_Execution_Time_s  : 3330.223
Mean_Query_Execution_Time_s        : 0.1254557543793558
Median_Predicate_Execution_Time_s  : 0.004
Percentile95_Ms                    : 271.0
Number_of_Predicates               : 26545

Release                            : v2.14.0
Platform                           : x86-linux
Language                           : cpp
Total_Serialized_Execution_Time_Ms : 3910243
Mean_Predicate_Execution_Time_Ms   : 97.5706906876934
Median_Predicate_Execution_Time_Ms : 2.0
Standard_Deviation_Ms              : 1583.9926359710646
Total_Serialized_Execution_Time_s  : 3910.243
Mean_Query_Execution_Time_s        : 0.0975706906876933
Median_Predicate_Execution_Time_s  : 0.002
Percentile95_Ms                    : 200.0
Number_of_Predicates               : 40076

Release                            : v2.14.0
Platform                           : x86-linux
Language                           : c
Total_Serialized_Execution_Time_Ms : 2886922
Mean_Predicate_Execution_Time_Ms   : 108.65344373353406
Median_Predicate_Execution_Time_Ms : 2.0
Standard_Deviation_Ms              : 1783.906509131339
Total_Serialized_Execution_Time_s  : 2886.922
Mean_Query_Execution_Time_s        : 0.108653443733534
Median_Predicate_Execution_Time_s  : 0.002
Percentile95_Ms                    : 237.0
Number_of_Predicates               : 26570

Release                            : v2.14.0
Platform                           : x86-windows
Language                           : cpp
Total_Serialized_Execution_Time_Ms : 4498723
Mean_Predicate_Execution_Time_Ms   : 112.41186906546729
Median_Predicate_Execution_Time_Ms : 3.0
Standard_Deviation_Ms              : 1841.844021324711
Total_Serialized_Execution_Time_s  : 4498.723
Mean_Query_Execution_Time_s        : 0.1124118690654672
Median_Predicate_Execution_Time_s  : 0.003
Percentile95_Ms                    : 224.0
Number_of_Predicates               : 40020

Release                            : 261
Platform                           : x86-linux
Language                           : cpp
Total_Serialized_Execution_Time_Ms : 3963224
Mean_Predicate_Execution_Time_Ms   : 97.16642149651858
Median_Predicate_Execution_Time_Ms : 2.0
Standard_Deviation_Ms              : 1482.4144369023234
Total_Serialized_Execution_Time_s  : 3963.224
Mean_Query_Execution_Time_s        : 0.0971664214965185
Median_Predicate_Execution_Time_s  : 0.002
Percentile95_Ms                    : 200.0
Number_of_Predicates               : 40788

Release                            : 261
Platform                           : x86-linux
Language                           : c
Total_Serialized_Execution_Time_Ms : 3227501
Mean_Predicate_Execution_Time_Ms   : 114.14277125477436
Median_Predicate_Execution_Time_Ms : 2.0
Standard_Deviation_Ms              : 1844.798175084509
Total_Serialized_Execution_Time_s  : 3227.501
Mean_Query_Execution_Time_s        : 0.1141427712547743
Median_Predicate_Execution_Time_s  : 0.002
Percentile95_Ms                    : 225.0
Number_of_Predicates               : 28276


🏁 Below are the slowest predicates for the last 2 releases vs this PR.


Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-windows
Language          : cpp
Suite             : autosar-default
Predicate         : UnusedIncludeDirectives::getANonLocalDependency#574b69bc#ff
Execution_Time_Ms : 35422

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-windows
Language          : cpp
Suite             : autosar-default
Predicate         : SSAConstruction::PhiInsertion::definitionHasRedefinition#2b11997e#fff
Execution_Time_Ms : 35551

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : #select#cpe#134#fff#shared
Execution_Time_Ms : 136465

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-windows
Language          : cpp
Suite             : autosar-default
Predicate         : NameInDependentBase::parentMemberFunctionCall#6fb39822#fff
Execution_Time_Ms : 145768

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-windows
Language          : cpp
Suite             : autosar-default
Predicate         : m##DataFlowUtil::localFlowStep#7572fbecPlus#bf
Execution_Time_Ms : 173664

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-windows
Language          : cpp
Suite             : autosar-default
Predicate         : Dependency::dependsOnTransitive#b0c9183e#ff
Execution_Time_Ms : 231282

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : #select#ff#antijoin_rhs
Execution_Time_Ms : 198068

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-windows
Language          : c
Suite             : misra-default
Predicate         : #select#cpe#134#fff#antijoin_rhs
Execution_Time_Ms : 64027

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-windows
Language          : c
Suite             : misra-default
Predicate         : #select#cpe#13456#fffff#shared
Execution_Time_Ms : 122933

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-windows
Language          : c
Suite             : misra-default
Predicate         : #select#cpe#134#fff#shared
Execution_Time_Ms : 171881

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-windows
Language          : c
Suite             : misra-default
Predicate         : #select#ff#antijoin_rhs
Execution_Time_Ms : 227605

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-windows
Language          : c
Suite             : misra-default
Predicate         : SSAConstruction::DefUse::hasNonPhiDefinition#2b11997e#ffff
Execution_Time_Ms : 34177

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-linux
Language          : cpp
Suite             : cert-default
Predicate         : SSAConstruction::DefUse::hasNonPhiDefinition#2b11997e#ffff
Execution_Time_Ms : 30531

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : NameInDependentBase::parentMemberFunctionCall#6fb39822#fff
Execution_Time_Ms : 130555

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : m##DataFlowUtil::localFlowStep#7572fbecPlus#bf
Execution_Time_Ms : 166984

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : Dependency::dependsOnTransitive#b0c9183e#ff
Execution_Time_Ms : 181966

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : #select#cpe#1#f#antijoin_rhs
Execution_Time_Ms : 28654

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : #select#cpe#134#fff#antijoin_rhs
Execution_Time_Ms : 58447

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : #select#cpe#13456#fffff#shared
Execution_Time_Ms : 93370

Release           : v2.14.0
Run               : 2023-03-02_02-14-06
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : UnusedIncludeDirectives::getANonLocalDependency#574b69bc#ff
Execution_Time_Ms : 30386

Release           : 261
Run               : 2023-03-17_15-28-20
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : #select#cpe#134#fff#shared
Execution_Time_Ms : 138103

Release           : 261
Run               : 2023-03-17_15-28-20
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : RepeatedInitializationOfAggregateObjectElement#76b22bdc::hasMultipleInitializerExprsForSameField#2#ff
Execution_Time_Ms : 66642

Release           : 261
Run               : 2023-03-17_15-28-20
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : #select#cpe#134#fff#antijoin_rhs
Execution_Time_Ms : 62716

Release           : 261
Run               : 2023-03-17_15-28-20
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : RepeatedInitializationOfAggregateObjectElement#76b22bdc::hasMultipleInitializerExprsForSameIndexInternal#4#ffff#join_rhs
Execution_Time_Ms : 58390

Release           : 261
Run               : 2023-03-17_15-28-20
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : PreProcessorShallOnlyBeUsedForCertainDirectivesPatterns#32e68f1c::getAGuard#1#ff
Execution_Time_Ms : 31852

Release           : 261
Run               : 2023-03-17_15-28-20
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : Dependency#b0c9183e::dependsOnTransitive#2#ff
Execution_Time_Ms : 177253

Release           : 261
Run               : 2023-03-17_15-28-20
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : UnusedIncludeDirectives#574b69bc::getANonLocalDependency#1#ff
Execution_Time_Ms : 32620

Release           : 261
Run               : 2023-03-17_15-28-20
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : SSAConstruction#2b11997e::DefUse::hasNonPhiDefinition#4#ffff
Execution_Time_Ms : 30569

Release           : 261
Run               : 2023-03-17_15-28-20
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : m##DataFlowUtil#7572fbec::localFlowStep#2Plus#bf
Execution_Time_Ms : 187081

Release           : 261
Run               : 2023-03-17_15-28-20
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : #select#ff#antijoin_rhs
Execution_Time_Ms : 213962


@jsinglet jsinglet requested review from a team March 17, 2023 18:08
@jsinglet jsinglet merged commit 8159166 into rc/2.15 Mar 17, 2023
@jsinglet jsinglet deleted the jsinglet/performance-hotfix-2-15-1 branch March 17, 2023 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

2 participants