Merge 4-0-267 to backup_sync #189
Open
+40,371
−36,583
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Sometimes AST trees would be thrown away, specifically when long chained expressions would be compressed into temporary cache variables. This only happens with expressions, and closures typically write directly to the diff graph. This PR minimizes what is written directly to the diff graph from function creation, and extends `x2cpg.Ast` to also include `CAPTURE` edges. This also introduces `closureToRefs` map to track which `Block` nodes already have live method ASTs in the diff graph.
…rnio#4983) This reverts commit 3c27bf6.
* [ruby] Changed handling of ForEach loops * [ruby] fixed double _astIn reference to identifier * [ruby] fix failing tests
…" (joernio#4985) * Revert "Revert "[ruby] Ignore "Throwaway" AST Structures (joernio#4982)" (joernio#4983)" This reverts commit 464480d. * [ruby] Re-implemented "Ignore "Throwaway" AST Structures (joernio#4982)" This correctly prevents re-use of nodes that are already being used elsewhere by ensuring deep copies.
`RubyExpression` nodes don't inherently consider the `span` in the calculation of its hash, so when put into any hashed context, nodes that only rely on `span` alone will collide in these contexts.
… dir (joernio#4956) The logic to guess the base dir of the installation is quite fiddly but works for our use cases for astgen. PhpParser implemented something similar, but not quite - and it failed for buildbot. On buildbot the installation path for php2cpg is `/worker/sptestV2-php2cpg/build/x2cpg-internal/php2cpg/target/universal/stage` which (prior to this PR) leads to an invalid derived php-parser name and the following error: ``` 2024-09-25 09:30:08.623 ERROR Invalid path for PhpParserBin: /worker/sptestV2-/php2cpg/bin/php-parser/php-parser.php ```
…nse header (joernio#4995) Turns out the HTTPServer expects an explicit "Connection: close" header which requests the connection to be closed after the transaction ends. Otherwise, it would wait for a 10sec timeout for the next thread to become available. In case we only allow for one thread (jimple2cpg) that would mean additional waiting which renders the whole server approach useless. This in now fixes as we immediately close the connection after the frontend is finished.
This happened for parameter and variable declarations in parentheses.
…ex for named arguments (joernio#5000)
* upgrade deps * trigger ci
…ed arguments (joernio#5003) * [dataflowengineoss] Fix PassThroughMapping criteria for named arguments * sort result for testing purposes
joernio#4998) DescriptorRendererImpl asserts that the options are locked upon init. This means that the type rendering will fail anywhere with assertions enabled (`-ea`) as the options weren't being locked after setup.
* ExternalCommand: more information in error conditions * report exit code if it's non-zero * pass on original error (if any) rather than disregarding it * log.warn stderr output (if any) * add tests * compiler warning fix * exit code `2` on linux, `1` on mac... * fix for mac * error msg is different on windows
Works with Java Process / ProcessBuilder now. No more scala.sys.process. ----------- Co-authored-by: Michael Pollmeier <[email protected]>
fixes the regression from the ExternalCommand refactor, but I couldn't leave the the env var handling as it was either. Fixing that by using a proper library for parsing CLI arguments
* [kotlin2cpg] Improve fullnames. Improve and cleanup the fullname calculation for methods and callsites. The most noteable changes here are: 1. The lambda function naming schema changed from `package.<fileName>.<lambda>counter:signature` to `package.scope.<lambda>counter:signature`. This give a better indication which lambda function is represented also gets rid of exception of using the filename in the method fullnames. 2. Fixed lambda function signatures. 3. The fullname schema of extension methods changed from `extendedClassFullName.extensionMethodName:signatureWithoutThisParameter` to `package.extensionMethodName:signatureWithThisParameter` The fact that the this parameter is part of the signature is an exception in our java fullnames but necessary in order to distinguish otherwise identical extension methods. 4. We avoid the recalculation of the binding context with every file processed. This is now done once at the begining. 5. Calls to `::class` are now represented as calls to `<operator>.class`. * Fix extension method arg and param indicies. The instance arg/param now have index 1 as they are not the receiver on which the call is dispatches. It is a static call after all. * Implement first sane version of lambda binding nodes. Additionally fixed the inheritance information for lambda type decl nodes. * Refactor NameRender. Split some methods into BindingContextUtils and renamed those methods. * fmt * Incorporate review comments. - No more special case for handling `kotlin.Function`. - Removed some unused code. - Stop using currentTimeMillis. * Remove typeInfoProvider.expressionType and replace it via nameRenderer. This results in expresssion types now being renderer in the new fassion. * Translate kotlin.Array type full name to java representation. * Removed multiple further type full name API methods from TypeInfoProvider. * Remove typeFullName for KtDestructuringDeclarationEntry from TypeInfoProvider. * Remove visibility and modality API from TypeInfoProvider. * Remove containingTypeDeclFullName. * Remove unnecessary code. * Remove inheritanceTypes API from TypeInfoProvider. * Remove destructuringEntries API from TypeInfoProvider. * Removed KtTypeAlias related TypeInfoProvider APIs. * Remove typeFullName API for binary expressions from TypeInfoProvider. * Remove typeFullName for annotations from TypeInfoProvider. * Remove unnecessary Option types from BindingContextUtils APIs. * Rework lambda implicit parameter handling. - Removed implicitParameterName and hasApplyOrAlsoScopeFunctionParent APIs from TypeInfoProvider. - Removed wrong parameter deconstruction in lambda. New implemention is missing. - Some lambda to builtin `apply` and `also` functions did not get return statements generated. That is now fixed. * Remove typeFullName and isCompanionObject APIs from TypeInfoProvider. * Remove typeFullName API for KtTypeReference from TypeInfoProvider. * Remove containingDeclType API from TypeInfoProvider. * Remove typeFullName and referenceTargetTypeFullName APIs from TypeInfoProvider. Also refactored assignmentAstForDestructuringEntry. The right hand side base is now created outside of this function allowing for other constructs than pure identifiers. The pure identifier were e.g. wrong in case of class member references. * Remove propertyType and typeFromImports API from TypeInfoProvider. * Remove TypeRenderer. * Improve comment. * Fmt. * Fix for `ScopeFunctionsTests` * Adjusted test expectations for more precise/sound outcomes * Left note about the lambda flows * Add better fallback handling methodFullName of calls. In case of incomplete type information calls with overloads can get ambiguous. We now use the ambiguous function descriptors to get as much of a methodFullName correct as possible. * Fix test fullname. --------- Co-authored-by: David Baker Effendi <[email protected]>
This bypasses tty buffering and read timeouts.
* [dataflowengineoss] add Operators.modulo semantics * sort results for deterministic outcome
* [pysrc2cpg] refactor <module>, ANY and __init__ constants * Replace Constants/Defines, cf. feedback suggestion
* fix - partial ast gen result in case of failure --------- Co-authored-by: PallaviShreshtha <[email protected]>
In the case where an unexpected string in placed in the `filename` property, default to attempting URI construction within a `Try` and returning a `None` on failure
* [c#] fix synthetic set_* method call signature * [c#] support setter assignments via += et al assignments
Lowering regex matches to reflect assignments made to global variables. The lowering becomes: ```ruby tmp = 'hello'.match(/h(el)lo/) if tmp $~ = tmp $& = tmp[0] tmp.begin(0) else $~ = nil $& = nil nil end ``` Some heuristics to assign group matching global variables is next.
This PR implements the other component of regex matching defining global variables. In Ruby, `$1`, `$2`, etc. correspond to the group matched in the last match. This is synonymous to how a `MatchData` object could refer to these matches. This PR models these `nref` objects to `$[1]` and, during a match lowering, defines them to the corresponding index position of the lowered temp match object, i.e., `$[1] = <tmp-0>[1]` where `N` is determined by the number of opening parenthesis (simple heuristic). Additionally, the lowered `match` calls have their `methodFullName` defined for convenient policy/semantic definition creation.
It was recently determined that bundled types were prefixed with the `__core.Kernel` prefix which was unintended, as it should only go as far as `__core`. This change remediates this, as well as logs a warning when a full name is attempted to be constructed in a way that will generate a bundled type with the `__core.Kernel` prefix, or if a non-bundled type will be created with a `__core` prefix.
* fix tab-completion in distribution the `removeModuleInfoFromJars` hack only get's triggered for `Universal/stage` - our release uses `Universal/packageBin` and that doesn't seem to reuse anything related that we could trigger on unfortunately... So I thought it's easiest to just add the same hack to the start script... re joernio#4625 (comment) * abs dir * handle both Universal/packageBin and stage: adapt mappings * remove hack from start script * refactor * fix copy pasta * simplify based on PR discussion
Fixed a bug introduced by converting all regex match methods by the `~=` binary statements. The bug is introduced is that this replaces ordinary calls (expressions) with binary statements leading to unexpected structures during AST creation. This was done to prevent infinite lowering whenever a regex method is created (as a `match` call is part of this lowering, which in of itself is a regex method). This change adds a "tag" to show when a call needs this lowering.
Addresses the technical debt introduced by joernio#5316
Replaced the check for the installed PHP version using Michael's `VersionHelper` instead of regex.
* added ExternalCommandUtil for buildfiles to remove the usage of scala.Process * Moved ExternalCommand to io.shiftleft.semanticcpg.utils, updated all references to new ExternalCommand * Remove old ExternalCommand. Some code cleanup * Removed ExternalCommandUtil, reverted build.sbt files to use scala.sys.Process. Re-added io.joern.x2cpg.utils.ExternalCommand with Deprecated annotation. Added io.shiftleft.semanticcpg.utils.ExternalCommandImpl trait * Removed ExternalCommandUtil, reverted build.sbt files to use scala.sys.Process. Re-added io.joern.x2cpg.utils.ExternalCommand with Deprecated annotation. Added io.shiftleft.semanticcpg.utils.ExternalCommandImpl trait * fmt * remove default values from call * remove default values from call * Change Some(var) to Option(var) for null-safety
* upgrade deps * ghidra 11.3.1
Persisting the lazily created JRuby execution environment and loaded global variables with cleaning up only happening at shutdown. This removes the overhead of JRuby instantiating a Ruby environment for each RubySrc2Cpg.
* Working on replacing better files * Replace all instances of better.files.File.newTemporaryDirectory with our own impl * Rename two functions, remove unused implicit class * Use Files.writeString, alias better.files instead of java.io.file * Replace all JFile to use Path instead * Replace Files.delete with FileUtil.delete * rename createIfNotExists function, added directory creation as well * Add linkOption for delete * remove println
Add modifiers to members --------- Co-authored-by: NextToMinus <[email protected]> Co-authored-by: David Baker Effendi <[email protected]> Co-authored-by: StackOF <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Created by Github action