Merge 4-0-280 to backup_sync #195
Open
+43,071
−38,038
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
fixes the regression from the ExternalCommand refactor, but I couldn't leave the the env var handling as it was either. Fixing that by using a proper library for parsing CLI arguments
* [kotlin2cpg] Improve fullnames. Improve and cleanup the fullname calculation for methods and callsites. The most noteable changes here are: 1. The lambda function naming schema changed from `package.<fileName>.<lambda>counter:signature` to `package.scope.<lambda>counter:signature`. This give a better indication which lambda function is represented also gets rid of exception of using the filename in the method fullnames. 2. Fixed lambda function signatures. 3. The fullname schema of extension methods changed from `extendedClassFullName.extensionMethodName:signatureWithoutThisParameter` to `package.extensionMethodName:signatureWithThisParameter` The fact that the this parameter is part of the signature is an exception in our java fullnames but necessary in order to distinguish otherwise identical extension methods. 4. We avoid the recalculation of the binding context with every file processed. This is now done once at the begining. 5. Calls to `::class` are now represented as calls to `<operator>.class`. * Fix extension method arg and param indicies. The instance arg/param now have index 1 as they are not the receiver on which the call is dispatches. It is a static call after all. * Implement first sane version of lambda binding nodes. Additionally fixed the inheritance information for lambda type decl nodes. * Refactor NameRender. Split some methods into BindingContextUtils and renamed those methods. * fmt * Incorporate review comments. - No more special case for handling `kotlin.Function`. - Removed some unused code. - Stop using currentTimeMillis. * Remove typeInfoProvider.expressionType and replace it via nameRenderer. This results in expresssion types now being renderer in the new fassion. * Translate kotlin.Array type full name to java representation. * Removed multiple further type full name API methods from TypeInfoProvider. * Remove typeFullName for KtDestructuringDeclarationEntry from TypeInfoProvider. * Remove visibility and modality API from TypeInfoProvider. * Remove containingTypeDeclFullName. * Remove unnecessary code. * Remove inheritanceTypes API from TypeInfoProvider. * Remove destructuringEntries API from TypeInfoProvider. * Removed KtTypeAlias related TypeInfoProvider APIs. * Remove typeFullName API for binary expressions from TypeInfoProvider. * Remove typeFullName for annotations from TypeInfoProvider. * Remove unnecessary Option types from BindingContextUtils APIs. * Rework lambda implicit parameter handling. - Removed implicitParameterName and hasApplyOrAlsoScopeFunctionParent APIs from TypeInfoProvider. - Removed wrong parameter deconstruction in lambda. New implemention is missing. - Some lambda to builtin `apply` and `also` functions did not get return statements generated. That is now fixed. * Remove typeFullName and isCompanionObject APIs from TypeInfoProvider. * Remove typeFullName API for KtTypeReference from TypeInfoProvider. * Remove containingDeclType API from TypeInfoProvider. * Remove typeFullName and referenceTargetTypeFullName APIs from TypeInfoProvider. Also refactored assignmentAstForDestructuringEntry. The right hand side base is now created outside of this function allowing for other constructs than pure identifiers. The pure identifier were e.g. wrong in case of class member references. * Remove propertyType and typeFromImports API from TypeInfoProvider. * Remove TypeRenderer. * Improve comment. * Fmt. * Fix for `ScopeFunctionsTests` * Adjusted test expectations for more precise/sound outcomes * Left note about the lambda flows * Add better fallback handling methodFullName of calls. In case of incomplete type information calls with overloads can get ambiguous. We now use the ambiguous function descriptors to get as much of a methodFullName correct as possible. * Fix test fullname. --------- Co-authored-by: David Baker Effendi <[email protected]>
This bypasses tty buffering and read timeouts.
* [dataflowengineoss] add Operators.modulo semantics * sort results for deterministic outcome
* [pysrc2cpg] refactor <module>, ANY and __init__ constants * Replace Constants/Defines, cf. feedback suggestion
* minor changes: code format * add "isInline" method for "CallMethods" * add a method "isInline" for CallTraversal
…esses. (joernio#5036) * [javasrc2cpg] Generate TYPE_REF for implicit base of static field accesses. For static field access like `String x = staticField1;` we used to generate a IDENTIFIER with the name of the class declaring the static field as base for the field access operator. Now we generate a TYPE_REF referencing the declaring type. This better represents the actual situation where there is no local variable for a class/type which in practice also resulted in a CPG format error because the IDENTIFIFER node was at least not in all cases linked to a LOCAL. While implementing this i reorganized the code a bit to avoid duplication. Also changes the `equals` semantic for TrackedTypeRef. Before it was based on the actual TYPE_REF node which does not make sense. Instead we need to compare the referenced TYPE itself to check if two different type references in code point to the same TYPE. * Handle statically imported field scenario.
Due to the performance and stability issues with a 2-phase LLK parser approach with ANTLR, this branch, developed over the past two weeks, introduces the usage of a [`parser`](https://github.com/whitequark/parser/tree/master) based approach wrapped in [`ruby_ast_gen`](https://github.com/joernio/ruby_ast_gen). This Gem is used by many popular Ruby linters and static analysis tools, and is compatible with Ruby versions across 1-3. We bundle `ruby_ast_gen` with the frontend under `resources`, and execute the Ruby code using JRuby to maximize portability. If the resource is detected to be in a JAR, `ruby_ast_gen` is written to a temporary directory, from where it is then executed. Both JRuby and native Ruby runs of `ruby_ast_gen` far outperform the old ANTLR version, which would often have worst-case performance for some set of language constructs encountered. Changes include: * Remove ANTLR implementation * Execute `ruby_ast_gen` and parse JSON output * Map the JSON output to the same intermediate AST nodes that ANTLR used to avoid changing `AstCreation` logic * Rename the method name for executable strings (`%x{}`-style strings) to `<operator>.backticks` instead of `exec` as `exec` technically has different semantics * Trim `.code` properties to 60 characters to avoid large `.code` properties. (this is done by `ruby_ast_gen`) * Added `db/migration` to default ignored directories Tests are largely kept the same, and `ruby:test` is green on `codescience` after some small `.code` and `.lineNumber` tweaks. See `dave/ruby/prep-for-ruby-parse-gem`. The `astGenResourceTask` task in `build.sbt` is tied to `Compile / resourceGenerators` as the task fetches the zipped Ruby code and places it under `resources` to be packaged alongside the classes in the JAR. --------- Co-authored-by: Andrei Dreyer <[email protected]> Co-authored-by: Andrei Dreyer <[email protected]>
…condition (joernio#5046) * [gosrc2cpg]handle the init code before if condition. * [gosrc2cpg]handle the init code before if condition, fix the test error. * [gosrc2cpg]after formatting.
We migrated to sonatype central due to publishing problems, as suggested by their support. Let's see if this works. https://central.sonatype.com/
* Moved lowering for ||= and &&= to AstCreator * Moved lowering func to AstCreatorHelper trait
* upgrade deps * latest
…nio#5071) * [kotlin2cpg] Dispose the environment as soon as it's not needed * scalafmt
…ode (joernio#5076) * [javasrc2cpg] support flows for field accesses with a TYPE_REF base node * scalafmt * simplify `.where(_.or...)` with `.or(...)` cf. review suggestion Co-authored-by: maltek <[email protected]> --------- Co-authored-by: maltek <[email protected]>
Replaced the check for the installed PHP version using Michael's `VersionHelper` instead of regex.
* added ExternalCommandUtil for buildfiles to remove the usage of scala.Process * Moved ExternalCommand to io.shiftleft.semanticcpg.utils, updated all references to new ExternalCommand * Remove old ExternalCommand. Some code cleanup * Removed ExternalCommandUtil, reverted build.sbt files to use scala.sys.Process. Re-added io.joern.x2cpg.utils.ExternalCommand with Deprecated annotation. Added io.shiftleft.semanticcpg.utils.ExternalCommandImpl trait * Removed ExternalCommandUtil, reverted build.sbt files to use scala.sys.Process. Re-added io.joern.x2cpg.utils.ExternalCommand with Deprecated annotation. Added io.shiftleft.semanticcpg.utils.ExternalCommandImpl trait * fmt * remove default values from call * remove default values from call * Change Some(var) to Option(var) for null-safety
* upgrade deps * ghidra 11.3.1
Persisting the lazily created JRuby execution environment and loaded global variables with cleaning up only happening at shutdown. This removes the overhead of JRuby instantiating a Ruby environment for each RubySrc2Cpg.
* Working on replacing better files * Replace all instances of better.files.File.newTemporaryDirectory with our own impl * Rename two functions, remove unused implicit class * Use Files.writeString, alias better.files instead of java.io.file * Replace all JFile to use Path instead * Replace Files.delete with FileUtil.delete * rename createIfNotExists function, added directory creation as well * Add linkOption for delete * remove println
Add modifiers to members --------- Co-authored-by: NextToMinus <[email protected]> Co-authored-by: David Baker Effendi <[email protected]> Co-authored-by: StackOF <[email protected]>
…detecting types for PHP in queryDB (joernio#5327) * fix: Add PHPJoern queries for SQL injection, command injection, code injection, unrestricted file upload, and XSS detection * fix: Improve PHP version comparison to handle version strings correctly * fix: Enhance PHP version parsing and update SQL injection query titles for clarity * Update querydb/src/main/scala/io/joern/scanners/php/PhpJoern.scala merge the suggestion for the source identification Co-authored-by: David Baker Effendi <[email protected]> * Update querydb/src/main/scala/io/joern/scanners/php/PhpJoern.scala merge the suggestion for the source identification Co-authored-by: David Baker Effendi <[email protected]> * Update querydb/src/main/scala/io/joern/scanners/php/PhpJoern.scala merge the suggestion for the description about the XSS attack Co-authored-by: David Baker Effendi <[email protected]> * Update querydb/src/main/scala/io/joern/scanners/php/PhpJoern.scala merge the suggestion for the description about the code execution attack Co-authored-by: David Baker Effendi <[email protected]> * Update querydb/src/main/scala/io/joern/scanners/php/PhpJoern.scala merge the suggestion for the description about the SQLi attack Co-authored-by: David Baker Effendi <[email protected]> * Update querydb/src/main/scala/io/joern/scanners/php/PhpJoern.scala merge the suggestion for the default iterator accessing Co-authored-by: David Baker Effendi <[email protected]> * fix: Add PHP security test suite and update dependencies * fix: Update PHPJoernTests to reflect changes in user input handling * refactor: Improve code formatting and readability in PHP-related files --------- Co-authored-by: David Baker Effendi <[email protected]>
* Make Gradle dependency fetcher init script more robust * Fetch class jars instead of source jars in init script
* parallelize delombok-ing * Update joern-cli/frontends/javasrc2cpg/src/main/scala/io/joern/javasrc2cpg/util/Delombok.scala Co-authored-by: maltek <[email protected]> --------- Co-authored-by: maltek <[email protected]>
* [jimple2cpg] if a member have ConstantValueTag in SootField info, add the tag value to the member possibleTypes. add tests in MemberTests. * [jimple2cpg] change the constantValue to the member code --------- Co-authored-by: NextToMinus <[email protected]>
To be consistent with other dynamic calls, adds a dynamic type hint to literals + sets `match` to be `DynamicCallUnknownFullName` to enable dynamic call resolution strategies to kick in.
…oernio#5332) * Removed last File.newTemporaryDirectory instances * code cleanup * fix jimple failing tests * fix jimple tests * Ensure all file streams are closed to allow file delete on windows * fmt * fix gradle destinationDir * code cleanup --------- Co-authored-by: Andrei Dreyer <[email protected]>
* Removed last File.newTemporaryDirectory instances * code cleanup * fix jimple failing tests * fix jimple tests * Ensure all file streams are closed to allow file delete on windows * fmt * fix gradle destinationDir * code cleanup * remove all usages of better.files.File.newTemporaryFile * remove all usages of better.files.File.newTemporaryFile * remove touch-like function --------- Co-authored-by: Andrei Dreyer <[email protected]>
…ntent via offset (joernio#5341) * Add sourceCode accessor to nodes and traversals for accessing source code via offset * Update semanticcpg/src/main/scala/io/shiftleft/semanticcpg/language/nodemethods/AstNodeMethods.scala Co-authored-by: Michael Pollmeier <[email protected]> * Separate offset and sourceCode tests * Improve offset test descriptions --------- Co-authored-by: Michael Pollmeier <[email protected]>
* ghidra: use our fork https://github.com/joernio/ghidra * use first release made from github actions runner https://github.com/joernio/ghidra/actions/runs/13589776349 * next try, this time with mac and win binaries
) * Removed last File.newTemporaryDirectory instances * code cleanup * fix jimple failing tests * fix jimple tests * Ensure all file streams are closed to allow file delete on windows * fmt * fix gradle destinationDir * code cleanup * remove all usages of better.files.File.newTemporaryFile * remove all usages of better.files.File.newTemporaryFile * remove touch-like function * removed all instances of better.files.File.usingTemporaryDirectory/usingTemporaryFile * code cleanup * fix file content for rubysrc2cpg * Attempting to fix windows tests on runner * remove system.gc * fmt * Revert to default scala Using, update writeBytes to use Iterable instead of iterator type for param * Removed default charset to all writeString calls, deleted commented code * fmt * missed a File.usingTemporaryDirectorry * revert retry delete * remove unused charset imports * remove commented out line in workrflow * code cleanup + fmt --------- Co-authored-by: Andrei Dreyer <[email protected]>
* Add init script that only works with gradle 8 * Fix init script for Gradle <8 * Add default app override * Clean up unnecessary TODOs
No custom operator call anymore. It's semantically just a block of expressions.
This speeds up parsing quite a lot. (OPTION_NO_IMAGE_LOCATIONS, OPTION_SKIP_TRIVIAL_EXPRESSIONS_IN_AGGREGATE_INITIALIZERS) Also cleaned up the logs: - do not log preprocessorErrorCount and parse problem numbers. These numbers might be nice for small code snippets but do not mean anything on real projects when they go to >100 or the like. - only log relative paths (easier to read logs)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Created by Github action