Snyk scanning pipeline fixes. #18499

mashhurs · 2025-12-17T05:40:34Z

With this PR, the improvements are:

fixes the bug where elastic_integration is in the elastic org
snyk CLI needs to be located in the cloned plugin repo. CI jobs are green but if we open it Snyk CLI is actually not available: /bin/bash: line 14: ./snyk: No such file or directory
introduces a feature to use Logstash branch in the matrix (e.g: logstash_branch: 'main') if requires (e.g: elastic_integration or date-filter require logstash-core gems)

With the matrix definition, the key points are:

requires_logstash_core: some plugins require LS core gems, we need to clone LS repo and define LS core path;
- Some plugins in the matrix require Logstash core gems, some do not. For example the date-filter, its gradle requires logstashCoreGemPath, eventually :vendor => "gradle.properties" rake task creates it.
branches: some plugins have multiple active branches (e.g: main, 11.x) to scan;
scan_branches: elastic_integration plugin requires 1:1 branch map

…ration is in other org and snyk CLI needs to be located in the repo.

github-actions · 2025-12-17T05:40:48Z

🤖 GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
/run exhaustive tests : Run the exhaustive tests Buildkite pipeline.

mergify · 2025-12-17T05:41:11Z

This pull request does not have a backport label. Could you fix it @mashhurs? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
If no backport is necessary, please add the backport-skip label

donoghuc · 2025-12-17T20:57:36Z

.buildkite/scripts/snyk/plugins-scan/generate-steps.py

-curl -sL --retry-max-time 60 --retry 3 --retry-delay 5 https://static.snyk.io/cli/latest/snyk-linux -o snyk
-chmod +x ./snyk
+export JAVA_HOME="/opt/buildkite-agent/.java/adoptiumjdk_21"
+export PATH="/opt/buildkite-agent/.java/bin:$JAVA_HOME:$PATH"


Can we just source https://github.com/elastic/logstash/blob/main/.buildkite/scripts/common/vm-agent-multi-jdk.sh like we do elsewhere for prepping java env?

Yup, thank you for pointing it, replaced with vm-agent-multi-jdk.sh

…nds to target only gradle belong to the plugin and ignores multi-projects

…s conflict.

…oth all-project and project-name

donoghuc

Instead of using special branches and logstash_branch values for elastic_integration, I think we should be more explicit. If we find that is hard to maintain or keep updated in the future we can talk about other optimizations/workarounds. I think we are optimizing too early here and its starting to make the step generation code hard to reason about.

donoghuc · 2026-01-05T17:32:11Z

.buildkite/scripts/snyk/plugins-scan/generate-steps.py

+if ! git clone --depth 1 --branch {logstash_branch} https://github.com/elastic/logstash.git; then
+    echo "Branch {logstash_branch} not found in logstash, skipping..."
+    rm -rf {work_dir}
+    exit 0


Should this exit non zero?

Yup, thanks for catching it. Removed if ... then if git clone workflow also fails (tested in the past)

donoghuc · 2026-01-05T17:56:56Z

.buildkite/scripts/snyk/plugins-scan/plugins-snyk-scan-matrix.yaml

    branches: [main, 'use-release-branches']
-    ignore_branches: [7.x]
+    ignore_branches: [7.x]
+    logstash_branch: 'match-with-plugin-branches'


I'm a bit hesitant to introduce this "magic" match-with-plugin-branches name. Fundamentally the only plugin that strives to sync up branch name with logstash is filter-elastic_integration. Clearly there is a case for mapping target plugin branch with logstash branch. Instead I think we should make this explicit. Specifically I think logstash_branch should be a map in which the keys are the branches in branches section and the values are the branch in logstash. We should add validation that when logstash_branch is present that branches is also present.

- logstash-input-dead_letter_queue: branches: [main] logstash_branch: main: main

There is the same sentiment for the "magic" use-release-branches name. I really dont think that having these special cases throughout the code is worth the overhead. I think that maintaining static lists is less cognitive burden and makes us be more intentional about exactly what we are scanning.

Yup, let's think about optimization. I have removed the magic use-release-branches entry 😁 and specified with plugin vs core branch pair.

… Use a precise {key, val} matrix to scan elastic_integration plugin with Logstash branch pair.

donoghuc · 2026-01-05T22:18:46Z

.buildkite/scripts/snyk/plugins-scan/plugins-snyk-scan-matrix.yaml

-  - logstash-filter-geoip
-  - logstash-filter-jdbc_static
+  - logstash-filter-date:
+      requires_logstash_core: true


is requires_logstash_core a shorthand for a scan_branches map with a single key/value?

I guess its a shorthand for:

- logstash-filter-date: scan_branches: - branch: main logstash: main

You effectively get the "default" branch for both the plugin and logstash (where default branch is the configured default in the git repository).

Ah, sorry I had to add such details in the PR description to explain. I will add.
Some plugins in the matrix require Logstash core gems, some do not. For example in this date-filter, its gradle requires logstashCoreGemPath, eventually rake task creates it.
In the past, I somehow used logstash_branch which doesn't make sense to me as we have introduced scan_branches map. So, replaced with requires_logstash_core to make it more understandable.

Basically, overall key points are:

some plugins require LS core gems, we need to clone LS repo and define LS core path;

some plugins have multiple active branches (e.g: main, 11.x) to scan;

elastic_integration plugin requires 1:1 branch map

I have updated the PR description as well!

That sounds like the same use case to me? the plugins (like date-filter) require the logstash repo to be cloned on the test runner. Presumably if there was a matrix of plugin version to logstash version we would want to do the same thing (make sure that logstash is available at a checked out version). I'm wondering why there are two different paths to effectively do the same thing.

Yup yup, depth-first search became breadth-first search :)

Thanks for asking clarifications.

donoghuc

Love this simplification f8da811

Thanks for iterating on the feedback. This is looking good to me! 🚀

elasticmachine · 2026-01-06T00:30:48Z

💚 Build Succeeded

Buildkite Build
Commit: f8da811

History

💛 Build #4045 was flaky d84da7c
💛 Build #4022 was flaky 4693fe8
💛 Build #4021 was flaky 52cdfcc
💚 Build #3999 succeeded 968a3b6
💚 Build #3996 succeeded c76d76c
💚 Build #3993 succeeded 0dd70a6

cc @mashhurs

Fixes the bugs of plugin snyk scan pipeline logic where elastic_integ…

4d380ab

…ration is in other org and snyk CLI needs to be located in the repo.

mashhurs requested a review from donoghuc December 17, 2025 05:40

mashhurs self-assigned this Dec 17, 2025

mashhurs added 4 commits December 16, 2025 22:11

Export JAVA_HOME and skip if branch doesn't exit when git cloning.

bab4ffa

Add Java path

370c419

Add agent to the CI job.

5eccb7b

Set Java home.

321f502

mashhurs marked this pull request as draft December 17, 2025 07:02

donoghuc reviewed Dec 17, 2025

View reviewed changes

mashhurs added 10 commits December 17, 2025 15:55

If root folder has gradle settings, snyk may detect. This commit inte…

2ecc61c

…nds to target only gradle belong to the plugin and ignores multi-projects

Try tempdir to avoid gradle root path settings conflict.

4d92079

Remove input-jdbc_static

0dd70a6

Use Logstash for elastic_integration plugin use case.

c76d76c

The plugins which needs 'logstashCoreGemPath' now uses Logstash.

6ae1ca9

Use static working dir insteam tesm generated to avoid gradle setting…

bd227e7

…s conflict.

geoip-filter requires 'logstashCoreGemPath' as well.

968a3b6

Let's try all projects scanning except ruby package manager.

0cc0346

Use common vm configs, remove project name since snyk doesn't allow b…

6b55bcc

…oth all-project and project-name

Avoid scanning test paths

52cdfcc

mashhurs requested a review from donoghuc December 22, 2025 21:19

mashhurs marked this pull request as ready for review December 22, 2025 21:19

Include integration-kafka 11.x branch

4693fe8

donoghuc requested changes Jan 5, 2026

View reviewed changes

Intentionally fail if Logstash branch doesn't exist when git cloning.…

d84da7c

… Use a precise {key, val} matrix to scan elastic_integration plugin with Logstash branch pair.

donoghuc reviewed Jan 5, 2026

View reviewed changes

Simplify the matrix to have only scan_branches param.

f8da811

donoghuc approved these changes Jan 6, 2026

View reviewed changes

donoghuc mentioned this pull request Jan 6, 2026

Stop new snyk pipeline schedule while being developed #18542

Closed

mashhurs merged commit f9d8b77 into elastic:main Jan 6, 2026
12 checks passed

Snyk scanning pipeline fixes. #18499

Snyk scanning pipeline fixes. #18499

Conversation

mashhurs commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 17, 2025

🤖 GitHub comments

Uh oh!

mergify bot commented Dec 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

donoghuc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

donoghuc left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jan 6, 2026

💚 Build Succeeded

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mashhurs commented Dec 17, 2025 •

edited

Loading