Retrieving results from CI tool

[shaqb]

Hi team! We're just exploring the concept of shift left testing and we came across your tool for sending our results data to as it gives us lovely rich dashboards!

One thing we want to do is pass or fail a build based on whether a performance test has deviated from previous runs (say 10% deviation on the 95th percentile response time for a particular API).

I can see JTL reporter can send alerts based on deviation but is there a way to call an API to get the same status which we can call at the end of our run as a pipeline stuff, and pass/fail the build accordingly?

Our pipeline would look something like:

Developer merges code -> build/deploy -> run performance test -> push to JTL Reporter -> check against baseline data -> pass/fail.

Thanks!

Shaq

[ludeknovy]

Hi @shaqb !
If I understood correctly your query - you would like to send the report status from JtlReporter to CI? And based on the response adjust the build status accordingly?

[shaqb]

Hi @ludeknovy - yes that's correct. Or alternatively, the CI can query the JTLReporter at the end of the build and the response from the JTLReporter will determine whether the build passes or fails. This is so we can stop builds from being progressed in our CI flow.

[shaqb]

That would be awesome!

[shaqb]

There is a performance plugin for Jenkins which does similar: http://jenkinsci.github.io/performance-plugin/Reporting.html

But:

• It's not CI agnostic (only works in Jenkins)
• The graphs are terrible
• Retention is not great

But I like the idea of passing/failing builds based on a deviation from a relative baseline or an absolute threshold. Also being able to fail builds if too high an error rate was seen.

[ludeknovy]

WIP 👁️

[ludeknovy]

@shaqb what would be the desired behavior in case a baseline report is deleted or changed?

[ludeknovy]

@shaqb Let me re-phrase my question - what should happen for the current reports?
Case 1: the baseline report is deleted - for the reports they contain regression alert, I was thinking of removing it. Of course for newly uploaded reports, it won't get calculated.
Case2: the baseline report is updated - I was thinking about updating the regression alerts asynchronously.

[shaqb]

That’s awesome!

…

On Sun, 2 Apr 2023 at 20:58, Ludek Novy ***@***.***> wrote: WIP 👁️ [image: Screenshot 2023-04-02 at 21 55 33] <https://user-images.githubusercontent.com/13610612/229375972-cfafe665-770d-4468-ac53-15958091a175.png> — Reply to this email directly, view it on GitHub <#207 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEWBYKMLKOOMWO2JUNYW3M3W7HK4TANCNFSM6AAAAAAWKI6WPY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[shaqb]

I think in that situation we can’t report against and maybe have a different return code or status?

…

On Mon, 3 Apr 2023 at 20:43, Ludek Novy ***@***.***> wrote: @shaqb <https://github.com/shaqb> what would be the desired behavior in case a baseline report is deleted or changed? — Reply to this email directly, view it on GitHub <#207 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEWBYKOXWSP3TAUXXZIQZQDW7MR6FANCNFSM6AAAAAAWKI6WPY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[shaqb]

For case 2 would you look to recalculate on an interval or at point of request for the baseline and we would continue to poll until it’s ready? What would the sequence look like?

…

On Tue, 4 Apr 2023 at 07:17, Ludek Novy ***@***.***> wrote: @shaqb <https://github.com/shaqb> Let me re-phrase my question - what should happen for the current reports? Case 1: the baseline report is deleted - for the reports they contain regression alert, I was thinking of removing it. Of course for newly uploaded reports, it won't get calculated. Case2: the baseline report is updated - I was thinking about updating the regression alerts asynchronously. — Reply to this email directly, view it on GitHub <#207 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEWBYKKPAWS2WR6GYO32KW3W7O4GPANCNFSM6AAAAAAWKI6WPY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ludeknovy]

This would get recalculated after selecting another report as a baseline (it's done through the report detail update).
It would have no impact on your flow (it could, but the changes are very little actually, as the baseline report change would need to happen during the new report processing). The case 2 is much more about whether the reports should be re-calculated for the past reports.

[ludeknovy]

ludeknovy/jtl-reporter-be#213
ludeknovy/jtl-reporter-fe#316

[shaqb]

Ok that makes sense.

…

On Tue, 4 Apr 2023 at 08:37, Ludek Novy ***@***.***> wrote: This would get recalculated after selecting another report as a baseline (it's done through the report detail update). It would have no impact on your flow (it could, but the changes are very little actually, as the baseline report change would need to happen during the new report processing). The case 2 is much more about whether the reports should be re-calculated for the past reports. — Reply to this email directly, view it on GitHub <#207 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEWBYKLELYIWNFRH3AAR2QTW7PFSHANCNFSM6AAAAAAWKI6WPY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[shaqb]

You’ve been busy!

…

On Tue, 4 Apr 2023 at 20:19, Ludek Novy ***@***.***> wrote: ludeknovy/jtl-reporter-be#213 <ludeknovy/jtl-reporter-be#213> ludeknovy/jtl-reporter-fe#316 <ludeknovy/jtl-reporter-fe#316> — Reply to this email directly, view it on GitHub <#207 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEWBYKKK5RYQNRBZUFYPGSDW7RX5HANCNFSM6AAAAAAWKI6WPY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ludeknovy]

released in v4.8.2
https://github.com/ludeknovy/jtl-reporter/blob/main/docker-compose.yml

[shaqb]

Hi @ludeknovy - I'm now in the process of getting this integrated. Is there any documentation on how to retrieve the data, and also how to set thresholds for the baseline comparison (% or absolute deviation, number of runs to base it on, etc)

[ludeknovy]

To retrieve the data, you will need an API token, and with that you will be able to call /api/projects/<project_name>/scenarios/<scenario_name>/items/<id>

To set up the thresholds go to Project->scenario->settings, and please keep in mind that you need to have the baseline set up.
You can do so from the report settings modal (accessible via report detail or test runs listing) and checking the "Set as base run" checkbox.

[shaqb]

Ah ok got it. Is it possible for the "base run" data to be automatically calculated based on the last x runs where x is a configurable number?

So as an example I want to see if my latest run has deviated by more than 10% of the last 10 runs. This way the test reporting is completely dynamic and we don't need to regularly go in and manually update the latest base run (which we would need to do if performance got better or worse in a given release).

[ludeknovy]

No, this is not currently possible. Actually, it was working similar to this in the past. But you can set the baseline just once. There is a disadvantage from the approach you are describing; over a long period of time, the performance could really degrade without noticing/ triggering the threshold. So I think it really makes much more sense to compare it to a baseline report in this case.

[shaqb]

Fair point. Anyway let me play around with it and see what the output looks like. I still can't believe you've made such a comprehensive tool like this!

[ludeknovy]

Keen to hear your feedback!

[shaqb]

Hey @ludeknovy - been playing around with this. Is it the threshold block I need to look at?

"thresholds": {
    "passed": false,
    "results": [
      {
        "label": "RQ-BrowsePDP",
        "passed": true,
        "result": {
          "errorRate": {
            "passed": true,
            "diffValue": 100
          },
          "percentile": {
            "passed": true,
            "diffValue": 57.72285966460724
          },
          "throughput": {
            "passed": true,
            "diffValue": null
          }
        }
      },
      {
        "label": "RQ-InventoryAvailability",
        "passed": true,
        "result": {
          "errorRate": {
            "passed": true,
            "diffValue": 100
          },
          "percentile": {
            "passed": true,
            "diffValue": 24.846446329336064
          },
          "throughput": {
            "passed": true,
            "diffValue": null
          }
        }
      },
      {
        "label": "RQ-ProductList",
        "passed": false,
        "result": {
          "errorRate": {
            "passed": true,
            "diffValue": 100
          },
          "percentile": {
            "passed": false,
            "diffValue": 107.27272727272728
          },
          "throughput": {
            "passed": true,
            "diffValue": null
          }
        }
      }
    ],
    "thresholds": {
      "errorRate": "5",
      "percentile": "5",
      "throughput": "5"
    }
  },

What's your view on having a separate API endpoint for this just to make it easier to parse for our CI tool...

[ludeknovy]

Hi @shaqb yes, that's it. How would a separate endpoint make the parsing easier? Honestly, I don't see any benefit here.

[shaqb]

Hi Ludeknovy. Poor choice of words on my part. What I meant was to have an endpoint specifically just for the pass/fail status. But I can see that is quite a specific use case and the information is already there.

…

On Wed, 24 May 2023 at 07:06, Ludek Novy ***@***.***> wrote: Hi @shaqb <https://github.com/shaqb> yes, that's it. How would a separate endpoint make the parsing easier? Honestly, I don't see any benefit here. — Reply to this email directly, view it on GitHub <#207 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEWBYKMEJFTAZ66ME7TFTZTXHWQN3ANCNFSM6AAAAAAWKI6WPY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ludeknovy]

Yes, from my point of view, it's redundant.

[shaqb]

Hey @ludeknovy so we've started integrating the JTLReporter into our CI pipelines and I think we might have an issue. When I submitted the result I got an itemId back. However, this test run is still processing so when I call the API to get the status I get back the following response:

{ "message": "Not Found" }

Is it possible to change the output to say 'processing' or something like that? I can then poll with the CI tool until the actual json of the test run is returned.

I can see the processing here:

But the API is showing the Not Found above.

[ludeknovy]

Hi @shaqb! Changing that API is not as easy as it sounds. You can achieve the desired behavior by using following endpoint:
/api/projects/<project_name>/scenarios/<scenario_name>/processing-items
in the response it returns:

{
    "failed": [],
    "inprogress": [
        {
            "id": "3215c173-ebc1-469a-b92f-5eddadc53dde",
            "reportStatus": "in_progress",
            "uploadTime": "2023-04-03T16:58:29.035Z"
        }
    ]
}

And once it's not there anymore, it means the processing finished.

[shaqb]

Nice! That will do the trick. I'll add that as a step in our pipeline.