Some insighst into report

[tomatizon]

Hi, @ludeknovy did You considered to rebuild Your report in a way that:
aggregated table from locust will not be a base for first visible graph/chart in test run results?
Usually, there is a main puropse of running performace test run (for examle - You are running a test scenario, that should measure against /placeOrder endpoint performance. To do that, you are running a path that include /login, /addToCart, etc.
Right now, locust (and Yours JTL reporter app) at the first step of presentation gives You an aggregated metrics for all of that endpoints, which is not that much useful, when You need a specific endpoint metrics).
It would be great, if after opening the test run report, at the top of a first visible dashboard You could choose which endpoint results should be presented in details, with an option to compare them with any other previous run (right now, You can find them in a section below, expanding graph section (available for all endpoints).
Is there any way to customize that, and Im missing it? Without messing up with app code customization of course:P
BTW: GREAT WORK!!!

[ludeknovy]

Hi @tomatizon.

That's an interesting idea. Currently, that's not possible. What could be doable (without some major re-work) is to introduce a new tab next to the custom chart tab in the report, where you could add any chart you want and save it as blueprint - let's say you upload the very first report, then you select the charts you want to appear them and save the settings. After uploading another report those charts would be automatically added (if found) to the tab. What do you think?

[tomatizon]

Next few days we will try to go deeper into JTL reporting, because our current approach (manual or half manual uploading results into excel is not most efficient way and costs a lot of time, preparing own resolution on Grafana/Kibana will add a lot of work with files flow, compatibility with previous runs etc). And JTL is out-of-the-box resolution, always better than in-house apps that need maintenance)
So, I will let You know the results of our investigations, and "If" we think that there are some areas that can be improve. But, remember -> right now we are using excel sheets, so any automation is upgrade:P
Personally, I think that this chart:

has not much use in most of the test runs. Giving aggregate values for random endpoints (/login, /addToCart, /placeOrder) during the test doesnt give You any valuable insight, just shows that tests are working:P Of course thats a representation of aggregated values from Locust output, which I always skip when analysing results. Of course - I can be wrong, and there is more value in that then I think.

If that chart will give You for examlple last 15 runs for one endpoint that is yours main focus, with the same, comparable metrics (p90, p95, p99 response times, errors i.e) it will serve guys that looks into this just to check if everything goes right direction (so, mainly management/team leaders).
BTW. Our setup for perf tests will be: creating a build for endpoint that needs to be tested, run from CI/CD tool (like jenkins or teamcity) triggered manually, sheduled or on action, with integrated JTL Reports linked in job artifacts. I hope at least:P Ive made POC locally, and trying to make comprehensive examples.

[ludeknovy]

Aha, I see. Yes, this could be done (Actually, you can get the data you ask for from the report detail - request stats table there is an icon on the right side which displays the trend of an individual label). The only problem I see for a test runs with many labels, that would be just a mess, I think. So some kind of limit would be needed with the option to specify which label should be displayed by default.

Btw I used to have similar setup CI/CI triggered the performance test -> uploaded the data into the JtlReporter which sent an notification to slack/ms teams with a link.

[ludeknovy]

#204

[ludeknovy]

I have some progress on the #204
https://user-images.githubusercontent.com/13610612/221402589-28fdc311-1f1a-4bc1-9784-844c158e22fc.mov

It will allow displaying two metrics, planning to add a switch so they could be configured as well. What metrics do you think would be helpful here? I was thinking about p90, throughput and error rate.

The number of labels displayed is limited to 20. Warning is shown if the data is truncated

[ludeknovy]

@tomatizon If you feel adventurous, you might try the latest tags to see the label charts in action.
Do not forget to backup your data (just in case): https://jtlreporter.site/docs/guides/backup-and-restore-database

[tomatizon]

Sure, i can see individual endpoints statistics, thats why JTL reporting is great:P

Thats the first and most important place when I can look for results. At least me. The charts like this one:

have a litlle value for me personally, becouse they show aggregated endpoints statistics, and can aggregate data from /SsoLogin (which takes lets say 1300 ms, becouse of external integration, and happens exactly 3000 times with a ratio 50r/s during test for /placeOrder) and data from placeOrder(that can be hit 12000times, with a ratio 200r/s, and expected p95 less than 900ms).
And that chart makes an average of both, which says me not much:P But thats basically the same issue with aggregated locust table.

So, I think that You do have all data available i JTL and even a some kind of 'base run' conception implemented. But somwhere somwhere at the end of the report, which makes it less accesible:P for less determined users). Moving order of the tables, to show detailed view first (or just let customize order for a user - will do great)

After some testing and using this awesome app, I've got new improvements requests - table chart in the section Request Statistics can be compared, but graph visualise only current test run.
And - there is no req/s metrics to create custom report to compare how response time is changing with increased req/s to define the point in which we cannot handle new users or req/s (because number of users in Locust is just a way of increasing max value of req/s as I think) without significant performance drop. But thats a lot of work:P

Tool aready looks GREAT. This week we will set up new flow around perf tests, with JTL reporter.

[ludeknovy]

I've got new improvements requests - table chart in the section Request Statistics can be compared, but graph visualise only current test run. not sure I got this part. What's the desired behaviour here?

[tomatizon]

Here I would like to see, on a graph, results of both:
current run
and run that I choose to compare

The same metrics applied. It will be a lot of metrics, but I think people usually compare the same metric p/99 of current with p/99 of chosen to compare.

[ludeknovy]

#205

[ludeknovy]

👀

Screen.Recording.2023-03-03.at.15.03.26.mov

[ludeknovy]

#205 merged

[tomatizon]

Yah, You are right, the comparision of r/s and response time is there:P