[CI] Reassess value of load tests

[djaglowski]

@open-telemetry/collector-contrib-approvers, @open-telemetry/collector-approvers, the load tests have always been a source of test failures for us. All we always do is increase the limits once we start hitting them more frequently. I don't remember ever seeing a true positive out of those tests. Are those tests providing value to us? Or are they just sources of noise?

Originally posted by @jpkrohling in #9094 (comment)

[codeboten]

We've talked about this in the past and there was some discussion of running the load tests less frequently, maybe on a daily basis. I'd prefer continuing to run the load tests on every build, record their results, and if possible, emit some sort of report (ie. codecov) instead of having intermittent failures in the build.

I think it's more important to be able to see the historical results of the load tests, since the intermittent failures are being worked around by bumping the resource requirements every few weeks and hide actual build failures.

I'd be happy to take on a task to produce a report with results from every build into main if folks think it's valuable. Producing the report that's published using github pages has been done in the python repo for some time, i don't know if there's an easy way to produce a report that can be added as a comment, i'd have to dig into that.

[tigrannajaryan]

That python repo is very nice! Would be great to publish our perf results in a similar way.

[djaglowski]

I fully agree with @codeboten's assessment and suggestion. It's valuable information but can be non-blocking without much risk.

[jpkrohling]

If we end up having something like the Python repo, the value from the load tests becomes huge.

[TylerHelmuth]

As part of an ongoing discussion in the Go SIG around performance tests, I have requested a machine so that we can test out if private hardware will solve test inconsistencies. I plan to experiment with the Contrib's load test as well.

If private hardware is a solution, then hopefully we can find a way to run the tests on the machines going forward. More discussion on long term plans can be found in the linked issue.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[dmitryax]

We ran into an issue when the load tests haven't detected a significant spike. I submitted another issue for the same problem with some suggested action items #20718 PTAL

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.