Provide options to limit concurrent ONTAP HTTP connections

[songlin-rgb]

Harvest currently does not limit its number of HTTP connections to ONTAP [1]. Since ONTAP limits the number of connection requests for a socket (e.g., up to 766), Harvest could use up that limitation and prevent connections from production traffic.

We have a multi-tenancy environment so we installed one Harvest for each tenancy (up to 10 in total). We recently saw Harvest connections saturating ONTAP in a prod-like environment.

We request the Harvest community to improve the HTTP connection logic. For example, use a HTTP pool with a configurable number of connections.

[1] https://github.com/NetApp/harvest/blob/main/cmd/tools/rest/client.go#L73-L78
[2] https://docs.netapp.com/us-en/ontap-ems/bsdsocket-queuelimit-events.html

[cgrinds]

Thanks for the request @songlin-rgb, we'll take a look. Did you explore any other ways to handle multi-tenancy? I agree that monitoring a single cluster with ten collectors may saturate the cluster. Do any of the previous discussions on multi-tenancy look promising for you? It seems like the problem you are trying to solve is how to handle multi-tenancy. Is that fair?

• https://github.com/NetApp/harvest/wiki/FAQ#multi-tenancy
• Restrict user accounts to specific clusters/SVMs #2441
• Add multi-tenant SVM support #355

[songlin-rgb]

Thank you so much for the questions @cgrinds!

The ONTAP cluster can be saturated by internal traffic. For this reason, it is important that individual components, such as Harvest, connect to it using a bounded number of connections. We can continue to learn and tune our configuration in this discussion, but these performance tuning efforts cannot resolve that foundational issue.

At this point, I believe we have gathered enough information to confirm that #3999 should be accepted. Do you agree, or shall we escalate this matter further?

Regarding your questions:

• I'm not sure if it's allowed or efficient to share logs. Could you please share instructions on what information to look for in the logs instead?
• How do I check whether I'm using SVM tunneling or legacy RBAC? I believe it is SVM tunneling, but I would like to confirm.

[cgrinds]

@songlin-rgb we'll take a look at adding concurrency limits after the 25.11 release (next week or two depending on burn down).

The logs tell us which collectors/templates are being used, the number of APIs, avg API response time to collect metrics, number of instances, number of metrics exported, and insight into the concurrency. Feel free to check the logs for personally identifiable information. There should not be any. Broadly speaking, the logs help the Harvest team understand how you are using Harvest and if there are any errors. Can you check if you're allowed to send them? In my experience, it's quite efficient to gain an understanding by visualizing the logs. It answers a lot of questions about which collectors you are using, permission problems, ONTAP errors, etc.

Harvest does not currently use SVM tunneling. ONTAP requires passing the X-Dot-SVM-Name or X-Dot-SVM-UUID header. Examples here. Unless you've modified Harvest to include the header, you're using legacy RBAC. Did you setup your harvest user/role via the CLI following the least-privileged approach?

This Discord link also describes a way to use RBAC to limit a role to a specific SVM. That might be an interesting option to try or maybe that is what you are already doing?

[cgrinds]

@songlin-rgb another way to limit the concurrency is from the cluster-side. Have you tried limiting the connections on the cluster with the session limit command? Something like this, replacing $Harvest-UA with your version of Harvest's user-agent.

security session limit application create -application $Harvest-UA -interface rest -max-active-limit 10

If you don't know your Harvest version's user-agent, you can find it in a couple of ways:

1. By running bin/harvest --version. It will print something like
   harvest version 25.11.05-1 (commit e5ce2c87) (build date 2025-11-05T01:47:16-0500) linux/amd64
   The version from this command is 25.11.05, so the the user agent string would be Harvest-25.11.05

2. You can ask ONTAP to show you the Harvest applications that have talked with your cluster by running:
   security session request-statistics show-by-application -application Harvest*
   From that list you can pick the user-agent.

If you want to reset the max-active count, you can do that by running the following diag command. Note: this will reset the count for all applications. I don't see a way to limit this command to only a single application.
security session request-statistics reset -content max-active

[songlin-rgb]

Thanks! Since https://github.com/NetApp/harvest/tree/main/cmd/tools/rest/client.go doesn't have unit tests, how do you usually test changes there?

[cgrinds]

integration tests

[rahulguptajss]

@songlin-rgb The changes for #3999 have been committed and are available in the nightly build. You can find documentation on limiting concurrent collectors here: https://netapp.github.io/harvest/nightly/configure-harvest-basic/#pool

[songlin-rgb]

Thank you so much!

[songlin-rgb]

@cgrinds, I understand we can run this with the nightly build. Will this change also be included in the 25.11 release? Thanks!

[cgrinds]

Hi @songlin-rgb no, it did not make the 25.11 release. It is in nightly and will be in the 26.02 release. Let us know how it works for you.