Systems Not Reliably Archiving to Remote Archiver

[bladepickering]

Hello,

We have several perfSONAR nodes in a mesh that are having trouble reliably archiving test results to our remote archiver. We are using IP-based authentication to send results to the archiver. While some tests are successfully archived and appear in Grafana, a significant number are missing. What steps would you recommend to help identify the cause of this issue?

I identified the missing tests by logging into one of the test nodes, running pscheduler schedule --filter-test=throughput -PT12H, and filtering for the IP addresses of the other nodes. I then used pscheduler result --full to view the test results. In the archivings section at the bottom, the output shows to http, finished - succeeded. I can find the result on that machine’s local Grafana web page, but not on our remote archiver.

I am not sure whether this is related or just a coincidence, but this setup was working before the Debian test authentication issues a couple of months ago. Since updating to iperf3 3.20, the authentication issue appears to be resolved, but we are now seeing this archiving problem.

I would be happy to provide logs from either the local machine or the remote archiver if that would be helpful. I can also be available for perfSONAR office hours or hop on a call. Please advise.

Thank you for your help,

Blade Pickering
Research Network Engineer
NCSA / University of Illinois

[bladepickering]

Hello,

I wanted to check on this ticket and see if there have been any updates.

If there is any additional information I can provide to help, please let me know. Happy to provide logs, configuration details, test results, or other troubleshooting data that would be useful from our side.

Thanks,

-Blade

[mfeit-internet2]

Just to close the loop on this, Blade and I spoke on Zoom the other day and debugged at least part of this.

@bladepickering please close this if it no longer needs to be open.

[bladepickering]

Hi Mark,

Thank you for your help last week!

I've implemented your suggestions - adding the archiving information in /usr/lib/perfsonar/web-config/ and removing the json file I created on the nodes that pointed to the remote archiver. I've also updated all the nodes to the latest version - 5.2.5-1. I no longer see the error in psconfig-pscheduler-agent.log, so we have made some progress.

The grafana dashboard is looking better, but we are still missing several test results. Is it worth us setting up another call between the two of us, or would it be worthwhile to involve some of the other developers that are more psconfig focused?

I'm happy to run commands, share logs, and so on if providing new information would be helpful.

I'll shoot you an email as well, but wanted to reply here first.

Thanks,

Blade