Implement XRootD restart/reconfig with thread-safe process management

[Copilot]

XRootD Restart/Reconfig Implementation Plan

• Understand current XRootD lifecycle management
  • Review launch.go for daemon startup
  • Review daemon/launch_unix.go for process management
  • Review launchers/origin_serve.go and cache_serve.go
  • Review commsocket and communication mechanisms
  • Identify PIDs storage in OriginServer and CacheServer
• Design restart/reconfig mechanism
  • Add thread-safe restart mutex to prevent concurrent restarts
  • Create restart functions in xrootd package
  • Handle SIGTERM properly to distinguish from shutdown
  • Implement reconfig of xrootd runtime directory on restart
  • Update monitoring goroutines to handle process restart
• Implement core restart functionality
  • Add RestartXrootd function with mutex protection
  • Implement graceful shutdown of existing processes
  • Reconfigure runtime directory (call ConfigXrootd)
  • Relaunch daemons with updated configuration
  • Update PIDs in server instances via RestartServer helper
• Update XRootDServer interface
  • Add Restart method to interface (Removed - abandoned approach)
  • Implement in OriginServer (Removed - abandoned approach)
  • Implement in CacheServer (Removed - abandoned approach)
• Handle monitoring goroutines
  • Ensure monitoring continues after restart (existing code handles this)
  • Handle SIGTERM from restart vs actual shutdown (graceful shutdown implemented)
• Add tests
  • Create unit test for restart mechanism
  • Test concurrent restart protection
  • Verify in federation tests (e2e_fed_tests)
  • Test monitoring continuation after restart (implicit through file access tests)
• Documentation
  • Add code comments explaining restart flow
  • Document thread-safety guarantees

Summary

This PR implements XRootD restart/reconfig functionality as requested in the issue. The implementation includes:

1. Thread-Safe Restart Mechanism: Added RestartXrootd() function with mutex protection to prevent concurrent restart attempts.

2. Graceful Shutdown: Processes are first sent SIGTERM for graceful shutdown, with a configurable timeout (via Xrootd.ShutdownTimeout). Any processes that don't respond are force-killed with SIGKILL.

3. Runtime Reconfiguration: On restart, the XRootD runtime directory is reconfigured by calling ConfigXrootd(), ensuring any configuration changes are applied.

4. Process Tracking: PIDs are properly updated in both OriginServer and CacheServer via the RestartServer() helper function, avoiding circular dependency issues.

5. Monitoring Continuity: Existing monitoring goroutines continue to function after restart without modification, as they monitor server health independently of PIDs.

6. Comprehensive Testing: Includes unit tests for mutex protection and basic restart flow, plus e2e federation tests that verify file access works before and after restart.

7. Windows Compatibility: Stub implementations for Windows indicate restart is not supported on that platform.

The implementation follows the existing patterns in the codebase and makes minimal changes to achieve the goal.

Usage

To restart XRootD from code:

// Direct restart with PID tracking (from xrootd package or higher-level code)
newPids, err := xrootd.RestartXrootd(ctx, oldPids)

// Or using the helper to update server PIDs automatically
err := xrootd.RestartServer(ctx, originOrCacheServer)

The restart function is thread-safe and will return an error if another restart is already in progress.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Allow XRootD restart/reconfig</issue_title>
<issue_description>In #2895, @copilot cowardly refused to implement a restart (and implicitly, a reconfig) of XRootD.

To reduce the complexity of that PR, let's carve out the restart as a separate line of work.

Make it so the XRootD server can be shutdown and restarted (this must be thread-safe to prevent concurrent restart requests). The xrootd monitoring goroutines should understand this has occurred (you'll need to SIGTERM the processes which could cause them to think that pelican itself should shutdown. On restart, we should go through and reconfig the xrootd runtime directory as well.

Make sure to use the built-in federation tests to ensure xrootd restarting works.</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Allow XRootD restart/reconfig #2915

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[bbockelm]

LGTM.

[bbockelm]

Last piece before the really fun one - dynamic log levels!