Add runtime log level management API with automatic restoration #2897

Copilot · 2025-12-19T18:48:09Z

Implements a web API and CLI to temporarily change server log levels at runtime without restart or config modification. Log levels automatically revert after a specified duration.

Implementation

Core Manager (logging/level_manager.go)

Singleton LogLevelManager tracks active temporary changes with expiration times
Background goroutine checks every 30 seconds for expired changes
Multiple concurrent changes resolve to most verbose level
Callback-based design avoids import cycles with config package
Accepts context.Context and errgroup.Group for clean shutdown
Integrated with server_utils.ResetTestState() for proper test cleanup

Web API (/api/v1.0/logging/level)

POST - Create temporary change (max 24h duration)
GET - Query current status and active changes
DELETE /:changeId - Cancel change early
Requires admin authorization via existing AuthHandler/AdminAuthHandler
Full OpenAPI schema documentation in swagger/pelican-swagger.yaml

CLI Command

pelican server set-logging-level debug 300 -s https://origin.example.com:8447

Organized under new server command for server management operations
Auto-generates admin token using server's issuer key when invoked from container
Intelligent server URL discovery:
- Uses explicit --server flag if provided
- Falls back to Server.ExternalWebUrl config if flag not provided
- When config URL has port 0, automatically discovers actual server address from file
- Searches well-known locations: /run/pelican/server-web-url, /var/run/pelican/server-web-url, temp directory, config directory
Structured output showing level, duration, change ID, and revert time
Optional --token flag for custom credentials
Comprehensive unit tests for URL discovery logic

Token Scope

Added pelican.logging_modify scope under "Logging Management Scopes" in docs/scopes.yaml

Example Usage

# Enable debug logging for 5 minutes
pelican server set-logging-level debug 300 -s https://my-origin.com:8447

# When running inside the server container (port 0 auto-discovery)
pelican server set-logging-level debug 300

# Check status via API
curl -H "Authorization: Bearer $TOKEN" https://my-origin.com:8447/api/v1.0/logging/level

CLI output:

Log level successfully changed to 'debug' for 300 seconds
Change ID: 550e8400-e29b-41d4-a716-446655440000
Will revert at: 2025-12-19T19:15:00Z

API response shows current level and active changes:

{
  "currentLevel": "debug",
  "baseLevel": "info",
  "activeChanges": [{
    "changeId": "550e8400-e29b-41d4-a716-446655440000",
    "level": "debug",
    "endTime": "2025-12-19T19:15:00Z",
    "remainingSeconds": 245
  }]
}

Notes

Changes don't persist across restarts
Recomputes effective level on each change/expiration to avoid clobbering concurrent modifications
Module-specific levels (including XRootD) reserved for future enhancement
Server URL discovery logic extracted into discoverLocalServerURL() for testability and clarity

Original prompt

This section details on the original issue you should resolve

<issue_title>Temporarily change log level</issue_title>
<issue_description>Add a new web API to change the logging configuration (global log level or specific modules) at runtime.

The API should require admin authorization. It should take one (or more) log levels to change and the time to restore them to normal. Do not persist it on process restart.

When the end time hits, do not clobber any other changes subsequently made. Recompute the levels from all the known active changes. If the levels change from the currently installed settings after recomputing then make the change back. The goroutine that you add to manage the log level should recompute every few minutes even if there were no changes to avoid race conditions.

Add code to the CLI so the API can be invoked from within the server's container (i.e., assume the CLI may need to find the server's private key at the configured location and sign a credential permitting it to make the change).</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes Temporarily change log level #2896

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

bbockelm

@copilot - please start addressing the requested changes.

bbockelm · 2025-12-19T20:16:17Z

cmd/logging.go

+
+func init() {
+	// Add the server URL as a persistent flag to all subcommands of loggingCmd
+	loggingCmd.PersistentFlags().StringVarP(&loggingServerURLStr, "server", "s", "", "Web URL of the Pelican server (e.g. https://my-origin.com:8447)")


Take this information from the config file if not specified by the CLI.

Done in a567fdd. Server URL now defaults to Server.ExternalWebUrl from config if --server flag is not provided.

bbockelm · 2025-12-19T20:18:21Z

cmd/logging_set_level.go

+Valid log levels: debug, info, warn, error, fatal, panic
+
+Examples:
+  pelican logging set-level debug 300 -s https://my-origin.com:8447


Having a top-level command for just the logging seems overkill. Maybe have a server command and set-logging-level can be the sub-command?

@patrickbrophy - for the API token work you're considering, we should align the "top level" command. Manipulating API tokens and manipulating server log levels seems to have some parallel.

Done in fc53ca0. Restructured to pelican server set-logging-level command structure. Created new server command as parent for server management operations.

bbockelm · 2025-12-19T20:19:08Z

cmd/logging_set_level.go

+	}
+	req.Header.Set("Content-Type", "application/json")
+	req.Header.Set("Authorization", "Bearer "+tok)
+	req.AddCookie(&http.Cookie{Name: "login", Value: tok})


Cookie shouldn't be needed if the Authorization is set.

Fixed in a567fdd. Removed cookie - Authorization header is sufficient.

bbockelm · 2025-12-19T20:19:52Z

cmd/logging_set_level.go

+	}
+
+	// Parse response
+	var response map[string]interface{}


Have a structured response, don't just print out JSON.

Done in a567fdd. Now outputs structured response:

Log level successfully changed to 'debug' for 300 seconds Change ID: 550e8400-e29b-41d4-a716-446655440000 Will revert at: 2025-12-19T19:15:00Z

bbockelm · 2025-12-19T20:20:36Z

cmd/logging_set_level.go

+	return targetURL, nil
+}
+
+func trimSuffix(s, suffix string) string {


If there's a standard library equivalent, use that. This seems fairly basic.

Fixed in a567fdd. Now using strings.TrimSuffix from standard library.

bbockelm · 2025-12-19T20:22:19Z

docs/scopes.yaml

 acceptedBy: ["*"]
 ---
 ############################
+#      Pelican Scopes      #


Call this "Logging Management Scopes"

Done in a567fdd. Section renamed to "Logging Management Scopes".

web_ui/ui.go

bbockelm · 2025-12-19T20:30:28Z

logging/level_manager.go

+type (
+	// LogLevelChange represents a temporary change to a log level
+	LogLevelChange struct {
+		Level      log.Level


Log level changes should not be only the global level but also the sub-levels, including those for XRootD.

Acknowledged. The current implementation supports global log level changes. Support for module-specific levels including XRootD will require additional design work to:

Identify and integrate with XRootD's logging system

Extend LogLevelChange structure for module-specific configurations

Modify applyChanges() to handle per-module level settings

Update API to accept module names

This is reserved for a follow-up PR given the scope. The ModuleName field is already present in the data structures as a placeholder for this future enhancement.

Since the original draft was done, we added the ability for xrootd to restart. This is now plausible.

logging/level_manager.go

bbockelm · 2025-12-21T03:38:24Z

logging/level_manager.go

+			setLevelFunc:    setFunc,
+			getLevelFunc:    getFunc,
+		}
+		go globalManager.manageLogLevels()


This should take a context.Context and an errgroup.ErrorGroup to allow for a clean shutdown where all background goroutines are stopped.

Done in a567fdd. LogLevelManager now accepts context.Context and errgroup.Group, with nil-safe fallbacks for backward compatibility.

bbockelm

Only got partly through the file. Here's my first piece of input thoguh.

cmd/logging_set_level.go

bbockelm · 2025-12-22T00:04:10Z

cmd/logging_set_level.go

+	// Well-known locations for the address file
+	// These should match the locations where the server drops the address file
+	possiblePaths := []string{
+		"/run/pelican/server-web-url",           // systemd standard location


The relevant logic is in config/address_file.go. The name of the file is pelican.addresses.

bbockelm · 2025-12-22T00:05:26Z

cmd/logging_set_level.go

+	possiblePaths := []string{
+		"/run/pelican/server-web-url",           // systemd standard location
+		"/var/run/pelican/server-web-url",       // alternative systemd location
+		filepath.Join(os.TempDir(), "pelican", "server-web-url"), // temp directory


How does this make sense? You're looking for a common directory that the daemon is using; generating a temporary directory won't help find the file.

bbockelm · 2025-12-22T00:05:47Z

cmd/logging_set_level.go

+	}
+
+	// Also check in the config directory if available
+	configDir := viper.GetString("ConfigDir")


Don't use viper directly. It'd be the runtime directory anyway.

bbockelm · 2025-12-22T00:06:15Z

cmd/logging_set_level.go

+	// Search for the address file
+	for _, path := range possiblePaths {
+		if data, err := os.ReadFile(path); err == nil {
+			urlStr := strings.TrimSpace(string(data))


Not the right format. Refer to config/address_file.go.

cmd/logging_set_level.go

bbockelm · 2025-12-22T01:27:13Z

logging/level_manager.go

+type (
+	// LogLevelChange represents a temporary change to a log level
+	LogLevelChange struct {
+		Level      log.Level


Since the original draft was done, we added the ability for xrootd to restart. This is now plausible.

swagger/pelican-swagger.yaml

bbockelm · 2025-12-22T01:28:55Z

web_ui/logging_api.go

+	}
+
+	// Module-specific log levels are not yet supported
+	if req.ModuleName != "" {


These will be added soon, I think.

web_ui/logging_api_test.go

- Added pelican.logging_modify token scope - Implemented LogLevelManager for tracking temporary log level changes - Added web API endpoints for setting, getting, and deleting log level changes - Created CLI command 'pelican logging set-level' for invoking the API - Includes background goroutine for automatic log level restoration

The test was experiencing a race condition where HTTP requests were made through the broker before the Director had processed the Origin's advertisement and registered the broker endpoint. Changes: - Add HasBrokerEndpoint() method to broker.BrokerDialer to check if an endpoint is registered - Add HasBrokerForAddr() function to director package to expose the check - Update TestBrokerApi to poll until the Director has registered the origin's broker endpoint before proceeding with the test This ensures the broker connection infrastructure is fully set up before the test makes requests, eliminating the race condition that caused sporadic TLS handshake timeout failures in CI (20% failure rate).

bbockelm · 2025-12-22T20:19:57Z

Alright, this is all ready!

Copilot AI assigned Copilot and bbockelm Dec 19, 2025

Copilot started work on behalf of bbockelm December 19, 2025 18:48 View session

Copilot AI changed the title ~~[WIP] Add API to temporarily change log level~~ Add runtime log level management API with automatic restoration Dec 19, 2025

Copilot AI requested a review from bbockelm December 19, 2025 19:09

Copilot finished work on behalf of bbockelm December 19, 2025 19:09

bbockelm requested changes Dec 21, 2025

View reviewed changes

Copilot started work on behalf of bbockelm December 21, 2025 03:40 View session

Copilot AI requested a review from bbockelm December 21, 2025 03:54

Copilot finished work on behalf of bbockelm December 21, 2025 03:54

bbockelm requested changes Dec 21, 2025

View reviewed changes

cmd/logging_set_level.go Show resolved Hide resolved

Copilot started work on behalf of bbockelm December 21, 2025 22:27 View session

Copilot AI requested a review from bbockelm December 21, 2025 22:38

Copilot finished work on behalf of bbockelm December 21, 2025 22:38

bbockelm reviewed Dec 22, 2025

View reviewed changes

bbockelm force-pushed the copilot/add-change-log-level-api branch from 5aedafa to 94ccfd5 Compare December 22, 2025 06:01

bbockelm added enhancement New feature or request cache Issue relating to the cache component origin Issue relating to the origin component director Issue relating to the director component registry Issue relating to the registry component labels Dec 22, 2025

bbockelm force-pushed the copilot/add-change-log-level-api branch from c671897 to 56f6af5 Compare December 22, 2025 19:17

Copilot AI and others added 5 commits December 22, 2025 13:42

Remove hand-created token; force CLI to create one

d23c42f

Make sure runtime dir is not the global one for tests

f3f18b6

Tuneup after invoking the set-logging-level from my terminal

f7cc62f

bbockelm force-pushed the copilot/add-change-log-level-api branch from 56f6af5 to 16ec843 Compare December 22, 2025 19:43

bbockelm marked this pull request as ready for review December 22, 2025 20:19

bbockelm merged commit da2e9f1 into main Dec 22, 2025
30 of 32 checks passed

brianaydemir mentioned this pull request Dec 31, 2025

Redact JWT signature blocks in logs #2836

Open

Add runtime log level management API with automatic restoration #2897

Add runtime log level management API with automatic restoration #2897

Uh oh!

Conversation

Copilot AI commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Implementation

Example Usage

Notes

Comments on the Issue (you are @copilot in this section)

Uh oh!

bbockelm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

bbockelm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bbockelm commented Dec 22, 2025

Uh oh!

Copilot AI commented Dec 19, 2025 •

edited

Loading