Add HTCondor integration tests for file transfer plugin #2900

Copilot · 2025-12-19T20:43:17Z

The v7.22 release was delayed by a plugin bug that unit tests missed due to differences in how HTCondor invokes the plugin versus test mocks.

Changes

Development container: Added HTCondor installation to pelican-dev image for running integration tests
Integration test: Created cmd/plugin_htcondor_test.go with full plugin lifecycle test:
- Starts mini HTCondor with Pelican plugin configured via wrapper script
- Spins up federation (director, registry, origin)
- Submits job with input/output transfers using pelican:// URLs
- Verifies job completion and file content
- Uses OS-assigned ports for parallel execution
- Skips automatically if condor_master not in PATH
Documentation: Added integration test section to README with run command

Test configuration

The integration test uses HTCondor's shared port feature and dynamic port allocation:

// HTCondor config with random ports
COLLECTOR_PORT = %d
SHARED_PORT = %d
USE_SHARED_PORT = True
FILETRANSFER_PLUGINS = $(LIBEXEC)/pelican_plugin

// Wrapper script for plugin invocation
#!/bin/bash
exec /path/to/pelican plugin transfer "$@"

Run with: go test -tags=integration ./cmd -run TestHTCondorPlugin

Original prompt

This section details on the original issue you should resolve

<issue_title>Add support for HTCondor-based integration tests.</issue_title>
<issue_description>The v7.22 release was delayed (partially) due to a bug that outright broke the HTCondor file transfer plugin. This was due to a slight difference between how HTCondor provides inputs to the plugin and what we had in unit tests.

Use https://github.com/bbockelm/golang-htcondor/blob/main/httpserver/integration_test.go as a template to create a new integration test that runs if condor_master is present in the path. Keep the use of random ports and shared port (to ensure multiple integration tests can be run simultaneously as needed). Configure the integration test to use the built plugin.

Update the development Dockerfile to ensure HTCondor is installed. If you need to see how to install HTCondor, consult https://github.com/bbockelm/golang-htcondor/blob/main/Dockerfile.

To complete this issue, we will need a new integration test that starts a mini condor with the Pelican plugin as a file transfer plugin, starts a data federation, submits a job with input/output transfers with the data federation, and ensures the job completes successfully.</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes Add support for HTCondor-based integration tests. #2899

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

This test validates that the Pelican file transfer plugin works correctly with HTCondor for transferring files using the pelican:// protocol. Key features: - Uses dynamic port allocation (COLLECTOR_HOST=127.0.0.1:0) to avoid race conditions - Tests pelican_plugin naming (not stash_plugin) - Validates PelicanCfg_ job classad attribute mechanism (+PelicanCfg_TLSSkipVerify) - Sets XRD_PLUGINCONFDIR so cache can handle pelican:// protocol - Creates mini HTCondor instance with all necessary daemons - Submits a job that downloads input file and uploads output file - Verifies successful job completion and file transfer The test starts a full Pelican federation (director, registry, origin, cache) and a mini HTCondor setup, then submits a job that transfers files through the plugin, validating end-to-end functionality. Addresses: #2900

bbockelm

Another batch of changes to work on.

cmd/plugin_htcondor_test.go

bbockelm · 2025-12-21T23:56:42Z

cmd/plugin_htcondor_test.go

+  EnableWrites: true
+`)
+
+	// Wait for federation to be ready


Poll for readiness; don't have an arbitrary sleep.

(the address file is dropped by the daemon when it is ready).

bbockelm · 2025-12-22T01:59:13Z

cmd/plugin_htcondor_test.go

+error = %s/test.err
+
+# Transfer the input file from the federation
+transfer_input_files = %s/test/%s


The last bug was only triggered if you had two input files coming from the data federation. Can you add a second?

cmd/plugin_htcondor_test.go

This test validates that the Pelican file transfer plugin works correctly with HTCondor for transferring files using the pelican:// protocol. Key features: - Uses dynamic port allocation (COLLECTOR_HOST=127.0.0.1:0) to avoid race conditions - Tests pelican_plugin naming (not stash_plugin) - Validates PelicanCfg_ job classad attribute mechanism (+PelicanCfg_TLSSkipVerify) - Sets XRD_PLUGINCONFDIR so cache can handle pelican:// protocol - Creates mini HTCondor instance with all necessary daemons - Submits a job that downloads input file and uploads output file - Verifies successful job completion and file transfer The test starts a full Pelican federation (director, registry, origin, cache) and a mini HTCondor setup, then submits a job that transfers files through the plugin, validating end-to-end functionality. Addresses: #2900

- Update development Dockerfile to install HTCondor package - Create plugin_htcondor_test.go with integration test - Test configures mini HTCondor with pelican plugin - Test starts data federation and verifies file transfers

The removeBufferedHook function was directly accessing log.StandardLogger().Hooks and iterating over it while the background log level manager goroutine could be triggering logging operations that modify the same map concurrently. Changed to use ReplaceHooks() which provides internal locking, preventing the race condition that caused 'fatal error: concurrent map iteration and map write' in TestGetOIDCProvider and other tests.

Replace test_utils.GetUniqueAvailablePorts anti-pattern with port 0 binding to eliminate race conditions where ports can be taken between check and use. Changes: - Set Origin_Port and Server_WebPort to 0 in all origin tests - OS automatically assigns available ports via net.Listen - Clear cached issuer URL after listener creation to ensure correct port - Move issuer URL retrieval in TestMultiExportOrigin to after server startup This prevents 'address already in use' errors in CI.

The test was experiencing a race condition where HTTP requests were made through the broker before the Director had processed the Origin's advertisement and registered the broker endpoint. Changes: - Add HasBrokerEndpoint() method to broker.BrokerDialer to check if an endpoint is registered - Add HasBrokerForAddr() function to director package to expose the check - Update TestBrokerApi to poll until the Director has registered the origin's broker endpoint before proceeding with the test This ensures the broker connection infrastructure is fully set up before the test makes requests, eliminating the race condition that caused sporadic TLS handshake timeout failures in CI (20% failure rate).

Copilot AI assigned Copilot and bbockelm Dec 19, 2025

Copilot started work on behalf of bbockelm December 19, 2025 20:43 View session

Copilot AI changed the title ~~[WIP] Add support for HTCondor-based integration tests~~ Add HTCondor integration tests for file transfer plugin Dec 19, 2025

Copilot AI requested a review from bbockelm December 19, 2025 20:56

Copilot finished work on behalf of bbockelm December 19, 2025 20:56

bbockelm added enhancement New feature or request test Improvements to the test suite plugin Issue affecting the file transfer plugin labels Dec 21, 2025

bbockelm reviewed Dec 22, 2025

View reviewed changes

bbockelm force-pushed the copilot/add-htcondor-integration-tests branch 2 times, most recently from effd06d to 355e564 Compare December 22, 2025 19:35

Add HTCondor integration test and update Dockerfile

86fecb8

- Update development Dockerfile to install HTCondor package - Create plugin_htcondor_test.go with integration test - Test configures mini HTCondor with pelican plugin - Test starts data federation and verifies file transfers

bbockelm force-pushed the copilot/add-htcondor-integration-tests branch from 355e564 to 86fecb8 Compare December 22, 2025 20:20

bbockelm added 3 commits December 22, 2025 14:36

bbockelm marked this pull request as ready for review December 22, 2025 21:15

bbockelm merged commit 6e5a178 into main Dec 22, 2025
28 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add HTCondor integration tests for file transfer plugin #2900

Add HTCondor integration tests for file transfer plugin #2900

Uh oh!

Copilot AI commented Dec 19, 2025 •

edited

Loading

Uh oh!

bbockelm left a comment

Uh oh!

Uh oh!

bbockelm Dec 21, 2025

Uh oh!

bbockelm Dec 22, 2025

Uh oh!

bbockelm Dec 22, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add HTCondor integration tests for file transfer plugin #2900

Add HTCondor integration tests for file transfer plugin #2900

Uh oh!

Conversation

Copilot AI commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Test configuration

Comments on the Issue (you are @copilot in this section)

Uh oh!

bbockelm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bbockelm Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

bbockelm Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

bbockelm Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Dec 19, 2025 •

edited

Loading