HPCDATAMGM-2165: Create a task to populate system or user metadata#150
Open
dinhys wants to merge 13 commits intoreleases/3.24.0from
Open
HPCDATAMGM-2165: Create a task to populate system or user metadata#150dinhys wants to merge 13 commits intoreleases/3.24.0from
dinhys wants to merge 13 commits intoreleases/3.24.0from
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a new scheduled migration task for applying “staged” metadata (stored in hpc_staged_metadata_attributes) onto iRODS collections/data objects, auditing the before/after metadata, and then moving processed rows to hpc_migrated_metadata_attributes. It also extends metadata validation to optionally allow updating reserved system-generated metadata attributes.
Changes:
- Add a new migration scheduler cron + task entrypoint to process staged metadata attributes.
- Add DAO/service plumbing and Oracle tables to read staged metadata rows and move them to a “migrated” table after processing.
- Extend
HpcMetadataValidator/HpcMetadataServiceAPIs with anallowSystemMetadataflag.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
src/hpc-server/hpc-scheduler-migration/.../hpc-scheduler-migration.properties |
Adds cron expression for the new staged-metadata scheduled task. |
src/hpc-server/hpc-scheduler-migration/.../HpcScheduledTasksImpl.java |
Adds a new @Scheduled method to invoke staged-metadata processing. |
src/hpc-server/hpc-domain-model/.../HpcDataMigration.xsd |
Adds HpcStagedMetadataAttribute complex type for domain model generation. |
src/hpc-server/hpc-dao-impl/.../hpc_staged_metadata_attributes.sql |
Creates hpc_staged_metadata_attributes and hpc_migrated_metadata_attributes tables. |
src/hpc-server/hpc-dao-impl/.../HpcDataMigrationDAOImpl.java |
Adds queries/mappers to fetch staged metadata and move/delete rows after processing. |
src/hpc-server/hpc-dao-api/.../HpcDataMigrationDAO.java |
Adds DAO API for staged metadata fetch/cleanup. |
src/hpc-server/hpc-bus-service-impl/.../HpcDataMigrationBusServiceImpl.java |
Adds business logic to apply staged metadata to collection/data objects and cleanup. |
src/hpc-server/hpc-bus-service-impl/.../HpcDataManagementBusServiceImpl.java |
Updates calls to metadata update APIs to pass the new boolean flag. |
src/hpc-server/hpc-bus-service-api/.../HpcDataMigrationBusService.java |
Adds new bus-service API method for staged metadata processing. |
src/hpc-server/hpc-app-service-impl/.../HpcMetadataValidator.java |
Adds allowSystemMetadata parameter and adjusts validation behavior. |
src/hpc-server/hpc-app-service-impl/.../HpcMetadataServiceImpl.java |
Threads allowSystemMetadata through collection/data-object metadata updates. |
src/hpc-server/hpc-app-service-impl/.../HpcDataMigrationServiceImpl.java |
Adds service methods to fetch/apply/cleanup staged metadata entries + audit. |
src/hpc-server/hpc-app-service-api/.../HpcMetadataService.java |
Updates metadata service API signatures with allowSystemMetadata. |
src/hpc-server/hpc-app-service-api/.../HpcDataMigrationService.java |
Adds new migration service API methods for staged metadata processing. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
...app-service-impl/src/main/java/gov/nih/nci/hpc/service/impl/HpcDataMigrationServiceImpl.java
Show resolved
Hide resolved
...er/hpc-app-service-impl/src/main/java/gov/nih/nci/hpc/service/impl/HpcMetadataValidator.java
Show resolved
Hide resolved
...-bus-service-impl/src/main/java/gov/nih/nci/hpc/bus/impl/HpcDataMigrationBusServiceImpl.java
Show resolved
Hide resolved
...-bus-service-impl/src/main/java/gov/nih/nci/hpc/bus/impl/HpcDataMigrationBusServiceImpl.java
Show resolved
Hide resolved
...-bus-service-impl/src/main/java/gov/nih/nci/hpc/bus/impl/HpcDataMigrationBusServiceImpl.java
Outdated
Show resolved
Hide resolved
...rver/hpc-dao-impl/src/main/java/gov/nih/nci/hpc/dao/oracle/impl/HpcDataMigrationDAOImpl.java
Show resolved
Hide resolved
...rver/hpc-dao-impl/src/main/java/gov/nih/nci/hpc/dao/oracle/impl/HpcDataMigrationDAOImpl.java
Show resolved
Hide resolved
...app-service-impl/src/main/java/gov/nih/nci/hpc/service/impl/HpcDataMigrationServiceImpl.java
Outdated
Show resolved
Hide resolved
…hpc/service/impl/HpcDataMigrationServiceImpl.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…hpc/bus/impl/HpcDataMigrationBusServiceImpl.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…hpc/service/impl/HpcDataMigrationServiceImpl.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Contributor
Co-authored-by: dinhys <31490208+dinhys@users.noreply.github.com>
Wrap cleanupStagedMetadataAttribute in a transaction
Contributor
Contributor
…allowSystemMetadata is true Co-authored-by: dinhys <31490208+dinhys@users.noreply.github.com>
Fix: enforce restrictMetadata for non-system attributes even when allowSystemMetadata is true
Fix: Add row-level claiming to prevent duplicate staged metadata processing in multi-node deployments
dinhys
commented
Mar 11, 2026
...rver/hpc-dao-impl/src/main/java/gov/nih/nci/hpc/dao/oracle/impl/HpcDataMigrationDAOImpl.java
Show resolved
Hide resolved
metadata Code review comments.
metadata Added code review item to reset in process flag upon server restart.
menons
approved these changes
Mar 13, 2026
dinhys
commented
Mar 13, 2026
| } | ||
|
|
||
| } else { | ||
| // Get the metadata for this collection. |
Collaborator
Author
There was a problem hiding this comment.
Update the comment to data object.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added a new migration scheduled task to add staged metadata attribute/values to a collection or data object path specified.
The metadata to be populated will be staged in hpc_staged_metadata_attributes table, and moves to hpc_migrated_metadata_attributes table once the scheduled task completes processing it. The metadata before and after the update is recorded into hpc_data_management_audit table.
Additional change made to pass a boolean flag, allowSystemMetadata for updating the metadata into the metadata validator to allow updates of system generated metadata.