Skip to content

Add row-level claiming to prevent duplicate staged metadata processing in multi-node deployments#152

Closed
Copilot wants to merge 3 commits intofeatures/HPCDATAMGM-2165from
copilot/sub-pr-150-again
Closed

Add row-level claiming to prevent duplicate staged metadata processing in multi-node deployments#152
Copilot wants to merge 3 commits intofeatures/HPCDATAMGM-2165from
copilot/sub-pr-150-again

Conversation

Copy link
Contributor

Copilot AI commented Mar 3, 2026

processStagedMetadataAttributes() had no locking mechanism, allowing multiple scheduler nodes to read and process the same HPC_STAGED_METADATA_ATTRIBUTES rows concurrently — resulting in duplicate metadata writes and conflicting deletes.

Changes

  • Schema: Added IN_PROCESS NUMBER(1) DEFAULT 0 NOT NULL to HPC_STAGED_METADATA_ATTRIBUTES
  • DAO: getStagedMetadataAttributes() now filters WHERE IN_PROCESS = 0; new claimStagedMetadataAttribute() issues a conditional update:
    UPDATE HPC_STAGED_METADATA_ATTRIBUTES
       SET IN_PROCESS = 1
     WHERE PATH = ? AND META_ATTR_NAME = ? AND IN_PROCESS = 0
    Oracle row-level locking serializes concurrent claims — only one node updates 1 row; the other gets 0 rows and skips.
  • Bus service: processStagedMetadataAttributes() claims each row before processing and skips any row already claimed by another node.

Follows the same conditional-update pattern used for HpcDataMigrationTask.IN_PROCESS elsewhere in this codebase.


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 2 commits March 3, 2026 16:45
…staged metadata rows in multi-node deployments

Co-authored-by: dinhys <31490208+dinhys@users.noreply.github.com>
Co-authored-by: dinhys <31490208+dinhys@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Mar 3, 2026

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • raw.github.com
    • Triggering command: /usr/lib/jvm/temurin-17-jdk-amd64/bin/java /usr/lib/jvm/temurin-17-jdk-amd64/bin/java --enable-native-access=ALL-UNNAMED -classpath /usr/share/apache-maven-3.9.12/boot/plexus-classworlds-2.9.0.jar -Dclassworlds.conf=/usr/share/apache-maven-3.9.12/bin/m2.conf -Dmaven.home=/usr/share/apache-maven-3.9.12 -Dlibrary.jansi.path=/usr/share/apache-maven-3.9.12/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/REDACTED/work/HPC_DME_APIs/HPC_DME_APIs/src/hpc-server org.codehaus.plexus.classworlds.launcher.Launcher compile -pl hpc-dao-api,hpc-dao-impl,hpc-app-service-api,hpc-app-service-impl,hpc-bus-service-impl -am -q (dns block)
    • Triggering command: /usr/lib/jvm/temurin-17-jdk-amd64/bin/java /usr/lib/jvm/temurin-17-jdk-amd64/bin/java --enable-native-access=ALL-UNNAMED -classpath /usr/share/apache-maven-3.9.12/boot/plexus-classworlds-2.9.0.jar -Dclassworlds.conf=/usr/share/apache-maven-3.9.12/bin/m2.conf -Dmaven.home=/usr/share/apache-maven-3.9.12 -Dlibrary.jansi.path=/usr/share/apache-maven-3.9.12/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/REDACTED/work/HPC_DME_APIs/HPC_DME_APIs/src org.codehaus.plexus.classworlds.launcher.Launcher -f pom.xml -B -V -e -Dfindbugs.skip -Dcheckstyle.skip -Dpmd.skip=true -Dspotbugs.skip -Denforcer.skip -Dmaven.javadoc.skip (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI changed the title [WIP] WIP to improve metadata population task based on feedback Add row-level claiming to prevent duplicate staged metadata processing in multi-node deployments Mar 3, 2026
@dinhys dinhys closed this Mar 3, 2026
dinhys added a commit that referenced this pull request Mar 16, 2026
Fix: Add row-level claiming to prevent duplicate staged metadata processing in multi-node deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants