Here are a bunch of ILDG and Bash scripts to help streamline the upload of HotQCD configurations to ILDG.

To use this software, you need

1. Python 3.9+ (For the AnalysisToolbox)
2. createQCDml
3. AnalysisToolbox
4. try-binary
5. try-client

The main scripts you will use are

1. packILDG.py and unpackILDG.py
2. convenience/get_token.bash
3. uploadILDG.bash

In the convenience folder there are other Bash wrappers for some of the try-client commands. These are just to make things easier for me. You don't need to use them.

Getting started

1. Have a look at env.bash. Those are the locations where you want to keep the auxiliary software above. Adjust as needed.
2. Have a look at setUp.bash. Adjust as needed to install whichever auxiliary software you are missing. Note that the try-binary software will need to be compiled.
3. Call ./setUp.bash. Besides installation, this is necessary to make sure scripts know where to look for auxiliary software. You may need to close and reopen your terminal

Packing configurations

To pack, call

python packILDG.py --ens <ensembleLabel> --series <stream> --dir <directory with to-be-packed configurations>

This will create the ensemble QCDml file on the current directory level, along with a lime file that contains all configurations inside the directory and a corresponding QCDml file for the lime.

Unpacking a lime file

To unpack the lime file and recover the configurations, call

python unpackILDG.py --lime <limeFile>

It will extract all configurations on the current directory level.

Uploading XML and lime files to ILDG

To upload to ILDG, you need a HotQCD bearer token. (I assume you already have the necessary permissions to get this token. If not, contact one of the HotQCD scientists.) You get this token by going into the convenience directory and calling

bash get_token.bash

following the instructions it gives you. Once getToken.bash gives you the token, export the token. Then call

bash uploadILDG.bash <limeXML>

where limeXML is the QCDml for the lime file you wish to upload.

Downloading a lime file

To reconstruct configurations using unpackILDG.py you need both the lime file and lime XML, as the XML contains some information needed to reconstruct configurations into NERSC format. Using the LFN of your lime file, you get download both with

bash downloadILDG.bash <LFN>

How was this tested?

First test (May 2026)

I checked that all 14 NERSC configurations in

l6420f21b7570m003946m01973_1_10010to10140

which is part of the f21_highTspf project, survived this round trip:

1. packILDG.py
2. uploadILDG.bash
3. downloadILDG.bash
4. unpackILDG.py
5. diff between all unpacked and original configurations

All configurations are the same according to this.

Second test (16 June 2026)

After uploading all ensembles from the f21_highTspf project, I tried downloadILDG.bash then unpackILDG.py on l6420f21b7570m003946m01973/9_2870to3710.lime. I then compiled checkConf of SIMULATeQCD and ran it on the configurations in the lime file. All configurations were readable and correct.

Are the configurations ILDG-compliant?

They should be. The code assumes input configurations are in NERSC format, which saves in the order mu, x, y, z, t. ILDG format also saves in the order mu, x, y, z, t. The lime file saves naked binaries along with a brief header. The unpackILDG.py script mentioned above separates this lime file back into NERSC format configurations, as this code is intended for use within the HotQCD collaboration, which works primarily in NERSC.

Why is this code part Bash part Python?

I tried to do everything in Python, I really did. But I couldn't get the upload to work in Python so I gave up and resorted to Bash. My Bash skills are not that great, so I needed help from Claude in a couple places. I marked where I used Claude.

More information

You can learn more about how to use some of the metadata catalogue, file catalogue, and storage element commands at https://slides.koutsou.net/ILDG-hands-on-2025-07-08/#1