HORT: Monocular Hand-held Objects Reconstruction with Transformers

Zerui Chen¹   Rolandos Alexandros Potamias²   Shizhe Chen¹   Cordelia Schmid¹

¹WILLOW, INRIA Paris, France
²Imperial College London, UK

This is the training code of HORT, an state-of-the-art hand-held object reconstruction algorithm.

Installation 👷

git clone https://github.com/zerchen/hort_train.git
cd hort_train

It is suggested to use an anaconda encironment to install the the required dependencies:

conda create --name hort python=3.12
conda activate hort

conda install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 pytorch-cuda=12.1 -c pytorch -c nvidia
# Install requirements
pip install -r requirements.txt
conda install pytorch3d-0.7.8-py312_cu121_pyt241.tar.bz2 # https://anaconda.org/pytorch3d/pytorch3d/files?page=2
cd common/networks/tgs/models/snowflake/pointnet2_ops_lib && python setup.py install

It is also required to download MANO model from MANO website. Create an account by clicking Sign Up and download the models (mano_v*_*.zip). Unzip and place the right hand model MANO_RIGHT.pkl under the mano_data/mano/ folder. Note that MANO model falls under the MANO license.

Data preparation 📈

Please organize the data into structures like this (take the ObMan dataset as an example):

   ${ROOT}/datasets/obman
   └── splits
       obman.py
       data
        ├── val
        ├── train
        └── test
            ├── rgb
            ├── mesh_hand
            ├── mesh_obj

Then, execute corresponding preprocessing code to generate training files:

python preprocess/cocoify_obman.py

Training 💻

Then, launch the training script as follows.

cd tools
# training, and testing will be launched automatically when training finishes
bash dist_train.sh 4 1234 -e ../playground/object_pc_dino/experiments/obman_141k.yaml --gpu 0-3
# evaluation
python eval.py -e ${OUTPUT_DIR}

Acknowledgements

Parts of the code are based on WiLoR, SnowflakeNet and gSDF.

License 📚

HORT is licensed under MIT License. This repository also depends on WiLoR, Ultralytics library and MANO Model, which are fall under their own licenses.

Citation 📝

If you find HORT useful for your research, please consider citing our paper:

@InProceedings{chen2025hort,
  title={{HORT}: Monocular Hand-held Objects Reconstruction with Transformers},
  author={Chen, Zerui and Potamias, Rolandos Alexandros and Chen, Shizhe and Schmid, Cordelia},
  booktitle={ICCV},
  year={2025}
}