I have built PhoenixOS from source. I am trying to checkpoint resnet example using pos_cli.
These are the steps I followed
- Built PhoenixOS using readme inside docker container
- Started PoS daemon using
pos_cli --start --target daemon
- Ran resnet inference using
env $phos python3 inference.py
- Tried pre-dumping using
pos_cli --pre-dump --dir /root/ckpt --pid 378
These are my configurations
- Docker container script used:
run_torch_cu113.sh
But I get error when trying to do pre-dump
root@ajay-OMEN-Laptop-15-en0xxx:~# pos_cli --pre-dump --dir /root/ckpt --pid 378
POS Warn [POSOobClient @ 0x55f2418e70b0] failed to bind out-of-band UDP client to "0.0.0.0:10086": Address already in use, try 1th time to switch port to 10087
POS Debug [POSOobClient @ 0x55f2418e70b0] out-of-band UDP client is binded to 0.0.0.0:10087
POS Warn no target and skip-target specified, default to pre-dump only stateful resources (e.g., device memory)
POS Log clean old assets under specified pre-dump dir: dir(/root/ckpt), nb_removed_files(2)
POS Log reuse pre-dump dir: /root/ckpt
Warn (criu/kerndat.c:1637): CRIU was built without libnftables support
Warn (criu/cr-check.c:1348): Nftables based locking requires libnftables and set concatenations support
Warn (criu/kerndat.c:1637): CRIU was built without libnftables support
Error (criu/proc_parse.c:116): handle_device_vma plugin failed: No such file or directory
Error (criu/proc_parse.c:631): Can't handle non-regular mapping on 378's map 200200000
Error (criu/cr-dump.c:1486): Collect mappings (pid: 378) failed with -1
Error (criu/cr-dump.c:1905): Pre-dumping FAILED.
``` `
I have built PhoenixOS from source. I am trying to checkpoint resnet example using pos_cli.
These are the steps I followed
pos_cli --start --target daemonenv $phos python3 inference.pypos_cli --pre-dump --dir /root/ckpt --pid 378These are my configurations
run_torch_cu113.shBut I get error when trying to do pre-dump