handle down remotes

[ofaaland]

Powering off a lustre server, or stopping halo_remote there, results in a crash in halo_management.

Error output:

halo_manager[2584124]: stack backtrace:
halo_manager[2584124]: TODO: handle error here.: Os { code: 111, kind: ConnectionRefused, message: "Connection refused" }
halo_manager[2584124]: thread 'main' panicked at src/host/non_ha.rs:58:18:

Reproducer:

1. Generate a cluster configuration
2. Start halo_remote on the lustre servers (in my case via systemd)
3. Start halo_manager on the management node (in my case via systemd)
4. Issue halo status and get expected output describing the state of the cluster
5. Stop halo_remote on one of the nodes
6. Wait a few seconds
7. Issue systemctl status halo_manager and see that halo_manager has crashed

Relevant code is one of the to-do items:

52 impl Host {
 53     pub async fn observe(&self, cluster: &Cluster) {
 54         loop {
 55             let client = self
 56                 .get_client(cluster)
 57                 .await
 58                 .expect("TODO: handle error here.");
 59 

Observed with stack::

* 21394c9 docs: add information on how to get ZFS and Lustre OCF resource agent scripts
* fccb1db docs: add troubleshooting section to admin guide
* d33c948 readme: add HALO_LOG environment variable

[ofaaland]

My halo.conf file is:

hosts:
- hostname: iron1
  resources:
    iron1:
      kind: llnl/zpool
      parameters:
        pool: iron1
      requires: null
    iron1/mdt1:
      kind: llnl/lustre
      parameters:
        target: iron1/mdt1
        type: mdt
        mountpoint: /mnt/lustre/lobster-MDT0000
      requires: iron1
    iron1/mgs:
      kind: llnl/lustre
      parameters:
        mountpoint: /mnt/lustre/MGS
        type: mgs
        target: iron1/mgs
      requires: iron1
  fence_agent: null
  fence_parameters: null
- hostname: iron2
  resources:
    iron2/mdt1:
      kind: llnl/lustre
      parameters:
        type: mdt
        mountpoint: /mnt/lustre/lobster-MDT0001
        target: iron2/mdt1
      requires: iron2
    iron2:
      kind: llnl/zpool
      parameters:
        pool: iron2
      requires: null
  fence_agent: null
  fence_parameters: null
- hostname: iron3
  resources:
    iron3:
      kind: llnl/zpool
      parameters:
        pool: iron3
      requires: null
    iron3/ost1:
      kind: llnl/lustre
      parameters:
        target: iron3/ost1
        type: ost
        mountpoint: /mnt/lustre/lobster-OST0000
      requires: iron3
  fence_agent: null
  fence_parameters: null
- hostname: iron4
  resources:
    iron4/ost1:
      kind: llnl/lustre
      parameters:
        type: ost
        target: iron4/ost1
        mountpoint: /mnt/lustre/lobster-OST0001
      requires: iron4
    iron4:
      kind: llnl/zpool
      parameters:
        pool: iron4
      requires: null
  fence_agent: null
  fence_parameters: null
- hostname: iron5
  resources:
    iron5/ost1:
      kind: llnl/lustre
      parameters:
        type: ost
        mountpoint: /mnt/lustre/lobster-OST0002
        target: iron5/ost1
      requires: iron5
    iron5:
      kind: llnl/zpool
      parameters:
        pool: iron5
      requires: null
  fence_agent: null
  fence_parameters: null
- hostname: iron6
  resources:
    iron6/ost1:
      kind: llnl/lustre
      parameters:
        target: iron6/ost1
        type: ost
        mountpoint: /mnt/lustre/lobster-OST0003
      requires: iron6
    iron6:
      kind: llnl/zpool
      parameters:
        pool: iron6
      requires: null
  fence_agent: null
  fence_parameters: null
failover_pairs: null

[bertschinger]

This should be resolved by #80

Thanks for reporting the bug! Let me know if you get a chance to try the fix and if it works, or not.

[ofaaland]

This should be resolved by #80

Thanks for reporting the bug! Let me know if you get a chance to try the fix and if it works, or not.

Looks good so far. Started up halo_manager, with two nodes powered off, and it started successfully, stayed up, and "halo status" correctly showed the resources on those two nodes was unknown.

[ofaaland]

Whoops, I think I spoke too soon. I'll get more information later this week.

[ofaaland]

@bertschinger I have your latest code on iron. Here is the sequence of events:

1. halo-remote is running on all the lustre server nodes (iron[1-6])
2. halo-manager is running
3. halo status reports everything is running
4. I stop halo-remote on one node (iron2 in this case)
5. halo-manager crashes, with the following output:

Jun 02 18:22:11 ironi halo_manager[3189840]: [2026-06-03T01:22:11Z WARN  halo_lib::resource] Updating status of resource iron2 from RunningOnHome to Unknow
Jun 02 18:22:11 ironi halo_manager[3189840]: [2026-06-03T01:22:11Z WARN  halo_lib::resource] Updating status of resource iron2/mdt1 from RunningOnHome to U
Jun 02 18:22:11 ironi halo_manager[3189840]: thread 'main' panicked at src/host/ha/observe.rs:149:49:
Jun 02 18:22:11 ironi halo_manager[3189840]: not yet implemented
Jun 02 18:22:11 ironi halo_manager[3189840]: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Jun 02 18:22:11 ironi halo_manager[3189840]: thread 'main' panicked at src/host/mod.rs:495:9:
Jun 02 18:22:11 ironi halo_manager[3189840]: Resource token ResourceToken { id: "iron2", location: Home } was illegally dropped!

I can do this again with RUST_BACKTRACE=1 if that would be helpful.

[bertschinger]

Thanks, that's enough to identify the problem, I don't need a backtrace.

Basically I refactored how the observe-only mode worked a while ago and didn't finish the new implementation - you're hitting the cases I left as todos. #81 should cover the rest of these cases. I'll try to finish that PR today and once I've done that, you can try again and things should work better.