openmpi 5.0.5 + lsf: The affinity file provided in LSB_AFFINITY_HOSTFILE could not be converted to a rankfile

[puneet336]

Hi Team
I have compiler openmpi 5.0.5 and PRRTE 3.0.8 on RHEL 9.5 (Linux login1 5.14.0-503.38.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Apr 13 22:01:49 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux
) with following options:

--with-slurm --with-lsf=/CHBS/apps/HPC/lsf/lsf10.1/10.1 --with-lsf-libdir=/CHBS/apps/HPC/lsf/lsf10.1/10.1/linux3.10-glibc2.17-x86_64/lib

but seems mpirun is unable to pick up node information from LSF:

[singhpuv@server1 ~]$ mpirun -np 1 /bin/hostname
--------------------------------------------------------------------------
The specified LSF affinity file contained a node (server1) that is not in your
allocation. We therefore cannot map a process rank to it. Please
check your allocation and affinity file to ensure the latter only
contains allocated nodes.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
The affinity file provided in LSB_AFFINITY_HOSTFILE could not be converted
to a rankfile:

  File:  /home/singhpuv/.lsbatch/1765772632.2986.hostAffinityFile

We cannot continue.
--------------------------------------------------------------------------
[singhpuv@server1 ~]$ echo $LSB_AFFINITY_HOSTFILE
/home/singhpuv/.lsbatch/1765772632.2986.hostAffinityFile
[singhpuv@glchst-st220045 ~]$ cat /home/singhpuv/.lsbatch/1765772632.2986.hostAffinityFile
server1 0
server2 0
[singhpuv@server1 ~]$ prte
prte       prted      prte_info  prterun
[singhpuv@server1 ~]$ prte_info |grep -i lsf
                          '--with-lsf=/CHBS/apps/HPC/lsf/lsf10.1/10.1'
                          '--with-lsf-libdir=/CHBS/apps/HPC/lsf/lsf10.1/10.1/linux3.10-glibc2.17-x86_64/lib'
                 MCA ess: lsf (MCA v2.1.0, API v3.0.0, Component v3.0.6)
                 MCA plm: lsf (MCA v2.1.0, API v2.0.0, Component v3.0.6)
                 MCA ras: lsf (MCA v2.1.0, API v2.0.0, Component v3.0.8)
[singhpuv@server1 ~]$

Please advice.

[rhc54]

Well, is "server1" in your allocation? Add --prtemca ras_base_verbose 10 to your cmd line and see what we found.

[puneet336]

Thank you for the response,
i am attaching the logs with suggested flag.

[user1@login1 ~]$ bsub -n 2 -R "span[ptile=1]" -Is  /bin/bash
WARNING: memory per core not specified for job - setting to default of 3000 MB
WARNING: duration not specified for job - setting to default of 8 hours
Job <3202> is submitted to queue <short>.
<<Waiting for dispatch ...>>
<<Starting on server2.example.com>>
[user1@server2 ~]$ module load OpenMPI/5.0.7-GCC-12.3.0
Loading OpenMPI/5.0.7-GCC-12.3.0
  Loading requirement: GCCcore/12.3.0 zlib/1.2.13-GCCcore-12.3.0 binutils/2.40-GCCcore-12.3.0 GCC/12.3.0 numactl/2.0.19-GCCcore-12.3.0 XZ/5.6.3-GCCcore-12.3.0 libxml2/2.13.4-GCCcore-12.3.0
    libpciaccess/0.18.1-GCCcore-12.3.0 hwloc/2.11.2-GCCcore-12.3.0 OpenSSL/1.1 libevent/2.1.12-GCCcore-12.3.0 UCX/1.18.0-GCCcore-12.3.0 libfabric/2.0.0-GCCcore-12.3.0
    PMIx/5.0.6-GCCcore-12.3.0 PRRTE/3.0.8-GCCcore-12.3.0 UCC/1.3.0-GCCcore-12.3.0
[user1@server2 ~]$ which mpirun
/CHBS/apps/EB/software/OpenMPI/5.0.7-GCC-12.3.0/bin/mpirun
[user1@server2 ~]$ prte_info --version
[user1@server2 ~]$ prte_info
                    PRTE: 3.0.8v3.0.8rc1-2-gf6f5c181c1
      PRTE repo revision: v3.0.8rc1-2-gf6f5c181c1
       PRTE release date: @PMIX_RELEASE_DATE@
                    PMIx: OpenPMIx 5.0.6 (PMIx Standard: 4.2, Stable ABI: 0.0, Provisional ABI: 0.0)
                  Prefix: /CHBS/apps/EB/software/PRRTE/3.0.8-GCCcore-12.3.0
 Configured architecture: x86_64-pc-linux-gnu
          Configure host: login1
           Configured by: user1
           Configured on: Mon Dec 15 04:19:38 UTC 2025
          Configure host: login1
  Configure command line: '--prefix=/CHBS/apps/EB/software/PRRTE/3.0.8-GCCcore-12.3.0' '--with-libevent=/CHBS/apps/EB/software/libevent/2.1.12-GCCcore-12.3.0'
                          '--with-hwloc=/CHBS/apps/EB/software/hwloc/2.11.2-GCCcore-12.3.0' '--with-pmix=/CHBS/apps/EB/software/PMIx/5.0.6-GCCcore-12.3.0' '--with-slurm'
                          '--with-lsf=/CHBS/apps/HPC/lsf/lsf10.1/10.1' '--with-lsf-libdir=/CHBS/apps/HPC/lsf/lsf10.1/10.1/linux3.10-glibc2.17-x86_64/lib'
                Built by: user1
                Built on: Mon Dec 15 04:20:48 AM UTC 2025
              Built host: login1
              C compiler: gcc
     C compiler absolute: /CHBS/apps/EB/software/GCCcore/12.3.0/bin/gcc
  C compiler family name: GNU
      C compiler version: "12" "." "3" "." "0"
          Thread support: posix
  Internal debug support: no
              dl support: yes
   prun default --prefix: no
     Symbol vis. support: yes
          Manpages built: yes
                 MCA mca: no components
                MCA prte: no components
               MCA hwloc: no components
                MCA pmix: no components
                 MCA rml: no components
              MCA routed: no components
              MCA errmgr: dvm (MCA v2.1.0, API v3.0.0, Component v3.0.6)
              MCA errmgr: prted (MCA v2.1.0, API v3.0.0, Component v3.0.6)
                 MCA ess: env (MCA v2.1.0, API v3.0.0, Component v3.0.6)
                 MCA ess: hnp (MCA v2.1.0, API v3.0.0, Component v3.0.6)
                 MCA ess: lsf (MCA v2.1.0, API v3.0.0, Component v3.0.6)
                 MCA ess: slurm (MCA v2.1.0, API v3.0.0, Component v3.0.6)
               MCA filem: raw (MCA v2.1.0, API v2.0.0, Component v3.0.6)
             MCA grpcomm: direct (MCA v2.1.0, API v3.0.0, Component v3.0.6)
                 MCA iof: hnp (MCA v2.1.0, API v2.0.0, Component v3.0.6)
                 MCA iof: prted (MCA v2.1.0, API v2.0.0, Component v3.0.6)
                MCA odls: default (MCA v2.1.0, API v2.0.0, Component v3.0.6)
                 MCA oob: tcp (MCA v2.1.0, API v2.0.0, Component v3.0.6)
                 MCA plm: slurm (MCA v2.1.0, API v2.0.0, Component v3.0.6)
                 MCA plm: ssh (MCA v2.1.0, API v2.0.0, Component v3.0.6)
                 MCA plm: lsf (MCA v2.1.0, API v2.0.0, Component v3.0.6)
           MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v3.0.6)
              MCA prtedl: dlopen (MCA v2.1.0, API v1.0.0, Component v3.0.6)
     MCA prteinstalldirs: env (MCA v2.1.0, API v2.0.0, Component v3.0.6)
     MCA prteinstalldirs: config (MCA v2.1.0, API v2.0.0, Component v3.0.6)
       MCA prtereachable: weighted (MCA v2.1.0, API v2.0.0, Component v3.0.6)
       MCA prtereachable: netlink (MCA v2.1.0, API v2.0.0, Component v3.0.6)
                 MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component v3.0.6)
                 MCA ras: testrm (MCA v2.1.0, API v2.0.0, Component v3.0.6)
                 MCA ras: pbs (MCA v2.1.0, API v2.0.0, Component v3.0.6)
                 MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v3.0.6)
                 MCA ras: lsf (MCA v2.1.0, API v2.0.0, Component v3.0.8)
               MCA rmaps: ppr (MCA v2.1.0, API v4.0.0, Component v3.0.8)
               MCA rmaps: rank_file (MCA v2.1.0, API v4.0.0, Component v3.0.6)
               MCA rmaps: round_robin (MCA v2.1.0, API v4.0.0, Component v3.0.6)
               MCA rmaps: seq (MCA v2.1.0, API v4.0.0, Component v3.0.6)
                 MCA rtc: hwloc (MCA v2.1.0, API v1.0.0, Component v3.0.6)
              MCA schizo: ompi (MCA v2.1.0, API v1.0.0, Component v3.0.6)
              MCA schizo: prte (MCA v2.1.0, API v1.0.0, Component v3.0.6)
               MCA state: dvm (MCA v2.1.0, API v1.0.0, Component v3.0.6)
               MCA state: prted (MCA v2.1.0, API v1.0.0, Component v3.0.6)
[user1@server2 ~]$

here are the requested logs:

[user1@server2 ~]$ echo $LSB_JOBID
3202
[user1@server2 ~]$ cat /home/user1/.lsbatch/1765781124.3202.hostAffinityFile
server2.example.com 0
server1.example.com 0
[user1@server2 ~]$ mpirun --prtemca ras_base_verbose 10 /bin/hostname
[server2:2788255] mca: base: component_find: searching NULL for ras components
[server2:2788255] mca: base: find_dyn_components: checking NULL for ras components
[server2:2788255] pmix:mca: base: components_register: registering framework ras components
[server2:2788255] pmix:mca: base: components_register: found loaded component simulator
[server2:2788255] pmix:mca: base: components_register: component simulator register function successful
[server2:2788255] pmix:mca: base: components_register: found loaded component testrm
[server2:2788255] pmix:mca: base: components_register: component testrm register function successful
[server2:2788255] pmix:mca: base: components_register: found loaded component pbs
[server2:2788255] pmix:mca: base: components_register: component pbs register function successful
[server2:2788255] pmix:mca: base: components_register: found loaded component slurm
[server2:2788255] pmix:mca: base: components_register: component slurm register function successful
[server2:2788255] pmix:mca: base: components_register: found loaded component lsf
[server2:2788255] pmix:mca: base: components_register: component lsf register function successful
[server2:2788255] mca: base: components_open: opening ras components
[server2:2788255] mca: base: components_open: found loaded component simulator
[server2:2788255] mca: base: components_open: found loaded component testrm
[server2:2788255] mca: base: components_open: found loaded component pbs
[server2:2788255] mca: base: components_open: component pbs open function successful
[server2:2788255] mca: base: components_open: found loaded component slurm
[server2:2788255] mca: base: components_open: component slurm open function successful
[server2:2788255] mca: base: components_open: found loaded component lsf
[server2:2788255] mca: base: components_open: component lsf open function successful
[server2:2788255] mca:base:select: Auto-selecting ras components
[server2:2788255] mca:base:select:(  ras) Querying component [simulator]
[server2:2788255] mca:base:select:(  ras) Querying component [testrm]
[server2:2788255] mca:base:select:(  ras) Querying component [pbs]
[server2:2788255] mca:base:select:(  ras) Querying component [slurm]
[server2:2788255] mca:base:select:(  ras) Querying component [lsf]
[server2:2788255] mca:base:select:(  ras) Query of component [lsf] set priority to 75
[server2:2788255] mca:base:select:(  ras) Selected component [lsf]
[server2:2788255] mca: base: close: unloading component simulator
[server2:2788255] mca: base: close: unloading component testrm
[server2:2788255] mca: base: close: unloading component pbs
[server2:2788255] mca: base: close: component slurm closed
[server2:2788255] mca: base: close: unloading component slurm
[server2:2788255] [prterun-server2-2788255@0,0] ras:base:allocate
[server2:2788255] ras/lsf: New Node (server2) [slots=1]
[server2:2788255] ras/lsf: New Node (server1) [slots=1]
[server2:2788255] [prterun-server2-2788255@0,0] ras:base:node_insert inserting 2 nodes
[server2:2788255] [prterun-server2-2788255@0,0] ras:base:node_insert updating HNP [server2] info to 1 slots
[server2:2788255] [prterun-server2-2788255@0,0] ras:base:node_insert node server1 slots 1

======================   ALLOCATED NODES   ======================
    server2: slots=1 max_slots=0 slots_inuse=0 state=UP
        Flags: DAEMON_LAUNCHED:LOCATION_VERIFIED:SLOTS_GIVEN
        aliases: server2
    server1: slots=1 max_slots=0 slots_inuse=0 state=UP
        Flags: SLOTS_GIVEN
        aliases: NONE
=================================================================
[server2:2788255] [prterun-server2-2788255@0,0] ras:base:allocate
[server2:2788255] [prterun-server2-2788255@0,0] ras:base:allocate allocation already read
--------------------------------------------------------------------------
The specified LSF affinity file contained a node (server2.example.com) that is not in your
allocation. We therefore cannot map a process rank to it. Please
check your allocation and affinity file to ensure the latter only
contains allocated nodes.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
The affinity file provided in LSB_AFFINITY_HOSTFILE could not be converted
to a rankfile:

  File:  /home/user1/.lsbatch/1765781124.3202.hostAffinityFile

We cannot continue.
--------------------------------------------------------------------------
[user1@server2 ~]$

[puneet336]

from the same job environment, openmpiv4 works as expected
so this appears to be an issue with the OpenMPIv5

[user1@server2 ~]$ echo $LSB_JOBID
3202
[user1@server2 ~]$
[user1@server2 ~]$ module load OpenMPI/4.1.5-GCC-12.3.0
Loading OpenMPI/4.1.5-GCC-12.3.0
  Loading requirement: GCCcore/12.3.0 zlib/1.2.13-GCCcore-12.3.0 binutils/2.40-GCCcore-12.3.0 GCC/12.3.0 numactl/2.0.19-GCCcore-12.3.0 XZ/5.6.3-GCCcore-12.3.0 libxml2/2.13.4-GCCcore-12.3.0
    libpciaccess/0.18.1-GCCcore-12.3.0 hwloc/2.11.2-GCCcore-12.3.0 OpenSSL/1.1 libevent/2.1.12-GCCcore-12.3.0 UCX/1.18.0-GCCcore-12.3.0 libfabric/1.18.0-GCCcore-12.3.0
    PMIx/5.0.6-GCCcore-12.3.0 PRRTE/3.0.8-GCCcore-12.3.0 UCC/1.3.0-GCCcore-12.3.0
[user1@server2 ~]$ mpirun /bin/hostname
glchst-st220046
glchst-st220045
[user1@server2 ~]$

[puneet336]

i checked further and noticed that when we remove the domain information from the affinity file, we are able to launch jobs.

a) without removing the fqdn from affinity file:

[user1@server2 ~]$ vim /home/user1/.lsbatch/1765781124.3202.hostAffinityFile
[user1@server2 ~]$ vim /home/user1/.lsbatch/1765781124.3202.hostAffinityFile
[user1@server2 ~]$ mpirun /bin/hostname
--------------------------------------------------------------------------
The specified LSF affinity file contained a node (server2.example.com) that is not in your
allocation. We therefore cannot map a process rank to it. Please
check your allocation and affinity file to ensure the latter only
contains allocated nodes.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
The affinity file provided in LSB_AFFINITY_HOSTFILE could not be converted
to a rankfile:

  File:  /home/user1/.lsbatch/1765781124.3202.hostAffinityFile

We cannot continue.
--------------------------------------------------------------------------
[user1@server2 ~]$ cat /home/user1/.lsbatch/1765781124.3202.hostAffinityFile
server2.example.com 0
server1.example.com 0
[user1@server2 ~]$ ping server1
PING server1.example.com (10.167.209.145) 56(84) bytes of data.
64 bytes from server1.example.com (10.167.209.145): icmp_seq=1 ttl=64 time=0.043 ms

--- server1.example.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.043/0.043/0.043/0.000 ms
[user1@server2 ~]$ ping server2
PING server2.example.com (10.167.209.146) 56(84) bytes of data.
64 bytes from server2.example.com (10.167.209.146): icmp_seq=1 ttl=64 time=0.011 ms
64 bytes from server2.example.com (10.167.209.146): icmp_seq=2 ttl=64 time=0.012 ms

--- server2.example.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1034ms
rtt min/avg/max/mdev = 0.011/0.011/0.012/0.000 ms
[user1@server2 ~]$ mpirun --version
mpirun (Open MPI) 5.0.7

Report bugs to https://www.open-mpi.org/community/help/
[user1@server2 ~]$

after removing fqdn information from the affinity file:

[user1@server2 ~]$ vim /home/user1/.lsbatch/1765781124.3202.hostAffinityFile
[user1@server2 ~]$ cat /home/user1/.lsbatch/1765781124.3202.hostAffinityFile
server2 0
server1 0
[user1@server2 ~]$ prt /bin/hostname
prte       prted      prte_info  prterun    prtstat
[user1@server2 ~]$ prterun /bin/hostname
server2
server1
[user1@server2 ~]$ mpirun /bin/hostname
server2
server1
[user1@server2 ~]$

[puneet336]

so i need following settings to fix the issue:

LSF_STRIP_DOMAIN=".example.com"

And reconfigure LSF to take effect:

lsadmin reconfig
badmin mbdrestart

i tried on a different environment and ran into same issues with openmpi v4 vs v5.
there seems to be a change in the way openmpi5 interacts with LSF.

[rhc54]

Try adding --prtemca prte_keep_fqdn_hostnames 1 to your mpirun cmd line

[puneet336]

Here is the output :

[user1@login1 ~]$ bsub  -n 2 -R"span[ptile=1]" -Is /bin/bash
WARNING: memory per core not specified for job - setting to default of 3000 MB
WARNING: duration not specified for job - setting to default of 8 hours
Job <3407> is submitted to queue <short>.
<<Waiting for dispatch ...>>
<<Starting on server2.example.com>>
[user1@server2 ~]$ module load OpenMPI/5.0.7-GCC-12.3.0
Loading OpenMPI/5.0.7-GCC-12.3.0
  Loading requirement: GCCcore/12.3.0 zlib/1.2.13-GCCcore-12.3.0 binutils/2.40-GCCcore-12.3.0 GCC/12.3.0
    numactl/2.0.19-GCCcore-12.3.0 XZ/5.6.3-GCCcore-12.3.0 libxml2/2.13.4-GCCcore-12.3.0 libpciaccess/0.18.1-GCCcore-12.3.0
    hwloc/2.11.2-GCCcore-12.3.0 OpenSSL/1.1 libevent/2.1.12-GCCcore-12.3.0 UCX/1.18.0-GCCcore-12.3.0
    libfabric/2.0.0-GCCcore-12.3.0 PMIx/5.0.6-GCCcore-12.3.0 PRRTE/3.0.8-GCCcore-12.3.0 UCC/1.3.0-GCCcore-12.3.0
[user1@server2 ~]$ prte_info |grep -i lsf
                          '--with-lsf=/CHBS/apps/HPC/lsf/lsf10.1/10.1'
                          '--with-lsf-libdir=/CHBS/apps/HPC/lsf/lsf10.1/10.1/linux3.10-glibc2.17-x86_64/lib'
                 MCA ess: lsf (MCA v2.1.0, API v3.0.0, Component v3.0.6)
                 MCA plm: lsf (MCA v2.1.0, API v2.0.0, Component v3.0.6)
                 MCA ras: lsf (MCA v2.1.0, API v2.0.0, Component v3.0.8)
[user1@server2 ~]$ mpirun  /bin/hostname
--------------------------------------------------------------------------
The specified LSF affinity file contained a node (server2.example.com) that is not in your
allocation. We therefore cannot map a process rank to it. Please
check your allocation and affinity file to ensure the latter only
contains allocated nodes.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
The affinity file provided in LSB_AFFINITY_HOSTFILE could not be converted
to a rankfile:

  File:  /home/user1/.lsbatch/1765803993.3407.hostAffinityFile

We cannot continue.
--------------------------------------------------------------------------
[user1@server2 ~]$ mpirun --prtemca prte_keep_fqdn_hostnames 1  /bin/hostname
--------------------------------------------------------------------------
The rankfile that was used claimed that a host was either not
allocated or oversubscribed its slots.  Please review your rank-slot
assignments and your host allocation to ensure a proper match.  Also,
some systems may require using full hostnames, such as
"host1.example.com" (instead of just plain "host1").

  Host: server2.example.com
--------------------------------------------------------------------------
[user1@server2 ~]$
[singhpuv@server2 ~]$ cat  /home/user1/.lsbatch/1765803993.3407.hostAffinityFile
server2.example.com 0
server1.example.com 0
[user1@server2 ~]$

[puneet336]

[user1@server2 ~]$ mpirun --prtemca prte_keep_fqdn_hostnames 1   --prtemca ras_base_verbose 10 /bin/hostname
[server2:2977159] mca: base: component_find: searching NULL for ras components
[server2:2977159] mca: base: find_dyn_components: checking NULL for ras components
[server2:2977159] pmix:mca: base: components_register: registering framework ras components
[server2:2977159] pmix:mca: base: components_register: found loaded component simulator
[server2:2977159] pmix:mca: base: components_register: component simulator register function successful
[server2:2977159] pmix:mca: base: components_register: found loaded component testrm
[server2:2977159] pmix:mca: base: components_register: component testrm register function successful
[server2:2977159] pmix:mca: base: components_register: found loaded component pbs
[server2:2977159] pmix:mca: base: components_register: component pbs register function successful
[server2:2977159] pmix:mca: base: components_register: found loaded component slurm
[server2:2977159] pmix:mca: base: components_register: component slurm register function successful
[server2:2977159] pmix:mca: base: components_register: found loaded component lsf
[server2:2977159] pmix:mca: base: components_register: component lsf register function successful
[server2:2977159] mca: base: components_open: opening ras components
[server2:2977159] mca: base: components_open: found loaded component simulator
[server2:2977159] mca: base: components_open: found loaded component testrm
[server2:2977159] mca: base: components_open: found loaded component pbs
[server2:2977159] mca: base: components_open: component pbs open function successful
[server2:2977159] mca: base: components_open: found loaded component slurm
[server2:2977159] mca: base: components_open: component slurm open function successful
[server2:2977159] mca: base: components_open: found loaded component lsf
[server2:2977159] mca: base: components_open: component lsf open function successful
[server2:2977159] mca:base:select: Auto-selecting ras components
[server2:2977159] mca:base:select:(  ras) Querying component [simulator]
[server2:2977159] mca:base:select:(  ras) Querying component [testrm]
[server2:2977159] mca:base:select:(  ras) Querying component [pbs]
[server2:2977159] mca:base:select:(  ras) Querying component [slurm]
[server2:2977159] mca:base:select:(  ras) Querying component [lsf]
[server2:2977159] mca:base:select:(  ras) Query of component [lsf] set priority to 75
[server2:2977159] mca:base:select:(  ras) Selected component [lsf]
[server2:2977159] mca: base: close: unloading component simulator
[server2:2977159] mca: base: close: unloading component testrm
[server2:2977159] mca: base: close: unloading component pbs
[server2:2977159] mca: base: close: component slurm closed
[server2:2977159] mca: base: close: unloading component slurm
[server2:2977159] [prterun-server2-2977159@0,0] ras:base:allocate
[server2:2977159] ras/lsf: New Node (server2.example.com) [slots=1]
[server2:2977159] ras/lsf: New Node (server1.example.com) [slots=1]
[server2:2977159] [prterun-server2-2977159@0,0] ras:base:node_insert inserting 2 nodes
[server2:2977159] [prterun-server2-2977159@0,0] ras:base:node_insert node server2.example.com slots 1
[server2:2977159] [prterun-server2-2977159@0,0] ras:base:node_insert node server1.example.com slots 1

======================   ALLOCATED NODES   ======================
    server2.example.com: slots=1 max_slots=0 slots_inuse=0 state=UP
        Flags: SLOTS_GIVEN
        aliases: server2
    server1.example.com: slots=1 max_slots=0 slots_inuse=0 state=UP
        Flags: SLOTS_GIVEN
        aliases: server1
=================================================================
[server2:2977159] [prterun-server2-2977159@0,0] ras:base:allocate
[server2:2977159] [prterun-server2-2977159@0,0] ras:base:allocate allocation already read
--------------------------------------------------------------------------
The rankfile that was used claimed that a host was either not
allocated or oversubscribed its slots.  Please review your rank-slot
assignments and your host allocation to ensure a proper match.  Also,
some systems may require using full hostnames, such as
"host1.example.com" (instead of just plain "host1").

  Host: server2.example.com
--------------------------------------------------------------------------
[user1@server2 ~]$

[rhc54]

Okay, so now it recognized the host names. It claims that there aren't enough slots for that affinity file. Can you output the affinity file as well?

Please understand - IBM no longer participates in this project, so there is no one supporting LSF environments over here. A few of us might try to resolve individual issues as they arise, but only in the case where the problem isn't too involved. Most importantly, nobody in the project has access to IBM machines, and so there is zero testing of LSF support and no real way to support development/maintenance work - and that has been the case for several years now.

If it is important to you/your org, I'd suggest raising it to your IBM/LSF salesperson.

[puneet336]

Thank you @rhc54
i will try raising support case with LSF as well

bsub  -n 2 -R"span[ptile=1]" -Is /bin/bash

[user1@server1 ~]$ module load OpenMPI/5.0.7-GCC-12.3.0
Loading OpenMPI/5.0.7-GCC-12.3.0
  Loading requirement: GCCcore/12.3.0 zlib/1.2.13-GCCcore-12.3.0 binutils/2.40-GCCcore-12.3.0 GCC/12.3.0 numactl/2.0.19-GCCcore-12.3.0 XZ/5.6.3-GCCcore-12.3.0 libxml2/2.13.4-GCCcore-12.3.0
    libpciaccess/0.18.1-GCCcore-12.3.0 hwloc/2.11.2-GCCcore-12.3.0 OpenSSL/1.1 libevent/2.1.12-GCCcore-12.3.0 UCX/1.18.0-GCCcore-12.3.0 libfabric/2.0.0-GCCcore-12.3.0
    PMIx/5.0.6-GCCcore-12.3.0 PRRTE/3.0.8-GCCcore-12.3.0 UCC/1.3.0-GCCcore-12.3.0
[user1@server1 ~]$ module load OpenMPI/5.0.7-GCC-12.3.0
[user1@server1 ~]$ mpirun --prtemca prte_keep_fqdn_hostnames 1   --prtemca ras_base_verbose 10 /bin/hostname
[server1:27923] mca: base: component_find: searching NULL for ras components
[server1:27923] mca: base: find_dyn_components: checking NULL for ras components
[server1:27923] pmix:mca: base: components_register: registering framework ras components
[server1:27923] pmix:mca: base: components_register: found loaded component simulator
[server1:27923] pmix:mca: base: components_register: component simulator register function successful
[server1:27923] pmix:mca: base: components_register: found loaded component testrm
[server1:27923] pmix:mca: base: components_register: component testrm register function successful
[server1:27923] pmix:mca: base: components_register: found loaded component pbs
[server1:27923] pmix:mca: base: components_register: component pbs register function successful
[server1:27923] pmix:mca: base: components_register: found loaded component slurm
[server1:27923] pmix:mca: base: components_register: component slurm register function successful
[server1:27923] pmix:mca: base: components_register: found loaded component lsf
[server1:27923] pmix:mca: base: components_register: component lsf register function successful
[server1:27923] mca: base: components_open: opening ras components
[server1:27923] mca: base: components_open: found loaded component simulator
[server1:27923] mca: base: components_open: found loaded component testrm
[server1:27923] mca: base: components_open: found loaded component pbs
[server1:27923] mca: base: components_open: component pbs open function successful
[server1:27923] mca: base: components_open: found loaded component slurm
[server1:27923] mca: base: components_open: component slurm open function successful
[server1:27923] mca: base: components_open: found loaded component lsf
[server1:27923] mca: base: components_open: component lsf open function successful
[server1:27923] mca:base:select: Auto-selecting ras components
[server1:27923] mca:base:select:(  ras) Querying component [simulator]
[server1:27923] mca:base:select:(  ras) Querying component [testrm]
[server1:27923] mca:base:select:(  ras) Querying component [pbs]
[server1:27923] mca:base:select:(  ras) Querying component [slurm]
[server1:27923] mca:base:select:(  ras) Querying component [lsf]
[server1:27923] mca:base:select:(  ras) Query of component [lsf] set priority to 75
[server1:27923] mca:base:select:(  ras) Selected component [lsf]
[server1:27923] mca: base: close: unloading component simulator
[server1:27923] mca: base: close: unloading component testrm
[server1:27923] mca: base: close: unloading component pbs
[server1:27923] mca: base: close: component slurm closed
[server1:27923] mca: base: close: unloading component slurm
[server1:27923] [prterun-server1-27923@0,0] ras:base:allocate
[server1:27923] ras/lsf: New Node (server1.example.com) [slots=1]
[server1:27923] ras/lsf: New Node (server2.example.com) [slots=1]
[server1:27923] [prterun-server1-27923@0,0] ras:base:node_insert inserting 2 nodes
[server1:27923] [prterun-server1-27923@0,0] ras:base:node_insert node server1.example.com slots 1
[server1:27923] [prterun-server1-27923@0,0] ras:base:node_insert node server2.example.com slots 1

======================   ALLOCATED NODES   ======================
    server1.example.com: slots=1 max_slots=0 slots_inuse=0 state=UP
        Flags: SLOTS_GIVEN
        aliases: server1
    server2.example.com: slots=1 max_slots=0 slots_inuse=0 state=UP
        Flags: SLOTS_GIVEN
        aliases: server2
=================================================================
[server1:27923] [prterun-server1-27923@0,0] ras:base:allocate
[server1:27923] [prterun-server1-27923@0,0] ras:base:allocate allocation already read
--------------------------------------------------------------------------
The rankfile that was used claimed that a host was either not
allocated or oversubscribed its slots.  Please review your rank-slot
assignments and your host allocation to ensure a proper match.  Also,
some systems may require using full hostnames, such as
"host1.example.com" (instead of just plain "host1").

  Host: server1.example.com
--------------------------------------------------------------------------
[user1@server1 ~]$ cat $LSB_AFFINITY_HOSTFILE
server1.example.com 0
server2.example.com 0
[user1@server1 ~]$

[puneet336]

another observation i have is that as soon as i change the hostname from server1 to server1.example.com

[user@server1 ~]$ hostnamectl
 Static hostname: server1
       Icon name: computer-server
         Chassis: server 🖳
      Machine ID: -
         Boot ID: -
Operating System: Red Hat Enterprise Linux 9.5 (Plow)
     CPE OS Name: cpe:/o:redhat:enterprise_linux:9::baseos
          Kernel: Linux 5.14.0-503.38.1.el9_5.x86_64
    Architecture: x86-64
 Hardware Vendor: Lenovo
  Hardware Model: ThinkSystem SR630 V3
[user@server1 ~]$ hostnamectl set-hostname server1.example.com
[user@server2 ~]$ hostnamectl set-hostname server2.example.com

openmpi v5 is able to dispatch jobs

[user1@server1 ~]$ mpirun /bin/hostname
server1.example.com
server2.example.com

[rhc54]

I believe I fixed one part of this problem: openpmix/prrte#2322. Looking at the naming issue.

[rhc54]

Just to clarify on the node naming issue you cited. We expect a logically consistent cluster - i.e., if you are using FQDN, then all node names will conform to that behavior. In the case you are citing, your scheduler is using FQDN - but your servers are not.

The problem this creates is that there is no way for us to know if your servers really belong to that domain name. You could have multiple servers with that same base name, but different domains - how are we to know which domain "server1" belongs to? Yes, we could just assume the two names are the same - but that is a big assumption that is definitely not universally true.

I'm sure we could do a better job of dealing with the FQDN scenario. However, I'm not convinced there is a defensible way to handle a mixed-scheme system where some nodes are FQDN and others are not. I'll continue to take a look as time permits, but for now I believe our code is probably doing the best anyone can do.

[puneet336]

Sorry for the confusion.
Currently we dont have fqdn on any of the servers - We dont have mixed hostname fqdn scenario.
What I meant is that once I set the FQDN on "all the compute servers" that participate in job allocation, OpenMPI is able to dispatch jobs.
In contrast, OpenMPI v4 works correctly in all scenarios without needing this.

ssh server1 'hostnamectl set-hostname server1.example.com'
ssh server2 'hostnamectl set-hostname server2.example.com'

here's what /etc/hosts looks like on all servers

...
10.167.209.145  server1.example.com         server1
10.167.209.146  server2.example.com         server2
...

[puneet336]

after applying fix, i still see -

--------------------------------------------------------------------------
[user1@server1 validate]$ prterun /bin/hostname
--------------------------------------------------------------------------
The specified LSF affinity file contained a node (server1.example.com) that is not in your
allocation. We therefore cannot map a process rank to it. Please
check your allocation and affinity file to ensure the latter only
contains allocated nodes.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
The affinity file provided in LSB_AFFINITY_HOSTFILE could not be converted
to a rankfile:

  File:  /home/singhpuv/.lsbatch/1765852822.207.hostAffinityFile

We cannot continue.
--------------------------------------------------------------------------