FEX x86_64 user-mode fails to enumerate .ned files correctly (SPEC CPU2017 620.omnetpp_s)

[Anon-233]

Summary

When running the SPEC CPU2017 benchmark 620.omnetpp_s (OMNeT++ 4.0) on an ARM64 host using FEX (x86_64 user-mode), OMNeT++ fails to load most .ned files from a directory, and then reports that the main network largeNet is missing. The same x86_64 binary and configuration run correctly under qemu-x86_64 on the same ARM64 machine.

This looks like a bug in how FEX implements directory enumeration (readdir/getdents-style syscalls) for x86_64 user-mode.

Environment

• Host architecture: ARM64
• Host OS: Ubuntu 24.04 (noble) arm64 (in a container)
• FEX: [main branch, commit hash: bf9ab7f], Rootfs download by FEXRootFSFetcher
• SPEC CPU version: SPEC CPU2017
• Benchmark: 620.omnetpp_s (omnetpp_s_base.x64_static-m64)
• OMNeT++ version: 4.0 (as shipped by SPEC2017)
• QEMU: qemu-user package (qemu-x86_64)

SPEC run directory (inside container / host FS):

• /host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000

NED directory:

• /host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned

ned/ contains 13 .ned files (no Cli.ned originally):

• EtherAppCli.ned
• EtherAppSrv.ned
• EtherBus.ned
• EtherEncap.ned
• EtherHost.ned
• EtherHub.ned
• EtherLLC.ned
• EtherMAC.ned
• EtherSwitch.ned
• LargeNet.ned
• MACRelayUnit.ned
• MACRelayUnitNP.ned
• MACRelayUnitPP.ned

Both on the host and inside FEX /bin/bash, ls shows the same 13 files.

Reproduction

From the SPEC run directory:

cd /host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000

1. Run with qemu-x86_64 (works)

qemu-x86_64 ../../exe/omnetpp_s_base.x64_static-m64 -f ./omnetpp.ini -c General -r 0

Output (abridged):

• Loading NED files from a SPEC directory 14
• Setting up network 'largeNet'...
• Simulation runs normally

2. Run with FEX (fails)

FEX ../../exe/omnetpp_s_base.x64_static-m64 -f ./omnetpp.ini -c General -r 0

Without any manual changes, OMNeT++ prints:

<!> Error: Error loading NED sources from `/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned': cannot stat file 'Cli.ned': No such file or directory.

Note: there is no Cli.ned in the ned/ directory; SPEC’s reference x86_64 run on a native x86_64 machine also works fine without such a file.

If I manually create an empty Cli.ned:

cd ned
touch Cli.ned
cd ..

and re-run under FEX:

FEX ../../exe/omnetpp_s_base.x64_static-m64 -f ./omnetpp.ini -c General -r 0

I get a bit further:

OMNeT++ Discrete Event Simulation  (C) 1992-2008 Andras Varga, OpenSim Ltd.
Version: 4.0, build: 090310-10709, edition: Academic Public License -- NOT FOR COMMERCIAL USE
Setting up Cmdenv...
Loading NED files from a SPEC directory 2

Preparing for running configuration General, run #0...
Scenario: $repetition=0
Assigned runID=speccpu-runid

<!> Error: Network `largeNet' not found, check .ini and .ned files.

End.

Key difference:

• Under qemu-x86_64, OMNeT++ reports: Loading NED files from a SPEC directory 14
• Under FEX, OMNeT++ reports: Loading NED files from a SPEC directory 2, and then fails with Network 'largeNet' not found

The only difference between the runs is the x86_64 emulator (qemu vs FEX); same working directory, same omnetpp.ini, same binary.

File system visibility check

On host / container:

cd /host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000
ls -l ned/ | wc -l
ls -l ned/

Output shows 13 .ned files as listed above.

Inside FEX /bin/bash:

FEX /bin/bash
cd /host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000
ls -l ned/ | wc -l
ls -l ned/

Output is identical: same 13 .ned files with same sizes and permissions. So the host FS is mapped correctly into FEX (at least for shell utilities).

strace comparison (qemu-x86_64 vs FEX)

I captured strace logs for both qemu-x86_64 and FEX runs.

qemu-x86_64

Command:

strace -f -o qemu.log \
  qemu-x86_64 ../../exe/omnetpp_s_base.x64_static-m64 -f ./omnetpp.ini -c General -r 0

From qemu.log, OMNeT++ clearly enumerates and opens many .ned files in ned/:

newfstatat(AT_FDCWD, "EtherAppCli.ned", {st_mode=S_IFREG|0775, st_size=1678, ...}, 0) = 0
readlinkat(AT_FDCWD, "/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/EtherAppCli.ned", 0xffffed9884f0, 1023) = -1 EINVAL (Invalid argument)
faccessat(AT_FDCWD, "/usr/gnemul/qemu-x86_64/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/EtherAppCli.ned", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/EtherAppCli.ned", O_RDONLY) = 4

newfstatat(AT_FDCWD, "EtherAppSrv.ned", {st_mode=S_IFREG|0775, st_size=1137, ...}, 0) = 0
openat(AT_FDCWD, "/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/EtherAppSrv.ned", O_RDONLY) = 4

newfstatat(AT_FDCWD, "EtherBus.ned", {st_mode=S_IFREG|0775, st_size=1843, ...}, 0) = 0
...

(newfstatat/openat for many other .ned files, including LargeNet.ned, MACRelayUnit*.ned, etc.)

So under qemu-x86_64 the x86_64 process successfully enumerates and opens all .ned files in the directory.

FEX

Command:

strace -f -o fex.log \
  FEX ../../exe/omnetpp_s_base.x64_static-m64 -f ./omnetpp.ini -c General -r 0

From fex.log (without my manually created empty Cli.ned):

openat2(5, "host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/package.ned", {flags=O_RDONLY, resolve=RESOLVE_IN_ROOT}, 24) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/package.ned", O_RDONLY) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "Cli.ned", 0xfffffc18a080, 0) = -1 ENOENT (No such file or directory)

Notably:

• There are no newfstatat/openat calls for the existing .ned files such as EtherAppCli.ned, LargeNet.ned, etc.
• Only package.ned and Cli.ned are ever attempted.
• This is consistent with OMNeT++ reporting Loading NED files from a SPEC directory 2 under FEX vs 14 under qemu.

So although a normal shell inside FEX can ls all files in ned/, the x86_64 process under FEX apparently does not see / enumerate those entries when scanning for NED files.

This strongly suggests an issue in FEX’s implementation of directory enumeration syscalls (getdents, getdents64, or related) or path resolution for the emulated x86_64 process.

Workarounds

I found two workarounds:

1. Manually creating an empty Cli.ned
   This bypasses the initial cannot stat file 'Cli.ned' error but still fails later with Network 'largeNet' not found, because most .ned files are not being loaded.

2. Explicitly specifying all .ned files in omnetpp.ini
   If I list all .ned files explicitly via ned-files = ned/EtherAppCli.ned ned/EtherAppSrv.ned ... in omnetpp.ini, OMNeT++ can open them directly (without relying on directory enumeration), and the simulation can run further under FEX.

In contrast, qemu-x86_64 works out of the box, without any changes to the SPEC-supplied configuration, which matches behavior on a native x86_64 host.

Expected behavior

• The x86_64 process running under FEX should see the same directory entries as under qemu-x86_64 / native x86_64.
• OMNeT++ should be able to enumerate and open all .ned files under ned/, resulting in Loading NED files from a SPEC directory 14, and the network largeNet should be found and the simulation should run normally.

Actual behavior

• Under FEX, OMNeT++ only sees 2 NED sources (vs 14 under qemu).

• It attempts to access package.ned and Cli.ned but never attempts to stat/open real existing NED files like EtherAppCli.ned, LargeNet.ned, etc.

• As a result, it fails with:

  <!> Error: Network `largeNet' not found, check .ini and .ned files.
  

Request

Could you please:

1. Confirm whether this is a known issue with x86_64 user-mode directory enumeration in FEX?
2. Suggest a proper fix or workaround (other than manually listing all .ned files)?
3. If helpful, I can provide the full qemu.log and fex.log files or further reduced repros.

Thanks!

[Sonicadvance1]

Is SPECCPU2017 able to get legally obtained without throwing a bunch of money at some company? Reproducing the problem otherwise is kind of a pain.