Windows platform support bringup

[ScottTodd]

We're aiming to support as much of the toolkit as possible on Windows.

• Our build target is "native" Windows (not WSL 1 or WSL 2), compiling using standard build tools (e.g. MSVC).
• We should support local builds, have regular CI builds (GitHub Actions), and eventually be able to distribute artifacts.
• We should also have any appropriate documentation, developer guides, etc. needed.

Initial experiments

Setup notes

• I'm on Windows 11 version 23H2, OS build 22631.4751
• I enabled Windows developer mode and granted my user account the ability to create symlinks
• I created a Dev Drive by following https://learn.microsoft.com/en-us/windows/dev-drive/ (500GB, shared with other projects)
• I installed standard software development tools and configured developer options:

  • git from https://git-scm.com/downloads (latest version 2.45.1) then enabled symlinks with

    git config --global core.symlinks true
    

  • Build Tools for Visual Studio 2022 from https://visualstudio.microsoft.com/downloads/ (latest version 17.12.4)

  • CMake from https://cmake.org/download/ (latest, version 3.31.5)

  • Ninja as a CMake generator from https://ninja-build.org/ (latest, version 1.12.1)

  • ccache from https://ccache.dev/ (latest, version 4.10.2)

Fetching sources

After cloning TheRock, I tried to checkout sources using python ./build_tools/fetch_sources.py. That needed the repo tool (https://gerrit.googlesource.com/git-repo/+/HEAD/README.md, docs at https://source.android.com/docs/setup/reference/repo), which is not distributed on Windows, so I downloaded the script manually. I did then have to edit build_tools/fetch_sources.py to exec python D:/path/to/repo instead of just repo. We can teach that script how to download the file on its own and run it in a portable way.

Configure and build

After fetching sources, I tried configuring and building with CMake (under VSCode):
[proc] Executing command: "C:\Program Files\CMake\bin\cmake.EXE" -DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=TRUE -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DTHEROCK_ENABLE_RCCL=OFF -DTHEROCK_ENABLE_MATH_LIBS=OFF -DTHEROCK_ENABLE_ML_FRAMEWORKS=OFF -DTHEROCK_AMDGPU_FAMILIES=gfx110X-dgpu -UTHEROCK_AMDGPU_TARGETS -UTHEROCK_AMDGPU_DIST_BUNDLE_NAME -DLLVM_BUILD_LLVM_DYLIB=OFF --no-warn-unused-cli -SD:/projects/TheRock -Bd:/projects/TheRock/build -G Ninja

I found a few Windows issues specific to TheRock:

• Quote paths in toolchain to support spaces in paths on Windows. #34
• Skip dlopen test on Windows for now. #35

Next, I'm finding issues in the subprojects that TheRock includes. Sample logs: https://gist.github.com/ScottTodd/0a6625c4502169c5d54535bf122fc7fd. I'm not sure how deep these issues go, and if some of them are fixed on development branches. Here are a few I've found so far:

• rocm-core __attribute__ is not portable to all compilers (specifically MSVC) in https://github.com/ROCm/rocm-core/blob/master/rocm_version.h.in, do something like

  #if defined(_WIN32) || defined(__CYGWIN__)
  #define LIB_API_PUBLIC __declspec(dllexport)
  #else
  #define LIB_API_PUBLIC __attribute__((visibility("default")))
  #endif

  Need a similar fix for __attribute__((nonnull)), maybe _Ret_nonnull (https://stackoverflow.com/a/78737973)

• rocm-core link.h and dlfcn.h headers are Unix-only in https://github.com/ROCm/rocm-core/blob/master/rocm_getpath.cpp (use something like LoadLibraryA() on a .dll instead of dlopen() on a .so). The headers should also only be included if BUILD_SHARED_LIBS is defined (see next bullet)

  • Setting -DBUILD_SHARED_LIBS=OFF here helps a bit:

    TheRock/base/CMakeLists.txt

    Line 22 in 96ebf46

    "-DBUILD_SHARED_LIBS=ON"

• rocm-core in the same file uses PATH_MAX which is nonstandard. There is a FILENAME_MAX that is standard but https://stackoverflow.com/a/65174437 says it is not to be trusted for memory allocation. Could do something like min(FILENAME_MAX, 4096) (choosing some sensible value that is not INT_MAX)

• llvm warns about Generating libLLVM is not supported on MSVC in https://github.com/ROCm/llvm-project/blob/amd-staging/llvm/tools/llvm-shlib/CMakeLists.txt so I set -DLLVM_BUILD_LLVM_DYLIB=OFF in my CMake configure command. Not sure if that was load bearing for anything else yet. Could auto-detect and flip that off by default in TheRock or one of the subprojects that includes it.

• rocm_smi_lib sets CMAKE_CXX_FLAGS that assume gcc/clang: https://github.com/ROCm/rocm_smi_lib/blob/e5c5a1d5b7cba41ab801a987fb6c466503411804/CMakeLists.txt#L79-L85. It seems like https://github.com/ROCm/amdsmi is the successor to rocm_smi, but it has the same issue: https://github.com/ROCm/amdsmi/blob/9872083b491c9b136496fd493414aca5c6c144cb/CMakeLists.txt#L99-L104. Both projects also only claim to support Linux.

[ScottTodd]

Got a build to succeed (run to completion with exit code 0) after disabling many subprojects and writing some local patches.

• Logs: https://gist.github.com/ScottTodd/3ad6ea3caae56c146856bfb8225f1844
• Branch with changes: https://github.com/ScottTodd/TheRock/tree/windows-bootstrap
• Still need to figure out how to export my patches into the https://github.com/nod-ai/TheRock/tree/main/patches to share them. Mostly working around the rocm-core issues described above

[amd-chrissosa]

For 3: git diff master Branch1 > ../patchfile and check that in should be fine. apply patches functionality seems pretty straightforward: https://github.com/nod-ai/TheRock/blob/main/build_tools/fetch_sources.py

[stellaraccident]

FYI - for managing the patches directory, I've been making commits to the repo checked out git dirs. Then you dump them all with git format-patch. There's a handy (Linux) script in build_tools/save_patches.sh, but it can just be a one liner:

git format-patch -o ~/TheRock/patches/rocm-6.3.1/rocm-core rocm-6.3.1

Then if you re-run fetch_sources.py it will reset to the base tag (rocm-6.3.1) and re-apply the patches, same as a fresh checkout on the CI.

I was doing it the manual way via git diff but it was absolutely murder trying to upstream them and such. This way, you have a nice stack of commits in your git repo and can cherry-pick/rebase to a new branch to make an upstream PR, etc.

[stellaraccident]

Sorry - I could have documented my flow on this better: key is that git format-patch is the inverse of git am (which is what fetch_sources.py does to re-apply)

[ScottTodd]

Thanks, the save_patches.sh script worked fine for me (using git bash under cmder)

[ScottTodd]

Made some progress on pushing patches needed for a minimal local build. Now trying to get that wrapped into a GitHub Actions workflow, using this dev branch: https://github.com/ScottTodd/TheRock/tree/windows-ci-setup (workflow file is here: https://github.com/ScottTodd/TheRock/blob/windows-ci-setup/.github/workflows/build_windows_packages.yml). Here are some logs with a bunch of debug steps included: https://github.com/ScottTodd/TheRock/actions/runs/13082919002/job/36509861561.

Stuck for today on

Run cmake --build build
[0/2] Re-checking globbed directories...
ninja: error: 'D:/a/TheRock/TheRock/base/rocm-cmake/CMakeLists.txt', needed by 'compile_commands_fragment_rocm-cmake.json', missing and no known rule to make it
Error: Process completed with exit code 1.

I saw that locally when symlinks were not enabled in the git repository or the fetch_sources.py file was not run. I think the right files are on the runner, but I'm still suspicious of the symlink / hardlink configuration.

[stellaraccident]

Made some progress on pushing patches needed for a minimal local build. Now trying to get that wrapped into a GitHub Actions workflow, using this dev branch: https://github.com/ScottTodd/TheRock/tree/windows-ci-setup (workflow file is here: https://github.com/ScottTodd/TheRock/blob/windows-ci-setup/.github/workflows/build_windows_packages.yml). Here are some logs with a bunch of debug steps included: https://github.com/ScottTodd/TheRock/actions/runs/13082919002/job/36509861561.

Stuck for today on

Run cmake --build build
[0/2] Re-checking globbed directories...
ninja: error: 'D:/a/TheRock/TheRock/base/rocm-cmake/CMakeLists.txt', needed by 'compile_commands_fragment_rocm-cmake.json', missing and no known rule to make it
Error: Process completed with exit code 1.

I saw that locally when symlinks were not enabled in the git repository or the fetch_sources.py file was not run. I think the right files are on the runner, but I'm still suspicious of the symlink / hardlink configuration.

That is a strange error: there should not be a direct dependency between those two files (the dependency crosses a couple of other commands). But I think this is cmake/ninja wonkiness where for custom targets, it makes them depend on the transitive set of input files. So there is probably a lot wrong, and you're just getting that one error out of the pile randomly.

One thing you could quickly try: on line 418 of therock_subproject.cmake, comment out the `"${_cmake_source_dir}/CMakeLists.txt" line. This will mean it won't retrigger on sub-project changes, but that is not a problem on CI.

Still feeling like there is something else wrong and this is just the error you are getting.

I can pull it down on my windows machine and try.

[ScottTodd]

One thing you could quickly try: on line 418 of therock_subproject.cmake, comment out the `"${_cmake_source_dir}/CMakeLists.txt" line. This will mean it won't retrigger on sub-project changes, but that is not a problem on CI.

https://github.com/ScottTodd/TheRock/actions/runs/13122880778/job/36612781302#step:18:60

 [1/34] Configure sub-project therock-eigen (in background)
FAILED: third-party/eigen/stamp/configure.stamp third-party/eigen/build/CMakeCache.txt third-party/eigen/build/cmake_install.cmake compile_commands_fragment_therock-eigen.json D:/a/TheRock/TheRock/build/third-party/eigen/stamp/configure.stamp D:/a/TheRock/TheRock/build/third-party/eigen/build/CMakeCache.txt D:/a/TheRock/TheRock/build/third-party/eigen/build/cmake_install.cmake D:/a/TheRock/TheRock/build/compile_commands_fragment_therock-eigen.json 
C:\Windows\system32\cmd.exe /C "cd /D D:\a\TheRock\TheRock\build\third-party\eigen\build && "C:\Program Files\CMake\bin\cmake.exe" -GNinja -BD:/a/TheRock/TheRock/build/third-party/eigen/build -SD:/a/TheRock/TheRock/build/third-party/eigen/source -DCMAKE_INSTALL_PREFIX=D:/a/TheRock/TheRock/build/third-party/eigen/stage -DCMAKE_TOOLCHAIN_FILE=D:/a/TheRock/TheRock/build/third-party/eigen/_toolchain.cmake -DCMAKE_PROJECT_TOP_LEVEL_INCLUDES=D:/a/TheRock/TheRock/build/third-party/eigen/_init.cmake && "C:\Program Files\CMake\bin\cmake.exe" -E touch D:/a/TheRock/TheRock/build/third-party/eigen/build/compile_commands.json && "C:\Program Files\CMake\bin\cmake.exe" -E touch D:/a/TheRock/TheRock/build/third-party/eigen/stamp/configure.stamp && "C:\Program Files\CMake\bin\cmake.exe" -E copy D:/a/TheRock/TheRock/build/third-party/eigen/build/compile_commands.json D:/a/TheRock/TheRock/build/compile_commands_fragment_therock-eigen.json"
CMake Error: The source directory "D:/a/TheRock/TheRock/build/third-party/eigen/source" does not appear to contain CMakeLists.txt.

Similar error...

[stellaraccident]

It seems to be saying pretty insistently that the file is not there. I've got nothing... The file must be not really there somehow

[ScottTodd]

🤦 okay, I can repro those eigen sources missing locally now. Probably some artifact of disabling subprojects that failed to build and that resulting in an incomplete build directory. As I was testing locally I configured a few times with different settings.

The dependency that you pointed me to might still need some tweaks:

TheRock/cmake/therock_subproject.cmake

Lines 430 to 431 in b3307ed

	DEPENDS
	"${_cmake_source_dir}/CMakeLists.txt"

At least now I can debug a bit locally.

[ScottTodd]

Double 🤦 , the eigen error was caused by removing that dependency link. It is load bearing. So I have local builds working reliably, but the GitHub-hosted runners are still having trouble seeing files, perhaps due to symlinks. Might be time to try using our runner cluster instead.

[ScottTodd]

One clue with the GitHub hosted runners is that there are two drives in use, C:/ and D:/

-- Found Python3: C:/hostedtoolcache/windows/Python/3.11.9/x64/python3.exe (found suitable version "3.11.9", minimum required is "3.9") found components: Interpreter
-- Including subproject therock-boost (from D:/a/TheRock/TheRock/third-party/boost/cmake_project)
--   PROVIDE Boost = lib/cmake/Boost-1.87.0 (from therock-boost)
--   PROVIDE boost_atomic = lib/cmake/boost_atomic-1.87.0 (from therock-boost)
--   PROVIDE boost_filesystem = lib/cmake/boost_filesystem-1.87.0 (from therock-boost)
--   PROVIDE boost_headers = lib/cmake/boost_headers-1.87.0 (from therock-boost)
--   PROVIDE boost_system = lib/cmake/boost_system-1.87.0 (from therock-boost)
--   CONFIGURE_DEPENDS:  
--   JOB_POOL: therock_background
-- Including subproject therock-eigen (from D:/a/TheRock/TheRock/build/third-party/eigen/source)

This happens in other projects too. Maybe our code in https://github.com/nod-ai/TheRock/blob/main/cmake/therock_subproject.cmake is somehow not handling that on Windows? I can test that theory locally... thus far I've only been using a single D:/ drive.

[stellaraccident]

Something is not right about that log: the PROVIDE paths should be absolute, not relative. Depending on what the current drive is, that could absolutely be a problem.

(edit: scratch that. PROVIDE should be relative. INJECT should be absolute)

[ScottTodd]

A few updates:

1. I got a successful run on GitHub Actions using standard GitHub hosted runners here: https://github.com/ScottTodd/TheRock/actions/runs/13142910967/job/36674106275 . I had to replace the symlinks with direct copies of the relevant files:

         - name: Replace symlinks with copies
           run: |
             rm base/half
             rm base/rocm-cmake
             rm base/rocm-core
   
             rm -rf sources/half/.git
             rm -rf sources/rocm-cmake/.git
             rm -rf sources/rocm-core/.git
   
             cp -r sources/half base
             cp -r sources/rocm-cmake base
             cp -r sources/rocm-core base

2. I started testing using our larger self-hosted Windows runners here: https://github.com/nod-ai/TheRock/actions/runs/13124975600/job/36619337677. Hit a SSL: CERTIFICATE_VERIFY_FAILED error trying to download repo. Can make more progress there by checking the file in to third-party instead of downloading on demand, or can try mirroring to a location with a different network/auth setup.

3. I also tested a bit on my other Windows machine (Windows 10) and found that fetching the files outside of a dev drive is noticeably slower on the same network. My setup there also ran into some bumps with the patching (due to my SSH key git signing config...), so if we could move to git submodules with forks that carry patches natively instead of using repo + patch files that would simplify quite a bit.