Flaky "failed to remove temporary directory" on Windows

[anforowicz]

Summary: on Windows rustc can intermittently/flakily fail with: failed to remove temporary directory: The process cannot access the file because it is being used by another process. (os error 32) at path "D:\\b\\s\\w\\ir\\cache\\builder\\src\\out\\f6b4-win32-official\\obj/base/test\\.tmpcZBp3F.temp-archive"

No repro steps, because this error seems to happen flakily :-(. Let me record command-line arguments of rustc that seem most relevant:

• --crate-type rlib
• `--emit=dep-info=obj/base/test/libbase_stest_ctest_urust_ulogger_uconsumer.rlib.d,link
• -o obj/base/test/libbase_stest_ctest_urust_ulogger_uconsumer.rlib
• --target=i686-pc-windows-msvc
• -Ccodegen-units=1

Actual behavior (from https://ci.chromium.org/ui/p/chromium/builders/ci/win32-official/10756/overview):

[19104/109285] 5m08.81s F RUST obj/base/test/libbase_stest_ctest_urust_ulogger_uconsumer.rlib	
FAILED: ff262c98-d6be-4fec-80c3-4043a0ff24d0 "./obj/base/test/libbase_stest_ctest_urust_ulogger_uconsumer.rlib" RUST obj/base/test/libbase_stest_ctest_urust_ulogger_uconsumer.rlib
err: exit=1
"D:/b/s/w/ir/cipd_bin_packages/cpython3/bin/python3.exe" "../../build/rust/gni_impl/rustc_wrapper.py" --target-windows --rustc=../../third_party/rust-toolchain/bin/rustc --depfile=obj/base/test/libbase_stest_ctest_urust_ulogger_uconsumer.rlib.d --rsp=obj/base/test/libbase_stest_ctest_urust_ulogger_uconsumer.rlib.rsp --  -Clinker=..\..\third_party\llvm-build\Release+Asserts\bin\lld-link.exe --crate-name base_stest_ctest_urust_ulogger_uconsumer ../../base/test/logging/test_rust_logger_consumer.rs --crate-type rlib --edition=2024 -Dunsafe_op_in_unsafe_fn -Dwarnings -Zsplit-lto-unit -Clinker-plugin-lto=yes -Cdefault-linker-libraries -Zdep-info-omit-d-target -Zmacro-backtrace -Zremap-cwd-prefix=. -Zexternal-clangrt --target=i686-pc-windows-msvc -Ccodegen-units=1 -Zunstable-options -Cpanic=immediate-abort -Zpanic_abort_tests --cfg "cr_rustc_revision=\"1920ab925646fae038b02bd462cd328ae9eef1639236-ab925646fae038b02bd462cd328ae9eef1639236-1-22-12326-g8a5f1533\"" -Copt-level=3 -Cllvm-args=-pick-merged-source-locations -g -Zallow-features= -Clinker-plugin-lto=yes -Ccontrol-flow-guard -Ctarget-feature=+crt-static --sysroot=local_rustc_sysroot --emit=dep-info=obj/base/test/libbase_stest_ctest_urust_ulogger_uconsumer.rlib.d,link -o obj/base/test/libbase_stest_ctest_urust_ulogger_uconsumer.rlib -Ldependency=obj/third_party/rust/log/v0_4/lib -Ldependency=local_rustc_sysroot/lib/rustlib/i686-pc-windows-msvc/lib -Ldependency=obj/build/rust/allocator -Ldependency=obj/third_party/rust/cxx/v1/lib -Ldependency=win_clang_x86_for_rust_host_build_tools -Ldependency=obj/third_party/rust/foldhash/v0_2/lib -Ldependency=obj/build/rust/chromium_prelude -Clink-arg=-Bdynamic -Clink-arg=obj/base/test/test_rust_logger_consumer_cxx_generated.lib -Clink-arg=obj/build/rust/cxx_cppdeps.lib -Clink-arg=obj/build/rust/allocator/alloc_error_handler_impl.lib -Clink-arg=obj/build/rust/allocator/allocator_impls.lib -Clink-arg=obj/base/allocator/partition_allocator/src/partition_alloc/allocator_base.lib -Clink-arg=obj/base/allocator/partition_allocator/src/partition_alloc/allocator_core.lib -Clink-arg=obj/base/allocator/partition_allocator/src/partition_alloc/allocator_shim.lib -Clink-arg=../../third_party/llvm-build/Release+Asserts/lib/clang/22/lib/windows/clang_rt.builtins-i386.lib -lwinmm.lib --extern log=obj/third_party/rust/log/v0_4/lib/liblog_lib.rlib --extern build_srust_sallocator_callocator=obj/build/rust/allocator/libbuild_srust_sallocator_callocator.rlib --extern cxx=obj/third_party/rust/cxx/v1/lib/libcxx_lib.rlib --extern chromium=obj/build/rust/chromium_prelude/libchromium.rlib LDFLAGS RUSTENV OUT_DIR=gen/base/test/test_rust_logger_consumer
build step: rust_rlib "./obj/base/test/libbase_stest_ctest_urust_ulogger_uconsumer.rlib"
siso_rule: rust_rlib
stderr:
error: failed to build archive at `obj/base/test/libbase_stest_ctest_urust_ulogger_uconsumer.rlib`: failed to remove temporary directory: The process cannot access the file because it is being used by another process. (os error 32) at path "D:\\b\\s\\w\\ir\\cache\\builder\\src\\out\\f6b4-win32-official\\obj/base/test\\.tmpcZBp3F.temp-archive"

error: aborting due to 1 previous error

IIUC the error above ("failed to remove temporary directory") comes from rustc itself:

• AFAICT Chromium's //build/rust/gni_impl/rustc_wrapper.py does not create temporary directories itself.
• I found an earlier report of a similar issue in https://users.rust-lang.org/t/the-process-cannot-access-the-file-because-it-is-being-used-by-another-process-os-error-32/129335

---

Given that this happens flakily, I understand that this bug report might be inactionable. Nevertheless, I thought that reporting it may be helpful (sharing details + awareness).

And I wonder if extra logging could help. I am not sure about this myself, because:

• Extra logging may disrupt timing (e.g. the other process using the directory can disappear by the time logging tries to discover it)
• It's unclear what to log:
  • contents of the directory?
  • which exact file path couldn't be deleted (maybe .tmpcZBp3F.temp-archive is "it"?)?
  • which other process prevents deletion (not sure if Windows provides an easy way to get this info)?

[bjorn3]

contents of the directory?

The only file we create in this directory is tmp.a:

rust/compiler/rustc_codegen_ssa/src/back/archive.rs

Line 524 in 292db4a

let archive_tmpfile_path = archive_tmpdir.path().join("tmp.a");

This file is moved to somewhere in the parent directory before the temporary directory gets removed.

which exact file path couldn't be deleted (maybe .tmpcZBp3F.temp-archive is "it"?)?

It tries to remove that temporary directory with remove_dir_all. This means that either the directory itself or it's contents couldn't be removed. remove_dir_all doesn't return any information about which of the two cases it is.

which other process prevents deletion (not sure if Windows provides an easy way to get this info)?

Fair chance this is your antivirus.

Unfortunately on Windows the default way of opening a file prevents deletion until the file is closed again, unlike on unix systems where you can remove the file and the file will just hang around without a filename until it is closed, thereby ensuring that the parent directory can be removed just fine.

I suspect the best we can do is either trying to remove it a couple of times with a short sleep in between or logging a warning and keeping the directory around until the user clears their target dir.

[ChrisDenton]

This would ideally be solved in the tempfile crate. If necessary this could also be worked around in the compiler (we did something similar for temp directory creation).

The standard library's remove_dir_all is not likely to add sleeps or too many retries. This has been proposed and rejected a few times.