Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Link object files that use #[used] #137426

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

Conversation

DianQK
Copy link
Member

@DianQK DianQK commented Feb 22, 2025

By directly linking the object files that use #[used], we ensure the linker can see them.

This approach allows #[used] to avoid modifying symbol visibility, preserving local symbols. A similar example in C would be:

// foo.c
__attribute__((constructor)) static void foo() {}
// main.c
void main(void) {}

If foo.c is placed in a static library, it will never be loaded unless the entire static library is fully loaded by --whole-archive.

This pull request removes some of the symbols in symbols.o. We can remove more symbols in a follow-up PR.

@rustbot rustbot added A-run-make Area: port run-make Makefiles to rmake.rs S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Feb 22, 2025
@rust-log-analyzer

This comment has been minimized.

@DianQK
Copy link
Member Author

DianQK commented Feb 22, 2025

Weird. The tests failed on macOS after I fixed various test cases. I'll look into it tomorrow.

@bjorn3
Copy link
Member

bjorn3 commented Feb 22, 2025

I think we should link all rlibs with --whole-archive. That would also make some currently cgu partitioning dependent linker errors when an unused function references an undefined symbol consistently produce errors independent of which cgu it ends up in.

@petrochenkov petrochenkov self-assigned this Feb 22, 2025
@DianQK
Copy link
Member Author

DianQK commented Feb 23, 2025

Weird. The tests failed on macOS after I fixed various test cases. I'll look into it tomorrow.

I found that I switched from traversing the object file to using the symbol table. I'm unsure why rustc doesn't add the symbol table when building static libraries for macOS. I have implemented a more robust approach that doesn't rely on the static library.

BTW, I included an experimental commit that doesn't need to be part of the PR.

@DianQK
Copy link
Member Author

DianQK commented Feb 23, 2025

I think we should link all rlibs with --whole-archive. That would also make some currently cgu partitioning dependent linker errors when an unused function references an undefined symbol consistently produce errors independent of which cgu it ends up in.

Beyond the overhead to compile time, this appears to subtly alter the semantics of rlibs during symbol resolution? In C, at least, we only typically load undefined symbols. I'm uncertain what impact this modification might have on real-world projects.

@bjorn3
Copy link
Member

bjorn3 commented Feb 23, 2025

Beyond the overhead to compile time, this appears to subtly alter the semantics of rlibs during symbol resolution? In C, at least, we only typically load undefined symbols. I'm uncertain what impact this modification might have on real-world projects.

You already can't depend on this behavior in Rust. Rustc changes the codegen unit partitioning between rustc versions, so if in one rustc version you wouldn't get a function pulled in, the next rustc version may cause it to be pulled in anyway if the function ends up co-located with another function that is used.

@rust-log-analyzer
Copy link
Collaborator

The job x86_64-gnu-tools failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)
tests/pass/float_nan.rs ... ok
tests/pass/0weak_memory_consistency_sc.rs ... ok

FAILED TEST: tests/pass/alloc-access-tracking.rs
command: MIRI_ENV_VAR_TEST="0" MIRI_TEMP="/tmp/miri-uitest-2o6LBg" RUST_BACKTRACE="1" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/miri" "--error-format=json" "--sysroot=/checkout/obj/build/x86_64-unknown-linux-gnu/miri-sysroot" "-Dwarnings" "-Dunused" "-Ainternal_features" "-Zui-testing" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-tools/miri_ui/0/tests/pass" "tests/pass/alloc-access-tracking.rs" "-Zmiri-track-alloc-id=21" "-Zmiri-track-alloc-accesses" "-Cpanic=abort" "--edition" "2021"
error: actual output differed from expected
Execute `./miri test --bless` to update `tests/pass/alloc-access-tracking.stderr` to the actual output
--- tests/pass/alloc-access-tracking.stderr
+++ <stderr output>
+++ <stderr output>
-note: tracking was triggered
-  --> tests/pass/alloc-access-tracking.rs:LL:CC
-   |
-LL |         let ptr = miri_alloc(123, 1);
-   |                   ^^^^^^^^^^^^^^^^^^ created Miri bare-metal heap allocation of 123 bytes (alignment ALIGN bytes) with id $ALLOC
-   = note: BACKTRACE:
-   = note: inside `miri_start` at tests/pass/alloc-access-tracking.rs:LL:CC
-
-note: tracking was triggered
-note: tracking was triggered
-  --> tests/pass/alloc-access-tracking.rs:LL:CC
-   |
-LL |         *ptr = 42; // Crucially, only a write is printed here, no read!
-   |         ^^^^^^^^^ write access to allocation with id $ALLOC
-   = note: BACKTRACE:
-   = note: inside `miri_start` at tests/pass/alloc-access-tracking.rs:LL:CC
-
-note: tracking was triggered
-note: tracking was triggered
-  --> tests/pass/alloc-access-tracking.rs:LL:CC
-   |
-LL |         assert_eq!(*ptr, 42);
-   |         ^^^^^^^^^^^^^^^^^^^^ read access to allocation with id $ALLOC
-   = note: BACKTRACE:
-   = note: inside `miri_start` at RUSTLIB/core/src/macros/mod.rs:LL:CC
-   = note: this note originates in the macro `assert_eq` (in Nightly builds, run with -Z macro-backtrace for more info)
-
-
-note: tracking was triggered
-  --> tests/pass/alloc-access-tracking.rs:LL:CC
-   |
-LL |         miri_dealloc(ptr, 123, 1);
-   |         ^^^^^^^^^^^^^^^^^^^^^^^^^ freed allocation with id $ALLOC
-   = note: BACKTRACE:
-   = note: inside `miri_start` at tests/pass/alloc-access-tracking.rs:LL:CC
-

---

Location:
   /cargo/registry/src/index.crates.io-1949cf8c6b5b557f/ui_test-0.28.0/src/lib.rs:369

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
error: test failed, to rerun pass `--test ui`
Caused by:
  process didn't exit successfully: `/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-tools/x86_64-unknown-linux-gnu/release/deps/ui-145608eb94c4e144 --quiet` (exit status: 1)
  process didn't exit successfully: `/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-tools/x86_64-unknown-linux-gnu/release/deps/ui-145608eb94c4e144 --quiet` (exit status: 1)
Command has failed. Rerun with -v to see more details.
  local time: Sun Feb 23 12:30:02 UTC 2025
  network time: Sun, 23 Feb 2025 12:30:02 GMT
##[error]Process completed with exit code 1.
Post job cleanup.

@DianQK
Copy link
Member Author

DianQK commented Feb 23, 2025

I think we should link all rlibs with --whole-archive. That would also make some currently cgu partitioning dependent linker errors when an unused function references an undefined symbol consistently produce errors independent of which cgu it ends up in.

I'm testing this at #137481.

bors added a commit to rust-lang-ci/rust that referenced this pull request Feb 23, 2025
[perf experiment] Link rlibs with `--whole-archive`

For rust-lang#137426 (comment).

r? ghost
@DianQK
Copy link
Member Author

DianQK commented Feb 23, 2025

Beyond the overhead to compile time, this appears to subtly alter the semantics of rlibs during symbol resolution? In C, at least, we only typically load undefined symbols. I'm uncertain what impact this modification might have on real-world projects.

You already can't depend on this behavior in Rust. Rustc changes the codegen unit partitioning between rustc versions, so if in one rustc version you wouldn't get a function pulled in, the next rustc version may cause it to be pulled in anyway if the function ends up co-located with another function that is used.

IIUC, if all libs are well-defined, no issue here? So this is a bad experience when encountering an undefined symbol error?

bors added a commit to rust-lang-ci/rust that referenced this pull request Feb 23, 2025
[perf experiment] Link rlibs with `--whole-archive`

For rust-lang#137426 (comment).

r? ghost
bors added a commit to rust-lang-ci/rust that referenced this pull request Feb 24, 2025
[perf experiment] Link rlibs with `--whole-archive`

For rust-lang#137426 (comment).

r? ghost
@DianQK
Copy link
Member Author

DianQK commented Feb 24, 2025

@bors try @rust-timer queue

@rust-timer
Copy link
Collaborator

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

@rustbot rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Feb 24, 2025
bors added a commit to rust-lang-ci/rust that referenced this pull request Feb 24, 2025
Link object files that use `#[used]`

By directly linking the object files that use `#[used]`, we ensure the linker can see them.

This approach allows `#[used]` to avoid modifying symbol visibility, preserving local symbols. A similar example in C would be:

```c
// foo.c
__attribute__((constructor)) static void foo() {}
// main.c
void main(void) {}
```

If `foo.c` is placed in a static library, it will never be loaded unless the entire static library is fully loaded by `--whole-archive`.

This pull request removes some of the symbols in `symbols.o`. We can remove more symbols in a follow-up PR.
@bors
Copy link
Contributor

bors commented Feb 24, 2025

⌛ Trying commit bb77554 with merge 46536f1...

@rust-log-analyzer
Copy link
Collaborator

The job dist-x86_64-linux failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)
file:.git/config remote.origin.url=https://github.com/rust-lang-ci/rust
file:.git/config remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
file:.git/config gc.auto=0
file:.git/config http.https://github.com/.extraheader=AUTHORIZATION: basic ***
file:.git/config branch.try.remote=origin
file:.git/config branch.try.merge=refs/heads/try
file:.git/config remote.upstream.fetch=+refs/heads/*:refs/remotes/upstream/*
file:.git/config submodule.library/backtrace.active=true
file:.git/config submodule.library/backtrace.url=https://github.com/rust-lang/backtrace-rs.git
file:.git/config submodule.library/stdarch.active=true
---
[RUSTC-TIMING] rustc_borrowck test:false 114.850
   Compiling rustc_driver v0.0.0 (/checkout/compiler/rustc_driver)
error: linking with `clang` failed: exit status: 1
  |
  = note:  "clang" "-Wl,--version-script=/tmp/rustc17As2h/list" "-Wl,--no-undefined-version" "-m64" "/tmp/rustc17As2h/symbols.o" "<253 object files omitted>" "<sysroot>-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_driver-774f694bdb7deb15.0c7313dw7euea77omqyi4yki5.rcgu.rmeta" "-Wl,--as-needed" "-Wl,-Bstatic" "/tmp/rustc17As2h/{librustc_codegen_llvm-6a81856ba97f4eba.rlib,librustc_llvm-8ed1c7633feca432.rlib,libblake3-c6c459bef0d0f74f.rlib,libpsm-e146f4bd642e37ac.rlib,libprofiler_builtins-c72ca7fe3dac98c5.rlib}.rlib" "<sysroot>/lib/rustlib/x86_64-unknown-linux-gnu/lib/{libcompiler_builtins-*}.rlib" "-Wl,-Bdynamic" "-lLLVM-20-rust-1.87.0-nightly" "-ldl" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-B<sysroot>/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld" "-fuse-ld=lld" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "<sysroot>-rustc/x86_64-unknown-linux-gnu/release/build/psm-bee8a26e2991f023/out" "-L" "<sysroot>-rustc/x86_64-unknown-linux-gnu/release/build/blake3-73e53664fd4755c1/out" "-L" "<sysroot>-rustc/x86_64-unknown-linux-gnu/release/build/blake3-73e53664fd4755c1/out" "-L" "<sysroot>-rustc/x86_64-unknown-linux-gnu/release/build/rustc_llvm-2f758f0fe20936e5/out" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/lib" "-L" "/rustroot/bin/../lib/gcc/x86_64-pc-linux-gnu/9.5.0/../../../../lib64" "-L" "<sysroot>/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "<sysroot>-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-774f694bdb7deb15.so" "-shared" "-Wl,-soname=librustc_driver-774f694bdb7deb15.so" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-nodefaultlibs" "-u" "__llvm_profile_runtime" "-Wl,-z,origin" "-Wl,-rpath,$ORIGIN/../lib" "-Wl,--icf=all"
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: rust-lld: error: version script assignment of 'global' to symbol '__addtf3' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__ashlti3' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__ashrti3' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__bswapdi2' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__bswapsi2' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__clzdi2' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__clzti2' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__ctzdi2' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__ctzti2' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__divmodti4' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__divtf3' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__eqtf2' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__extenddftf2' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__extendsftf2' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__fixsfti' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__fixtfdi' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__fixtfsi' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__fixtfti' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__fixunsdfdi' failed: symbol not defined
          rust-lld: error: version script assignment of 'global' to symbol '__fixunssfdi' failed: symbol not defined
          rust-lld: error: too many errors emitted, stopping now (use --error-limit=0 to see all errors)
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

[RUSTC-TIMING] rustc_driver test:false 103.696
error: could not compile `rustc_driver` (lib) due to 1 previous error
Build completed unsuccessfully in 0:13:49
---
Caused by:
    Command RUST_BACKTRACE=full python3 /checkout/x.py build --target x86_64-unknown-linux-gnu --host x86_64-unknown-linux-gnu --stage 2 library/std --rust-profile-generate /tmp/tmp-multistage/opt-artifacts/rustc-pgo --set llvm.thin-lto=false --set llvm.link-shared=true [at /checkout/obj] has failed with exit code Some(1)

Stack backtrace:
   0: <anyhow::Error>::msg::<alloc::string::String>
             at /rust/deps/anyhow-1.0.95/src/backtrace.rs:27:14
   1: <opt_dist::exec::CmdBuilder>::run
             at /rustc/46536f154414de0c4e28d0e9297a5ac10ef00174/src/tools/opt-dist/src/exec.rs:80:17
   2: <opt_dist::exec::Bootstrap>::run
             at /rustc/46536f154414de0c4e28d0e9297a5ac10ef00174/src/tools/opt-dist/src/exec.rs:181:9
             at /rustc/46536f154414de0c4e28d0e9297a5ac10ef00174/src/tools/opt-dist/src/main.rs:222:13
             at /rustc/46536f154414de0c4e28d0e9297a5ac10ef00174/src/tools/opt-dist/src/main.rs:222:13
   4: <opt_dist::timer::TimerSection>::section::<opt_dist::execute_pipeline::{closure#1}::{closure#0}, ()>
             at /rustc/46536f154414de0c4e28d0e9297a5ac10ef00174/src/tools/opt-dist/src/timer.rs:111:22
             at /rustc/46536f154414de0c4e28d0e9297a5ac10ef00174/src/tools/opt-dist/src/main.rs:211:9
             at /rustc/46536f154414de0c4e28d0e9297a5ac10ef00174/src/tools/opt-dist/src/main.rs:211:9
   6: <opt_dist::timer::TimerSection>::section::<opt_dist::execute_pipeline::{closure#1}, opt_dist::training::RustcPGOProfile>
             at /rustc/46536f154414de0c4e28d0e9297a5ac10ef00174/src/tools/opt-dist/src/timer.rs:111:22
             at /rustc/46536f154414de0c4e28d0e9297a5ac10ef00174/src/tools/opt-dist/src/main.rs:208:29
   8: opt_dist::main
             at /rustc/46536f154414de0c4e28d0e9297a5ac10ef00174/src/tools/opt-dist/src/main.rs:399:18
   9: <fn() -> core::result::Result<(), anyhow::Error> as core::ops::function::FnOnce<()>>::call_once
   9: <fn() -> core::result::Result<(), anyhow::Error> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/f0cb41030579cd1a6f72bd23f38e677052d5d485/library/core/src/ops/function.rs:250:5
  10: std::sys::backtrace::__rust_begin_short_backtrace::<fn() -> core::result::Result<(), anyhow::Error>, core::result::Result<(), anyhow::Error>>
             at /rustc/f0cb41030579cd1a6f72bd23f38e677052d5d485/library/std/src/sys/backtrace.rs:152:18
  11: std::rt::lang_start::<core::result::Result<(), anyhow::Error>>::{closure#0}
             at /rustc/f0cb41030579cd1a6f72bd23f38e677052d5d485/library/std/src/rt.rs:199:18
  12: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
  13: std::panicking::try::do_call
             at /rustc/f0cb41030579cd1a6f72bd23f38e677052d5d485/library/std/src/panicking.rs:587:40
  14: std::panicking::try
             at /rustc/f0cb41030579cd1a6f72bd23f38e677052d5d485/library/std/src/panicking.rs:550:19

@bors
Copy link
Contributor

bors commented Feb 24, 2025

💔 Test failed - checks-actions

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 24, 2025
@bjorn3
Copy link
Member

bjorn3 commented Feb 24, 2025

Beyond the overhead to compile time, this appears to subtly alter the semantics of rlibs during symbol resolution? In C, at least, we only typically load undefined symbols. I'm uncertain what impact this modification might have on real-world projects.

You already can't depend on this behavior in Rust. Rustc changes the codegen unit partitioning between rustc versions, so if in one rustc version you wouldn't get a function pulled in, the next rustc version may cause it to be pulled in anyway if the function ends up co-located with another function that is used.

IIUC, if all libs are well-defined, no issue here? So this is a bad experience when encountering an undefined symbol error?

Correct. If all libs are well-defined, adding --whole-archive shouldn't cause any difference other than forcing #[used] statics to be kept even if they are in an object file that would otherwise not be pulled in. It won't grow the executable as we use --gc-sections anyway. When someone tries to use an undefined symbol this will however force the error to be reported consistently across rustc versions, making it less likely that someone opens a bug report with a linker "regression".

@ChrisDenton
Copy link
Member

If we do --whole-archive for all platforms then I think we should split Windows raw-dylib generated imports into a separate staticlib file. While it should now work with whole-archive, I still don't think it's a great idea.

@bjorn3
Copy link
Member

bjorn3 commented Feb 24, 2025

If we do --whole-archive for all platforms then I think we should split Windows raw-dylib generated imports into a separate staticlib file.

When creating a staticlib we need to keep it merged as for those there is no way to tell the build system to link a separate import library.

@bors
Copy link
Contributor

bors commented Feb 25, 2025

☔ The latest upstream changes (presumably #135726) made this pull request unmergeable. Please resolve the merge conflicts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-run-make Area: port run-make Makefiles to rmake.rs S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. S-waiting-on-perf Status: Waiting on a perf run to be completed. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants