relaxing traditional TLS GD

[LukeZhuang]

As previously discussed in #308, relaxing traditional TLSGD is impossible in the current psABI because of the lack of the connection between la.tls.gd and call __tls_get_addr.
In my understanding, TLSDESC is a theoretically perfect alternative solution. However, it requires dynamic loader support, and its implementation in GLIBC (https://patchwork.sourceware.org/project/glibc/list/?series=24625) is still too far from productization. If TLSDESC is not translated to IE/LE, for example when we are linking shared objects, it would crash immediately when we get there. Even if GLIBC support is checked-in one day, people who are using old-versioned GLIBC are still not available to this excellent feature. And that's one reason why LLVM still prefer not to turn TLSDESC on by default (llvm/llvm-project#166540).

So, I would like to bring up the discussion again. This kind of relaxation (traditional GD -> LE/IE) seems more robust:

1. It is a compiler/linker-only work, does not depend on any specific GLIBC version (once __tls_get_addr is supported).
2. Even if the relaxation does not happen, the program still runs well.

Currently I can think of two possible solutions:

Proposal 1 (from #308)

We can add a new relocation type called R_RISCV_TLS_CALL_LINK which contains the label of R_RISCV_TLS_GD_HI20, and append it to the call __tls_get_addr:

label:
    auipc a0,0                    # R_RISCV_TLS_GD_HI20 (symbol), R_RISCV_RELAX
    addi  a0,a0,0                 # R_RISCV_PCREL_LO12_I (label), R_RISCV_RELAX
    call  __tls_get_addr@plt      # R_RISCV_TLS_CALL_LINK(label), R_RISCV_CALL_PLT(__tls_get_addr), R_RISCV_RELAX

This idea is more like the idea of TLSDESC. It's more flexible for insn scheduling, allows intervening instructions, while may need more work on the psABI/compiler/assembler/linker I guess.

Proposal 2

We only do this translation when the three relocations (or insns) are strictly adjacent by address, which means by checking the offset strictly matches this:

      +0x0: auipc a0, 0x0
        +0x0: R_RISCV_TLS_GD_HI20
        +0x0: R_RISCV_RELAX
      +0x4: mv a0, a0
        +0x4: R_RISCV_PCREL_LO12_I
        +0x4: R_RISCV_RELAX
      +0x8: auipc ra, 0x0
        +0x8: R_RISCV_CALL_PLT __tls_get_addr
        +0x8: R_RISCV_RELAX
      +0xc: jalr ra

The idea comes from x86-64 psABI (http://man6.org/lib/pdfjs/web/viewer.html?file=/blog/PdfFile/x86-64-psABI-1.0.pdf, chapter 10.3.3), looks like they already forces tlsgd and the call to be together in psABI (chapter 10.3.1).
And we can add an option in the compiler which forces the combination of la.tls.gd and call __tls_get_addr to allow more relaxation chances in the linker.
The advantage is that it's more straightforward, the linker support is more trivial, and no major changes to the psABI (just add the GD->IE/LE translation rule).
The drawback is that it may be a little bit tricky, and may lose some flexibility in instruction scheduling.

May I ask does this motivation make sense? Which solution is more preferable, or more proposals?

[rui314]

Wait, I thought that TLSDESC support had been added to glibc in 2023, but is it still pending? If that’s the case, I think you should focus on adding TLSDESC support to glibc as soon as possible instead of standardizing something else. If it had been added in 2023, it would be widely available by now.

[LukeZhuang]

Wait, I thought that TLSDESC support had been added to glibc in 2023, but is it still pending? If that’s the case, I think you should focus on adding TLSDESC support to glibc as soon as possible instead of standardizing something else. If it had been added in 2023, it would be widely available by now.

Sadly looks like not :(
I just checked the GLIBC repo, untill today (the latest sha is dfb150f3aadd931), I still cannot find any tlsdesc support in sysdeps/riscv: https://sourceware.org/git/?p=glibc.git;a=tree;f=sysdeps/riscv;h=08628bc9edd9220ce0262ac77d6dffb9bf25d3df;hb=HEAD. Only the series of patches I mentioned above is available. Please correct me if I have missed anything.

[LukeZhuang]

While besides that, from a toolchain supplier point of view, I'm afraid if the user somehow refused to upgrade their GLIBC (since it is usually bound to several linux versions), we can still not use TLSDESC for shared libraries, may I ask is that right?

[rui314]

Realistically, standardizing TLSGD relaxation here, implementing it across all major toolchains, and making these changes generally available to users would take a few years. On the other hand, TLSDESC is already supported in most places, including the psABI, compiler, and linker; the only missing piece is glibc which is just neglected. I'm not sure it's worth pursuing the former path. At the very least, someone needs to submit the pending changes to glibc, which I believe shouldn’t be too difficult.

[aswaterman]

Agreed with @rui314; let's focus on the long game, which is TLSDESC.

(As a separate matter, I like that Proposal 2 is self-contained and would probably suffice in practice, but it will probably become moot in a few years when distros can start assuming TLSDESC support.)

[LukeZhuang]

I have no objections on proceeding with glibc TLSDESC support, excepting that @rui314 TLSDESC relaxation support on GNU-LD is not ready as well (https://patchwork.sourceware.org/project/binutils/list/?series=31139). I have briefly looked at this patch, the main skeleton is okay but seems has some small issues (TLSDESC->IE looks buggy, lack of tests etc.).
For the GLIBC patch, sadly I'm not an expert in GLIBC, but I can volunteer to learn and help with it.
The patches (GNU-LD and GLIBC support) are both authored by @ishitatsuyuki , I'm not sure if he's still working on RISCV. Please tell me if I can help with that.
And may I ask could we also have the opinion from others @kito-cheng @jrtc27 @topperc etc.

[rui314]

I don't think @ishitatsuyuki is still working on these patches, so you can take them over.

[kito-cheng]

Yeah, I agree we should just pushing TLSDESC forward instead of keep improving traditional TLS, thanks for raising this issue :)

[LukeZhuang]

Thanks folks for the opinions. I guess I want to volunteer to take over the GLIBC TLSDESC support and the GNU-LD TLSDESC relaxation support :)
While the GLIBC work looks related to Jessica's issue here: #488, Florian also mentioned about this here: https://patchwork.sourceware.org/project/glibc/patch/20230914084033.222120-4-ishitatsuyuki@gmail.com/. Seems we need to resolve that first

Updated: I just noticed that there were more discussions on libc-alpha here since then: https://inbox.sourceware.org/libc-alpha/09A14C77-9EEA-4707-9E36-D3AAF8A98B24@gmail.com/T/#u

[LukeZhuang]

Hi @ilovepi , I remember in the last psABI meeting you said Fuchsia has moved completely from traditional TLS GD to TLSDESC (or maybe I've misheard), may I ask is that right?

[ilovepi]

That is correct. Fuchsia will no longer support anything other than TLSDESC in our ABI. We've updated all the compilers to stop emitting references to __tls_get_addr altogether. So for the 3 architectures we support (x86_64, Aarch64, RISC-V) we always emit TLSDESC references. We have one downstream user on an old toolchain, but as soon as they update, we will delete __tls_get_addr from our libc permanently. We've held off on updating our docs on this until we finalize the removal, which should be in the next couple of weeks.

[LukeZhuang]

That is correct. Fuchsia will no longer support anything other than TLSDESC in our ABI. We've updated all the compilers to stop emitting references to __tls_get_addr altogether. So for the 3 architectures we support (x86_64, Aarch64, RISC-V) we always emit TLSDESC references. We have one downstream user on an old toolchain, but as soon as they update, we will delete __tls_get_addr from our libc permanently. We've held off on updating our docs on this until we finalize the removal, which should be in the next couple of weeks.

Thanks @ilovepi for the more detailed info! looks like this is another reason for the complete removal of trad GD as well as its future optimization :)