Skip to content

Conversation

stanhu
Copy link
Collaborator

@stanhu stanhu commented Jan 20, 2025

This avoids the need to hard-code specific patch-level versions.

See https://github.com/rake-compiler/rake-compiler-dock/releases/tag/v1.9.1.

@mudge
Copy link
Owner

mudge commented Jan 21, 2025

Precompilation seems to be failing in the vast majority of cases with the following error:

/usr/local/rbenv/versions/3.4.1/lib/ruby/3.4.0/psych/visitors/to_ruby.rb:388:in 'Psych::Visitors::ToRuby#deduplicate': undefined method 'untaint' for an instance of String (NoMethodError)

Have you seen this error elsewhere (given the only change here is the rake-compiler-dock version)?

cc @flavorjones

@stanhu
Copy link
Collaborator Author

stanhu commented Jan 21, 2025

This code is failing: https://github.com/ruby/psych/blob/56051344d1c399d860f07bbab4720e7086254d8f/lib/psych/visitors/to_ruby.rb#L383-L388

String#untaint was removed in Ruby 2.7. Somehow RUBY_VERSION is < 2.7, but the stacktrace shows this is Ruby 3.4.1?

@flavorjones
Copy link

Have you seen this error elsewhere

No! I've upgraded nokogiri, sqlite3-ruby, and ruby-c-extensions-explained with no problem. And the rake-compiler-dock suite (which has its own test gem) is green.

Somehow RUBY_VERSION is < 2.7, but the stacktrace shows this is Ruby 3.4.1

I bet this is coming from the reset in rake-compiler's "fake.rb" rbconfig. Without spending more time I can't be sure what combination of things is causing this issue, especially since I haven't changed how we build the 2.7.8 cross ruby.

Maybe it's because the default Ruby in the container is now 3.4.1? Maybe you can try to run rbenv shell 3.1 in your RakeCompilerDock.sh script to determine if that works around it.

@flavorjones
Copy link

@mudge Also, and I know you don't want to hear it, but this is the kind of problem that tends to come up when building against older versions of Ruby. AKA the "too much effort" problem.1

Don't get me wrong, ideally this should work, and if I had an infinite amount of time I would absolutely continue to make old rubies work perfectly even as we make changes to accommodate new rubies. But I don't have that kind of time, and in my opinion it's not worth anyone's effort to precompile EOL rubies in native gems. See rake-compiler/rake-compiler-dock#77

Footnotes

  1. https://github.com/rake-compiler/rake-compiler-dock/issues/134#issuecomment-2571645343

@mudge
Copy link
Owner

mudge commented Jan 21, 2025

Using rbenv shell 3.1.6 seems to fix that issue but now the musl builds are broken with a missing libstdc++.

@stanhu
Copy link
Collaborator Author

stanhu commented Jan 21, 2025

But I don't have that kind of time, and in my opinion it's not worth anyone's effort to precompile EOL rubies in native gems.

I agree with this. It also makes the precompiled gems a lot bigger.

@flavorjones
Copy link

flavorjones commented Jan 21, 2025

@mudge

now the musl builds are broken with a missing libstdc++

I don't think this is related to the rake-compiler-dock changes. The most recent ruby:alpine images have had a few of the -dev packages removed which may have caused this library file to also be removed.

Edit: here's the commit that removed those packages: docker-library/ruby@7f078b1

@stanhu
Copy link
Collaborator Author

stanhu commented Jan 21, 2025

There appears to be some inconsistency:

ruby:3.4-alpine

$ docker run -it ruby:3.4-alpine sh
/ # ls -al /usr/lib/libstd*
lrwxrwxrwx    1 root     root            19 Jan 15 17:51 /usr/lib/libstdc++.so.6 -> libstdc++.so.6.0.33
-rwxr-xr-x    1 root     root       2771336 Oct 17 20:43 /usr/lib/libstdc++.so.6.0.33
/ # apk info --who-owns /usr/lib/libstdc++.so.6
/usr/lib/libstdc++.so.6 is owned by libstdc++-14.2.0-r4
/ # apk list libstdc++
WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.21/main: No such file or directory
WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.21/community: No such file or directory
libstdc++-14.2.0-r4 x86_64 {gcc} (GPL-2.0-or-later AND LGPL-2.1-or-later) [installed]

ruby:3.3-alpine

$ docker run -it ruby:3.3-alpine sh
/ # ls -al /usr/lib/libstd*
ls: /usr/lib/libstd*: No such file or directory
/ # apk list libstdc++
WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.21/main: No such file or directory
WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.21/community: No such file or directory

It looks like running apk update && apt add libstdc++ solves the problem.

@stanhu stanhu force-pushed the sh-set-ruby-cc-version-via-rake branch from c21e7e1 to 4d191b2 Compare January 21, 2025 18:40
Recent images appear to have removed libstdc++ from some Ruby
versions. Add it to ensure the re2 shared library can be loaded.
@stanhu stanhu force-pushed the sh-set-ruby-cc-version-via-rake branch from 4d191b2 to ec2fd57 Compare January 21, 2025 18:55
@flavorjones
Copy link

I added a small feature to make it easier to choose the ruby being used, and would appreciate any thoughts on it: rake-compiler/rake-compiler-dock#151

@stanhu
Copy link
Collaborator Author

stanhu commented Jan 21, 2025

Maybe it's because the default Ruby in the container is now 3.4.1? Maybe you can try to run rbenv shell 3.1 in your RakeCompilerDock.sh script to determine if that works around it.

@flavorjones That looks like it worked here. But it seems that fake RUBY_VERSION could cause all sorts of weird bugs in the future, even for supported Ruby versions. Right now I mostly see RUBY_VERSION < 2.7 in the Ruby code base, but if someone introduces a Ruby version-specific feature, then we might hit a similar issue again due to the mismatch.

I don't fully understand tjschuck/rake-compiler-dev-box#5, but I wonder if this workaround is still needed. I should hope most gem authors don't ever need to know about rake-compiler/rake-compiler-dock#151. 😄

@flavorjones
Copy link

flavorjones commented Jan 21, 2025

@stanhu To be clear, the comment referencing the rake-compiler-dev-box issue is referring to the 'resolv' require from rake-compiler/rake-compiler@c83393c2, not the fake_rb method. IIUC, fake_rb is fundamental to how rake-compiler works, but I also don't fully understand rake-compiler.

Maybe this is an opportunity to open an issue on rake-compiler to talk about this use case and the errors we saw in the build?

@stanhu stanhu merged commit 3723e60 into mudge:main Feb 1, 2025
153 checks passed
@larskanis
Copy link

larskanis commented Jul 26, 2025

That looks like it worked here. But it seems that fake RUBY_VERSION could cause all sorts of weird bugs in the future, even for supported Ruby versions. Right now I mostly see RUBY_VERSION < 2.7 in the Ruby code base, but if someone introduces a Ruby version-specific feature, then we might hit a similar issue again due to the mismatch.

This is true, but unfortunately there's no cross compiling support or standard in the ruby language. When cross compiling ruby itself, then it's fake.rb defines CROSS_COMPILING here with the native RUBY_DESCRIPTION and redefines all RUBY_* variables to the values of the target system. We do similar in rake-compiler here. If we wouldn't change RUBY_VERSION we would probably get errors in the cross build instead.

There are different options to solve these kind of issues:

  1. Choose a native ruby version which is "near" to the cross ruby version, so that there's no incompatibility. This is how you solved it by choosing native ruby-3.1 instead of 3.4.
  2. Change the offending code to check for the particular feature rather than relying on RUBY_VERSION. In the pysch case something like if "".respond_to?(:untaint) instead of if RUBY_VERSION < '2.7'. Although this is not the same since untaint was removed some ruby versions later.
  3. require "json" before require "mkmf". This is because mkmf is the entry point of the fake.rb that is used in rake-compiler. Everything required before mkmf sees the native RUBY_* variables.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants