Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot build my GDExtension with Godot 4.4 #1729

Open
ze2j opened this issue Mar 5, 2025 · 6 comments · May be fixed by #1731
Open

Cannot build my GDExtension with Godot 4.4 #1729

ze2j opened this issue Mar 5, 2025 · 6 comments · May be fixed by #1731
Labels
bug This has been identified as a bug needs testing topic:buildsystem Related to the buildsystem or CI setup topic:gdextension This relates to the new Godot 4 extension implementation

Comments

@ze2j
Copy link

ze2j commented Mar 5, 2025

Tested versions

Reproducible with Godot 4.4
Not reproducible with Godot 4.3

System information

Godot v4.4.1.rc (afaa0cd4b) - Ubuntu 22.04.5 LTS 22.04 on X11 - X11 display driver, Multi-window, 2 monitors - Vulkan (Forward+) - dedicated NVIDIA GeForce GTX 1070 (nvidia; 535.183.01) - Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz (8 threads)

Issue description

I encountered an unusual build error while porting my project from Godot 4.3 to 4.4. My project consists of several custom modules registering 94 classes to ClassDB and a small GDExtension written in C++.

Godot 4.4 itself builds without errors, and godot-cpp for 4.4 also compiles successfully. However, building my extension fails with the following error:

$ scons platform=linux dev_build=yes compiledb=true precision=double disable_exceptions=false verbose=true
scons: Reading SConscript files ...
Auto-detected 8 CPU cores available for build parallelism. Using 7 cores by default. You can override it with the -j argument.
Building for architecture x86_64 on platform linux
scons: done reading SConscript files.
scons: Building targets ...
ar rc <godot-cpp>/bin/libgodot-cpp.linux.template_debug.dev.double.x86_64.a <skipping a very long list of absolute paths to *.o files>
scons: *** [<godot-cpp-private-fork>/bin/libgodot-cpp.linux.template_debug.dev.double.x86_64.a] sh: Argument list too long
scons: building terminated because of errors.

In the ar rc command, 1050 object files are listed, including the 94 from my project. The total length of the arguments is 131,107 characters. I believe the error occurs because the arguments are interpreted as a single entity, exceeding the system's limit of MAX_ARG_STRLEN (131,072).

To confirm this, I commented out some of my GDREGISTER_CLASS calls to reduce the total argument length below the threshold, and the build succeeded. Another potential workaround could be relocating my project to a directory with a shorter path (though I haven’t tested this; my current project path is only 47 characters long).

As Godot continues to grow, I suspect more users on similar Unix-like systems will encounter this limitation.

I'm unsure whether to report this issue to godot-cpp, or even scons directly, given its potential impact on Godot’s documentation. Please let me know if I should transfer this bug report elsewhere.

Notes:

  • The issue occurs with both scons 4.0 and 4.9.
  • Enabling the scons stack trace (--debug=stacktrace) produces the following error when the build fails:
scons: internal stack trace:
  File "/home/mla/.local/lib/python3.10/site-packages/SCons/Taskmaster/Job.py", line 737, in _work
    task.execute()
  File "/home/mla/.local/lib/python3.10/site-packages/SCons/Script/Main.py", line 224, in execute
    SCons.Taskmaster.OutOfDateTask.execute(self)
  File "/home/mla/.local/lib/python3.10/site-packages/SCons/Taskmaster/__init__.py", line 263, in execute
    raise buildError

Steps to reproduce

Increase the number of classes registered to ClassDB then build the GDextension C++ example

Minimal reproduction project (MRP)

Let me know if you need one (it's not a trivial task)

@AThousandShips AThousandShips added bug This has been identified as a bug topic:buildsystem Related to the buildsystem or CI setup needs testing topic:gdextension This relates to the new Godot 4 extension implementation labels Mar 5, 2025
@akien-mga
Copy link
Member

Moving to https://github.com/godotengine/godot-cpp.

@akien-mga akien-mga transferred this issue from godotengine/godot Mar 5, 2025
@dsnopek
Copy link
Collaborator

dsnopek commented Mar 5, 2025

Thanks for the report!

Just to make sure I'm understanding correctly: the issue is caused by the classes that were added to Godot in your custom module, not the classes added in your GDExtension? That does make sense, given that all the generated classes are dumped into the same directory, and so scons would put them in the same .a file

I'm not sure what tools (if any) scons gives us to deal with this. A potential solution is just making more directories for the generated .cpp files, such that we ensure that each directory only holds a maximum of 50 (?) files?

@ze2j
Copy link
Author

ze2j commented Mar 5, 2025

Just to make sure I'm understanding correctly: the issue is caused by the classes that were added to Godot in your custom module, not the classes added in your GDExtension?

Yes that's correct.

That does make sense, given that all the generated classes are dumped into the same directory, and so scons would put them in the same .a file
I'm not sure what tools (if any) scons gives us to deal with this. A potential solution is just making more directories for the generated .cpp files, such that we ensure that each directory only holds a maximum of 50 (?) files?

I don’t think this would work, because the issue is with the number of files passed to the command that creates the library, and we want to reduce that number. I am trying a different approach: create small intermediate static libraries and then link them together.

Edit:
Here is an attempt trying to implement this approach (to apply in godotcpp.py in the _godot_cpp method) but it doesn't work... The resulting library is empty:

-        library = env.StaticLibrary(target=env.File("bin/%s" % library_name), source=sources)
+        libraries = []
+        remaining = sources[:]
+        n = 0
+        while len(remaining) > 0:
+            libraries.append(env.StaticLibrary(target="bin/libgodot-cpp-part_%d%s" % (n, env["suffix"] + env["LIBSUFFIX"]), source=remaining[:100]))
+            remaining = remaining[100:]
+            n += 1
+
+        library = env.StaticLibrary(target=env.File("bin/%s" % library_name), source=[], LIBS=libraries)
+        for lib in libraries:
+            env.Depends(env.File("bin/%s" % library_name), lib)
+

@ze2j
Copy link
Author

ze2j commented Mar 6, 2025

I can't find a single out-of-the-box method that is portable:

  • Linking smaller static libraries together doesn’t work as I thought because the object files they contain are not extracted and repackaged.
  • Incrementally building a large static library seems possible using ar with the right options, but this is not portable.
  • Extracting object files from smaller static libraries and then merging them should work, but this is also not portable.

As far as I can tell, we are left with two possible approaches, but one requirement must be dropped:

  • Instead of building a single static library, build smaller static libraries and let the GDExtension shared library link against them.
  • Create a custom builder, specific to Unix-like systems, to incrementally build the single large static library.

@enetheru
Copy link
Collaborator

enetheru commented Mar 6, 2025

does using a file for the options work?
https://www.man7.org/linux/man-pages/man1/ar.1.html

@file
           Read command-line options from file.  The options read are
           inserted in place of the original @file option.  If file does
           not exist, or cannot be read, then the option will be treated
           literally, and not removed.

           Options in file are separated by whitespace.  A whitespace
           character may be included in an option by surrounding the
           entire option in either single or double quotes.  Any
           character (including a backslash) may be included by prefixing
           the character to be included with a backslash.  The file may
           itself contain additional @file options; any such options will
           be processed recursively.

@ze2j
Copy link
Author

ze2j commented Mar 6, 2025

Nice idea!

It works, I am making some more tests and will provide a PR in a few hours if everything is fine.

@ze2j ze2j linked a pull request Mar 6, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This has been identified as a bug needs testing topic:buildsystem Related to the buildsystem or CI setup topic:gdextension This relates to the new Godot 4 extension implementation
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants