-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support rebuild
and build
for cross-compiling Node-API module to wasm on Windows
#2974
Conversation
The lint failing here will be resolved by: |
@cclauss Branch updated. Once nodejs/gyp-next#222 and nodejs/gyp-next#240 land in gyp-next 0.17.0 and this PR get merged, excited to see the next release of node-gyp can support build wasm! |
gyp 0.17.0 released, anyone can review this PR? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM % a nit.
Also I think it will be helpful to add an integration test like https://github.com/nodejs/node-gyp/blob/main/test/test-addon.js#L45-L55. |
But integration test for wasm requires latest gyp. Should I include the commit of The test with this change is here https://github.com/toyobayashi/emnapi-node-gyp-test/actions/runs/9403090113 |
b60aa42
to
cb8bfc1
Compare
Or this PR should wait latest gyp get merged and then add the integration test? |
@toyobayashi would you mind adding integration tests to this PR? In this way, we can merge #3039 just for gyp updating first. Thanks! |
@legendecas OK, cherry picked b4cad44 |
ignore emsdk temporarily remove wasi test add wasi test back --experimental-wasi-unstable-preview1 test separate wasm test run test in bash on windows fix python path
b4cad44
to
664dbb6
Compare
package.json
Outdated
"bindings": "^1.5.0", | ||
"cross-env": "^7.0.3", | ||
"emnapi": "^1.2.0", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should use clang or gcc that is also available on github action https://github.com/actions/runner-images/blob/main/images/windows/Windows2022-Readme.md instead of introducing these dependencies.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As you can see https://github.com/nodejs/node-gyp/actions/runs/9483655441, the make generator does not support native windows, I just do the minimum work in gyp repo to make it work for wasm (require emnapi's common.gypi
and some sources code).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Short to say, to let the windows && make && rebuild
test work, we need to build it to wasm (require emnapi as dependency)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My local native test shows: clang: error: unsupported option '-fPIC' for target 'x86_64-pc-windows-msvc'
gyp info it worked if it ends with ok
gyp verb cli [
gyp verb cli 'C:\\Users\\toyobayashi\\app\\nodejs\\node.exe',
gyp verb cli 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\bin\\node-gyp.js',
gyp verb cli 'configure',
gyp verb cli '-C',
gyp verb cli 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\test\\node_modules\\hello_napi\\',
gyp verb cli '--verbose',
gyp verb cli '--',
gyp verb cli '-f',
gyp verb cli 'make-linux'
gyp verb cli ]
gyp info using [email protected]
gyp info using [email protected] | win32 | x64
gyp info chdir C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi\
gyp verb download using dist-url https://npm.taobao.org/mirrors/node
gyp verb find Python Python is not set from command line or npm configuration
gyp verb find Python Python is not set from environment variable PYTHON
gyp verb find Python checking if the py launcher can be used to find Python 3
gyp verb find Python - executing "py.exe" to get Python 3 executable path
gyp verb find Python - executable path is "C:\Users\toyobayashi\AppData\Local\Programs\Python\Python310\python.exe"
gyp verb find Python - executing "C:\Users\toyobayashi\AppData\Local\Programs\Python\Python310\python.exe" to get version
gyp verb find Python - version is "3.10.6"
gyp info find Python using Python version 3.10.6 found at "C:\Users\toyobayashi\AppData\Local\Programs\Python\Python310\python.exe"
gyp verb get node dir no --target version specified, falling back to host node version: 16.18.1
gyp verb download using dist-url https://npm.taobao.org/mirrors/node
gyp verb install input version string "16.18.1"
gyp verb install installing version: 16.18.1
gyp verb install --ensure was passed, so won't reinstall if already installed
gyp verb install version is already installed, need to check "installVersion"
gyp verb got "installVersion" 11
gyp verb needs "installVersion" 11
gyp verb install version is good
gyp verb on Windows; need to check node.lib
gyp verb get node dir target node version installed: 16.18.1
gyp verb build dir attempting to create "build" dir: C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi\build
gyp verb build dir "build" dir needed to be created? No
gyp verb build/config.gypi creating config file
gyp verb build/config.gypi writing out config file: C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi\build\config.gypi
gyp verb config.gypi checking for gypi file: C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi\config.gypi
gyp verb common.gypi checking for gypi file: C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi\common.gypi
gyp info spawn C:\Users\toyobayashi\AppData\Local\Programs\Python\Python310\python.exe
gyp info spawn args [
gyp info spawn args 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\gyp\\gyp_main.py',
gyp info spawn args 'binding.gyp',
gyp info spawn args '-f',
gyp info spawn args 'make-linux',
gyp info spawn args '-I',
gyp info spawn args 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\test\\node_modules\\hello_napi\\build\\config.gypi',
gyp info spawn args '-I',
gyp info spawn args 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\addon.gypi',
gyp info spawn args '-I',
gyp info spawn args 'C:\\Users\\toyobayashi\\AppData\\Local\\node-gyp\\Cache\\16.18.1\\include\\node\\common.gypi',
gyp info spawn args '-Dlibrary=shared_library',
gyp info spawn args '-Dvisibility=default',
gyp info spawn args '-Dnode_root_dir=C:\\Users\\toyobayashi\\AppData\\Local\\node-gyp\\Cache\\16.18.1',
gyp info spawn args '-Dnode_gyp_dir=C:\\Users\\toyobayashi\\Projects\\node-gyp',
gyp info spawn args '-Dnode_lib_file=C:\\\\Users\\\\toyobayashi\\\\AppData\\\\Local\\\\node-gyp\\\\Cache\\\\16.18.1\\\\<(target_arch)\\\\node.lib',
gyp info spawn args '-Dmodule_root_dir=C:\\Users\\toyobayashi\\Projects\\node-gyp\\test\\node_modules\\hello_napi',
gyp info spawn args '-Dnode_engine=v8',
gyp info spawn args '--depth=.',
gyp info spawn args '--no-parallel',
gyp info spawn args '--generator-output',
gyp info spawn args 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\test\\node_modules\\hello_napi\\build',
gyp info spawn args '-Goutput_dir=.'
gyp info spawn args ]
gyp info ok
gyp info it worked if it ends with ok
gyp verb cli [
gyp verb cli 'C:\\Users\\toyobayashi\\app\\nodejs\\node.exe',
gyp verb cli 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\bin\\node-gyp.js',
gyp verb cli 'build',
gyp verb cli '-C',
gyp verb cli 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\test\\node_modules\\hello_napi\\',
gyp verb cli '--verbose'
gyp verb cli ]
gyp info using [email protected]
gyp info using [email protected] | win32 | x64
gyp info chdir C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi\
gyp verb build type Release
gyp verb architecture x64
gyp verb node dev dir C:\Users\toyobayashi\AppData\Local\node-gyp\Cache\16.18.1
gyp verb python C:\Users\toyobayashi\AppData\Local\Programs\Python\Python310\python.exe
gyp verb `which` succeeded for `make` C:\Users\toyobayashi\app\make\make.EXE
gyp info spawn make
gyp info spawn args [ 'V=1', 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory 'C:/Users/toyobayashi/Projects/node-gyp/test/node_modules/hello_napi/build'
"C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/Llvm/x64/bin/clang.exe" -o Release/obj.target/hello/hello.o ../hello.c '-DNODE_GYP_MODULE_NAME=hello' '-DUSING_UV_SHARED=1' '-DUSING_V8_SHARED=1' '-DV8_DEPRECATION_WARNINGS=1' '-DV8_DEPRECATION_WARNINGS' '-DV8_IMMINENT_DEPRECATION_WARNINGS' '-D_GLIBCXX_USE_CXX11_ABI=1' '-D_LARGEFILE_SOURCE' '-D_FILE_OFFSET_BITS=64' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_NO_PINSHARED' '-DOPENSSL_THREADS' '-DBUILDING_NODE_EXTENSION' -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/include/node -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/src -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/openssl/config -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/openssl/openssl/include -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/uv/include -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/zlib -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/v8/include -fPIC -pthread -Wall -Wextra -Wno-unused-parameter -m64 -O3 -fno-omit-frame-pointer -MMD -MF ./Release/.deps/Release/obj.target/hello/hello.o.d.raw -c
clang: error: unsupported option '-fPIC' for target 'x86_64-pc-windows-msvc'
make: *** [hello.target.mk:111: Release/obj.target/hello/hello.o] Error 1
make: Leaving directory 'C:/Users/toyobayashi/Projects/node-gyp/test/node_modules/hello_napi/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.<anonymous> (C:\Users\toyobayashi\Projects\node-gyp\lib\build.js:216:23)
gyp ERR! stack at ChildProcess.emit (node:events:513:28)
gyp ERR! stack at Process.ChildProcess._handle.onexit (node:internal/child_process:293:12)
gyp ERR! System Windows_NT 10.0.22631
gyp ERR! command "C:\\Users\\toyobayashi\\app\\nodejs\\node.exe" "C:\\Users\\toyobayashi\\Projects\\node-gyp\\bin\\node-gyp.js" "build" "-C" "C:\\Users\\toyobayashi\\Projects\\node-gyp\\test\\node_modules\\hello_napi\\" "--verbose"
gyp ERR! cwd C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi
gyp ERR! node -v v16.18.1
gyp ERR! node-gyp -v v10.1.0
gyp ERR! not ok
After I remove -fPIC
manually in makefile then output some link errors
gyp info it worked if it ends with ok
gyp verb cli [
gyp verb cli 'C:\\Users\\toyobayashi\\app\\nodejs\\node.exe',
gyp verb cli 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\bin\\node-gyp.js',
gyp verb cli 'build',
gyp verb cli '-C',
gyp verb cli 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\test\\node_modules\\hello_napi\\',
gyp verb cli '--verbose'
gyp verb cli ]
gyp info using [email protected]
gyp info using [email protected] | win32 | x64
gyp info chdir C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi\
gyp verb build type Release
gyp verb architecture x64
gyp verb node dev dir C:\Users\toyobayashi\AppData\Local\node-gyp\Cache\16.18.1
gyp verb python C:\Users\toyobayashi\AppData\Local\Programs\Python\Python310\python.exe
gyp verb `which` succeeded for `make` C:\Users\toyobayashi\app\make\make.EXE
gyp info spawn make
gyp info spawn args [ 'V=1', 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory 'C:/Users/toyobayashi/Projects/node-gyp/test/node_modules/hello_napi/build'
"C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/Llvm/x64/bin/clang.exe" -o Release/obj.target/hello/hello.o ../hello.c '-DNODE_GYP_MODULE_NAME=hello' '-DUSING_UV_SHARED=1' '-DUSING_V8_SHARED=1' '-DV8_DEPRECATION_WARNINGS=1' '-DV8_DEPRECATION_WARNINGS' '-DV8_IMMINENT_DEPRECATION_WARNINGS' '-D_GLIBCXX_USE_CXX11_ABI=1' '-D_LARGEFILE_SOURCE' '-D_FILE_OFFSET_BITS=64' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_NO_PINSHARED' '-DOPENSSL_THREADS' '-DBUILDING_NODE_EXTENSION' -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/include/node -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/src -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/openssl/config -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/openssl/openssl/include -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/uv/include -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/zlib -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/v8/include -pthread -Wall -Wextra -Wno-unused-parameter -m64 -O3 -fno-omit-frame-pointer -MMD -MF ./Release/.deps/Release/obj.target/hello/hello.o.d.raw -c
"C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/Llvm/x64/bin/clang++.exe" -o Release/obj.target/hello.node -shared -pthread -rdynamic -m64 -Wl,-soname=hello.node -Wl,--start-group Release/obj.target/hello/hello.o -Wl,--end-group
clang++: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
clang++: warning: argument unused during compilation: '-rdynamic' [-Wunused-command-line-argument]
LINK : warning LNK4044: 无法识别的选项“/soname=hello.node”;已忽略
LINK : warning LNK4044: 无法识别的选项“/-start-group”;已忽略
LINK : warning LNK4044: 无法识别的选项“/-end-group”;已忽略
正在创建库 Release/obj.target/hello.lib 和对象 Release/obj.target/hello.exp
hello.o : error LNK2019: 无法解析的外部符号 __imp_napi_module_register,函数 _register_hello 中引用了该符号
hello.o : error LNK2019: 无法解析的外部符号 __imp_napi_create_function,函数 napi_register_module_v1 中引用了该符号
hello.o : error LNK2019: 无法解析的外部符号 __imp_napi_set_named_property,函数 napi_register_module_v1 中引用了该符号
hello.o : error LNK2019: 无法解析的外部符号 __imp_napi_create_string_utf8,函数 hello 中引用了该符号
Release\obj.target\hello.node : fatal error LNK1120: 4 个无法解析的外部命令
clang++: error: linker command failed with exit code 1120 (use -v to see invocation)
make: *** [hello.target.mk:137: Release/obj.target/hello.node] Error 96
make: Leaving directory 'C:/Users/toyobayashi/Projects/node-gyp/test/node_modules/hello_napi/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.<anonymous> (C:\Users\toyobayashi\Projects\node-gyp\lib\build.js:216:23)
gyp ERR! stack at ChildProcess.emit (node:events:513:28)
gyp ERR! stack at Process.ChildProcess._handle.onexit (node:internal/child_process:293:12)
gyp ERR! System Windows_NT 10.0.22631
gyp ERR! command "C:\\Users\\toyobayashi\\app\\nodejs\\node.exe" "C:\\Users\\toyobayashi\\Projects\\node-gyp\\bin\\node-gyp.js" "build" "-C" "C:\\Users\\toyobayashi\\Projects\\node-gyp\\test\\node_modules\\hello_napi\\" "--verbose"
gyp ERR! cwd C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi
gyp ERR! node -v v16.18.1
gyp ERR! node-gyp -v v10.1.0
gyp ERR! not ok
More effort need to be done to the gyp's make generator, it's out of scope. I personally don't care whether it work for native, it's enough that work for wasm.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since gyp
can generate correct makefiles on windows now, could emnapi call gyp
directly if it has it's own gypi and configurations?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think most of the config in https://github.com/nodejs/node-gyp/blob/main/addon.gypi would not be applicable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since gyp can generate correct makefiles on windows now, could emnapi call gyp directly if it has it's own gypi and configurations?
Yes, indeed can do this. However, all my efforts on gyp are ultimately aimed at making it easier for Node.js native module developers to use node-gyp to compile to both native and wasm at the same time, instead of introducing another gyp wrapper or something like additional way to reuse there binding.gyp
configuration.
I added a minimal wasm gypi config to make the tests work, only targeting the change of this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you take another look at these changes, they are the final piece of the puzzle
rebuild
and build
when configure with -- -f make
on Windowsrebuild
and build
for cross-compiling Node-API module to wasm on Windows
} | ||
|
||
if (ext === '.js') { | ||
const emnapi = require('@emnapi/runtime') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could this section be removed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
okay, it has been removed
8d34941
to
0bb4f2a
Compare
Checklist
npm install && npm run lint && npm test
passesDescription of change
node-gyp rebuild
andnode-gyp build
doesn't work on windows if using-f make
to perform cross-compiling, this PR add support for it.Related: