Question about cross compile Node-API addon to wasm with emnapi and emscripten on Windows/macOS #2823

toyobayashi · 2023-04-01T17:29:12Z

Hi! I'm the creator of emnapi which is a Node-API implementation for emscripten/wasi-sdk, it can be used for porting existing Node-API native addon to WebAssembly. I'm wondering if node-gyp can be used for building wasm with emscripten, then I have a test repo https://github.com/toyobayashi/emnapi-node-gyp-test here, seems that node-gyp cross compiling with emscripten only works on Linux, both Windows and macOS are failed.

Node.js version: 18
npm version: 8

Steps to reproduce:

git clone https://github.com/toyobayashi/emnapi-node-gyp-test
cd emnapi-node-gyp-test
npm install --ignore-scripts
node-gyp clean
node-gyp configure --nodedir=./wasm -- -f make
emmake make -C build

Logs and reason:

Windows 11 22H2 22621.1413

Failed due to the generated Makefile still use unix tools, even if execute make under WSL 2 then failed due to \\ path seperator.

> npm install --ignore-scripts

add 3 packages, and audited 4 packages in 8s

found 0 vulnerabilities

> node-gyp clean
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | win32 | x64
gyp info ok

> node-gyp configure --nodedir=./wasm -- -f make
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | win32 | x64
gyp info find Python using Python version 3.9.7 found at "C:\Users\toyobayashi\AppData\Local\Programs\Python\Python39\python.exe"
gyp info find VS using VS2022 (17.2.32516.85) found at:
gyp info find VS "C:\Program Files\Microsoft Visual Studio\2022\Community"
gyp info find VS run with --verbose for detailed information
gyp WARN read config.gypi ENOENT: no such file or directory, open 'C:\Users\toyobayashi\Projects\emnapi-node-gyp-test\wasm\include\node\config.gypi'
gyp info spawn C:\Users\toyobayashi\AppData\Local\Programs\Python\Python39\python.exe
gyp info spawn args [
gyp info spawn args   'C:\\Users\\toyobayashi\\app\\nvm\\v18.12.1\\node_modules\\node-gyp\\gyp\\gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\toyobayashi\\Projects\\emnapi-node-gyp-test\\build\\config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\toyobayashi\\app\\nvm\\v18.12.1\\node_modules\\node-gyp\\addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\toyobayashi\\Projects\\emnapi-node-gyp-test\\wasm\\common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=./wasm',
gyp info spawn args   '-Dnode_gyp_dir=C:\\Users\\toyobayashi\\app\\nvm\\v18.12.1\\node_modules\\node-gyp',
gyp info spawn args   '-Dnode_lib_file=wasm\\\\$(Configuration)\\\\node.lib',
gyp info spawn args   '-Dmodule_root_dir=C:\\Users\\toyobayashi\\Projects\\emnapi-node-gyp-test',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'C:\\Users\\toyobayashi\\Projects\\emnapi-node-gyp-test\\build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
gyp info ok

> emmake make -C build
make: C:\Users\toyobayashi\app\make\mingw32-make.exe -C build
mingw32-make: Entering directory 'C:/Users/toyobayashi/Projects/emnapi-node-gyp-test/build'
process_begin: CreateProcess(NULL, printf %s\n "  CC(target) Release/obj.target/binding/node_modules\emnapi\src\js_native_api.o", ...) failed.
make (e=2): 系统找不到指定的文件。
mingw32-make: *** [binding.target.mk:110: Release/obj.target/binding/node_modules\emnapi\src\js_native_api.o] Error 2
mingw32-make: Leaving directory 'C:/Users/toyobayashi/Projects/emnapi-node-gyp-test/build'
emmake: error: 'C:\Users\toyobayashi\app\make\mingw32-make.exe -C build' failed (returned 2)

> wsl
$ emmake make -C build
make: make -C build
make: Entering directory '/mnt/c/Users/toyobayashi/Projects/emnapi-node-gyp-test/build'
make: *** No rule to make target '../wasm\common.gypi', needed by 'Makefile'.  Stop.
make: Leaving directory '/mnt/c/Users/toyobayashi/Projects/emnapi-node-gyp-test/build'
emmake: error: 'make -C build' failed (returned 2)

WSL 2 Ubuntu 20.04 (Success)

$ npm install --ignore-scripts

added 3 packages in 418ms

$ node-gyp clean
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | linux | x64
gyp info ok

$ node-gyp configure --nodedir=./wasm -- -f make
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | linux | x64
gyp info find Python using Python version 3.8.10 found at "/usr/bin/python3"
gyp WARN read config.gypi ENOENT: no such file or directory, open '/home/toyobayashi/Projects/emnapi-node-gyp-test/wasm/include/node/config.gypi'
gyp info spawn /usr/bin/python3
gyp info spawn args [
gyp info spawn args   '/home/toyobayashi/.nvm/versions/node/v18.14.0/lib/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/home/toyobayashi/Projects/emnapi-node-gyp-test/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/toyobayashi/.nvm/versions/node/v18.14.0/lib/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/toyobayashi/Projects/emnapi-node-gyp-test/wasm/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=./wasm',
gyp info spawn args   '-Dnode_gyp_dir=/home/toyobayashi/.nvm/versions/node/v18.14.0/lib/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=wasm/$(Configuration)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/home/toyobayashi/Projects/emnapi-node-gyp-test',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
gyp info ok

$ emmake make -C build
make: make -C build
make: Entering directory '/home/toyobayashi/Projects/emnapi-node-gyp-test/build'
  CC(target) Release/obj.target/binding/node_modules/emnapi/src/js_native_api.o
  CC(target) Release/obj.target/binding/node_modules/emnapi/src/node_api.o
  CC(target) Release/obj.target/binding/node_modules/emnapi/src/emnapi.o
  CC(target) Release/obj.target/binding/node_modules/emnapi/src/async_cleanup_hook.o
  CC(target) Release/obj.target/binding/node_modules/emnapi/src/async_context.o
  CC(target) Release/obj.target/binding/node_modules/emnapi/src/async_work.o
  CC(target) Release/obj.target/binding/node_modules/emnapi/src/threadsafe_function.o
  CC(target) Release/obj.target/binding/node_modules/emnapi/src/uv/uv-common.o
  CC(target) Release/obj.target/binding/node_modules/emnapi/src/uv/threadpool.o
  CC(target) Release/obj.target/binding/node_modules/emnapi/src/uv/unix/loop.o
  CC(target) Release/obj.target/binding/node_modules/emnapi/src/uv/unix/thread.o
  CC(target) Release/obj.target/binding/node_modules/emnapi/src/uv/unix/async.o
  CC(target) Release/obj.target/binding/node_modules/emnapi/src/uv/unix/core.o
  CC(target) Release/obj.target/binding/src/binding.o
  LINK(target) Release/binding.js
em++: warning: USE_PTHREADS + ALLOW_MEMORY_GROWTH may run non-wasm code slowly, see https://github.com/WebAssembly/design/issues/1271 [-Wpthreads-mem-growth]
cache:INFO: generating system asset: symbol_lists/7f16a0cf42f34d504060010d577a8705337deb0d.txt... (this will be cached in "/home/toyobayashi/Projects/emsdk/upstream/emscripten/cache/symbol_lists/7f16a0cf42f34d504060010d577a8705337deb0d.txt" for subsequent builds)
cache:INFO:  - ok
make: Leaving directory '/home/toyobayashi/Projects/emnapi-node-gyp-test/build'

macOS 13.2.1 (M2 Pro)

Failed due to emcc doesn't support -arch arm64 compile flag.

$ npm install --ignore-scripts

added 3 packages in 593ms

$ node-gyp clean
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | darwin | arm64
gyp info ok

$ node-gyp configure --nodedir=./wasm -- -f make
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | darwin | arm64
gyp info find Python using Python version 3.9.6 found at "/Library/Developer/CommandLineTools/usr/bin/python3"
gyp WARN read config.gypi ENOENT: no such file or directory, open '/Users/toyobayashi/code/github/emnapi-node-gyp-test/wasm/include/node/config.gypi'
gyp info spawn /Library/Developer/CommandLineTools/usr/bin/python3
gyp info spawn args [
gyp info spawn args   '/Users/toyobayashi/.nvm/versions/node/v18.12.1/lib/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/toyobayashi/code/github/emnapi-node-gyp-test/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/toyobayashi/.nvm/versions/node/v18.12.1/lib/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/toyobayashi/code/github/emnapi-node-gyp-test/wasm/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=./wasm',
gyp info spawn args   '-Dnode_gyp_dir=/Users/toyobayashi/.nvm/versions/node/v18.12.1/lib/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=wasm/$(Configuration)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/Users/toyobayashi/code/github/emnapi-node-gyp-test',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
gyp info ok

$ emmake make -C build
make: make -C build
  CC(target) Release/obj.target/binding/node_modules/emnapi/src/js_native_api.o
emcc: error: arm64: No such file or directory ("arm64" was expected to be an input file, based on the commandline arguments provided)
make: *** [Release/obj.target/binding/node_modules/emnapi/src/js_native_api.o] Error 1
emmake: error: 'make -C build' failed (returned 2)

Is there any way to workaround such gyp default behavior on different OS? Does it require upstream gyp code change to support? Or is there any way to solve these problem via configuration or some command line flags?

The text was updated successfully, but these errors were encountered:

toyobayashi · 2024-01-08T12:48:02Z

anyone could help?

mhdawson · 2024-01-08T21:58:02Z

@toyobayashi just noticed this issue. May not be the right group to discuss the issues around node-gyp but if you come to the Node-api team meeting on Friday at 11 ET (https://zoom.us/j/363665824
https://hangouts.google.com/hangouts/_/event/c0eevtrlajniu7h8cjrdk0f56c8

I think we'd be intersted in better understanding what you are trying to do and the current issues. Might be one way to find people who might be interested in helping.

toyobayashi · 2024-01-09T07:02:17Z

@mhdawson Thanks for your invitation! But I'm sorry I think my English listening and speaking skills are not enough for me to attend the meeting.

I just want to find a way that make node-gyp can work fine on win/linux/mac for emscripten target, currently it only works on linux via --nodedir and -- -f make.

mhdawson · 2024-01-09T23:20:19Z

I'm not sure if its a reasonable suggestion but I think @legendecas is in the same region as you and maybe you two could talk more easily? Just a suggestions as I do know there are still different languages in the region.

One thing I wondered was how hard the dependency on node-gyp was versus using something like cmake which I know is also a possiblity for node-addon-api modules.

toyobayashi · 2024-01-10T03:07:46Z

@mhdawson CMake is easy to integrate cause emscripten has already provided the toolchain file, while node-gyp no. emnapi support user to use CMake build Node-API wasm module. On the other hand, not only does node-gyp not have a toolchain config file for emscripten, the generator will also generate platform-specific compilation options based on the operating system, for example on Windows it will generate VS solution by default, even if I tell it generate makefile, the path separator is incorrect, and on macOS it generates -arch arm64 which doesn't work in emcc. I don't know how to control these behavior.

Although CMake can indeed easily build Node-API wasm modules, most native modules in Node.js community are built using node-gyp. For example, sharp is using emnapi to build wasm that can run on StackBlitz, they still prefer using node-gyp (https://github.com/lovell/sharp/blob/main/src/emscripten/common.gypi) instead of maintaining another CMake configuration for emnapi. So I think it is so nice if node-gyp can work on all platform cause this can reduce the cost for native module maintainers to compile Node-API into wasm modules.

@legendecas Are you interested in taking a look?

legendecas · 2024-01-10T05:52:36Z

I am not familiar with windows builds but I would try this out on mac. Also, please feel free to message me on OpenJS Slack.

toyobayashi · 2024-01-11T10:58:06Z

@legendecas The link seems to be broken.

Just now I found out how to make it work on mac, pushed emnapi-node-gyp-test. I added these environment variables and place link flags in xcode_settings.OTHER_LDFLAGS then it work.

GYP_CROSSCOMPILE=1
AR_host=ar
CC_host=clang
CXX_host=clang++
AR_target=emar
CC_target=emcc
CXX_target=em++

emmake node-gyp rebuild --verbose --nodedir=./wasm -- -f make

legendecas · 2024-01-12T03:38:36Z

Does the same trick address the problem on Windows as well?

legendecas · 2024-01-12T03:39:49Z

For what it's worth, I can build with https://github.com/toyobayashi/emnapi-node-gyp-test/blob/main/build.sh with following patch:

export GYP_CROSSCOMPILE=1
export AR_host=ar
export CC_host=clang
export CXX_host=clang++
export AR_target=emar
export CC_target=emcc
export CXX_target=em++

emmake node-gyp rebuild --verbose --nodedir=./wasm -- -f make

toyobayashi · 2024-01-12T09:43:25Z

Does the same trick address the problem on Windows as well?

No, makefile generator generates linux commands which is not supported on windows. I tried to use Cygwin but the path separator "" generated in makefile doesn't work.

toyobayashi · 2024-01-15T14:14:25Z

After some hard trying, finally build successfully on Windows with some tricks. Pushed emnapi-node-gyp-test.

Install make and cygwin
npm install -D [email protected]

Modify node_modules/node-gyp/gyp/pylib/gyp/generator/make.py, line 862

- self.output = self.output_binary = self.ComputeOutput(spec)
+ self.output = self.output_binary = self.ComputeOutput(spec).replace('\\', '/')

build.bat

@echo off

set GYP_CROSSCOMPILE=1
set AR_target=emar.bat
set CC_target=emcc.bat
set CXX_target=em++.bat

call npx.cmd node-gyp clean
call npx.cmd node-gyp configure --nodedir=./wasm -- -f make

node ./scripts/replace-sep.js

call emmake.bat make -C build V=1

scripts/replace-sep.js

const path = require('path')
const fs = require('fs')

const buildDir = path.join(__dirname, '../build')

fs.readdirSync(buildDir)
  .filter(p => p.endsWith('.target.mk'))
  .map((p) => path.join(buildDir, p))
  .map((p) => {
    const content = fs.readFileSync(p, 'utf8').replace(/\\|\\\\/g, '/').replace(/\/(\r?\n)/g, '\\$1')
    fs.writeFileSync(p, content, 'utf8')
  })

Run build.bat in Cygwin terminal.

This is just for such a simple example, if for a more complex project, I think there will be more gyp source code that needs to be modified.

Can gyp consider make generator/make.py generate correct path on windows? Or add nmake generator?

toyobayashi · 2024-01-28T16:31:43Z

Opened these PR:

My test https://github.com/toyobayashi/emnapi-node-gyp-test/actions looks ok with the patches.

rey-ms · 2024-04-04T08:06:31Z

Hello @toyobayashi I had a similar issue trying to install emnapi on Windows. At the end I just installed it with Windows Powershell and I had not issue. It's a workaround, but it works.

This was referenced Jan 15, 2024

node-gyp support? toyobayashi/emnapi#23

Closed

fix: support cross compiling for wasm with make generator nodejs/gyp-next#222

Merged

This was referenced Jan 26, 2024

feat: define common_gypi_dir when running node-gyp configure #2973

Closed

Support rebuild and build for cross-compiling Node-API module to wasm on Windows #2974

Merged

cclauss added the wasm label Mar 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Question about cross compile Node-API addon to wasm with emnapi and emscripten on Windows/macOS #2823

Question about cross compile Node-API addon to wasm with emnapi and emscripten on Windows/macOS #2823

toyobayashi commented Apr 1, 2023

toyobayashi commented Jan 8, 2024

mhdawson commented Jan 8, 2024

toyobayashi commented Jan 9, 2024

mhdawson commented Jan 9, 2024

toyobayashi commented Jan 10, 2024

legendecas commented Jan 10, 2024 •

edited

Loading

toyobayashi commented Jan 11, 2024

legendecas commented Jan 12, 2024

legendecas commented Jan 12, 2024

toyobayashi commented Jan 12, 2024

toyobayashi commented Jan 15, 2024 •

edited

Loading

toyobayashi commented Jan 28, 2024

rey-ms commented Apr 4, 2024

Question about cross compile Node-API addon to wasm with emnapi and emscripten on Windows/macOS #2823

Question about cross compile Node-API addon to wasm with emnapi and emscripten on Windows/macOS #2823

Comments

toyobayashi commented Apr 1, 2023

toyobayashi commented Jan 8, 2024

mhdawson commented Jan 8, 2024

toyobayashi commented Jan 9, 2024

mhdawson commented Jan 9, 2024

toyobayashi commented Jan 10, 2024

legendecas commented Jan 10, 2024 • edited Loading

toyobayashi commented Jan 11, 2024

legendecas commented Jan 12, 2024

legendecas commented Jan 12, 2024

toyobayashi commented Jan 12, 2024

toyobayashi commented Jan 15, 2024 • edited Loading

toyobayashi commented Jan 28, 2024

rey-ms commented Apr 4, 2024

legendecas commented Jan 10, 2024 •

edited

Loading

toyobayashi commented Jan 15, 2024 •

edited

Loading