Site larger than 2GB can't be exported using All In One WP Migration

[wojtekn]

Quick summary

After upgrading Playground to 1.0.29 to add 64bit int support, I've tried exporting a site larger than 2GB using All In One WP Migration. It failed with error.

Steps to reproduce

1. Use a branch with Playground 1.0.29
2. Create a Studio site
3. Open WP admin
4. Install All In One WP Migration plugin
5. Upload >2GB files to the site
6. Export site to file using the plugin

What you expected to happen

Site is exported and file is downloaded.

What actually happened

Nothing happens in UI, console shows Array buffer allocation failed error.

Impact

Some (< 50%)

Available workarounds?

No but the app is still usable

Platform

Mac Silicon

Logs or notes

Exception trace:

Trace: Error: Array buffer allocation failed
    at PHP.ae (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:197223:8479)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async PHP.run (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:197223:1726)
    at async PHPRequestHandler.pe (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:197229:6613)
    at async PHPRequestHandler.ce (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:197229:6221) {
  cause: Error: Rethrown
      at UnhandledRejectionsTarget.e (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:197223:7927)
      at [nodejs.internal.kHybridDispatch] (node:internal/event_target:820:20)
      at UnhandledRejectionsTarget.dispatchEvent (node:internal/event_target:755:26)
      at t.wasmExports.<computed> (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:197174:12288)
      at invoke_iiiii (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:159590:14)
      at php.wasm.php_execute_script (wasm://wasm/php.wasm-038659a6:wasm-function[11334]:0x6489d1)
      at php.wasm.dynCall_ii (wasm://wasm/php.wasm-038659a6:wasm-function[13262]:0x720f3a)
      at ret.<computed> (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:158897:22)
      at t.wasmExports.<computed> (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:197174:12070)
      at t.wasmExports.<computed> (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:197174:12070)
      at invoke_ii (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:159502:14)
      at php.wasm.wasm_sapi_handle_request (wasm://wasm/php.wasm-038659a6:wasm-function[9835]:0x5cbc94)
      at ret.<computed> (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:158897:22)
      at t.wasmExports.<computed> (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:197174:12070)
      at t.wasmExports.<computed> (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:197174:12070)
      at Module._wasm_sapi_handle_request (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:159403:164) {
    cause: RangeError: Array buffer allocation failed
        at new ArrayBuffer (<anonymous>)
        at new Uint8Array (<anonymous>)
        at Object.expandFileStorage (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:153415:23)
        at Object.write (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:153559:15)
        at Object.write (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:155276:44)
        at doWritev (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:157968:21)
        at _fd_write (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:157984:17)
        at php.wasm.write (wasm://wasm/php.wasm-038659a6:wasm-function[960]:0x8f5c8)
        at php.wasm.wasm_sapi_ub_write (wasm://wasm/php.wasm-038659a6:wasm-function[9826]:0x5cb25f)
        at php.wasm.php_output_op (wasm://wasm/php.wasm-038659a6:wasm-function[3923]:0x2931cc)
        at php.wasm.php_output_write (wasm://wasm/php.wasm-038659a6:wasm-function[172]:0xe1b0)
        at php.wasm.ZEND_ECHO_SPEC_CV_HANDLER (wasm://wasm/php.wasm-038659a6:wasm-function[14657]:0x78898a)
        at php.wasm.execute_ex (wasm://wasm/php.wasm-038659a6:wasm-function[13071]:0x7068ed)
        at php.wasm.zend_execute (wasm://wasm/php.wasm-038659a6:wasm-function[3520]:0x248779)
        at php.wasm.zend_execute_scripts (wasm://wasm/php.wasm-038659a6:wasm-function[9439]:0x5b37f5)
        at php.wasm.dynCall_iiiii (wasm://wasm/php.wasm-038659a6:wasm-function[13259]:0x720ce2),
    betterMessage: 'Array buffer allocation failed'
  }
}
    at /Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:80778:99
    at Generator.throw (<anonymous>)
    at rejected (/Users/USER/Sites/studio/.webpack/main/siteServerProcess.js:80683:65)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

[adamziel]

Nothing in expandFileStorage points at Playground 64bit support issues.

However, this line points at a few other potential problems:

node.contents = new Uint8Array(newCapacity);

1. We are doubling the in-memory file storage. Studio should run such space-intense these operations on disk, not in memory. We need to identify why MEMFS is targeted instead of NODEFS. Perhaps allinone migrations create a tmp directory somewhere like /tmp which is a MEMFS mount?
2. Unrelated – Studio may be running a 32bit version of Node.js where you can't allocate more than 4GB of memory. Once that's addressed we may need to rebuild Playground in MEMORY64 mode to allow more than 4GB of memory allocation, but I want to clearly state that's not the exception we're seeing right now.