fix: add Large Payload Support via Temporary File Transport in call-step-file and node-runner #776

riturajFi · 2025-10-06T06:15:31Z

Overview

This PR introduces a robust solution to handle large request payloads (JSON or otherwise) without changing existing data types or introducing environment flags.
Previously, Motia serialized the entire request body into a single JSON string argument passed via argv to the runner. This caused OS-level E2BIG errors for large payloads (typically > 2–8MB).

The new design uses temporary files to transfer large data between the main process and language runners (Node, Python, Ruby), while preserving backward compatibility for smaller payloads.

Key Changes
🧠 call-step-file.ts

Added detection for payload size before spawning the runner.

When data exceeds a threshold (e.g., >1MB):

Writes the full event object to a secure temporary JSON file (in os.tmpdir()).

Passes only the path of this file to the runner via argv.

On completion or error, the temp directory is safely deleted to prevent disk bloat.

Maintains existing behavior for smaller requests to minimize overhead.

⚙️ node-runner.ts

Detects whether the received argument is:

A file path → loads JSON from file, executes handler, and deletes the file.

A JSON string → uses legacy flow (unchanged behavior).

Adds cleanup for temporary files even if the process is interrupted.

Maintains identical interface for user handlers — no breaking changes.

Benefits

✅ Removes OS argument size limitations (supports 100MB+ payloads).
✅ Preserves existing APIs, handlers, and request/response structures.
✅ Reduces memory duplication from repeated JSON serialization.
✅ Ensures secure, auto-cleaned temp directories (mode 0700, file mode 0600).
✅ Fully backward compatible and cross-language ready.

Testing Performed

Verified:

Small payloads (<1MB) → use legacy in-memory mode.

Large JSON payloads (5MB–200MB) → run successfully without E2BIG.

Automatic cleanup of temp files after step completion.

Regression-tested standard steps and event flows.

Next Steps

Add optional streaming support for extremely large binary bodies.

Integrate with multipart/form-data in future versions.

🎥 Demo:

Before -

Screencast.from.2025-10-06.10-33-33.webm

After -

Screencast.from.2025-10-06.10-39-10.webm

vercel · 2025-10-06T06:15:36Z

@riturajFi is attempting to deploy a commit to the motia Team on Vercel.

A member of the Team first needs to authorize it.

sergiofilhowz · 2025-10-06T12:06:32Z

packages/core/src/call-step-file.ts

      .catch((error) => {
+        try {
+          // spawn failed before handlers attached — still clean up
+          if (tempDir) fs.rmSync(tempDir, { recursive: true, force: true })


this can lead to concurrency issues, one process deleting the temp dir with files of other processes

sergiofilhowz · 2025-10-06T12:07:32Z

packages/core/src/call-step-file.ts

+    const cleanupTemp = () => {
+      if (!tempDir) return
+      try {
+        fs.rmSync(tempDir, { recursive: true, force: true })


we need to make it async, we're having some issues with synchronous methods already that I need to fix

packages/core/src/node/node-runner.ts

sergiofilhowz

Thanks @riturajFi!!

I like the approach, but we need a few things before we merge:

We need to implement it in all runtimes we have in codebase, such as Python and Ruby
We need automated test to validate it

riturajFi · 2025-10-07T03:07:10Z

Thanks @sergiofilhowz for your comments! I just want to validate whether the approach is correct. I shall implement this for all the runners (+ write an implementation prompt for future runners)
And write automted tests.

riturajFi · 2025-10-14T06:24:25Z

Hi @sergiofilhowz , implemented the changes you asked for -

made the file deletion async
made the cleaning step idempotent to prevent race condition of two processes deleting and accessing the same file
wrote tests for the feature

sergiofilhowz · 2025-10-16T12:48:55Z

This is good! I'm going to merge soon

packages/core/src/__tests__/call-step-file.large-payload.python.test.ts

packages/core/src/call-step-file.ts

packages/core/src/node/node-runner.ts

sergiofilhowz · 2025-10-16T12:52:06Z

@riturajFi thanks for the PR, I want to merge it, but first we need to fix the pipeline issues

riturajFi · 2025-10-20T10:05:16Z

@sergiofilhowz can you please review this? I fixed some minor concerns that were there. Can we merge it now?

sergiofilhowz · 2025-10-21T09:26:30Z

packages/core/src/call-step-file.ts

+        processManager.onProcessClose(async (code) => {
          if (timeoutId) clearTimeout(timeoutId)
          processManager.close()
+          await cleanupTemp()


this will lead to race conditions:

When two requests happen at the same time, one can delete the folder while the other tries to read it

sergiofilhowz · 2025-10-21T09:27:02Z

packages/core/src/call-step-file.ts

+    const cleanupTemp = async () => {
+      if (isCleaning || isCleaned || !tempDir) return
+      const dir = tempDir
+      tempDir = undefined
+      isCleaning = true
+
+      try {
+        await fs.promises.rm(dir, { recursive: true, force: true })
+      } catch {}
+
+      isCleaning = false
+      isCleaned = true
+
+    }


it needs to delete only the file that was created for the process, otherwise there will be race conditions

sergiofilhowz · 2025-10-21T14:16:41Z

packages/core/src/call-step-file.ts

+      tempDir = path.join(os.tmpdir(), `motia-${uniqueId}`)
+      fs.mkdirSync(tempDir, { recursive: true, mode: 0o700 })
+      metaPath = path.join(tempDir, 'meta.json')
+      fs.writeFileSync(metaPath, jsonData, { mode: 0o600 })


why not create a single file instead?

sergiofilhowz · 2025-10-21T14:17:14Z

packages/core/src/call-step-file.ts

+    let argvPayload = jsonData
+    let tempDir: string | undefined
+    let metaPath: string | undefined
+    // This is for keeping the cleanup idempotent
+    let isCleaned: boolean = false
+    let isCleaning: boolean = false


avoid usage of let

sergiofilhowz · 2025-10-21T14:17:59Z

packages/core/src/call-step-file.ts

+        // spawn failed before handlers attached — still clean up (async, best-effort)
+        if (tempDir) {
+          const dir = tempDir
+          tempDir = undefined
+          void fs.promises.rm(dir, { recursive: true, force: true })
+          .catch((err) => {logger.debug(`temp cleanup: ${dir}: ${err.message ?? err}`)})
+        }


why not run it at finally?

github-actions · 2025-10-23T14:06:36Z

📦 This PR is large (>500 lines). Please ensure it has been properly tested.

packages/core/src/call-step-file.ts

+      const uniqueId = `${process.pid}-${crypto.randomBytes(6).toString('hex')}`
+      const metaPath = path.join(baseTempDir, `meta-${uniqueId}.json`)
+      payloadState.tempFilePath = metaPath
+      fs.writeFileSync(metaPath, jsonData, { mode: 0o600 })


packages/e2e/scripts/global-setup.ts

packages/e2e/scripts/pr-global-setup.ts

packages/core/src/node/node-runner.ts

packages/core/src/call-step-file.ts

riturajFi · 2025-11-05T06:38:25Z

Hi @sergiofilhowz , the changes have been implemented according to your suggestion. Can you kindly review this?

sergiofilhowz reviewed Oct 6, 2025

View reviewed changes

packages/core/src/node/node-runner.ts Show resolved Hide resolved

sergiofilhowz requested changes Oct 6, 2025

View reviewed changes

riturajFi force-pushed the feat/large-file-upload-support branch from 057f3f6 to bad4567 Compare October 13, 2025 17:33

github-actions bot added the size/M label Oct 13, 2025

riturajFi force-pushed the feat/large-file-upload-support branch from bad4567 to 91289ea Compare October 14, 2025 06:22

riturajFi requested a review from sergiofilhowz October 14, 2025 06:24

github-advanced-security bot found potential problems Oct 16, 2025

View reviewed changes

packages/core/src/__tests__/call-step-file.large-payload.python.test.ts Fixed Show fixed Hide fixed

packages/core/src/call-step-file.ts Fixed Show fixed Hide fixed

packages/core/src/node/node-runner.ts Fixed Show fixed Hide fixed

riturajFi force-pushed the feat/large-file-upload-support branch from 64a8820 to 7385044 Compare October 19, 2025 19:54

riturajFi changed the title ~~Add Large Payload Support via Temporary File Transport in call-step-file and node-runner~~ fix: Add Large Payload Support via Temporary File Transport in call-step-file and node-runner Oct 19, 2025

riturajFi changed the title ~~fix: Add Large Payload Support via Temporary File Transport in call-step-file and node-runner~~ fix: add Large Payload Support via Temporary File Transport in call-step-file and node-runner Oct 20, 2025

sergiofilhowz reviewed Oct 21, 2025

View reviewed changes

riturajFi force-pushed the feat/large-file-upload-support branch from e46df4f to f0a7bea Compare October 21, 2025 11:15

sergiofilhowz reviewed Oct 21, 2025

View reviewed changes

riturajFi force-pushed the feat/large-file-upload-support branch 2 times, most recently from 14f0fc6 to 55edeb2 Compare October 23, 2025 14:06

github-actions bot added the size/L label Oct 23, 2025

github-advanced-security bot found potential problems Oct 23, 2025

View reviewed changes

packages/core/src/node/node-runner.ts Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Oct 23, 2025

View reviewed changes

packages/core/src/call-step-file.ts Fixed Show fixed Hide fixed

github-actions bot added the size/XL label Oct 24, 2025

riturajFi added 25 commits November 1, 2025 14:38

feat: allow large files to be in the request

eb012fc

minor changes

dde4e19

minor change

18e0c26

async fix

f108f92

tests

6165ae5

minor changes to be deleted later

64048dd

minor test fixes

f57a85b

minor fix

da4ca69

minor fix

116e36d

fixed race condition

9c40221

fix cleanup

35391a5

fixed some clean temp file

0019b54

race condition fix

3b360ac

minor

ae3ec2c

minor fixes

8db0e3d

temp change

4c2dc25

automated tests

6116e5d

minro change

6da00ab

lint issue solved

2dc7939

erro handling

99ab0d6

CodeQL fixes

a8e8c55

minor fix

96dd4d1

test fixes

6be4243

minor

f6b8580

Ruby RPC fix

7b9535a

riturajFi force-pushed the feat/large-file-upload-support branch from ffd4323 to 7b9535a Compare November 1, 2025 14:01

riturajFi requested a review from sergiofilhowz November 3, 2025 12:20

fix: add Large Payload Support via Temporary File Transport in call-step-file and node-runner #776

Are you sure you want to change the base?

fix: add Large Payload Support via Temporary File Transport in call-step-file and node-runner #776

Uh oh!

Conversation

riturajFi commented Oct 6, 2025

Uh oh!

vercel bot commented Oct 6, 2025

Uh oh!

sergiofilhowz Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

sergiofilhowz Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sergiofilhowz left a comment

Choose a reason for hiding this comment

Uh oh!

riturajFi commented Oct 7, 2025

Uh oh!

riturajFi commented Oct 14, 2025

Uh oh!

sergiofilhowz commented Oct 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sergiofilhowz commented Oct 16, 2025

Uh oh!

riturajFi commented Oct 20, 2025

Uh oh!

sergiofilhowz Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

sergiofilhowz Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

sergiofilhowz Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

sergiofilhowz Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

sergiofilhowz Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 23, 2025

Uh oh!

Check warning

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

riturajFi commented Nov 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants