Nodejs StartSocketConnection is not a function #3195

phawxby · 2025-02-17T15:16:28Z

Inquiry

We're using esbuild to bundle and minify handler code for AWS Lambdas producing this kind of output.

-rw-r--r--@ 1 paulhawxby  staff  1852129 Jan  1  1980 eventData.js
-rw-r--r--@ 1 paulhawxby  staff  4081730 Feb 17 13:24 eventData.js.map
-rwxr-xr-x@ 1 paulhawxby  staff  4800864 Feb 17 13:24 glide-rs.darwin-arm64.node
-rwxr-xr-x@ 1 paulhawxby  staff  4949752 Feb 17 13:24 glide-rs.linux-arm64-gnu.node
-rwxr-xr-x@ 1 paulhawxby  staff  4986584 Feb 17 13:24 glide-rs.linux-arm64-musl.node
-rwxr-xr-x@ 1 paulhawxby  staff  5756848 Feb 17 13:24 glide-rs.linux-x64-gnu.node
-rwxr-xr-x@ 1 paulhawxby  staff  5752728 Feb 17 13:24 glide-rs.linux-x64-musl.node

When executed on AWS we get the following error.

{
  "name": "TypeError",
  "location": "/var/task/eventData.js:13",
  "message": "(0 , nT.StartSocketConnection) is not a function",
  "stack": "TypeError: (0 , nT.StartSocketConnection) is not a function\n    at GlideClient.<anonymous> (/var/task/eventData.js:13:66296)\n    at Generator.next (<anonymous>)\n    at /var/task/eventData.js:13:29846\n    at new Promise (<anonymous>)\n    at F (/var/task/eventData.js:13:29598)\n    at GlideClient.createClientInternal (/var/task/eventData.js:13:66224)\n    at GlideClient.<anonymous> (/var/task/eventData.js:13:68101)\n    at Generator.next (<anonymous>)\n    at /var/task/eventData.js:13:67321\n    at new Promise (<anonymous>)"
}

I absolutely cannot figure out where this error is coming from.

    "@valkey/valkey-glide": "^1.2.1",
    "@valkey/valkey-glide-darwin-arm64": "^1.2.1",
    "@valkey/valkey-glide-linux-musl-arm64": "^1.2.1",
    "@valkey/valkey-glide-linux-arm64": "^1.2.1",
    "@valkey/valkey-glide-linux-x64": "^1.2.1",
    "@valkey/valkey-glide-linux-musl-x64": "^1.2.1",

Language

Typescript

Language Version

5.1.6

Engine Version

Node 22

Operating System

ARM & x86

Additional Technical Information

No response

The text was updated successfully, but these errors were encountered:

avifenesh · 2025-02-17T15:38:45Z

Hi @phawxby :)
Are you external glide in the serverless.yml file?
Can you please share it?

avifenesh · 2025-02-17T15:49:28Z

Having all those:

-rw-r--r--@ 1 paulhawxby  staff  1852129 Jan  1  1980 eventData.js
-rw-r--r--@ 1 paulhawxby  staff  4081730 Feb 17 13:24 eventData.js.map
-rwxr-xr-x@ 1 paulhawxby  staff  4800864 Feb 17 13:24 glide-rs.darwin-arm64.node
-rwxr-xr-x@ 1 paulhawxby  staff  4949752 Feb 17 13:24 glide-rs.linux-arm64-gnu.node
-rwxr-xr-x@ 1 paulhawxby  staff  4986584 Feb 17 13:24 glide-rs.linux-arm64-musl.node
-rwxr-xr-x@ 1 paulhawxby  staff  5756848 Feb 17 13:24 glide-rs.linux-x64-gnu.node
-rwxr-xr-x@ 1 paulhawxby  staff  5752728 Feb 17 13:24 glide-rs.linux-x64-musl.node

Means that you are bundling them for browser env, while those are native packages, and you can't bundle all, they have to run on node env only. Or deno/bun etc., but on server env.

Your severless.yml file should look something like this:

service: "esbuild-with-glide"
provider:
  name: aws
  runtime: nodejs20.x
build:
  esbuild: false
plugins:
  - serverless-esbuild
custom:
  esbuild:
    bundle: true
    minify: true
    sourcemap: true
    target: node20
    platform: node
    packager: 'npm'
    keepOutputDirectory: false
    loader:
      ".node": "copy"
    external:
      - "@valkey/valkey-glide"
      - "@valkey/valkey-glide-linux-musl-x64"
      - "@valkey/valkey-glide-linux-x64"
      - "glide-rs"
    exclude:
      - 'node_modules/@valkey/valkey-glide-linux-musl-x64/*'
    tsconfig: './tsconfig.json'
    outdir: '.esbuild/.serverless'
functions:
  hello:
    handler: index.handler
package:
      individually: true

avifenesh · 2025-02-17T16:00:32Z

@raczmarton do you have the same issue?

phawxby · 2025-02-17T16:04:45Z

@avifenesh thanks for the quick response. This is our esbuild config.

  const config = {
    entryPoints,
    keepNames: true,
    bundle: true,
    minify: false,
    sourcemap: true,
    platform: 'node',
    target: 'node22',
    outdir,
    loader: {
      '.node': 'file'
    },
    assetNames: '[name]',
    external: [
      '@aws-sdk/client-*',
      '@mikro-orm/mongodb',
      '@mikro-orm/mysql',
      '@mikro-orm/mariadb',
      '@mikro-orm/entity-generator',
      '@mikro-orm/migrations',
      '@mikro-orm/sqlite',
      'typeorm',
      'slonik',
      'tedious',
      'better-sqlite3',
      'oracledb',
      'pg-native',
      'mysql',
      'mysql2',
      'sqlite3',
      'pg-query-stream',
      'p-limit'
    ],
    plugins: [ddPlugin]
  };

I'll add glide to the externals and set the pipeline going and get back to you.

avifenesh · 2025-02-17T16:08:57Z

@phawxby Thanks. Note that if you build on ARM architecture, and deploy to Lambda, which is x86, you are going to have it fail.
The core is rust-based, and it compiles base on arch and being pull base on your OS and arch behind the scene.
As you can see here:

    "@valkey/valkey-glide": "^1.2.1",
    "@valkey/valkey-glide-darwin-arm64": "^1.2.1",
    "@valkey/valkey-glide-linux-musl-arm64": "^1.2.1",
    "@valkey/valkey-glide-linux-arm64": "^1.2.1",
    "@valkey/valkey-glide-linux-x64": "^1.2.1",
    "@valkey/valkey-glide-linux-musl-x64": "^1.2.1",

If this is your case, please let me know and ill give you a workaround.

phawxby · 2025-02-17T16:27:37Z

@avifenesh ah so our monorepo has a mixture of x86 and ARM lambda's but we have general purpose build process we use for everything that runs on x86 in CircleCI. What do you suggest?

avifenesh · 2025-02-17T16:46:41Z

The easiest way is to download separately the parts with Rust:

    // For ARM
    "@valkey/valkey-glide-linux-arm64": "^1.2.1",
    // you will have those anyway when you will install glide on x86:
    "@valkey/valkey-glide-linux-x64": "^1.2.1",
    // You will have it, but you dont need it unless you run on alpine:
    "@valkey/valkey-glide-linux-musl-x64": "^1.2.1",

Then on each platform all of them will be available, and the code is knowing to choose base on the env it runs on.
(BTW there's 1.3 already)
But since Rust binaries (in general) are heavy, I recommend stripping all the ones you don't need.
First — if you don't run musl, strip it.
If you know to distinguish between the lambda's that will run on ARM:

    exclude:
      - 'node_modules/@valkey/valkey-glide-linux-musl-x64/*'

And vice versa.
On generic server It's not so much of a problem, but I guess on lambda you prefer to be thin.

If you absolutely don't have how to separate, have both @valkey/valkey-glide-linux-musl-x64 and @valkey/valkey-glide-linux-arm64. But still exclude MUSL.
Until NPM 11, which just released, NPM didn't know to distinguish between musl and glibc, so you get both when you install glide.

phawxby · 2025-02-17T17:00:35Z

@avifenesh that's what I thought we were doing with this.

-rw-r--r--@ 1 paulhawxby  staff  1852129 Jan  1  1980 eventData.js
-rw-r--r--@ 1 paulhawxby  staff  4081730 Feb 17 13:24 eventData.js.map
-rwxr-xr-x@ 1 paulhawxby  staff  4800864 Feb 17 13:24 glide-rs.darwin-arm64.node
-rwxr-xr-x@ 1 paulhawxby  staff  4949752 Feb 17 13:24 glide-rs.linux-arm64-gnu.node
-rwxr-xr-x@ 1 paulhawxby  staff  4986584 Feb 17 13:24 glide-rs.linux-arm64-musl.node
-rwxr-xr-x@ 1 paulhawxby  staff  5756848 Feb 17 13:24 glide-rs.linux-x64-gnu.node
-rwxr-xr-x@ 1 paulhawxby  staff  5752728 Feb 17 13:24 glide-rs.linux-x64-musl.node

I was just including all the variants so that the code to decide which to import base on the environment

avifenesh · 2025-02-17T17:03:45Z

That's fine, it works, but it's heavy, that's all.
If it's not a problem, great.
Some users find it problematic.

It's heavy compared to node packages, not GB.
Maybe 5 MB.
Hopefully until 1.4 we will cut most of it, it is under work.
But Node.js using rust binary is always heavier than usual node packages.

phawxby · 2025-02-17T17:15:09Z

@avifenesh so that's the thing then. We're including all the native .node files yet it's still not running with the error above.

avifenesh · 2025-02-17T17:16:42Z

Want to jump to valkey slack to have faster debugging?

phawxby · 2025-02-19T12:47:24Z

For public awareness of what this specific issue appears to come down to.

Valkey glide is structured in such a way that there is an entry package which based on the architecture loads specific sub dependency packages, for example @valkey/valkey-glide-linux-arm64. Unfortunately those sub dependency packages also include the full JS wrapper around the rust native binary and not just the binary. This means that if you're trying to bundle the js with esbuild (and especially in our case where we want to bundle for multiple architectures at the same time) you end up with the same wrapper code bundled in multiple times which inflates the output bundle size and also appears to break things.

There are possible workarounds with lambda layering, manual esbuild steps to copy files around, but that would require a lot of rework of our build pipeline. Also @avifenesh has had success using serverless-esbuild.

It's worth noting most other libraries where native binaries are required are not structured this way, the arch specific binary is in a package on its own and the wrapper code it self acts as the parent entry package.

avifenesh · 2025-02-19T18:17:01Z

Adding some context -
The solution we know is working is to put the glide, as a native package, in the external section of the bundle.
It needs to be sent as part of zip, but not in the bundled code, but as part of the node_modules.
If you use pure esbuild, there is no node_modules being built automatically, so you'll need to config glide as external, to add it manually.
Using esbuild-serveless, the tool is smart enough to do it for you, so you'll get the external as part of the zip outside the bundled code.
This should be sufficient.

If you use more than one target, you need to install the specific valkey-glide-{target} you want to have available, and the package knows how to select the right one in runtime.
This comes with the cost of having two native binaries, which are comparably heavy as they are compiled rust code and not just JS code.

The restructuring is under development and likely to be released soon.
It will also include minimizing the binary size to be more suitable to serverless envs.
This will make the whole Node.js code as the valkey-glide main package, and the target-specific code, which is a native binary, a separate package, as an optional dependency which is pulled base on target.
Valkey-glide will still be a server only code, and can't run in browsers, meaning the env on which glide runs need to be the server (node, bun, deno etc.). Lambda, despite being “serverless” is a server env. The non-server env refers to browser env.
But the entire JS code can be bundled, and the binary only needs to be added aside from the bundled code.

avifenesh · 2025-02-19T20:16:52Z

@Muhammad-awawdi-amazon Another issue that will benefit from the restructuring. No pressure 😝
I closed all the others that are relevant for the restructure for cleaning, but linked them as branches of this one for tracking.
Error: Cannot find module 'protobufjs' #2812
Node: Error when installing via yarn #2680

avifenesh self-assigned this Feb 17, 2025

avifenesh added question Further information is requested node Node.js wrapper Users Pain An issue known to cause users pain, generaly open by the user. User issue Issue openned by users labels Feb 17, 2025

avifenesh mentioned this issue Feb 17, 2025

[Add "how to" for bundles] encountering lot of users having challenges with js bundles, which address the browser, we need to have a guide of how to bundle with "server only" packages. #3196

Open

avifenesh assigned Muhammad-awawdi-amazon Feb 19, 2025

This was referenced Feb 19, 2025

Error: Cannot find module 'protobufjs' #2812

Closed

Node: Error when installing via yarn #2680

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nodejs StartSocketConnection is not a function #3195

Nodejs StartSocketConnection is not a function #3195

phawxby commented Feb 17, 2025

avifenesh commented Feb 17, 2025 •

edited

Loading

avifenesh commented Feb 17, 2025 •

edited

Loading

avifenesh commented Feb 17, 2025

phawxby commented Feb 17, 2025

avifenesh commented Feb 17, 2025 •

edited

Loading

phawxby commented Feb 17, 2025

avifenesh commented Feb 17, 2025 •

edited

Loading

phawxby commented Feb 17, 2025

avifenesh commented Feb 17, 2025 •

edited

Loading

phawxby commented Feb 17, 2025

avifenesh commented Feb 17, 2025

phawxby commented Feb 19, 2025

avifenesh commented Feb 19, 2025

avifenesh commented Feb 19, 2025 •

edited

Loading

Nodejs StartSocketConnection is not a function #3195

Nodejs StartSocketConnection is not a function #3195

Comments

phawxby commented Feb 17, 2025

Inquiry

Language

Language Version

Engine Version

Operating System

Additional Technical Information

avifenesh commented Feb 17, 2025 • edited Loading

avifenesh commented Feb 17, 2025 • edited Loading

avifenesh commented Feb 17, 2025

phawxby commented Feb 17, 2025

avifenesh commented Feb 17, 2025 • edited Loading

phawxby commented Feb 17, 2025

avifenesh commented Feb 17, 2025 • edited Loading

phawxby commented Feb 17, 2025

avifenesh commented Feb 17, 2025 • edited Loading

phawxby commented Feb 17, 2025

avifenesh commented Feb 17, 2025

phawxby commented Feb 19, 2025

avifenesh commented Feb 19, 2025

avifenesh commented Feb 19, 2025 • edited Loading

avifenesh commented Feb 17, 2025 •

edited

Loading

avifenesh commented Feb 17, 2025 •

edited

Loading

avifenesh commented Feb 17, 2025 •

edited

Loading

avifenesh commented Feb 17, 2025 •

edited

Loading

avifenesh commented Feb 17, 2025 •

edited

Loading

avifenesh commented Feb 19, 2025 •

edited

Loading