feat(backend): Introduce machine authentication #5689

wobsoriano · 2025-04-22T00:16:24Z

Description

This PR adds machine authentication support (atm only in the backend SDK) by introducing support for 4 token types: api_key, oauth_token, machine_token, and session_token. To maintain backwards compatibility, session_token remains the default authentication method when no specific token type is specified. This ensures existing apps continue to work without modification while allowing new applications to opt-in to machine authentication methods through the acceptsToken option.

Note: The Nextjs part will be handled in #5710. Changesets to follow.

Key changes:

Deprecated SignedInState and SignedOutState in favor of AuthenticatedState and UnauthenticatedState to better represent both session and machine authentication states. They still return the same properties, with an added tokenType and isAuthenticated properties (deprecating isSignedIn).
The toAuth() method now returns a different value if the tokenType is not a session_token. For now, we landed on the id, name, subject, claims and scopes property for machine auth tokens.
Added two new internal functions in authenticateRequest: authenticateAnyRequestWithTokenInHeader and authenticateMachineRequestWithTokenInHeader to handle machine authentication.
The internal signedIn and signedOut functions have been updated to accommodate machine auth.
Added new error types and codes specific to machine token verification (MachineTokenVerificationErrorCode)
Added new APIs (APIKeysApi, IdPOAuthAccessTokenApi, and MachineTokensApi) used inside a new verifyMachineAuthToken function to validate tokens against their respective endpoints
Added test for various scenarios for token validation, handling different token types, token mismatch, and proper error responses when verification fails

Here's an example usage pattern with API key:

Say C1 wants to protect their endpoints in a Hono app:

import { serve } from '@hono/node-server'
import { createMiddleware } from 'hono/factory'
import { Hono } from 'hono'
import { clerkClient } from './client'
import { HTTPException } from 'hono/http-exception'

const app = new Hono()

const clerkMiddleware = createMiddleware(async (c, next) => {
  const authReq = await clerkClient.authenticateRequest(c.req.raw, {
    acceptsToken: 'api_key'
  })

  if (!authReq.isAuthenticated) {
    throw new HTTPException(401, { message: 'Unauthorized' })
  }

  await next()
})

app.post('/api/protected', clerkMiddleware, async (c, next) => {
  return c.text('Hello from /api/protected')
})

Then C2 can access it by passing the api_key:

const resp = await fetch('http://localhost:3000/api/protected', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': `Bearer ${process.env.API_KEY}`
  },
})

const data = await resp.text()

P.S. I attempted to break this down into smaller PRs but the changes are tightly coupled 😞. So sorry and thank you in advance reviewer! I believe 30-40% of the total changes are from the test files.

Resolves ROBO-36

Checklist

pnpm test runs as expected.
pnpm build runs as expected.
(If applicable) JSDoc comments have been added or updated for any package exports
(If applicable) Documentation has been updated

Type of change

vercel · 2025-04-22T00:16:28Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
clerk-js-sandbox	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 2, 2025 5:21pm

changeset-bot · 2025-04-22T02:52:33Z

⚠️ No Changeset found

Latest commit: 572854e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

jescalan

Looking great so far!

packages/backend/src/jwt/types.ts

jescalan · 2025-04-24T21:19:47Z

packages/backend/src/tokens/__tests__/machine.test.ts

+  });
+
+  it('returns false for tokens without a recognized prefix', () => {
+    expect(isMachineToken('unknown_prefix_token')).toBe(false);


Just wanna note that we do plan to allow custom prefixes in the future - likely these end up being prepended to the token type prefix so i think it should be a fairly straightforward change

Added a note to cover this in the future.

jescalan · 2025-04-24T21:22:02Z

packages/backend/src/tokens/__tests__/request.test.ts

+    });
+
+    // Test each token type with parameterized tests
+    const tokenTypes = ['api_key', 'oauth_token', 'machine_token'] as const;


I'm mildly confused by the typecasting here

Yeah the as const is needed here so TS knows these are literal types that match the keys in our mock objects, otherwise it would just see it as string[]

jescalan · 2025-04-24T21:37:09Z

packages/backend/src/tokens/request.ts

+    const { sessionTokenInHeader } = authenticateContext;
+    if (!sessionTokenInHeader) {
+      return handleError(new Error('No token in header'), 'header');
+    }


Something seems weird about this logic...

I believe in practice this shouldn't be hit, as we check the existence of the header token before calling this method.

Yeah either that or we remove and do non-null assertions

packages/backend/src/fixtures/machine.ts

wobsoriano · 2025-05-01T22:53:46Z

!snapshot

clerk-cookie · 2025-05-01T22:56:31Z

Hey @wobsoriano - the snapshot version command generated the following package versions:

Package	Version
@clerk/agent-toolkit	0.0.29-snapshot.v20250501225444
@clerk/astro	2.6.11-snapshot.v20250501225444
@clerk/backend	1.31.0-snapshot.v20250501225444
@clerk/chrome-extension	2.3.5-snapshot.v20250501225444
@clerk/clerk-js	5.63.3-snapshot.v20250501225444
@clerk/clerk-expo	2.10.5-snapshot.v20250501225444
@clerk/express	1.4.12-snapshot.v20250501225444
@clerk/fastify	2.2.12-snapshot.v20250501225444
@clerk/nextjs	6.18.3-snapshot.v20250501225444
@clerk/nuxt	1.5.12-snapshot.v20250501225444
@clerk/react-router	1.3.3-snapshot.v20250501225444
@clerk/remix	4.6.3-snapshot.v20250501225444
@clerk/tanstack-react-start	0.14.3-snapshot.v20250501225444
@clerk/testing	1.6.2-snapshot.v20250501225444

Tip: Use the snippet copy button below to quickly install the required packages.
@clerk/agent-toolkit

npm i @clerk/[email protected] --save-exact

@clerk/astro

npm i @clerk/[email protected] --save-exact

@clerk/backend

npm i @clerk/[email protected] --save-exact

@clerk/chrome-extension

npm i @clerk/[email protected] --save-exact

@clerk/clerk-js

npm i @clerk/[email protected] --save-exact

@clerk/clerk-expo

npm i @clerk/[email protected] --save-exact

@clerk/express

npm i @clerk/[email protected] --save-exact

@clerk/fastify

npm i @clerk/[email protected] --save-exact

@clerk/nextjs

npm i @clerk/[email protected] --save-exact

@clerk/nuxt

npm i @clerk/[email protected] --save-exact

@clerk/react-router

npm i @clerk/[email protected] --save-exact

@clerk/remix

npm i @clerk/[email protected] --save-exact

@clerk/tanstack-react-start

npm i @clerk/[email protected] --save-exact

@clerk/testing

npm i @clerk/[email protected] --save-exact

brkalow

Looking good! Nice work

brkalow · 2025-05-02T02:04:11Z

packages/backend/src/api/factory.ts

+    // Using "/" instead of an actual version since they're bapi-proxy endpoints.
+    // bapi-proxy connects directly to C1 without URL versioning,
+    // while API versioning is handled through the Clerk-API-Version header.


Huh, this is surprising to me, I would expect the path structure at edge to mirror origin 🤔 I'll start a thread in Slack

I saw the thread. Yeah, I also thought of creating a separate buildRequest function for bapi-proxy but that'd be too much considering the logic we already have when building a request. For now I'm passing "/" to bypass the api version during URL normalization

brkalow · 2025-05-02T02:04:29Z

packages/backend/src/api/endpoints/index.ts

 export * from './BetaFeaturesApi';
 export * from './BlocklistIdentifierApi';
 export * from './ClientApi';
 export * from './DomainApi';
 export * from './EmailAddressApi';
+export * from './IdPOAuthAccessTokenApi';


Why does this one start with IdP?

So.. this was hard to name. We already have:

OauthAccessToken which is used for retrieving the OAuth access token of a user

OAuthApplication for OAuth applications

I named it like that based on the object name used (clerk_idp_oauth_access_token) for response deserialization but suggestions welcome!

brkalow · 2025-05-02T02:14:22Z

packages/backend/src/tokens/authObjects.ts

+type MachineObjectExtendedProperties<TAuthenticated extends boolean> = {
+  api_key: {
+    name: TAuthenticated extends true ? string : null;
+    claims: TAuthenticated extends true ? Claims | null : null;
+  };
+  machine_token: {
+    name: TAuthenticated extends true ? string : null;
+    claims: TAuthenticated extends true ? Claims | null : null;
+  };
+  oauth_token: object;
+};


clever 👀

brkalow · 2025-05-02T02:21:23Z

packages/backend/src/tokens/authStatus.ts

+  /**
+   * @deprecated Use `isAuthenticated` instead.
+   */
  isSignedIn: true;


isSignedIn feels more natural when dealing with session tokens 🤔 I'm wondering if we're weighing machine token usage too heavily here when considering renaming this API...

Great point! I think we relied too much on having unified property, but most of our users are still thinking in terms of user sessions so it's best to still have it for sessions 🤔

I think we also named it like that for reasons that a tokenType property can have an array of tokens, and that would mean both properties (isSignedIn and isAuthenticated) will show up in the result

brkalow · 2025-05-02T02:51:02Z

packages/backend/src/tokens/request.ts

@@ -69,13 +72,64 @@ function isRequestEligibleForRefresh(
  );
 }

-export async function authenticateRequest(
+function maybeHandleTokenTypeMismatch(


Why is this maybe?

I originally named it to match the handleMaybeHandshakeStatus pattern, but looking back, that function actually returns either a signed out or handshake state, while maybeHandleTokenTypeMismatch only returns a signed out state or null. It's vague 😬 - renamed it to checkTokenTypeMismatch. Suggestions welcome!

brkalow · 2025-05-02T02:55:34Z

packages/backend/src/tokens/request.ts

+    if (!sessionOrMachineTokenInHeader) {
+      return handleError(new Error('Missing token in header'), 'header');
+    }


This shouldn't be possible to hit right? I see it's called inside a conditional header token check.

Correct! This or we do non-null assertions which I'm trying to avoid mostly

brkalow · 2025-05-02T02:56:50Z

packages/backend/src/tokens/request.ts

+    const { sessionOrMachineTokenInHeader } = authenticateContext;
+    // Use session token error handling if no token in header (default behavior)
+    if (!sessionOrMachineTokenInHeader) {
+      return handleError(new Error('Missing token in header'), 'header');


consider renaming handleError to handleSessionTokenError() to distinguish between the machine handler.

brkalow · 2025-05-02T02:57:11Z

packages/backend/src/tokens/request.ts

+    const { sessionTokenInHeader } = authenticateContext;
+    if (!sessionTokenInHeader) {
+      return handleError(new Error('No token in header'), 'header');
+    }


I believe in practice this shouldn't be hit, as we check the existence of the header token before calling this method.

brkalow · 2025-05-02T03:00:19Z

packages/backend/src/tokens/verify.ts

+    const verifiedToken = await client.machineTokens.verifySecret(secret);
+    return { data: verifiedToken, tokenType: TokenType.MachineToken, errors: undefined };
+  } catch (err: any) {
+    return handleClerkAPIError('machine_token', err, 'Machine token not found');


Use the enum here

brkalow · 2025-05-02T03:01:57Z

packages/nextjs/src/app-router/server/auth.ts

@@ -131,7 +132,7 @@ export const auth: AuthFn = async () => {
  };

  return Object.assign(authObject, { redirectToSignIn, redirectToSignUp });
-};
+}) as AuthFn;


Why do we need to cast here now?

Same as my comment here, but for this PR, it's just to make TS happy as getAuthDataFromRequest returns a AuthObject type and we want to force it to just the session types.

The full context is in the Nextjs companion PR #5710

clerk-cookie added nextjs backend agent-toolkit labels Apr 22, 2025

wobsoriano mentioned this pull request Apr 22, 2025

feat(backend): WIP M2M #5564

Closed

9 tasks

vercel bot deployed to Preview April 22, 2025 02:53 View deployment

vercel bot deployed to Preview April 22, 2025 03:01 View deployment

vercel bot deployed to Preview April 22, 2025 23:46 View deployment

vercel bot deployed to Preview April 23, 2025 18:12 View deployment

vercel bot deployed to Preview April 24, 2025 14:31 View deployment

vercel bot deployed to Preview April 24, 2025 14:52 View deployment

vercel bot deployed to Preview April 24, 2025 18:17 View deployment

jescalan reviewed Apr 24, 2025

View reviewed changes

vercel bot deployed to Preview April 25, 2025 04:22 View deployment

vercel bot deployed to Preview April 25, 2025 17:26 View deployment

vercel bot deployed to Preview April 25, 2025 19:25 View deployment

github-advanced-security bot found potential problems Apr 25, 2025

View reviewed changes

packages/backend/src/fixtures/machine.ts Fixed Show fixed Hide fixed

vercel bot deployed to Preview April 25, 2025 19:49 View deployment

vercel bot deployed to Preview April 25, 2025 19:53 View deployment

vercel bot deployed to Preview April 25, 2025 20:13 View deployment

vercel bot deployed to Preview April 28, 2025 15:16 View deployment

vercel bot temporarily deployed to Preview April 28, 2025 15:32 Inactive

clerk-cookie added express astro labels Apr 28, 2025

vercel bot temporarily deployed to Preview April 28, 2025 16:02 Inactive

clerk-cookie added nuxt and removed express astro nuxt labels Apr 28, 2025

wobsoriano added 8 commits May 1, 2025 09:11

chore: introduce TokenType const

1e2441c

chore: fix incorrect import

c4cab10

chore: export other token types

da5260d

chore: add remark about token prefixes

f61d668

chore: update doc regarding versioning

651b01a

chore: update doc regarding versioning

576d7c4

chore: update to use new token prefixes

b8eb01f

chore: address conflicts

Loading
Loading status checks…

58c0c97

wobsoriano force-pushed the rob/robo-36-sdk-m2m branch from 26ea907 to 58c0c97 Compare May 1, 2025 16:17

vercel bot deployed to Preview May 1, 2025 16:18 View deployment

chore: move some types to types file

Loading
Loading status checks…

e42fa3c

vercel bot deployed to Preview May 1, 2025 16:30 View deployment

wobsoriano changed the title ~~feat(backend): Add machine authentication support~~ feat(backend): Introduce machine authentication May 1, 2025

chore: accommodate update properties and new prefixes

Loading
Loading status checks…

837fa37

vercel bot deployed to Preview May 1, 2025 18:22 View deployment

chore: type adjustments

Loading
Loading status checks…

848d053

vercel bot deployed to Preview May 1, 2025 18:42 View deployment

chore: remove todo comments in nextjs

Loading
Loading status checks…

01a67a0

vercel bot temporarily deployed to Preview May 1, 2025 21:56 Inactive

brkalow reviewed May 2, 2025

View reviewed changes

wobsoriano added 3 commits May 2, 2025 09:56

chore: update remaining strings to use token type enums

9c65ad0

chore: make parsed token in header property name simple

ff52b64

chore: update remaining strings to use token type enums

Loading
Loading status checks…

86580ab

vercel bot deployed to Preview May 2, 2025 17:00 View deployment

chore: rename session token error handling function

Loading
Loading status checks…

4ca8a76

vercel bot deployed to Preview May 2, 2025 17:03 View deployment

chore: rename mismatched token checker name to checkTokenTypeMismatch

Loading
Loading status checks…

572854e

vercel bot deployed to Preview May 2, 2025 17:21 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(backend): Introduce machine authentication #5689

feat(backend): Introduce machine authentication #5689

wobsoriano commented Apr 22, 2025 •

edited

Loading

vercel bot commented Apr 22, 2025 •

edited

Loading

changeset-bot bot commented Apr 22, 2025 •

edited

Loading

jescalan left a comment

jescalan Apr 24, 2025

wobsoriano Apr 29, 2025

jescalan Apr 24, 2025

wobsoriano Apr 24, 2025

jescalan Apr 24, 2025

brkalow May 2, 2025

wobsoriano May 2, 2025

wobsoriano commented May 1, 2025

clerk-cookie commented May 1, 2025

brkalow left a comment

brkalow May 2, 2025

wobsoriano May 2, 2025

brkalow May 2, 2025

wobsoriano May 2, 2025 •

edited

Loading

brkalow May 2, 2025

brkalow May 2, 2025

wobsoriano May 2, 2025 •

edited

Loading

brkalow May 2, 2025

wobsoriano May 2, 2025

brkalow May 2, 2025

wobsoriano May 2, 2025

brkalow May 2, 2025

wobsoriano May 2, 2025

brkalow May 2, 2025

brkalow May 2, 2025

wobsoriano May 2, 2025

brkalow May 2, 2025

wobsoriano May 2, 2025 •

edited

Loading

feat(backend): Introduce machine authentication #5689

Are you sure you want to change the base?

feat(backend): Introduce machine authentication #5689

Conversation

wobsoriano commented Apr 22, 2025 • edited Loading

Description

Checklist

Type of change

vercel bot commented Apr 22, 2025 • edited Loading

changeset-bot bot commented Apr 22, 2025 • edited Loading

⚠️ No Changeset found

jescalan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wobsoriano commented May 1, 2025

clerk-cookie commented May 1, 2025

brkalow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wobsoriano May 2, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wobsoriano May 2, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wobsoriano May 2, 2025 • edited Loading

Choose a reason for hiding this comment

wobsoriano commented Apr 22, 2025 •

edited

Loading

vercel bot commented Apr 22, 2025 •

edited

Loading

changeset-bot bot commented Apr 22, 2025 •

edited

Loading

wobsoriano May 2, 2025 •

edited

Loading

wobsoriano May 2, 2025 •

edited

Loading

wobsoriano May 2, 2025 •

edited

Loading