PostHog email -> Keycloak sub migration: backfill $create_alias for all existing users

[Will-Howard]

Description

Doesn't need to be merged, just needs to be run locally

#2767 added aliasing of email -> sub on login (meaning that after this alias is set up a user can be safely identified by either). This script will add this alias for all existing users in PostHog, meaning it will be safe to cut over to using sub only, and existing users will have consistent histories.

The full migration plan for PostHog is in this comment. This is "PR 3", which will unlock "PR 4":

• PR 3 (doesn't actually need to be a PR, can just be run locally): Run a script to backfill the posthog.alias(auth.sub, auth.email) call for all existing users. It will then be safe to identify all existing and newly created users as either posthog.identity(auth.email) or posthog.identify(auth.sub)
• PR 4: Flip the 'identify_applicants' event definition from distinctId: user.email to distinctId: user.keycloakIdentifier. This submits identify events for both new users and new applications by existing users. It's safe to flip without any alias step because:
  a. For new users: They will have already hit the posthog.alias(auth.sub, auth.email) in step 2
  b. For new applications against existing users: They will have hit the posthog.alias(auth.sub, auth.email) from the backfill in step 3

Issue

#2713

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: eb36c7c6-0eec-4dae-a06e-8009f23df96d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a one-off TypeScript runner for linking Keycloak sub identifiers to email identities in PostHog. It supports CLI configuration, Postgres or JSON user input, HogQL queries for existing identities and aliases, detailed classification and skip counts, dry-run execution, batched event submission, progress persistence, resuming, throttling, and completion or error handling.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 22.22% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely summarises the backfill of PostHog aliases for existing users.
Description check	✅ Passed	The description covers the purpose, migration context, and linked issue, with only the checklist left unfilled.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch wh-2713-posthog-alias-backfill

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Will-Howard]

I will leave this as a draft since it doesn't need to be merged, and close it once I've run this script locally.

@coderabbitai review @greptileai review

[coderabbitai]

@Will-Howard Understood — I’ll review the draft with the local, one-time backfill use case in mind.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[greptile-apps]

Greptile Summary

This PR adds a one-off backfill script that reads all existing users from the Postgres replica and sends PostHog $create_alias events linking each user's Keycloak sub to their email distinct_id, replicating the login-time alias introduced in #2767 for users who existed before that change shipped.

• Fetches existing PostHog email-like distinct_ids and already-aliased subs via HogQL cursor pagination; classifies users into send/skip buckets (exact match, case-insensitive match, ambiguous, not in PostHog, already aliased, invalid) and skips duplicate subs to ensure idempotency.
• Sends events in configurable batches to the PostHog /batch/ endpoint with per-batch progress files, a --dry-run mode, and a --from <autoNumberId> resume flag; never logs emails or subs.

Confidence Score: 4/5

Safe to run locally as a dry-run first; the idempotency mechanism (aliasedSubs check + PostHog server-side no-op on duplicate alias) means re-runs are harmless. The two arg-parsing edge cases are unlikely to surface in a normal execution.

The script is well-guarded: dry-run mode, idempotent by design, no production writes to Postgres, and a progress file for resuming. Two small issues exist — a misleading --from start error hint on first-batch failure (which would produce NaN and an unhelpful subsequent error if followed literally), and a falsy-string check that silently ignores --from 0 — but neither affects the happy path or a normal resume.

apps/website/scripts/posthog-alias-backfill/backfill-alias.ts — the only changed file; the arg-parsing and error-hint issues are both here.

Important Files Changed

Filename	Overview
apps/website/scripts/posthog-alias-backfill/backfill-alias.ts	New one-off backfill script that reads users from Postgres and sends $create_alias events to PostHog. Well-structured with dry-run, resumable pagination, idempotency via aliasedSubs check, and per-batch progress file. Two minor arg-parsing/error-message issues noted.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Script
    participant Postgres
    participant PostHog HogQL
    participant PostHog Batch

    Script->>Postgres: "SELECT users WHERE keycloakIdentifier IS NOT NULL AND autoNumberId > from"
    Postgres-->>Script: BackfillUser[]

    Script->>PostHog HogQL: SELECT DISTINCT distinct_id FROM person_distinct_ids WHERE LIKE '%@%'
    PostHog HogQL-->>Script: posthogEmails[]

    Script->>PostHog HogQL: SELECT DISTINCT properties.alias FROM events WHERE event='$create_alias'
    PostHog HogQL-->>Script: aliasedSubs[]

    Note over Script: classify() — exact/lowercase match,
    Note over Script: skip already aliased / no person / invalid

    loop For each batch
        Script->>PostHog Batch: POST /batch/ { $create_alias events }
        PostHog Batch-->>Script: 200 OK
        Script->>Script: write progress file
    end

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Script
    participant Postgres
    participant PostHog HogQL
    participant PostHog Batch

    Script->>Postgres: "SELECT users WHERE keycloakIdentifier IS NOT NULL AND autoNumberId > from"
    Postgres-->>Script: BackfillUser[]

    Script->>PostHog HogQL: SELECT DISTINCT distinct_id FROM person_distinct_ids WHERE LIKE '%@%'
    PostHog HogQL-->>Script: posthogEmails[]

    Script->>PostHog HogQL: SELECT DISTINCT properties.alias FROM events WHERE event='$create_alias'
    PostHog HogQL-->>Script: aliasedSubs[]

    Note over Script: classify() — exact/lowercase match,
    Note over Script: skip already aliased / no person / invalid

    loop For each batch
        Script->>PostHog Batch: POST /batch/ { $create_alias events }
        PostHog Batch-->>Script: 200 OK
        Script->>Script: write progress file
    end

Loading

_{Reviews (1): Last reviewed commit: "[chore] posthog: add email->sub $create_..." | Re-trigger Greptile}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@apps/website/scripts/posthog-alias-backfill/backfill-alias.ts`:
- Line 187: Update the validation condition in the alias backfill flow to handle
missing user.email or user.sub values before calling includes. Treat records
with null or undefined values as invalid and tally them through the existing
invalid-record path, while preserving the current checks for populated strings.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 32b84e72-4273-45e5-b693-6093c35868ff

📥 Commits

Reviewing files that changed from the base of the PR and between 4f4ca03 and 01fe874.

📒 Files selected for processing (1)

• apps/website/scripts/posthog-alias-backfill/backfill-alias.ts

[Will-Howard]

I ran this, closing the PR now