Skip to content

Drizzle ORM migration with dual MySQL/PostgreSQL support#22

Closed
dnplkndll wants to merge 19 commits intoChurchApps:mainfrom
dnplkndll:feat/drizzle-orm
Closed

Drizzle ORM migration with dual MySQL/PostgreSQL support#22
dnplkndll wants to merge 19 commits intoChurchApps:mainfrom
dnplkndll:feat/drizzle-orm

Conversation

@dnplkndll
Copy link
Contributor

Summary

Complete migration from raw SQL (TypedDB/ConfiguredRepo) to Drizzle ORM with full dual-dialect support for MySQL and PostgreSQL.

This supersedes PR #17 (Kysely migrations) by providing the same self-hosting database migration capability, plus a full ORM layer.

What's included

  • 98/101 repository classes migrated to Drizzle ORM query builder (97% coverage)
    • 3 intentionally kept on TypedDB: AttendanceRepo (raw analytics SQL), ReportRepo (generic SQL runner), MembershipRepo in doing module (cross-module read-only)
  • Dual-dialect schema definitionssrc/db/schema/ (MySQL) and src/db/schema/pg/ (PostgreSQL), with a runtime resolver.ts that returns the correct schema based on DB_DIALECT env var
  • drizzle-kit migrations — generated SQL files in drizzle/mysql/ and drizzle/postgresql/ for all 6 modules (membership, attendance, content, giving, messaging, doing)
  • Programmatic migration runner (tools/migrate.ts) — applies pending migrations per module, supports --status flag
  • Replaced tools/dbScripts/ DDL — all CREATE TABLE SQL files removed; DDL now managed by drizzle-kit migrations. Demo data and stored procs retained.
  • initdb.ts rewritten — uses drizzle-orm migrators instead of raw SQL file iteration. MySQL→PG translation simplified to DML-only conversions.
  • initdb.sh updated — Helm hook script reads drizzle migration SQL files directly (no tsx needed in prod)
  • 299 integration tests across 6 test suites, all passing on both MySQL and PostgreSQL
  • CI workflow (.github/workflows/test.yml) — lint, typecheck, and integration tests against both MySQL and PostgreSQL

Architecture

DB_DIALECT=mysql|postgres (env var, default: mysql)
    ↓
src/db/schema/resolver.ts → returns dialect-correct table definitions
    ↓
src/db/drizzle.ts → lazy-creates drizzle instances per module
    ↓
DrizzleRepo / GlobalDrizzleRepo → base classes with CRUD, soft-delete, dialect-aware executeRows()
    ↓
Per-module repos (98 classes) → use query builder, with raw SQL only for complex analytics

3-tier base class hierarchy:

  • BaseDrizzleRepo — core: executeRows(), convertToModel()
  • DrizzleRepo (extends Base) — for tables with id + churchId: load(), loadAll(), save(), delete(), optional softDelete
  • GlobalDrizzleRepo (extends Base) — for tables with id only (no churchId): bible data, OAuth entities

Breaking changes

  • DB_DIALECT env var required for PostgreSQL (defaults to mysql for backward compatibility)
  • Per-module *_CONNECTION_STRING env vars now required (e.g., MEMBERSHIP_CONNECTION_STRING)
  • tools/dbScripts/ no longer contains CREATE TABLE SQL — use npm run migrate or npm run initdb instead

New npm scripts

Script Description
npm run migrate Apply pending drizzle-kit migrations (all modules)
npm run migrate:status Show applied/pending migration status
npm run migrate:generate Generate migration for DB_MODULE (requires DB_MODULE env var)
npm run migrate:generate:all Generate migrations for all modules, both dialects
npm run initdb -- --schema-only Create tables without stored procs or demo data

Known limitations

  • Schema defines events.registrationEnabled and events.capacity columns not yet in production DB
  • devices schema includes contentType/contentId columns (may only exist in deviceContents table)
  • contentProviderAuths table defined in schema but may not exist in production
  • PostgreSQL stored procs (cleanup, updateConversationStats, deleteForChurch) are handled in application code, not as DB procedures

Test plan

  • 299 integration tests passing (MySQL)
  • 299 integration tests passing (PostgreSQL)
  • TypeScript typecheck clean (tsc --noEmit)
  • npm run initdb works on fresh MySQL database
  • npm run initdb works on fresh PostgreSQL database
  • Helm deployment verified on MySQL (b1-test namespace)
  • Helm deployment verified on PostgreSQL (b1-postgres namespace)
  • docker compose up local dev workflow verified

🤖 Generated with Claude Code

dnplkndll and others added 19 commits March 14, 2026 20:11
Add drizzle-orm and drizzle-kit as dependencies. Define typed table schemas
for all 7 modules (attendance, content, doing, giving, membership, messaging,
reporting). Add connection factory in src/db/drizzle.ts with per-module
singleton caching using the existing mysql2 pool from @churchapps/apihelper.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 3-tier base class hierarchy:
- BaseDrizzleRepo: db connection + executeRows() helper
- DrizzleRepo: standard CRUD for tables with id + churchId columns,
  with opt-in soft-delete support (protected readonly softDelete = true)
- GlobalDrizzleRepo: CRUD for global tables with id only (no churchId)

Remove ConfiguredRepo and GlobalConfiguredRepo (replaced by DrizzleRepo).
Update barrel exports in shared/infrastructure/index.ts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert all 13 doing module repositories from raw SQL (TypedDB) to Drizzle
query builder: ActionRepo, AssignmentRepo, AutomationRepo, BlockoutDateRepo,
ConditionRepo, ConjunctionRepo, ContentProviderAuthRepo, PlanItemRepo,
PlanRepo, PlanTypeRepo, PositionRepo, TaskRepo, TimeRepo.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert all 13 messaging repositories from raw SQL to Drizzle query builder.
Remove no-op convertToModel/convertAllToModel overrides from repos.
Inline converter calls in 7 controllers and DeliveryHelper — repos now
return models directly, eliminating the unnecessary conversion layer.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert all 10 giving repositories from raw SQL to Drizzle query builder.
FundRepo uses softDelete=true, removing manual removed-column handling.
Remove no-op converter from EventLogRepo and inline in EventLogController.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert all 21 content repositories from raw SQL to Drizzle query builder.
Bible repos (Book, Chapter, Verse, VerseText, Translation, Lookup) and
SongDetail/SongDetailLink use GlobalDrizzleRepo (no churchId column).
Remove passthrough converters from LinkRepo and SettingRepo.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert 7 attendance repositories from raw SQL to Drizzle query builder:
CampusRepo, ServiceRepo, ServiceTimeRepo (all with softDelete=true),
GroupServiceTimeRepo, SessionRepo, VisitRepo, VisitSessionRepo.
AttendanceRepo kept as standalone (multi-table raw SQL queries).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert all 21 membership repositories from raw SQL to Drizzle query builder.
Global tables (ChurchRepo, UserRepo, ClientErrorRepo, OAuth* repos) use
GlobalDrizzleRepo. GroupRepo, PersonRepo, FormRepo, QuestionRepo use
softDelete=true.

Bug fixes:
- GroupRepo.loadByIds: add missing removed=false filter
- PersonRepo.loadByIds/loadByIdsOnly: add missing removed=false filter
- QuestionRepo.delete: fix CONCAT('d', sort) on int column

Remove passthrough converter overrides from 10 membership repos.
Inline converter calls in ChurchController.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 299 integration tests across 6 test suites covering all migrated repos:
- attendance (36 tests): Campus, Service, ServiceTime, Session, Visit CRUD
- content (62 tests): Block, Calendar, Event, File, Page, Sermon, Song/Arrangement
- doing (59 tests): Automation, Plan, Position, Assignment, Task, Blockout
- giving (58 tests): Fund, Donation, DonationBatch, Subscription, Gateway
- membership (51 tests): Person, Group, Form, Question, Household, Role, OAuth
- messaging (33 tests): Connection, Conversation, Message, Notification, Device

Includes shared db-helper.ts for test database setup/teardown,
jest.integration.config.cjs, and tsconfig.test.json.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Runs on PRs to main and pushes to main. Two jobs:
1. lint-and-typecheck: eslint + tsc
2. integration-tests: spins up MySQL 8.0 service container,
   creates databases, runs initdb, then executes 299 Drizzle
   integration tests via jest with --experimental-vm-modules.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace ~70 methods using raw sql tagged templates with MySQL-specific
functions (NOW(), CURDATE(), DATE_ADD(), IFNULL(), DATE_FORMAT()) with
Drizzle query builder equivalents (eq, between, gte, lt, count, sum,
innerJoin, leftJoin, etc.) and new DateHelper utilities. This prepares
the codebase for PostgreSQL portability — only ~17 methods with complex
reporting/stored procs remain as raw SQL.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add postgres.js driver, pg-core schema files, SqlDialect helper, and
dialect-branching in all ~17 raw SQL methods. Connection strings auto-derive
from DB_DIALECT env var. All 299 integration tests pass on MySQL; PG path
ready for CloudNativePG deployment.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Run 299 integration tests against both MySQL 8.0 and PostgreSQL 16 in
parallel. Uses DB_DIALECT env var to switch schema resolution and
connection strings. Both jobs must pass for the PR to be mergeable.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Auto-serialize Date params for postgres.js unsafe() calls
- PG schema: date mode, COALESCE, quoted identifiers in repos
- Dialect-aware identifier quoting in integration tests
- Restore sendInviteEmail endpoint, isNewUser flag, LinkRepo photo URL,
  PrivateMessageRepo rowToModel
- Restore security checks in MessageController and PersonController
- Strip id/churchId from .set() in all messaging repo save() methods
- Fix FormRepo loadAll filter, QuestionRepo sort on delete
- Add missing PG type translations (float, decimal, FOREIGN_KEY_CHECKS)
- CI: timeout-minutes 15, initdb double-precision regex fix

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add drizzle-kit migration infrastructure supporting both MySQL and PostgreSQL:
- drizzle.config.ts: dialect-aware (DB_DIALECT) + per-module (DB_MODULE)
- tools/migrate.ts: programmatic runner using drizzle-orm migrators
- npm scripts: migrate, migrate:status, migrate:generate, migrate:generate:all
- Initial schema migrations generated for all 6 modules on both dialects

Fix UserChurchHelper.createForNewUser: replace MySQL-only raw SQL (backtick-
quoted `groups`, boolean = 0) with Drizzle query builder that works on both
dialects. Fix duplicate index names in content schema (sections table) for
both MySQL and PG schemas.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove ~100 per-table CREATE TABLE SQL files from tools/dbScripts/ — table
creation is now handled by drizzle-kit migrations in drizzle/<dialect>/<module>/.
Drizzle schemas are the single source of truth for DDL.

Retained in tools/dbScripts/:
- demo.sql / populateData.sql (seed data, still uses mysqlToPgSql for PG)
- cleanup.sql, deleteForChurch.sql, updateConversationStats.sql (MySQL stored
  procs — PG equivalents are inlined in ConversationRepo)

Rewrote tools/initdb.ts to call drizzle-orm migrators instead of reading SQL
files. Stripped DDL-only regex rules from mysqlToPgSql (kept DML rules for
demo data). Simplified moduleDefinitions to moduleExtras (demo + stored procs).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
VisitRepo.toDateOnly used local-timezone Date methods (via DateHelper),
causing date-shift when server TZ != UTC. Now uses getUTC* methods
directly. Also fixes initdb loadDemoData to detect and route stored
procs/CALL statements through executeDDL (skipped on PG).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix duplicate index names in MySQL schemas: positions and
  subscriptionFunds tables had index name collisions with planItems
  and fundDonations respectively (PG schemas were already correct)
- Remove verbose debug logging from Environment.ts (was added for
  troubleshooting, not appropriate for upstream)
- Wire up --schema-only flag in initdb.ts (was parsed but unused)
- Add drizzle-kit migration files for the index renames

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Regenerate doing and giving MySQL migrations so the index name
corrections (idx_pos_church_plan, idx_sub_church_fund) are in the
0000 initial migration instead of a separate 0001 fix migration.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dnplkndll
Copy link
Contributor Author

@stephen-dahl I saw you where working on DB migrations with Kysely on #17
I added the migrations here. My goal was an option to use the pg stack I am more familiar writing.

@stephen-dahl
Copy link

@stephen-dahl I saw you where working on DB migrations with Kysely on #17 I added the migrations here. My goal was an option to use the pg stack I am more familiar writing.

Ya I haven't been happy with any of the ORMs I have used in the past including Drizzle. Kysely looks like I will be happy with it but @jzongker is the decision maker. So far I am just the guy throwing way too many library and refactor ideas at him.

Overall, I would say this MR is way too big and would take days to properly review, but it does give @jzongker a good idea of another ORM option.

@jzongker
Copy link
Contributor

I'm just getting comfortable with the idea of switching to Kysely. Switching to a full ORM is going to be too big of a change.

I just switched LessonsApi over to Kysely this weekend and it'll go live in a few hours. This is both migration scripts and all of the repository code. Assuming no major issues come from this, I think we can switch the main API to Kysely immediately after Easter.

@jzongker jzongker closed this Mar 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants