Drizzle ORM migration with dual MySQL/PostgreSQL support#22
Drizzle ORM migration with dual MySQL/PostgreSQL support#22dnplkndll wants to merge 19 commits intoChurchApps:mainfrom
Conversation
Add drizzle-orm and drizzle-kit as dependencies. Define typed table schemas for all 7 modules (attendance, content, doing, giving, membership, messaging, reporting). Add connection factory in src/db/drizzle.ts with per-module singleton caching using the existing mysql2 pool from @churchapps/apihelper. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 3-tier base class hierarchy: - BaseDrizzleRepo: db connection + executeRows() helper - DrizzleRepo: standard CRUD for tables with id + churchId columns, with opt-in soft-delete support (protected readonly softDelete = true) - GlobalDrizzleRepo: CRUD for global tables with id only (no churchId) Remove ConfiguredRepo and GlobalConfiguredRepo (replaced by DrizzleRepo). Update barrel exports in shared/infrastructure/index.ts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert all 13 doing module repositories from raw SQL (TypedDB) to Drizzle query builder: ActionRepo, AssignmentRepo, AutomationRepo, BlockoutDateRepo, ConditionRepo, ConjunctionRepo, ContentProviderAuthRepo, PlanItemRepo, PlanRepo, PlanTypeRepo, PositionRepo, TaskRepo, TimeRepo. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert all 13 messaging repositories from raw SQL to Drizzle query builder. Remove no-op convertToModel/convertAllToModel overrides from repos. Inline converter calls in 7 controllers and DeliveryHelper — repos now return models directly, eliminating the unnecessary conversion layer. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert all 10 giving repositories from raw SQL to Drizzle query builder. FundRepo uses softDelete=true, removing manual removed-column handling. Remove no-op converter from EventLogRepo and inline in EventLogController. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert all 21 content repositories from raw SQL to Drizzle query builder. Bible repos (Book, Chapter, Verse, VerseText, Translation, Lookup) and SongDetail/SongDetailLink use GlobalDrizzleRepo (no churchId column). Remove passthrough converters from LinkRepo and SettingRepo. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert 7 attendance repositories from raw SQL to Drizzle query builder: CampusRepo, ServiceRepo, ServiceTimeRepo (all with softDelete=true), GroupServiceTimeRepo, SessionRepo, VisitRepo, VisitSessionRepo. AttendanceRepo kept as standalone (multi-table raw SQL queries). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert all 21 membership repositories from raw SQL to Drizzle query builder.
Global tables (ChurchRepo, UserRepo, ClientErrorRepo, OAuth* repos) use
GlobalDrizzleRepo. GroupRepo, PersonRepo, FormRepo, QuestionRepo use
softDelete=true.
Bug fixes:
- GroupRepo.loadByIds: add missing removed=false filter
- PersonRepo.loadByIds/loadByIdsOnly: add missing removed=false filter
- QuestionRepo.delete: fix CONCAT('d', sort) on int column
Remove passthrough converter overrides from 10 membership repos.
Inline converter calls in ChurchController.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 299 integration tests across 6 test suites covering all migrated repos: - attendance (36 tests): Campus, Service, ServiceTime, Session, Visit CRUD - content (62 tests): Block, Calendar, Event, File, Page, Sermon, Song/Arrangement - doing (59 tests): Automation, Plan, Position, Assignment, Task, Blockout - giving (58 tests): Fund, Donation, DonationBatch, Subscription, Gateway - membership (51 tests): Person, Group, Form, Question, Household, Role, OAuth - messaging (33 tests): Connection, Conversation, Message, Notification, Device Includes shared db-helper.ts for test database setup/teardown, jest.integration.config.cjs, and tsconfig.test.json. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Runs on PRs to main and pushes to main. Two jobs: 1. lint-and-typecheck: eslint + tsc 2. integration-tests: spins up MySQL 8.0 service container, creates databases, runs initdb, then executes 299 Drizzle integration tests via jest with --experimental-vm-modules. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace ~70 methods using raw sql tagged templates with MySQL-specific functions (NOW(), CURDATE(), DATE_ADD(), IFNULL(), DATE_FORMAT()) with Drizzle query builder equivalents (eq, between, gte, lt, count, sum, innerJoin, leftJoin, etc.) and new DateHelper utilities. This prepares the codebase for PostgreSQL portability — only ~17 methods with complex reporting/stored procs remain as raw SQL. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add postgres.js driver, pg-core schema files, SqlDialect helper, and dialect-branching in all ~17 raw SQL methods. Connection strings auto-derive from DB_DIALECT env var. All 299 integration tests pass on MySQL; PG path ready for CloudNativePG deployment. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Run 299 integration tests against both MySQL 8.0 and PostgreSQL 16 in parallel. Uses DB_DIALECT env var to switch schema resolution and connection strings. Both jobs must pass for the PR to be mergeable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Auto-serialize Date params for postgres.js unsafe() calls - PG schema: date mode, COALESCE, quoted identifiers in repos - Dialect-aware identifier quoting in integration tests - Restore sendInviteEmail endpoint, isNewUser flag, LinkRepo photo URL, PrivateMessageRepo rowToModel - Restore security checks in MessageController and PersonController - Strip id/churchId from .set() in all messaging repo save() methods - Fix FormRepo loadAll filter, QuestionRepo sort on delete - Add missing PG type translations (float, decimal, FOREIGN_KEY_CHECKS) - CI: timeout-minutes 15, initdb double-precision regex fix Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add drizzle-kit migration infrastructure supporting both MySQL and PostgreSQL: - drizzle.config.ts: dialect-aware (DB_DIALECT) + per-module (DB_MODULE) - tools/migrate.ts: programmatic runner using drizzle-orm migrators - npm scripts: migrate, migrate:status, migrate:generate, migrate:generate:all - Initial schema migrations generated for all 6 modules on both dialects Fix UserChurchHelper.createForNewUser: replace MySQL-only raw SQL (backtick- quoted `groups`, boolean = 0) with Drizzle query builder that works on both dialects. Fix duplicate index names in content schema (sections table) for both MySQL and PG schemas. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove ~100 per-table CREATE TABLE SQL files from tools/dbScripts/ — table creation is now handled by drizzle-kit migrations in drizzle/<dialect>/<module>/. Drizzle schemas are the single source of truth for DDL. Retained in tools/dbScripts/: - demo.sql / populateData.sql (seed data, still uses mysqlToPgSql for PG) - cleanup.sql, deleteForChurch.sql, updateConversationStats.sql (MySQL stored procs — PG equivalents are inlined in ConversationRepo) Rewrote tools/initdb.ts to call drizzle-orm migrators instead of reading SQL files. Stripped DDL-only regex rules from mysqlToPgSql (kept DML rules for demo data). Simplified moduleDefinitions to moduleExtras (demo + stored procs). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
VisitRepo.toDateOnly used local-timezone Date methods (via DateHelper), causing date-shift when server TZ != UTC. Now uses getUTC* methods directly. Also fixes initdb loadDemoData to detect and route stored procs/CALL statements through executeDDL (skipped on PG). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix duplicate index names in MySQL schemas: positions and subscriptionFunds tables had index name collisions with planItems and fundDonations respectively (PG schemas were already correct) - Remove verbose debug logging from Environment.ts (was added for troubleshooting, not appropriate for upstream) - Wire up --schema-only flag in initdb.ts (was parsed but unused) - Add drizzle-kit migration files for the index renames Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Regenerate doing and giving MySQL migrations so the index name corrections (idx_pos_church_plan, idx_sub_church_fund) are in the 0000 initial migration instead of a separate 0001 fix migration. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@stephen-dahl I saw you where working on DB migrations with Kysely on #17 |
Ya I haven't been happy with any of the ORMs I have used in the past including Drizzle. Kysely looks like I will be happy with it but @jzongker is the decision maker. So far I am just the guy throwing way too many library and refactor ideas at him. Overall, I would say this MR is way too big and would take days to properly review, but it does give @jzongker a good idea of another ORM option. |
|
I'm just getting comfortable with the idea of switching to Kysely. Switching to a full ORM is going to be too big of a change. I just switched LessonsApi over to Kysely this weekend and it'll go live in a few hours. This is both migration scripts and all of the repository code. Assuming no major issues come from this, I think we can switch the main API to Kysely immediately after Easter. |
Summary
Complete migration from raw SQL (
TypedDB/ConfiguredRepo) to Drizzle ORM with full dual-dialect support for MySQL and PostgreSQL.This supersedes PR #17 (Kysely migrations) by providing the same self-hosting database migration capability, plus a full ORM layer.
What's included
AttendanceRepo(raw analytics SQL),ReportRepo(generic SQL runner),MembershipRepoin doing module (cross-module read-only)src/db/schema/(MySQL) andsrc/db/schema/pg/(PostgreSQL), with a runtimeresolver.tsthat returns the correct schema based onDB_DIALECTenv vardrizzle/mysql/anddrizzle/postgresql/for all 6 modules (membership, attendance, content, giving, messaging, doing)tools/migrate.ts) — applies pending migrations per module, supports--statusflagtools/dbScripts/DDL — all CREATE TABLE SQL files removed; DDL now managed by drizzle-kit migrations. Demo data and stored procs retained.initdb.tsrewritten — uses drizzle-orm migrators instead of raw SQL file iteration. MySQL→PG translation simplified to DML-only conversions.initdb.shupdated — Helm hook script reads drizzle migration SQL files directly (no tsx needed in prod).github/workflows/test.yml) — lint, typecheck, and integration tests against both MySQL and PostgreSQLArchitecture
3-tier base class hierarchy:
BaseDrizzleRepo— core:executeRows(),convertToModel()DrizzleRepo(extends Base) — for tables withid+churchId:load(),loadAll(),save(),delete(), optionalsoftDeleteGlobalDrizzleRepo(extends Base) — for tables withidonly (no churchId): bible data, OAuth entitiesBreaking changes
DB_DIALECTenv var required for PostgreSQL (defaults tomysqlfor backward compatibility)*_CONNECTION_STRINGenv vars now required (e.g.,MEMBERSHIP_CONNECTION_STRING)tools/dbScripts/no longer contains CREATE TABLE SQL — usenpm run migrateornpm run initdbinsteadNew npm scripts
npm run migratenpm run migrate:statusnpm run migrate:generateDB_MODULE(requiresDB_MODULEenv var)npm run migrate:generate:allnpm run initdb -- --schema-onlyKnown limitations
events.registrationEnabledandevents.capacitycolumns not yet in production DBdevicesschema includescontentType/contentIdcolumns (may only exist indeviceContentstable)contentProviderAuthstable defined in schema but may not exist in productionTest plan
tsc --noEmit)npm run initdbworks on fresh MySQL databasenpm run initdbworks on fresh PostgreSQL databasedocker compose uplocal dev workflow verified🤖 Generated with Claude Code