Skip to content

Conversation

simolus3
Copy link
Contributor

@simolus3 simolus3 commented Aug 26, 2025

This adds a compatibility option that, when enabled, will map custom postgres types into appropriate structures instead of syncing the raw representation.
Today, we support a static list of builtin Postgres types. Any other type will be synced as the underlying text representation as receive from the replication stream.

This PR improves that to add support for custom types. These types are resolved by querying their description from pg_type. Specifically, this supports:

  • Domain types, which wrap an underlying type and add constraints. There isn't much to parse here, Postgres already encodes them as the underlying type. We just have to recognize this alias to apply the decoder for the underlying type oid.
  • Composite types, which group multiple named values. We receive them as a tuple of raw values that we have to associate with field names when decoding to get an object back.
  • Range and multirange types, which are mainly parsed in case we may have to format the inner type (but it's also useful to represent them as JSON to make them easier to deal with in SQLite).

We also need to support nested types composed from these (e.g. a composite type could reference an array of a domain type that is itself another composite type). So we perform a BFS traversal through types when fetching them, and the decoding logic is recursive to handle these structures.

To decode the tuples that composite values are encoded as, this also refactors how arrays are parsed (since the structure is quite similar). The new decoder also supports arrays with a non-standard delimiter (which is also fetched from pg_type).

Closes #299.

TODOS:

  • Domain types
  • Composite types
  • Enums (these are always strings, so they work already)
  • Ranges
  • Test with actual sync
  • Use this information in getConnectionSchema().

Copy link

changeset-bot bot commented Aug 26, 2025

⚠️ No Changeset found

Latest commit: 1291598

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@simolus3 simolus3 force-pushed the better-postgres-types branch from 16b4268 to fdc70db Compare August 26, 2025 15:04
@rkistner
Copy link
Contributor

The new decoder also supports arrays with a non-standard delimiter (which is also fetched from pg_type).

I didn't even know that was possible...

Probably worth adding an explicit test for box[] to test this weird edge case.

@simolus3 simolus3 force-pushed the better-postgres-types branch 3 times, most recently from 67fed1f to db0151b Compare August 28, 2025 09:20
table.columns.push({
name: column.attname,
sqlite_type: sync_rules.expressionTypeFromPostgresType(pg_type).typeFlags,
sqlite_type: sync_rules.ExpressionType.fromTypeText(knownType.sqliteType()).typeFlags,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is worth noting that this doesn't take the current sync rules / compatibility options into account (I'm not sure if they're available here). So a DOMAIN type wrapping FLOAT would get represented as TYPE_REAL instead of TYPE_TEXT, even though we might replicate values as strings.

I think this may be fine because types are implemented as casts on the client-side, so the actual value returned by the view will be correct.

@simolus3 simolus3 force-pushed the better-postgres-types branch from d10c33e to 1291598 Compare August 28, 2025 12:08
@simolus3 simolus3 requested a review from rkistner August 28, 2025 12:08
@simolus3 simolus3 marked this pull request as ready for review August 28, 2025 12:08
@simolus3 simolus3 changed the title WIP: Better decoding of custom postgres types Better decoding of custom postgres types Aug 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Schema generations doesn't take well custom postgres types
2 participants