Types are incorrect for schema, where input & output are different

[pawelblaszczyk5]

Hello 👋

It seems like currently intended way is that schema is used for validating, but isn't impacting parsed return values:

db/packages/db/src/collection.ts

Lines 1616 to 1618 in bbed2f2

	// Return the original update data, not the merged data
	// We only used the merged data for validation
	return data as T

Is it set in stone? That limits a lot of usages for schemas, I'd love to see it work.

If it's not possible, is it feasible to make it work on a type level? Currently it just silently works and user won't know he has invalid types.

[samwillis]

Not set in stone - it would be good to understand the use case here a little more, could use explain where and how you would like it to work?

My colleague who worked on the schema implementation is on leave, so I don't have the full context. I believe the types are set up so that when you use a schema it uses the output types for the main collection types, and the input types for the insert/update functions. Is that what you are seeing?

I suspect that implies we should be storing the parsed/transformed data returned from the schema parse, is that not what is happening?

(I'm skimming the code and trying to get myself back up to speed on this bit)

[pawelblaszczyk5]

My initial thought was to reuse backend schemas that I'm using for database modeling with @effect/sql. Effect heavily relies on bidirectional schemas so they inherently have stuff going inside. For example, converting nullable field to Effect's Option, or native JS dates to Effect's DateTime. Currently types looks like everything is fine and the validators run, but under the hood I'm getting the stuff without transformations.

But maybe that's me approach it wrongly and frontend shape schema shouldn't be so tightly coupled to the model of db? I'm not sure here. Curious about your feelings/opinions there. Also I'm aware that most of different schema libraries aren't so heavily into transformations/bidirectional etc.

Also it seems like the behavior you're describing isn't fully correct? Or I'm missing something - currently for everything it looks like output schema is used? At least that what I'm observing. For schema like this:

Notice DateTimeFromDate usage, or OptionFromNullOr.

The inferred types has the information about both directions:

But insert/update & selecting both use the output:

[pawelblaszczyk5]

Update - types are lying a bit on update/insert, you were partially right. So if I make an update/insert according to suggested types after transformations - it throws validation errors. But if I made insert with "invalid" types, but matching input schema validation - it works correctly:

This works but has invalid types:

This doesn't work but types pass:

[KyleAMathews]

👋 popping my head in from parental leave — here's the PR that added this #209

It's only for inserts (not updates/deletes). The idea is you can insert something and have the schema fill in defaults e.g. id, created_at, etc. Which seemed reasonable/useful to me but perhaps it's causing trouble with Effect?

[samwillis]

Thanks @KyleAMathews!

I think there is a bug where it doesn't actually use the defaults or apply the transforms on insert - I'm looking into it.

I'm also suspicious we want it to apply the same changes on update too and the schema can make intended transformations.

Essentially I think the "output" types are what we hold internally in the collection, and output when it's consumed somewhere, and the "input" types are what insert/update should take.

[pawelblaszczyk5]

Okay, I think that makes my understanding here a lot clearer. So I'd probably want separate frontend shape schema, and backend model schema, because each of these uses defaults differently. And I should treat the schema input/output as providing e.g. defaults, not transforming values completely. That clears a lot of issues already.

I'm thinking about one more use case, maybe it's more specific to Electric (collection) now. Let's say I want to transform timestamptz (or any different data type) to Effect DateTime on frontend - that's easily doable on the frontend by using parser:

			parser: { timestamptz: (date: string) => DateTime.unsafeMake(date) },

If I understand correctly that's strictly mapping to pg types. How I could approach transforming data shape though? Let's say I want to represent nullable fields as Option from Effect, or transform e.g. object into class instance or normal array into fancy data structure? Is that possible/encouraged? I think with TanStack Query collection I could just use queryFn to map values before returning, right? But no obvious way of doing this with Electric Collection?

[KyleAMathews]

Updates can only update a validated insert which would already have all required fields so the defaults don't run as there's nothing to change. Right?

[samwillis]

My understanding is that the schema isn't just doing validation + settings defaults, it can also transform a value. So turning a string into a date or number, or one enum representation into another. I don't particularly like the idea then that the insert and update methods have different type signatures (might except a string for a date on insert, but not update).

[KyleAMathews]

Ah ok sure, we don't prevent people from doing transforms and that's a useful thing to do too 👍

[samwillis]

I've got a draft with the types tweeks, and I've take the opportunity to also address #226: #401

Will take a look at the transforms in insert and update next.

[pawelblaszczyk5]

So what’s the expected final behavior in a few words? Because I’m not sure now 😄 Let’s say I have a field which in schema is transformed from string to BigInt (sounds like reasonable use case). Should I receive/use string anywhere across the frontend?

[samwillis]

I think "normalise" is the best way to describe this. You can configure a scheme to use transform to allow accepting both BitInt or a string, this then is normalised to a BigInt.

This allows you to accept user input, and have the schema validate and normalise it on both insert and update, but you know you will always get a BigInt out of the collection - or at least thats my thinking.

[pawelblaszczyk5]

Hmm, that's a bit counterintuitive for me - can't imagine reason why I'd like to accept both instead of just consistently use transformed value on insert/update in business logic 😄