[Feature Request] Inherited Annotations

[binaryben]

This proposal is to add a mechanism to support the inheritance of annotations with the ability to merge annotations from multiple field definitions

A Partial Mixin would serve as a complement to a Type Mixin, which are used to enable the inheritance of fields.

The type mixin documentation currently states:

A model can use multiple mixins as long as their field names don't conflict.

This would alter that paradigm by providing a second mixin option that functions similar-ish to the abstract modifier available to Typescript classes. Unlike a Typescript abstract, it is proposed to be implemented as a standalone concept rather than as a modifier.

In contrast to a type mixin, a partial mixin:

• Would not require a field to be fully defined — LSP and other tools should assume that the field will be defined and finalized elsewhere
• Cannot include a data type definition — extends fields with annotations only
• Multiple definitions against fields with the same name are merged — fields can, and are expected to be duplicated across partial definitions

The type mixin would retain the current requirement that multiple type mixins applied to a type or model cannot cause conflicts due to having duplicated field entries.

Example

Allow the core definition of a field in a type or model to focus on the essentials ...

// Very clean and readable definition of the core data structures

type Base with BaseAcmeEditor, BaseBetterEditor, BaseGolang {
  /// Primary Key
  id  Int @id @default(cuid())

  /// Creation Date
  created Datetime

  /// Updated Date
  updated Datetime
}

... whilst functions are split into independent and reusuable blocks

// Currently running the Acme CMS ... but it keeps blowing up.

partial BaseAcmeEditor {
  id
    @acme.volatitle('Primary Key')
    @acme.icon('dynamite')
  created
    @acme.volatitle('Created on')
  updated
    @acme.volatitle('Updated on')
}

// We are replacing it, but need both in parallel for now ...
// I really hope we can easily remove those buggy
// Acme entries when the migration is done

partial BaseBetterEditor {
  id
    @better.title('Primary Key')
    @better.icon('key')
    @better.hidden()
  created
    @better.title('Created on')
  updated
    @better.title('Updated on')
}

// At least we can use ZenStack with a plugin to
// generate structs for our caching microservice ...

partial BaseGolang {
  id
    // Hypothetical need to transform type
    @golang.type('String')
    @golang.parse( ... )

  created
    // .. or to use a different field name
    @golang.map('CreatedUTC')

    // .. or to ensure working with UTC immediately
    @golang.default('time.Now().UTC()')

  updated
    @golang.map('UpdatedUTC')
}

More details ...

Rationale

Primary Motivation

The most salient aspect of this proposal is the notion to extend the newly introduced concept of a mixin¹ beyond only being a type mixin. It is also to encourage doing that early to avoid later confusion on what new concepts are.

I believe considering mixins as a more, "generic container that can be remixed and applied to different entities" concept, will be a needed conceptual and technical building block for implementing ambitious ideas such as #563. The current type mixin alone may not accommodate some of the challenges involved, especially without muddying the definition of what a "type" is further.

In an issue tracking the implementation of inheritance abstractions (#783)², @AmruthPillai commented:

[Would] it be wise to use another keyword other than type as Prisma might have plans on using the same keyword for relational databases, but for a different construct?

[Direct link]

As ZenStack diverges further away from PSL, the topic of what Prisma does with a keyword becomes somewhat less relevant. There are still strong conceptual ideas associated with what a "type" is to developers to consider though.

The current challenge is that there is currently no clear way to "mix-in" groups of annotations to final field definitions. Especially not in any way that emulates the composability of models which can be comprised of multiple type definitions. Field definitions are all or nothing once defined.

This is an important feature of type mixins however, as it directly supports having Strongly Typed JSON Fields. It is very important to maintain that approach of having a single place where the final "shape of the data" is defined. If a type could be defined in multiple places, it risks becoming inconsistent and unreliable.

Conflating abstraction and inheritance features as a generic concept too tightly with the notion of a "type" may also increase friction with developers understanding those features, and slow adoption. Instead, we can move the association of the mixin concept to the with keyword instead of the current emphasis on the type keyword. This allows the concept of "remixing" to be applied more liberally.

My current view is that a type mixin is, and should remain, tightly coupled to the Strongly Typed JSON Field feature, which largely defines data structures (the final shape). The addition of a partial mixin would support creating modular definitions of how data structures are interacted with (how the final shape can move). This would be done by focusing on creating a container that defines related groups of annotations as "mix-ins".

Benefits

Immediate Benefits

There are two broad areas the addition of partial mixin to ZenStack would benefit:

1. Separation of concerns — maintain integration of different functions independently
2. DX — improved readability and maintainability

Notable highlights:

• DRY-er schema definitions

  Consider two model definitions that have the same field with the same name, policy and other annotated requirements, but use a different data type. An id field could an auto-incremented Int on some models and an auto-generated UUID String on the others.

  Currently, a developer would have to maintain two individual type mixins to support this. This is regardless of if all other requirements are the same. The addition of a partial mixin would provide a way to prevent needing to duplicate shared attribute annotations.

• Security benefit: Composable definition of access policies

  Continuing with the example above, the id field in both types would likely need a policy that allows creation but prevents further updates. Entire policies can be described in a partial mixin and repeatably applied across multiple type mixins and models.

• Improved flexibility

  Can choose to run operations against a schema with only a subset of the available defined attributes applied. This can be done just by removing a mixin rather than commenting out entire blocks. Enables configuring different behaviors based on contexts (edge vs. origin, private vs public, dev vs prod, aws vs gcp, etc.)

• Improved mobility

  Similar to flexibility to change how a schema is deployed, the same flexibility can help improve mobility between different solutions. Locating all annotations per vendor/application/format in it's own partial means developers don't need to go looking for every single line to enable, or disable relevant annotations.

Future Benefits

There are also opportunities for further benefits that adding partial mixins would enable exploration of in the future. Prioritizing adding the concept of a partial in a basic form would be beneficial to:

• provide the above immediate benefits in terms of DX (readability especially) and composibility (increase flexibility whilst staying DRY)
• introduce the concept of having multiple mixin options to developers in one shot to avoid later confusion
• adding another differentiator over PSL whilst remaining a superset syntax

As such, the following ideas may be best left out of scope for any first implementation of partial mixin support.

Possible future opportunities include:

• Apply partial mixins based on conditions defined in the schema

  If a @condition attribute is introduced in the future, the benefits listed under "Improved flexibility" could be applied automatically. For example:

  partial Example {
    // Oh look, an example
  
    @@condition(flag: '')
    @@condition(dbEngine: '')
  }

  This could include filtering based on factors such as:

  • Feature flags
  • Environment variables
  • Database type or config
    • Use alternative @db.x annotations for NoSQL vs Postgres vs SQLite
  • Deployment context

• Apply annotations to multiple fields using regex (or similar) field names

  Example: Apply access annotations to all fields prefixed with something (eg. internal_*). Security is improved simply by following internal naming conventions.

  Example: Add default annotations to all fields in models that inherit the partial mixin

  Example: Add a plugin annotation to all fields including xyz

• Templates and transformers support

  If any form of templates support is introduced in the future, it may be possible to create basic macro-like functionality by combining various features of ZenStack with partial mixins and templates.

  This may even allow definition functionality directly in a schema to achieve outputs that would otherwise need a custom plugin. Particularly beneficial if regex field names are supported in a partial mixin

Implementation

Avoiding errors

To prevent unintentional errors by developers, it should only be possible to apply a partial mixin where every field in the partial also exists in the type or model it is applied to.

For example, the following should result in an error:

partial Wompinator2000 {
  womp_womp_2 @icon('not-found')
}

model Wompinator with Wompinator2000 {
  womp_womp String
}

Future iterations could include opt-out mechanisms to allow only applying annotations for fields that exist in a model and ignoring missing fields. This should be explicitly opt-out though if implemented as an option.

Conflict resolution

It is probably reasonable to document this as an advanced feature. Developers will need to be aware of how they use plugins and partial mixins, both when writing:

• schema definitions — i.e. may need to be mindful of the order that mixins are applied
• plugins — i.e. avoid assuming knowing the field datatype unless absolutely required

Some basic safeguards may need to be put in place to catch conflicts and bugs early. For example:

• Error or warn if duplicate annotations are found — i.e. multiple @default annotations applied
• Allow plugins to specify when included custom annotations require a specific datatype, and validate that is reflected when a partial mixin is applied to a model

Considerations

Limitations, breaking changes or other impacts

None identified currently, besides development time required. Please add any as they are identified.

Alternative implementation options

It could be possible to just allow the current type mixin to support merged fields. I personally believe the benefits of ensuring a type mixin is strongly defined by default outweigh having a reduced number of options to learn (see Rationale section).

It could also be possibly to reuse the abstract keyword from V2 to modify the behavior of type mixins. I think it is easier to conceptualize the idea of there being multiple mixins available, than how and when to modify mixins.

Reusing a keyword may also introduce confusion for developers migrating between versions. It is also for this reason that a new keyword (partial) has been proposed instead of reusing abstract as the name of a new mixin option.

Alternative syntax options

The provided example assumes that a partial mixin would be applied using the same syntax and keyword (with) as a type mixin. Consideration should be given to whether that could lead to any drawbacks. There are two other options (and a third hybrid, combining both options):

• Use a different keyword to apply a partial mixin to a type or model

  I am not particularly keen on this option, as it dilutes the singular concept of a mixin. Using the same keyword (with) for all mixins ensures it becomes a familiar concept, regardless of what is being mixed in.

  partial Buttercup {
    id // Adding nice stuff ...
  }
  
  type World {
    id // Adding boilerplate stuff ...
  }
  
  type Hello with World and Buttercup {
                      // ^
                      // |-- No idea what the hypothetical
                      // |-- keyword should even be ...
  
    id Int @id @default(autoincrement())
  }

• Add partial mixins after the type or model definition

  This would have the benefit of ensuring any mixins that include typed field definitions (a final shape) have priority when viewing a schema. It would also allude to the fact it is applied after the core model structure has been defined.

  partial Demo {
    id // ...
  }
  
  partial Lition {
    id // ...
  }
  
  model Darkness with Hello {
    friend String @default('old')
  } with Demo, Lition
  
  // Wow, this poor `id` field is getting a real workout

Footnotes

1. A mixin may already be intended by the ZenStack team as a more generic concept. If so, clarification may be required in the beta documentation and relevant comms. ↩

2. I have made a massive assumption here on the purpose of this open issue, and could be wrong. Apologies if I have! Although it is titled "Non-model types", it appears to me a loosely coupled collection of abstraction requirements that may need similar, but slightly tailored solutions. My identification of its' importance is on the basis it remains in the 2.x milestone and is tagged for "feedback" and "in-progress", but I also recognize internal direction or views may have diverged from the public facing records associated with V2 as V3 is developed. ↩

[ymc9]

Thanks for creating this @binaryben . I agree that being able to split models into multiple pieces is very useful, especially when you have lots of annotations on them.

I'm thinking about this along a slightly different direction. How do you feel about making models and types "partial" by default? Basically, if you define a model/type multiple times, fields are merged. It's a bit like how TypeScript automatically merges interface declarations.

[binaryben]

Thanks @ymc9, happy to hear the identified problem is valid and being considered 😄

How do you feel about making models and types "partial" by default? Basically, if you define a model/type multiple times, fields are merged. It's a bit like how TypeScript automatically merges interface declarations.

This was my original thought on how to allow splitting up where annotated attributes are applied to a field as well. In particular, I wanted to be mindful of suggesting any solutions that would need more development effort than absolutely required to solve the problem.

View detailed response ...

Allowing duplicated type and model definitions to be merged along with enabling partial definitions (not requiring the data type to be declared) by default, appeared like it would be a straightforward approach to solving some of the current challenges.

I ended up landing on the view that enforcing having a single place in a given schema to define the included fields with their data types is very beneficial though:

This is an important feature of type mixins, as it directly supports having Strongly Typed JSON Fields. It is very important to maintain that approach of having a single place where the final "shape of the data" is defined. If a type could be defined in multiple places, it risks becoming inconsistent and unreliable.

To clarify, I think it is too risky to allow otherwise. Personally, I think the importance of this is even higher than with a Typescript class or interface for example, since the schemas defined here affect the database layer indirectly via migrations. It is easy to forget having already defined a type or model called XYZ elsewhere, being unaware someone else defined a type or model with the same name, or even to just make a typo (now a new table is added, or the wrong one was modified).

Accidentally altering YXZ when you meant to alter XYZ would become a common and difficult to identify bug. The possibility of these bugs being introduced to a schema become more likely as we define types and models across multiple files (a feature I am incredibly grateful for). Requiring a uniquely named partial to be "mixed in" via the with keyword enables an IDE to highlight these kinds of bugs immediately.

Of course, as developers, we would want to implement regression and fuzz testing to prevent this kind of issue ever landing in production. We also want to have backups, so if a migration unintentionally deletes data it shouldn't have, we can recover. Recovering from backups takes time and sometimes backups fail regardless of best plans and intentions though. If problems that can occur from typos or forgetfulness can be prevented via the design of the DSL, I believe it should be prevented that way first.

---

My thoughts on why I would recommend against this approach to the syntax are largely the same reasons that I set the typescript-eslint no-unsafe-declaration-merging rule¹ to be treated as an error (part of the recommended-type-checked configuration). It might be nice as an option — ESLint allows developers to add a comment to disable a rule after all. I think it would be dangerous as the only or default way though.

Conceptually, it would also require the implementation to be more like making a property (ie. a ZenStack field) or method partial by default using the Typescript analogy, rather than an interface (type) or class (model) partial by default. Typescript's declaration merging doesn't support extending an already defined method or property (results in a compiler error). Once a property or method is added, the data structure of the property/method is considered relatively immutable².

This aspect is already supported by using the with keyword in Zenstack V3 anyway.

---

Partial by default would also mean the additional benefits around having composable attribute annotations on type and model definitions would be lost. We would still be repeating attribute annotations against every type or model that uses a subset of the same attributes on named fields where we need a few variations. I believe access policies in particular will benefit here.

Edit: On further reflection, I can see how if a type can be partial by default, it could theoretically be possible to still have nearly the same level of composability. It assumes a type can remain partial though (ie. is never fully defined to have a set data type) or used as a Strongly Defined JSON field. Variations (partial types) could then be mixed into several definitions of "real" types and models. This seems messy to me though. How would ZenStack treat a type that isn't fully defined then by default? A partial mixin would be both conceptually simpler and far safer in my mind.

Footnotes

1. I personally use Biome more lately, which enables the rule by default. I referenced the ESLint variant specifically since it is still the defacto linting tool currently. ↩

2. I am blatantly disregarding what is actually possible with the Javascript it is transpiled to or the various ways to break out of Typescript's guard rails ↩

[ymc9]

Got it. I can understand that an implicit auto-merge can be dangerous if used without awareness. If we go down that path may be we should make it a configuration, or a warning that needs to be explicitly suppressed, things like that, to improve safety.

I think a simple way to allow partial-by-default, is to require all fields to always carry there types, so each instance by itself is always "complete":

model Foo {
  id Int
}

model Foo {
  id Int @id @default(autoincrement())
}

The field types need to be consistent to have a successful merge. It comes with the cost of duplicated information.

[binaryben]

Closing as you're clearly angling for an easier solution to a different problem (ie. merge duplicate definitions during AST). Thank you for considering this request anyway.