Data Model - Normalization and Syntax(Typescript)

[romulocintra]

This discussion is a placeholder to concatenate 2 different tasks :

1- Definition in how do we express our schema #139
2- How we work as a group to normalize and merge Data Models #140

[eemeli]

My data model draft is now available here, along with a few examples of it in use.

I'm really quite happy with how it looks like it ought to be possible to keep the model conceptually pretty simple; I at least ended up only needing to define the following: Bundle, MessageSet, Message, Placeholder, Reference, Select.

[romulocintra]

Great work @eemeli !!! and i think that is a different POV, definitions are pretty neat, in the example implementation I have some doubts but looks good.

I'm not sure if transform could be and implicit select.

[mihnita]

My model here, using TypeScript:
https://github.com/mihnita/msgfmt_experiments/blob/master/ts/

It reflects the model proposed previously in slides.

The model itself:
https://github.com/mihnita/msgfmt_experiments/blob/master/ts/src/imessageformat.ts

And example:
https://github.com/mihnita/msgfmt_experiments/blob/master/ts/test/messageformat.test.ts

[eemeli]

For the sake of completeness, even though they're Rust rather than TypeScript, here are the two other data models:

@zbraniecki
https://github.com/zbraniecki/message-format-2.0-rs/blob/ast/src/ast.rs

@echeran
https://github.com/echeran/message-format-2.0-rs/blob/echeran-initial-work/src/data.rs

[zbraniecki]

My unordered comments, comparing the other ASTs to mine:

Echeran

I don't understand why Elango's model lists Plural and Gender as "magical" PlaceholderType. I believe we should treat all selectors the same way.
I'm confused why MessageBase stores PHValsMap which seems to be conflating AST with runtime contexts.
I don't see a way to pass arguments to a PlaceholderType::OTHER, which means I don't understand how calls to say, date and time formatting are meant to work. Something that in Fluent would be:

key = Today is { DATETIME($date, dateStyle: "long") }

I think they keys for variants can only be strings, and I believe we should allow them to be numbers as well.
I'm also struggling with the complexity of the MessageGroup and selector AST in this model.

Mihai

It shares with Elango's the idea that the AST will encode the type of the selector. I don't understand what's the reason for that. I come to the conversation with the proposal to treat any selector as a function call with arguments that can return a value that we can match to keys of variants. Nothing is "special".

I don't see how it passes variables to a ISwitch, or arguments, and the distinction between ISwitch and IPlaceholder is only preserved in the idea that ISwitch have to match against keys, and we may want to enforce some meta-information on what are possible return values to aid tooling, while IPlaceholder may return more free-form data.
I'm still not sure how much we want to clamp ISwitch, because I see a value in having the information on what values ISwitch can return, but I also see a scenario where we want to allow for more open-ended ISwitch that matches some variant keys, and goes to default variant otherwise.

By the way, I don't see any information in any of our ASTs on which variant should be used in case none matches. I think it's important to either have it encoded in AST or standardized that the "last one" is (and then enforce at least one variant).

Eemeli

This model is closest to my mental model. It's interesting that it encodes a nested MessageSet model, which allows for arbitrary depth of messages to exist, effectively closing the gap between what Fluent calls compound messages or we named localization units. I like it I think.

The keys of the select are strings, I think we should allow for numbers.
If I understand correctly, the Placeholder doesn't allow for multiple variables to be passed to a single "transform".

=======

Reviewing the other models and mine, I see the following shortcomings of my proposed model:

• I think we should encode default variant of a selector in AST
• The FunctionReference should allow for kwargs aligning it with ECMA-402 Options Bags, much like Fluent does: DATE($date, key: "value", key2: 2.5) or DATE($date_start, $date_end, approximate: "true") style.
• The flattened multi key select will require a lot of thinking on fallbacks

I'd love to hear people's take on the other shortcomings of my model.

[eemeli]

The keys of the select are strings, I think we should allow for numbers.

Probably? I mean, there clearly are situations in which at least exact integer numbers will need to be expressible as keys of selector cases/variants, but I'm not sure that this actually requires for those keys to be parsed from their source into numbers. I agree that doing so would make semantic sense, but on the other hand it slightly complicates the procedure for selecting one of a number of cases. Once we have at least a sketch of the data model, figuring out that procedure should be one of our next questions, and its resolution may well feed back into specifying our data model further.

I sketched out one possibility of such a procedure in #101, and there's another I've thought of since (resolve a Placeholder into a string[] of keys to test for in order, so with plural the value 1 would in English resolve as ['1', 'one', 'other']), and both of those are actually simpler if keys are always strings.

So my position on this is that we have a demonstrated need to allow for string keys, but we do not necessarily have a demonstrated need for allowing number keys. If/once we can demonstrate that need, then we should expand the data model to accommodate for that.

If I understand correctly, the Placeholder doesn't allow for multiple variables to be passed to a single "transform".

Yes, that needs to be fixed. Situations do exist which require more than one variable to be passed (e.g plural ranges, #125), and that needs a solution. Preferably one that doesn't unduly complicate the ergonomics of more common single-variable arguments.

By the way, I don't see any information in any of our ASTs on which variant should be used in case none matches. I think it's important to either have it encoded in AST or standardized that the "last one" is (and then enforce at least one variant).
[...] I think we should encode default variant of a selector in AST

I agree. I also think that having this enforced by an ordering rule or by making a key like other always magical would be unnecessarily opinionated of the data model. Otherwise, we could easily lose forward compatibility from at least Fluent, as well as any other source format that allows for the default to be user-defined.

The place in my own data model where I'd add this would probably be as a field default?: string in the Placeholder, a bit like Elango has in his. It probably should not be next to the key/message pair, as a default: true value there would not be sufficiently precise to deal with multi-argument selectors. That would also work for Placeholders in messages, to provide a fallback string in case e.g. the transform call somehow fails.

The FunctionReference should allow for kwargs aligning it with ECMA-402 Options Bags, much like Fluent does: DATE($date, key: "value", key2: 2.5) or DATE($date_start, $date_end, approximate: "true") style.

Would Record<string, string | number | boolean> be sufficient as a type for the options bag?

I'd love to hear people's take on the other shortcomings of my model.

I think using just a string identifier for VariableReference won't be enough. Or that some other means is required in order to make other messages either in the current bundle, or even a different bundle, addressable.

I have reservations about not making a clear difference between the inputs and the options that are being fed to what you call a FunctionReference and I a transform. If the options bag is independent of any placeholder values, it becomes significantly easier to construct and cache a formatter for it.

For example, MF1 currently supports skeleton strings for formatting numbers and dates. If that skeleton string were to be user-definable, the runtime environment where such a message needed to be formatted would need to have a skeleton string parser available. On the other hand, if the skeleton needs to be a literal value, it may be parsed during build-time, allowing for the generation of a substantially leaner and faster runtime formatter function that does not itself depend on a skeleton string parser.

[zbraniecki]

So my position on this is that we have a demonstrated need to allow for string keys, but we do not necessarily have a demonstrated need for allowing number keys. If/once we can demonstrate that need, then we should expand the data model to accommodate for that.

My take is that matching a numeric output of a formatter, to a string key is a bag of worms and I'd like to avoid that. I'm not convinced that it complicates the model, and I believe that Fluent approach works quite well here, but my opinion is weakly held here.

Would Record<string, string | number | boolean> be sufficient as a type for the options bag?

We spent a lot of time in Fluent debating if there should be boolean type. We ended up not including it. I'm open to change that, as I think options in ECMA-402 have boolean flags in reasonable quantity and for a reason. Coalescing that to string "true" and "false" is similarly undesirable for me as it is for numbers coalescing above.

I think using just a string identifier for VariableReference won't be enough. Or that some other means is required in order to make other messages either in the current bundle, or even a different bundle, addressable.

VariableReference is different from MessageReference in my model. One can, in the MF2.0 syntax, reference a variable (Fluent's $userName) or another message (Fluent's key2).

Two observations:

1. String indeed may not be enough, if we want to allow for structured variables to be passed. Instead of {userName: "John", userEmail: "[email protected]"} we may want to allow for {user: {name: "John", email: "[email protected]"}}. We didn't end up doing it in Fluent, but I can see a reason to revisit that. In that case, we may want to allow for VariableReference to be Vec<String> for user.name encoding.
2. There is a difference between MessageReference and dynamic message reference, which is a VariableReference where the variable's type is MessageReference.

I have reservations about not making a clear difference between the inputs and the options that are being fed to what you call a FunctionReference and I a transform. If the options bag is independent of any placeholder values, it becomes significantly easier to construct and cache a formatter for it.

I'm easily convinced by this argument. I suggest that we consider FunctionReference to be:

struct FunctionReference {
    arguments: Vec<InlineExpression>,
    options: Vec<(String, OptionValue)>
}

This would encode DATE_RANGE($dateStart, $dateEnd, style: "long", grouping: true) to be encoded as:

FuntionReference {
    arguments: vec![
        InlineExpression::VariableReference("dateStart"),
        InlineExpression::VariableReference("dateEnd"),
    ],
    options: vec![
        ("style", OptionValue::String("long")),
        ("grouping", OptionValue::Boolean(true)
    ]
}

I would encourage the group to consider:

1. Arguments don't have to be just variables. Someone may want to pass a message reference to a function: { UPPERCASE(another_message) }.
2. Option values may also be variables or functions or other expressions: { TITLE($variable, prefix: UPPERCASE(another_message)) }

I dislike the term transform, because it indicates that the only function a function (sic!) may serve is transformation of some value. But it doesn't!
Examples:

• key = You are using { PRODUCT_NAME() }
• key = Click on { MESSAGE($another_message_key) }

[mihnita]

I've also "ported" Zibi's and Elango's models to TS, to the best of my understanding.
I just didn't sent a note out, because I wanted to review them once more:

• https://github.com/mihnita/msgfmt_experiments/blob/master/ts_zibi
• https://github.com/mihnita/msgfmt_experiments/blob/master/ts_elango

[mihnita]

I come to the conversation with the proposal to treat any selector as a function
call with arguments that can return a value that we can match to keys of variants.
Nothing is "special".

And that is what I have in my model.
But I didn't put there a "function" or anything like that, because it has to be

• programming language independent
• need to be able to "register" some of them as "well known selectors" (in a "registry")
• move it through xliff and back

In other words, one has to be able to serialize it in some standard way.

So in my mind the type: string in IPlaceholder is really a function name.
Or a name of a function, of functor class, or whatever, that you "bind" to whatever you need.

We can discuss how to represent this in TS to make it more clear.
But not all programming languages have clear concepts for a "function" that you can store in a date model.

For example I see a use case for a class + special method(s).
Imagine this:

GramarService gs = new GramarService(user, password, ...);
MessageFormat.register("grammar", gs::resolve);
...
mf = MessageFormat.createInstance(locale, ".... {color, grammar, gender}...");
mf.format({color:"red"});

Anyway... not a simple function.

For Java I can thing of soo many ways to do that: an interface or class with a fixed method name (old style), functional interface (new style), lambda, method reference (since 8), java.util.function.Function

And I think that the function should be well defined in our standard (what is the input, and what is the output)

We can discuss how to represent this in TS to make it more clear.
I don't think there is a conflict.

---

I don't see how it passes variables to a ISwitch, or arguments, and the distinction between ISwitch and IPlaceholder is only preserved in the idea that ISwitch have to match against keys, and we may want to enforce some meta-information on what are possible return values to aid tooling, while IPlaceholder may return more free-form data.

Yes, needs refining.
I have some Java code, and some JS code. Need to "port" it to TS, and see how to "capture" that in TS definitions.

By the way, I don't see any information in any of our ASTs on which variant should be used in case none matches.
I think it's important to either have it encoded in AST or standardized that the "last one" is (and then enforce at least one variant).

I think it should be standardized, but it is useful to have more than "one default"

Example (crazy syntax that does not look like anything, to make it clear is it not about syntax :-):

switch ( [ {foo::gender, bar::plural, grammar::case] )
    [feminine, one, genitive] : "Some message ..."
    [feminine, one, *] : "..."
    [feminine, *, genitive] : "..."
    [f*, one, genitive] : "..."
    [*, one, *] : "Some message fallback / default if I don't have grammar info / services and I don't have gender info"
    [*, *, *] : "Some message, ultimate fallback / default"

The exact example does not matter, and the "*" is silly, I'm not pushing for that.
It is just the idea that it can be handy to have "partial defaults"

In my TS I added a comment to the map saying "The order matters..."
Because [feminine, *, genitive] and [feminine, one, *] have the same "score" (2/3 match). So which one wins?

Note: AST != Data Model :-)

[eemeli]

I've refactored my data model & examples to account for the reference discussion above with Zibi. I have not added default case/value indicators at least yet because I don't know of a single straightforward manner in which that ought to be done. We'll need to talk in more depth about how case selection really works, as well as how error fallbacking works, because that's all yet another interconnected mess for us to detangle.

Further replies to @zbraniecki:

My take is that matching a numeric output of a formatter, to a string key is a bag of worms and I'd like to avoid that. I'm not convinced that it complicates the model, and I believe that Fluent approach works quite well here, but my opinion is weakly held here.

To be clear, allowing for numbers doesn't complicate the model, it complicates the procedure for selecting a case, and it may complicate the permissible return types of a function, when one is being used as a selector.

1. String indeed may not be enough, if we want to allow for structured variables to be passed. Instead of {userName: "John", userEmail: "[email protected]"} we may want to allow for {user: {name: "John", email: "[email protected]"}}. We didn't end up doing it in Fluent, but I can see a reason to revisit that. In that case, we may want to allow for VariableReference to be Vec<String> for user.name encoding.

I hadn't thought of that previously, but you're right. This also happens to provide a rather nice type disambiguation between literals and variable references.

2. There is a difference between MessageReference and dynamic message reference, which is a VariableReference where the variable's type is MessageReference.

Yes. And my model was conflating those two in a manner that it shouldn't have.

I would encourage the group to consider:

1. Arguments don't have to be just variables. Someone may want to pass a message reference to a function: { UPPERCASE(another_message) }.

Also, arguments shouldn't have to be references. If/as we have good reason to allow for function calls with more than one argument, in some cases some of those arguments will be desirable to take literal values.

2. Option values may also be variables or functions or other expressions: { TITLE($variable, prefix: UPPERCASE(another_message)) }

This one I'm not (yet...) so convinced by. To me, TITLE here looks like it really ought to be a message rather than a function, and for that at least my model would allow for the prefix to be passed as a scope variable. I think the case for allowing functions to take in named as well as positional arguments (in addition to named options) isn't strong enough to balance its cost in terms of complexity.

I dislike the term transform, because it indicates that the only function a function (sic!) may serve is transformation of some value. But it doesn't!

I actually didn't like it either, so I'm happy to have refactored it away.

Finally, replying briefly to @mihnita:

The exact example does not matter, and the "*" is silly, I'm not pushing for that.
It is just the idea that it can be handy to have "partial defaults"

I actually rather like * as a fallback-value indicator. Compared to other, it far more clearly communicates that it covers any remaining cases. Fluent uses it as a decorator of sorts for the default case, but I was thinking that it could be used much like other in MF1 (as your example does). If there's a reason to use something different for some specific case, then that could be done by wrapping the selector value in a FunctionReference that takes in an option like default: 'other'.

[mihnita]

Copied all 4 current models to the experiments branch:
https://github.com/unicode-org/message-format-wg/tree/experiments/experiments/data_model