Are terminal fragments supported?

[paulfrench]

Langium seems to accept them but I don't know if I can use them. Also they seem to mess up the parser anyhow.
e.g.

terminal fragment INT: /[0-9]+/;

terminal NUMBER returns number:
	/(\-|\+)?[0-9]+(\.[0-9]+)?((e|E)(\+|\-)[0-9]+)?/;

My terminal NUMBER no longer works if I introduce the fragment but the grammar generates fine?

I've only just realized Langium is using regular expressions for it's terminals and XText uses Extended Backus-Naur Form-like (EBNF) expressions.

I keep learning :)

[msujew]

Hi @paulfrench

First of all, compared to Antlr, our lexing/parsing library Chevrotain does not take the longest match while tokenizing. Therefore, terminal order in your grammar is really important. I assume that you use NUMBER in your rules, but INT is likely to be lexed instead, if you have input like 123. Generally, you should avoid overlapping terminal definitions, since their subject to ordering issues. Additionally, terminal fragments currently behave in the same way as normal terminals.

Secondly, EBNF style syntax for terminals and terminal fragments will be supported when 0.3 lands. I implemented this in #288. In our first draft we oriented ourselves at how Chevrotain accepts token definitions, which is in Regex format.

[spoenemann]

Is there an issue about terminal fragments? They should be ignored when creating token types and using them in parser rules should be forbidden, right?

[msujew]

Currently, the fragment property is completely ignored, both for validation and token type generation purposes.