Code actions: pushMode/popMode/...

[Lotes]

Antlr4 has this feature 'lexer modes', where you can set, push and pop a mode as extra state. This enables the user to define tokens for WITHIN a string or a comment, and at the end switch back to the previous mode.

You can have it somehow in Langium when you implement your own TokenBuilder.

My question is: Should Langium also have this option (new code actions)? Is there a need by Langium's users?

//I am also asking to get this idea out of my head :3 tehee

[montymxb]

Are you thinking about adding this kind of support to the syntax of Langium? If so I've thought about it as well, similar to how Monarch structures which tokens can be recognized by state. By default the behavior would just be the current state, but it could get tricky if a user wants multiple states to still recognize the same token.

[msujew]

I have two thoughts on this, one in favor of this idea and one against it:

• [+] It would make information about lexer modes statically available. That is both easier to read and easier to statically verify. Currently, if you build a multi mode lexer using the TokenBuilder, you only know whether it works correctly (or works at all), once you start parsing code. With static availability, we can already verify the lexer behavior when running langium generate or maybe even earlier in the grammar validation phase in the editor.
• [-] It increases complexity of the grammar language. My original idea was to improve the API of the TokenBuilder to facilitate easier multi mode lexing instead of adding more complexity to the grammar language.

@spoenemann Your thoughts on that? Do you think the feature is worth the complexity/effort?

[Lotes]

I have 4 proposals…

1. take the same approach as Antlr4 and introduce code actions pushMode(name) and popMode. The consequence is that you have to introduce regions of tokens. Each mode has a set of tokens. Potentially duplicating existing tokens from other modes. This is a very pragmaatic approach.

2. you can create the same mechanism without polluting the syntax of Langium. By adding a custom TokenBuilder. Where you have to again, define your modes of tokens and care about key tokens that switch between modes using push and pop semantics.

3.1) introducing annotations like in Java to add metadata to your rules. Within the token builder, you can lookup this data inside of a Langium grammar instance.
3.2) you can also introduce JavaDoc as annotation. Within the token builder you can lookup the modes using reflection and setup modes again.

4. The actual goal is to mix grammars. To do that, you have to switch the lexer mode/token set via push/pop. What I actually would love to see is:

grammar Xml;

import Dtd; //the name of a different grammar!!!

entry XmlDocument: decl=XmlDecl dtd+=Dtd* root=XmlElement;
//I hope the DTD position is correct

XmlDecl…

So when XmlDecl ends and a grammar call might be expected, we could push the mode to the DTD lexer. When the DTD parser reaches EOF, we pop the mode.

[spoenemann]

The parsing approach that we're using separates lexing and parsing into separate phases, so we can't mix lexer mode switches into parser rules.

Of all the options you described, I currently like 1. the most. Something like this should work?

lexermode M1
terminal T1: {push mode M1} ...;
terminal T2: {pop mode} ...;

[Lotes]

4.) could work when we request tokens from a tokenizer using JS generators. But I am not sure if that is sufficient in cases where it has to fetch enough tokens to make a decision. It would indeed complicate Langium.

1.) is the good and way that is already tested by ANTLR4. And yes, you would have...

mode A:
terminal X: ...

mode B:
terminal Y: ...
terminal X: ...

Suggestion: We could put the terminals for one mode into parentheses mode XXX{ terminal A:...} to have it in a more clean way.
In Antlr4 you can mix parser and terminal rule, which can be overwhelming. But that's just details...

[spoenemann]

4.) could work when we request tokens from a tokenizer using JS generators.

I think Chevrotain doesn't support that:

export interface IMultiModeLexerDefinition {
  modes: MultiModesDefinition
  defaultMode: string
}

export type TokenTypeDictionary = { [tokenName: string]: TokenType }

export declare type TokenVocabulary =
  | TokenTypeDictionary
  | TokenType[]
  | IMultiModeLexerDefinition

Suggestion: We could put the terminals for one mode into parentheses mode XXX{ terminal A:...} to have it in a more clean way.

Some terminals should be available in multiple modes, so the modes should be a property of a terminal?

[Lotes]

Some terminals should be available in multiple modes, so the modes should be a property of a terminal?

That is also an idea. It was already proposed with 3.1 and 3.2 ...

One has to provide a method that introduces a n:m mapping between terminals and modes, where terminals must be in a specific order (a keyword can be shadowed by ID, when ID is defined first).

So terminals can be in multiple modes, and a mode contains multiple terminals. Annotating the terminals with their modes as a list of strings can be a nice solution. Maybe I dropped too much information, to make my proposals fully understandable?!

I think Chevrotain doesn't support that

Ok, let's do not reinvent the wheel. I mean in theory you can do so much, but when Chevrotain does not provide such a on-the-fly token generation, let's live with it...

[TheJim03]

Hi everyone,

I wanted to revisit this discussion.
I'm currently migrating a production DSL from ANTLR4 to Langium where lexer mode support is a hard requirement. The grammar uses multiple modes with pushMode, popMode, and mode() (stack replace) actions, including multi-push cases where a single token triggers two or three consecutive mode pushes.

I've implemented a workaround using a custom TokenBuilder with Chevrotain's IMultiModeLexerDefinition and custom token patterns for the multi-push cases, but it requires considerable boilerplate and is fragile to maintain compared to the original ANTLR4 grammar.

Is there any update on whether native lexer mode support is planned? I am working on a proof-of-concept and would be happy to contribute.

Thanks!

[msujew]

Hey @TheJim03,

The grammar uses multiple modes with pushMode, popMode, and mode() (stack replace) actions, including multi-push cases where a single token triggers two or three consecutive mode pushes.

I'm not sure this is even possible with Chevrotain? At least the stack replace and multi-push actions. Does Chevrotain's implementation purely using pushMode and popMode work for you?

Is there any update on whether native lexer mode support is planned? I am working on a proof-of-concept and would be happy to contribute.

Not planned directly, though we would accept proposals in that direction. I don't think we currently have an open issue for this topic. I would recommend to create a feature proposal issue, in which you outline the syntax/API changes/improvements you have in mind. That way, we can discuss some things in advance, before having to do this discussion in a PR.