Custom lexer for Python-like languages

[vlkv]

Hello,

I am trying to figure out how to use some more sophisticated lexer instead of simple declaration of several terminal regexps. More specifically, I want lexer to produce into the resulting token stream additional tokens INDENT and DEDENT (which will mark out beginning of scope and ending of scope correspondingly). I found that in pure Chevrotain this task is accomplished with a bunch of JavaScript logic. But how to integrate such a logic (or similar) into Languim?

Thank you.

[msujew]

Hey @vlkv,

you can override the TokenBuilder service to implement the Chevrotain pattern matching function in Langium.

[aabounegm]

Update: the IndentationAwareTokenBuilder has been released in v3.2.0, and there is now a recipe explaining how to use it.