Possible Langium parser bug

[Greglar4]

Hello,

We are using Langium in our application, and have noticed a scenario where the parser throws an error that does not seem to be correct.

In our application we have syntax very similar to that shown in the following repository:

https://github.com/Greglar4/duplicateKeywordBug

If you follow the instructions in the readme it should demonstrate the issue:

This repository demonstrates a potential bug in the Langium parser. To see it, do the following:

1. Open this project in VS Code
   

2. Run npm install
   

3. Run npm run langium:generate
   

4. Run npm run build
   

5. Run the extension in the development host (e.g. by using F5)
   

6. Create a file with .menu extension
   

7. Place the following contents into the file
   

vegan food []

vegan drinks []

8. There will be a parser error on the food token which says "Expecting token of type 'drinks' but found 'food'"
9. I would expect that it would complain about the fact that there are no string tokens in between the hard brackets.

I do not know enough about parsers to know if this is due to a limitation of parsers in general in relation to the desired syntax, or if this is a bug with the Langium parser, so I did not want to raise a bug directly.

It seems to be important that the entire vegan food and vegan drinks sections are optional.

Thoughts on this would be appreciated!

Thanks,

Greg

[msujew]

Hey @Greglar4,

the issue is likely a limitation in the lookahead. You have a few options in the grammar which are somewhat ambiguous, which result in the lookahead engine trying to scan ahead, but due to missing STRING tokens in your input, it essentially says that vegan food [] is not valid in terms of the parser lookahead. I might have accidentally fixed that by fixing a different bug in the lookahead engine. You might want to try to force-update the transitive chevrotain-allstar dependency to 0.4.2. Alternatively, you can probably change the maxLookahead value in your language config to something like 3. That should work quite well.

Please note that I wouldn't recommend designing your grammar in such a way. Ideally, parser rules should be small and flexible. I.e. unless absolutely necessary, I would split the grammar like shown in here.

[Greglar4]

Thanks for the information! This indeed doesn't happen on chevrotain-allstar 0.4.2

[msujew]

FYI, chevrotain-allstar 0.4.2 broke a few other things, so I had to revert those changes again. See #2146. My other recommendations are still valid though.

[Greglar4]

Alright.

For your recommendation about the grammar design, we would prefer if there could only be one declaration of each of the parser rules (vegan food/nonvegan food/vegan drinks/nonvegan drinks) in a Menu. Would it be possible to improve on our grammar, making the parser rules smaller and more flexible while still enforcing a single occurrence of the parser rules?

Thanks

[msujew]

Would it be possible to improve on our grammar, making the parser rules smaller and more flexible while still enforcing a single occurrence of the parser rules?

Well, yes and no. My recommendation is to parse more flexibly and to flag erroneous state using the validation feature.

Currently, you're trying to ensure in the parser that the input can only be parsed, if it is semantically valid - however, that's a task that becomes exponentially more difficult the more complicated your language becomes. Instead, it is usually better to use validations/diagnostics to present a fitting error to the user like "Only a single list of vegan food items is allowed", instead of a parser error like "Expected token X, but received Y". In short: Keep the language more flexible, allow for semantic errors, and use the validation to flag them.