Syntax errors are challenging to parse

[aaronjensen]

I'm not sure if this is a prism concern or a Ruby concern, but the syntax errors that Ruby prints are relatively challenging to parse for something like flycheck (flycheck/flycheck#2111). It's easy enough if we only want to highlight the very first syntax error, but if we would like to highlight all of the lines that there are errors along with the message it would require a multi-line regular expression (and counting spaces to ascertain the column). For a human they look great, so my concern is only about machine parsing.

$ echo "do\ndo" | ruby -wc
/redacted-path/ruby: -:1: syntax errors found (SyntaxError)
> 1 | do
    | ^~ unexpected 'do', ignoring it
> 2 | do
    | ^~ unexpected 'do', ignoring it

I'm curious if there could be a mode where Ruby printed the syntax errors each on their own line, similar to how warnings are printed:

$ echo "1 +x 2 do\ndo" | ruby -wc
-:1: warning: '+' after local variable or literal is interpreted as binary operator even though it seems like unary operator
-:1: warning: possibly useless use of + in void context
-:1: warning: possibly useless use of a literal in void context
/redacted-path/ruby: -:1: syntax errors found (SyntaxError)
> 1 | 1 +x 2 do
    |      ^ unexpected integer, expecting end-of-input
    |        ^~ unexpected 'do', ignoring it
    |        ^~ unexpected 'do', expecting end-of-input
> 2 | do
    | ^~ unexpected 'do', ignoring it

It's also unclear why the interpreter path is included in the syntax error output but not for the warnings. It's a seemingly odd inconsistency, but perhaps there is a reason for it.

If I should post this message to https://bugs.ruby-lang.org/ instead, please let me know. Thank you!

[bbatsov]

It's also unclear why the interpreter path is included in the syntax error output but not for the warnings. It's a seemingly odd inconsistency, but perhaps there is a reason for it.

I'm curious about the reasoning behind this as well. Especially, given that it wasn't like this before Ruby 3.4. It seems to me it'd be nice to either have detailed diagnostics for both warnings and errors or not for both of them.

I agree with @aaronjensen that ideally there should be a mode for humans (detailed diagnostics) and something more compact for editor lint plugins and similar tools, that is easier to parse.

[kddnewton]

Hey there. I definitely see what you're saying here, we really shouldn't be parsing formatted syntax errors, that's difficult.

From Prism's perspective (this repo specifically) all we do is bundle up the errors and warnings in the same kind of objects. In C, it's a linked list of objects that contain location information, a type, and a message. So from here, I think this is about what you would expect from the parser.

From CRuby's perspective, things get a little weirder. Warnings are gathered up and printed out one line at a time, and errors are grouped together into a singular syntax error and raised all as one. I've raised issues before (https://bugs.ruby-lang.org/issues/20024) about trying to get more detailed error messaging information, but haven't gotten too far.

Let me back up and ask your specific use case. Is a human or a computer running this code? Is this editor feedback? Are you in a place where you could manually parse the source Prism.parse(...) or is this basically parsing stderr?

In the simplest case, it might be easiest to add an environment variable that's like PRISM_ERRORS_PLAIN or something like that.

[aaronjensen]

The thing that led me here was attempting to add a computer-based parsing of Ruby's errors in a lint error displaying plugin for Emacs called Flycheck. It invokes Ruby with -wc. A different checker could potentially be written that invokes Prism directly, but I'm not sure how we'd do that and also get the Ruby warnings. I suppose we could syntax check first with Prism and skip the warning checker if the syntax check fails.

It does seem like from the Ruby issue you linked that you've considered this, so that's good. I'm not quite sure why the format changed so much in Ruby with the inclusion of the ruby path. Seems like that may have been a mistake. Not sure if that could be corrected.

What I'm hearing though is that it may be a lift at this point for Ruby to emit more than one syntax error, so I guess consider this me expressing interest in that functionality for Ruby. If it would be helpful for me to chime in on that Ruby issue, I'd be happy to.

[kddnewton]

@aaronjensen do you have the ability to use Prism itself? You could parse the code with Prism.parse and then will have access to an array of errors with location information and all of that. You will be able to format them as you like. Doing it as a one-liner would be something like:

echo "do\ndo" | ruby -rprism -e "pp Prism.parse_stream(STDIN).errors"

That would give you far more control. If you can't do that for whatever reason, I will see about adding an environment variable to get more parse-able error messages.

[aaronjensen]

@kddnewton I could, yes, it would require a new flycheck checker in this instance and I believe the Ruby-specific one would still be in play to deal with warnings. I probably won't do this in flycheck because the first syntax error is enough for me, personally.

It does seem like the inconsistency in Ruby should be addressed as a matter of principle, but I can understand the lift that that would take and why it may not be worth it.

[kddnewton]

You can get the warnings off the parse result as well, they're in there on Prism.parse_stream(STDIN).warnings. So in theory that should alleviate some of it?

[aaronjensen]

That’s handy. After thinking about this more it’d also require a version check since not every version has Prism. I’ll likely live with what we have unless you end up making upstream changes.

Thanks.

[kddnewton]

I fear you'll need a version check either way, as whatever changes I made wouldn't be backported to 3.4 since they wouldn't be considered a "bug". So even if I get it fixed for 3.5, we're stuck with this for 3.4 going forward.

[aaronjensen]

That tends to be handled with additional regular expressions, rather than version checks. The current checker can support pre and post prism error messages based solely on a set of regular expressions.