fix: remove double parenthesis as valid opening to arithmetic expasion (#311)

* chore: checkpoint with formatted tests

No changes for now, we only re-formatted tests to make it easier to diff
them when we actually do changes.

* feat: avoid spurious parsing of arithmetic expansion

* test: add test for parenthesis in quote

* chore: update known failures with master

Update the `script/known-failures.txt` document with the latest grammar
in `master`, to make it easier to check what our changes are impacting.

* refactor: move (( expression )) from test_command to compound_statement

With the introduced changes, some of the examples were failing with a
specific pattern, namely the files:

- `examples/wild-corpus/esoteric/lishp/evaluator.sh`
- `examples/wild-corpus/esoteric/lishp/test.sh`
- `examples/wild-corpus/esoteric/lishp/variables.map.sh`
- `examples/wild-corpus/esoteric/lishp/variables.sh`

When taking a closer look it was noted that the following program was
failing to parse:

```
(( size=1, max_index=10 )
```

After taking a closer look at bash's manual, it seems this should be
possible and it should be considered a compound command instead. As
such, this commit moves this pattern from `test_command` to
`compound_statement`, updating it so that we can have multiple
expressions separated by a comma.

Note that these changes will very very very likely break the existing
tests, as we recently updated some to expect `test_command`. However,
for the time being, I'm keeping those failures since I'd like to have
someone else take another look at the changes I'm introducing with this
commit to see if they make sense.

* refactor: revert changes to subscript

This commit reverts the changes that had been done to `subscript` in
order to fix failing examples for the following files:

- `examples/gentoo/dev-lang/mlton/mlton-20180207.ebuild`
- `examples/gentoo/eclass/cuda.eclass`
- `examples/wild-corpus/cloud/kubernetes/cluster/lib/logging.sh`
- `examples/wild-corpus/cloud/kubernetes/hack/update-godep-licenses.sh`
- `examples/wild-corpus/git/t/t9902-completion.sh`
- `examples/wild-corpus/shell/bashdb/test/unit/test-columns.sh`
- `examples/wild-corpus/shell/bashdb/lib/frame.sh`
- `examples/gentoo/dev-qt/qt-docs/qt-docs-6.10.0_p202510021201.ebuild`
- `examples/gentoo/dev-qt/qt-docs/qt-docs-6.9.3_p202509261208.ebuild`
- `examples/gentoo/eclass/linux-mod-r1.eclass`

Namely, with the introduced changes, the following program could not be parsed:

```
echo "${array[${index}+1]}"
```

Even though I've confirmed this is valid bash syntax and, if the
variables were set, this would actually be evaluated to the value in
that specific index of the array.

Once again, not touching tests right now as I want someone to
double-check that my changes make sense.

* test: commit updated good tests

Add the updated tests for both `test/corpus/literals.txt` and
`test/corpus/statements.txt`, while leaving the failure in
`test/corpus/commands.txt`, as the two first simply updated the
`test_command` node to `compound_statement`, which is expected.

The latter broke the binary expression in a parenthesized expression in
a subscript, which is simply returning word instead of actually parsing
the binary or unary expression, still need to figure that out.

* refactor: replace parenthesized_expression for compound_statement

Replace `parenthesized_expression` for `compound_statement` in
`subscript`, as that fixes the issue with patterns like
`${array[(($number+1))]}` where `(($number+1))` was being parsed as a
`(parenthesized_expression(word))`.

This removes even more known failures, while introducing a new single
on, which needs to be looked into.

* fix: add subshell to possible options in subscript

Not sure if we should add `subshell` but, since the last commit removed
`parenthesized_expression`, we now need another one to allow `( ... )`
in subscript.

* refactor: remove unnecessary optional

Co-authored-by: Max Brunsfeld <[email protected]>

* ci: disable node tests

Co-authored-by: Max Brunsfeld <[email protected]>

---------

Co-authored-by: Max Brunsfeld <[email protected]>

Author: dinocosta

.github/workflows/ci.yml

@@ -48,7 +48,7 @@ jobs:
         with:
           generate: false
           test-rust: true
-          test-node: true
+          test-node: false
           test-python: true
           test-go: true
           test-swift: false

grammar.js

@@ -386,10 +386,22 @@ module.exports = grammar({
       field('redirect', optional($._redirect)),
     )),
 
-    compound_statement: $ => seq(
-      '{',
-      optional($._terminated_statement),
-      token(prec(-1, '}')),
+    compound_statement: $ => choice(
+      seq(
+        '{',
+        optional($._terminated_statement),
+        token(prec(-1, '}')),
+      ),
+      seq(
+        '((',
+        repeat(
+          seq(
+            $._arithmetic_expression,
+            ',',
+          ),
+        ),
+        $._arithmetic_expression,
+        '))'),
     ),
 
     subshell: $ => seq(
@@ -435,7 +447,6 @@ module.exports = grammar({
           ),
           ']]',
         ),
-        seq('((', optional($._expression), '))'),
       ),
     ),
 
@@ -508,7 +519,7 @@ module.exports = grammar({
     subscript: $ => seq(
       field('name', $.variable_name),
       '[',
-      field('index', choice($._literal, $.binary_expression, $.unary_expression, $.parenthesized_expression)),
+      field('index', choice($._literal, $.binary_expression, $.unary_expression, $.compound_statement, $.subshell)),
       optional($._concat),
       ']',
       optional($._concat),
@@ -701,7 +712,7 @@ module.exports = grammar({
     ),
 
     arithmetic_expansion: $ => choice(
-      seq(choice('$((', '(('), commaSep1($._arithmetic_expression), '))'),
+      seq('$((', commaSep1($._arithmetic_expression), '))'),
       seq('$[', $._arithmetic_expression, ']'),
     ),
 
@@ -731,6 +742,7 @@ module.exports = grammar({
       $._simple_variable_name,
       $.variable_name,
       $.string,
+      $.raw_string,
     )),
 
     _arithmetic_binary_expression: $ => {
@@ -1108,7 +1120,7 @@ module.exports = grammar({
       $.variable_name,
     ),
 
-    _special_variable_name: $ => alias(choice('*', '@', '?', '!', '#', '-', '$', '0', '_'), $.special_variable_name),
+    _special_variable_name: $ => alias(choice('*', '@', '?', '!', '#', '-', '$', '_'), $.special_variable_name),
 
     word: _ => token(seq(
       choice(

script/known-failures.txt

@@ -1,16 +1,20 @@
-examples/bash-it/completion/available/aliases.completion.bash
 examples/bash-it/completion/available/bundler.completion.bash
 examples/bash-it/completion/available/sqlmap.completion.bash
 examples/bash-it/plugins/available/colors.plugin.bash
 examples/bash-it/plugins/available/history-eternal.plugin.bash
 examples/bash-it/themes/hawaii50/hawaii50.theme.bash
 examples/bash/examples/functions/notify.bash
 examples/bash/examples/shellmath/shellmath.sh
+examples/bash/tests/arith-for.tests
 examples/bash/tests/arith.tests
 examples/bash/tests/array.tests
 examples/bash/tests/assoc.tests
+examples/bash/tests/braces.tests
 examples/bash/tests/case.tests
+examples/bash/tests/casemod.tests
 examples/bash/tests/comsub-posix.tests
+examples/bash/tests/comsub.tests
+examples/bash/tests/comsub2.tests
 examples/bash/tests/cond.tests
 examples/bash/tests/errors.tests
 examples/bash/tests/extglob.tests
@@ -20,6 +24,7 @@ examples/bash/tests/histexp.tests
 examples/bash/tests/misc/dev-tcp.tests
 examples/bash/tests/more-exp.tests
 examples/bash/tests/new-exp.tests
+examples/bash/tests/parser.tests
 examples/bash/tests/posixexp.tests
 examples/bash/tests/posixexp2.tests
 examples/bash/tests/printf.tests
@@ -29,23 +34,29 @@ examples/bash/tests/redir.tests
 examples/bash/tests/test.tests
 examples/gentoo/app-misc/editor-wrapper/files/editor-wrapper-4.sh
 examples/gentoo/dev-lang/rust/rust-1.74.1-r101.ebuild
+examples/gentoo/dev-libs/libffi/libffi-3.5.1.ebuild
+examples/gentoo/dev-libs/libffi/libffi-3.5.2.ebuild
+examples/gentoo/dev-libs/libffi/libffi-9999.ebuild
 examples/gentoo/eclass/rust.eclass
 examples/gentoo/eclass/tests/toolchain-funcs.sh
 examples/gentoo/eclass/texlive-common.eclass
 examples/gentoo/eclass/toolchain-funcs.eclass
 examples/gentoo/mail-client/thunderbird-bin/files/thunderbird-bin-r1.sh
 examples/gentoo/mail-client/thunderbird/files/thunderbird-r1.sh
-examples/gentoo/net-misc/asterisk/asterisk-18.26.1-r1.ebuild
-examples/gentoo/net-misc/asterisk/asterisk-20.12.0.ebuild
-examples/gentoo/net-misc/asterisk/asterisk-20.13.0.ebuild
-examples/gentoo/net-misc/asterisk/asterisk-21.7.0.ebuild
-examples/gentoo/net-misc/asterisk/asterisk-21.8.0.ebuild
-examples/gentoo/net-misc/asterisk/asterisk-22.2.0.ebuild
-examples/gentoo/net-misc/asterisk/asterisk-22.3.0.ebuild
+examples/gentoo/net-misc/asterisk/asterisk-18.26.2.ebuild
+examples/gentoo/net-misc/asterisk/asterisk-18.26.4.ebuild
+examples/gentoo/net-misc/asterisk/asterisk-20.15.2.ebuild
+examples/gentoo/net-misc/asterisk/asterisk-21.10.2.ebuild
+examples/gentoo/net-misc/asterisk/asterisk-22.5.2.ebuild
 examples/gentoo/sys-apps/dcfldd/dcfldd-1.9.2-r1.ebuild
 examples/gentoo/sys-apps/dcfldd/dcfldd-1.9.3.ebuild
 examples/gentoo/sys-apps/less/files/lesspipe-r3.sh
 examples/gentoo/sys-apps/less/files/lesspipe-r4.sh
+examples/gentoo/sys-apps/locale-gen/locale-gen-3.8.ebuild
+examples/gentoo/sys-apps/locale-gen/locale-gen-3.9-r1.ebuild
+examples/gentoo/sys-apps/locale-gen/locale-gen-9999.ebuild
+examples/gentoo/sys-fs/bcachefs-tools/bcachefs-tools-1.25.3.ebuild
+examples/gentoo/sys-fs/bcachefs-tools/bcachefs-tools-1.31.5.ebuild
 examples/gentoo/www-client/firefox-bin/files/firefox-bin-r1.sh
 examples/gentoo/www-client/firefox/files/firefox-r1.sh
 examples/wild-corpus/cloud/chef-bcpc/legacy_scripts/make_databag.sh
@@ -79,8 +90,6 @@ examples/wild-corpus/distro/woof-CE/woof-code/rootfs-skeleton/usr/local/petget/i
 examples/wild-corpus/distro/woof-CE/woof-code/rootfs-skeleton/usr/local/petget/pkg_chooser.sh
 examples/wild-corpus/esoteric/CmdlineGL/share/examples/FlightSim.sh
 examples/wild-corpus/esoteric/CmdlineGL/share/examples/Robot.sh
-examples/wild-corpus/esoteric/lishp/callable.lambda.sh
-examples/wild-corpus/esoteric/lishp/parser.sh
 examples/wild-corpus/esoteric/make-a-lisp-bash/core.sh
 examples/wild-corpus/esoteric/shasm/shasm.sh
 examples/wild-corpus/exp/shootout/timing.sh
@@ -97,8 +106,6 @@ examples/wild-corpus/freebsd-11.1/crypto/openssl/util/bat.sh
 examples/wild-corpus/freebsd-11.1/gnu/usr.bin/binutils/ld/genscripts.sh
 examples/wild-corpus/freebsd-11.1/share/examples/drivers/make_device_driver.sh
 examples/wild-corpus/freebsd-11.1/share/examples/drivers/make_pseudo_driver.sh
-examples/wild-corpus/freebsd-11.1/sys/dev/bhnd/tools/bus_macro.sh
-examples/wild-corpus/freebsd-11.1/sys/tools/bus_macro.sh
 examples/wild-corpus/freebsd-11.1/tools/tools/mctest/mctest_run.sh
 examples/wild-corpus/freebsd-11.1/tools/tools/nanobsd/fill_pkg.sh
 examples/wild-corpus/freebsd-11.1/tools/tools/shlib-compat/test/regress.sh
@@ -276,17 +283,17 @@ examples/wild-corpus/oil/test/smoke.sh
 examples/wild-corpus/sdk/tools/clang/scripts/update.sh
 examples/wild-corpus/shell/ast/src/cmd/3d/3d.sh
 examples/wild-corpus/shell/ast/src/cmd/3d/features/syscall.sh
-examples/wild-corpus/shell/ast/src/cmd/html/mm2bb.sh
-examples/wild-corpus/shell/ast/src/cmd/html/mm2html.sh
-examples/wild-corpus/shell/ast/src/cmd/html/mm2twiki.sh
-examples/wild-corpus/shell/ast/src/cmd/ie/ie.sh
 examples/wild-corpus/shell/ast/src/cmd/INIT/ditto.sh
 examples/wild-corpus/shell/ast/src/cmd/INIT/hurl.sh
 examples/wild-corpus/shell/ast/src/cmd/INIT/iffe.sh
 examples/wild-corpus/shell/ast/src/cmd/INIT/mktest.sh
 examples/wild-corpus/shell/ast/src/cmd/INIT/package.sh
 examples/wild-corpus/shell/ast/src/cmd/INIT/regress.sh
 examples/wild-corpus/shell/ast/src/cmd/INIT/rt.sh
+examples/wild-corpus/shell/ast/src/cmd/html/mm2bb.sh
+examples/wild-corpus/shell/ast/src/cmd/html/mm2html.sh
+examples/wild-corpus/shell/ast/src/cmd/html/mm2twiki.sh
+examples/wild-corpus/shell/ast/src/cmd/ie/ie.sh
 examples/wild-corpus/shell/ast/src/cmd/ksh93/data/bash_pre_rc.sh
 examples/wild-corpus/shell/ast/src/cmd/ksh93/tests/append.sh
 examples/wild-corpus/shell/ast/src/cmd/ksh93/tests/arith.sh
@@ -353,5 +360,4 @@ examples/wild-corpus/shell/mksh/Build.sh
 examples/wild-corpus/shell/mksh/test.sh
 examples/wild-corpus/shell/modernish/install.sh
 examples/wild-corpus/shell/mwc-sh/shql.sh
-examples/wild-corpus/shell/posixcube/posixcube.sh
 examples/wild-corpus/shell/posixcube/test.sh