Improved software engineering quality suggestions

[Timmmm]

Hi, I was wondering if you would be open to PRs to improve the general software engineering quality of this repo. This is the sort of thing I'm thinking of in increasing order of invasiveness:

• Add pre-commit, tell it to format all the Python code using Black or Ruff and enforce in CI.
• Add Pylint to pre-commit. Enable Pylint in CI and fix its errors #311
• Restructure the code to turn it into an actual Python package with pyproject.toml etc. Publish it on Pypi. Reorganise Python code using pyproject.toml #294
• Add static type hints.
• Add Pyright (static type checker) to pre-commit to enforce them.

There's also some low hanging fruit in the actual code:

• Use argparse (Typer is a better option but it's beneficial not to add a third party dependency.) Use argparse for argument parsing #314
• Switch from ad-hoc dictionaries to dataclasses.
• Don't reparse the files 3 times.

I can do all that fairly easily but I thought I'd check if you would actually accept those improvements first. Seems like you might a bit review bottlenecked based on the open PRs (who isn't?).

[aswaterman]

Neel Gala (@neelgala) is the author of the bulk of the Python code, so I'll defer to him. But at first blush this seems reasonable.

[IIITM-Jay]

Commenting here to have just a glimpse of the improvements in software engineering qualities such as Refactorization, Modularization and using as much Optimization techniques as we can for better maintainability and enhancing the readability as well. Tagging a PR also with reference to above.

Refactor and Optimization:: constants.py

/CC Andrew Waterman (@aswaterman) and Rafael Sene (@rpsene)

[ethindp]

The use of other linters/type checkers/formatters other than ruff seems quite redundant. Is there a reason we don't just use ruff for everything? I'm pretty sure it already incorporates all the lints/type checking/formatting that the other major linters/type checkers/formatters do.

[Timmmm]

I think it's just because Ruff wasn't really mature enough at the time, but I agree we could probably replace Black, isort and Pylint with Ruff now.

It doesn't do type checking though so we'll still need Pyright (or Ty/Pyrefly when they are mature).

[IIITM-Jay]

I somewhat agree with Tim Hutt (@Timmmm) at some places. But, to add on, Pylint has a robust plugin system, allowing users to write custom checkers to enforce highly specific, in-house rules. Ruff implements all its rules natively in Rust and does not currently support third-party plugins.
Additionally, Pylint still catches a category of deeper, semantic bugs as compared to Ruff currently.

But yeah, no doubt, Ruff does fast syntactic linting.

[Timmmm]

Yeah probably worth still running Pylint. Unfortunately the only time I tried to add a lint to it (to detect accidental concatenation of strings) I found that it was impossible because its parser threw away that information. 🤷🏻‍♂️