refactor: move various file generation from tests to a separate crate

[magistau]

No description provided.

[kaikalii]

Can you briefly explain this change?

[magistau]

Pretty much what it says in the commit message. Instead of using tests for generating files like site/primitives.json, a new executable crate named uiua-helper is used.

[kaikalii]

What is the motivation?

[magistau]

Tests are intended for, well, testing, not for generating arbitrary files inside the repository. Especially since these tests are all in different places, it is not clear how and which files are generated. For example, creation of files inside ./site/text/ when testing the main crate which doesn't involve the website is pretty surprising. And likewise, it is surprising that to build some part of Uiua you need to run tests of a certain crate first. Everything generated by build/test process should be in either designated directories (like ./target/ or ~/.cargo/) or in temporary directories somewhere in /tmp/ which the user doesn't need to care about.

[magistau]

Also, files which can be fully generated from existing files shouldn't be tracked by Git, since they bear zero entropy and can easily go out of sync. site/primitives.json, for example, is fully determined by uiua implementation and so it should be considered a build artifact untracked by Git and only generated when building the website. Cargo.lock, however, is not fully determined by Cargo.toml files, as different lockfiles can correspond to the same manifest, so Cargo.lock should be tracked.

[kaikalii]

site/primitives.json needs to be available in the git repo to be useful externally.

[magistau]

By the way, site/blog/feed.rss is most likely a generated file, but I couldn't find where it is generated.

[kaikalii]

I like having generation in tests so I don't forget to run it.
Does this create any specific problems?

[magistau]

site/primitives.json needs to be available in the git repo to be useful externally.

Isn't it hosted by the website as https://www.uiua.org/primitives.json? And it would be available on the site branch anyway.

[magistau]

Does this create any specific problems?

Other than being unintuitive, this caused some troubles with Nix packaging (due to Nix making a lot of files read-only), but that was circumvented by disabling those "tests".

[magistau]

I like having generation in tests so I don't forget to run it.

Why would you need to "not forget to run it"? It can be done automatically by CI, for example.

[kaikalii]

I think they folder should probably be called generate rather than helper.

[magistau]

I think they folder should probably be called generate rather than helper.

Renamed to generator

[magistau]

Also, files which can be fully generated from existing files shouldn't be tracked by Git, since they bear zero entropy and can easily go out of sync. site/primitives.json, for example, is fully determined by uiua implementation and so it should be considered a build artifact untracked by Git and only generated when building the website. Cargo.lock, however, is not fully determined by Cargo.toml files, as different lockfiles can correspond to the same manifest, so Cargo.lock should be tracked.

In fact, site/primitives.json is actually out of sync with the docs. See backward docs and the relevant entry in site/primitives.json.

[kaikalii]

Why have feature gates in the pad?

[magistau]

To make site and uiua-editor usable as libraries without pulling unnecessary stuff. site::markdown::markdown_html is hard to decouple from the site crate, so generator needs to depend on site. For that, a more or less minimal version of site should be available, and that depends on uiua-editor. IIRC I also faced site and uiua-editor depending on the generated files, which doesn't seem adequate for a task as simple as converting markdown to HTML.