Testing! (with VHS)

[PThorpe92]

For those of you not aware... Eza is in the process of migrating to a new testing framework and setup, particularly we are interested in using VHS for something a little different than it's intended purpose.

Essentially how this will work, is you define a .tape file...

# Render the output TXT
Output example.txt

Require eza
# Set up frame and font size
Set FontSize 46
Set Height 600
Set Width 1200
Set DisableRender true

Type "eza -A -h --grid --no-permissions --git --no-user"
Sleep 1s
Type "eza -a -D -B -H --time-style --no-filesize"
Sleep 1s

and we can define a number of commands that can be run on the individual's computer to verify the outcome of whatever changes made.
There was some discussion of this in:
#147
as well as in our matrix channel.

So far, I have submitted a PR upstream to VHS that enables us to disable the rendering/ FFMPEG, that would be timely and expensive, and forego the .gif files all together. The text output would allow us to match the expected outcome to the outcome from VHS.

Since we haven't heard from VHS in about three weeks, and I believe we are already using my fork at this point anyway. I wanted to start a discussion for any additional features that would be practical for us to add to VHS, that probably dont fit in with them upstream (as we have a completely different use).

I will be maintaining the fork over here in eza-community, and will start to work on stripping it down from some of the things we do not need. So please, any thoughts or features that could be added (or removed) can be discussed here. Also, any thoughts or feelings about testing in general, as that is an issue we have been more or less putting off, and one we will have to eventually figure out. 👍

[cafkafk]

one thing I would really like is if you could leave in gif generation so we could (in the future) have it generate gifs from CI, and also, I'd prefer if it didn't need sleep commands, but rather could just execute commands one after the other when they were done.

[PThorpe92]

I am thinking of maybe a program that generates .tape files for our testing purposes, and then compares the output to an existing output tape from the main branch?

What would the ideal input be for testing? I believe sleep does have to exist somewhere, but if we can abstract that away and sleep for as little time as necessary between commands.
So if you could define a file with all the commands you would like eza to run, maybe I'll write a program that will parse it and generate the optimized .tape file for us? Or is it basically going to be the same commands every time anyway?

I'm just trying to get a picture of the ideal testing DX, and then I'll think about how to make it work.

[cafkafk]

One problem I've noticed with VHS is that it's not necessarily deterministic.

I'm tempted to write a library for this in rust, that just runs some commands and returns the output, and then write our test runner as a binary using that library tbh. Idk how you'd feel about this...?

[PThorpe92]

so the output will vary slightly enough where it cant be automated?

I wonder if we can just grab one frame from each output after the command is run and write that out to a file instead of diffing the whole output.txt

or yeah Idk I'm pretty much always down to rewrite it in rust ;) I'd be just as happy to contribute to that

[gierens]

I think this is a timing issue, you just need one frame where the original runner displayed a single line or even char faster and that's it ... so just comparing the last frame might work better.

[PThorpe92]

Closing in obvious favor of powertest + trycmd