RFC-005: Test crate implementation

[JSerFeng]

Motivation

Testing is of vital importance to RSPack. We want it to be clear and highly readable. In Rspack, what we test most are the bundle result, the snapshot test just takes each plain text for snapshot result, which is not suitable for us, as bundle result contains many files and directories which are not easy to present using snapshots.
Webpack plugin test is a good practice for our use cases. It creates a simple project, defines a directory called expect and manually stores the expected bundle result in it, then uses webpack to compile that project, and compares the compile result in the actual directory with the expected bundle result inside expect directory, tell if the two directories are totally the same.
So we want a test tool crate to handle this conveniently.

Naming

RST stands for rspack test (temporary name)*

Configuration

expect: &str

Define the name of the directory that you put things you expect inside it.
Default: "expect"

actual: &str

Define the name of the directory that contains actual result.
Default: "actual"

Mode: Enum

Mode::Partial
Every file and directory inside expect directory can be found in the actual directory
Mode::Strict
actual directory is the same with expect directory
Default: Mode::Partial

Env

MUTE: It env::var("MUTE") is Ok(), no output.

Usage

Example

#[test]
#[fixture("fixtures/**")]
fn test(p: PathBuf) {
   // Simplest way 😎
   rspack_test::assert(p);

   // Or if you want it more configurable 🧐
   let rst = rspack::RstBuilder::default()
      .fixture(&p)
      .actual("dist")
      .mode(Mode::Strict)
      .build();

  assert!(rst.test().is_ok());
}

#[fixture] derive macro is from swc testing_macro

RstBuilder is the builder of rst configuration

cli

Deprecated
cargo rst review
Review every failed test case, print each inconsistent file and the lines that cause inconsistence. Then user can accept the changes in each file, which will update the file to match the actual file

If the comparing file is code or text file, then print diff stats, otherwise just tell if two files are the same

/project/fixtures/print/expect/main.ts
----------------------------
+ 3 | cnosole.log("Hello")
+ 4 | console.log("World")
- 3 | print("World")
----------------------------
Press a to accept, c to continue reviewing

If user presses 'a', we update the expect/main.ts to actual/main.ts, if presses 'c', then go review next file

cargo rst update
Update the failed test case

cargo rst test
Run all the tests

Under the hood

Run by Runtime

#[test]
#[fixture("fixtures/**")]
fn test(p: PathBuf) {
  let rst = if p.as_path().to_str().unwrap() == "bar" {
    RstBuilder::default().fixture(&p).actual("dist").build();
  } else {
    RstBuilder::default().fixture(&p).mode(Mode::Strict).build();
  }
  rst.test();
}

Let's say If our fixtures directory have two directories: foo and bar

1. Generate 2 config records in a directory in the current working directory (binary format):
   fixture: fixtures/foo, expect: fixtures/xxx/expect, actual: fixtures/xxx/actual, mode: strict
   fixture: fixtures/bar, expect: fixtures/xxx/expect, actual: fixtures/xxx/dist, mode: partial
   The reason why storing the records is that, we could run test by CLI, but we still want each fixture's configuration is right, so we save each fixture's configuration.
2. Checking those 2 directory using each configuration, If test failed, store the record to a failed directory, so we can review them later using command line tools.

Run by Command Line

For example:

cargo rst update

Find all the failed records in the disk and check, then update, review or test using those records' config . If there is no record, using default config instead.

Related issue

#436

Conclusion

Cancel review feature.

Only save the failed records.

Add argument to control which fixture to update using CLI

[IWANABETHATGUY]

Ignore Pattern should be Option<Vector> instead of Option<&str> since we may want to ignore multiple file and directory pattern.

[JSerFeng]

That str is glob pattern, wait a sec, I will clarify more clearly in the rfc

[IWANABETHATGUY]

Of course,you could write multiple pattern in a single &str but it will lost some readability. However, we will not use this feature in most cases, so it is not big deal i guess.

[JSerFeng]

Oh right, got it, it should be vec

[hyf0]

Are there any necessary reasons that we have to use macros to implement this? If not please use functions instead.

[hyf0]

Maybe use builder pattern?
Maintaining macros is a pain.

[hyf0]

I also suggest that we could learn from the testing in SWC. The current RFC is complicated and does not work well with #431.

We need the ability that is described in https://github.com/swc-project/swc/blob/b21cccfea2e9e6a44146b8b329b64e3ba79cc8b7/crates/testing_macros/src/lib.rs#L8-L49

[IWANABETHATGUY]

I agree with @iheyunfei , builder pattern with a Default trait would be enough here.

[JSerFeng]

Thanks, I may make some changes soon

[JSerFeng]

I make some changes, do I understand correctly🐷

[hyf0]

So far, I don't think we need the configuration for Actual Dir and mode, we could assume that the actual dir is dist and the Expected Dir is expected.

[JSerFeng]

Yes, so I make "dist" and "expected" as default options, and "partial compare" too. Rst::default().test() could satisfy our needs, but leaving them configurable might be better for more usages in the future I suppose.🤔

[hyf0]

Get it. I think there is no confusion that need to discuss in this thread. Let's add it to the conference on Friday. Fy ***@***.***>于2022年7月13日 周三10:15写道：

…

Yes, so I make "dist" and "expected" as default options, and "partial compare" too. Rst::default().test() could satisfy our needs, but leaving them configurable might be better for more usages in the future I suppose. 🤔 — Reply to this email directly, view it on GitHub <#441 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALZVPWSQXR7J2LNLUKBUX3LVTYRETANCNFSM53JR72UA> . You are receiving this because you were mentioned.Message ID: ***@***.***>