Skip to content

smite-ir: Add unit tests for OperationParamMutator#55

Merged
morehouse merged 1 commit into
morehouse:masterfrom
Chand-ra:param_mutator_tests
May 19, 2026
Merged

smite-ir: Add unit tests for OperationParamMutator#55
morehouse merged 1 commit into
morehouse:masterfrom
Chand-ra:param_mutator_tests

Conversation

@Chand-ra

Copy link
Copy Markdown

This PR is a follow-up to #44, adding targeted, deterministic unit tests for the OperationParamMutator's helper functions.

Current tests for this mutator rely primarily on top-level integration loops. Directly testing the helper functions with seeded RNGs guarantees that every individual mutation branch is structurally sound and prevents future regressions (like off-by-one errors).

Comment thread smite-ir/Cargo.toml Outdated
Comment thread smite-ir/src/mutators/operation_param.rs Outdated
Comment thread smite-ir/Cargo.toml Outdated
Comment thread smite-ir/src/mutators/operation_param.rs Outdated
Comment thread smite-ir/src/mutators/operation_param.rs Outdated
Comment thread smite-ir/src/mutators/operation_param.rs Outdated
Comment thread smite-ir/Cargo.toml
@Chand-ra Chand-ra force-pushed the param_mutator_tests branch from f6ba3c6 to 90fe0e6 Compare May 3, 2026 07:11

@ekzyis ekzyis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GitHub is having issues (again). Couldn't create review comments for individual lines, sorry.

L340:

-        assert_ne!(shuffled, sorted, "Shuffle must shuffle elements");
+        assert_ne!(shuffled, sorted, "shuffle_subrange must shuffle elements");

or

-        assert_ne!(shuffled, sorted, "Shuffle must shuffle elements");
+        assert_ne!(shuffled, sorted, "must shuffle elements");

L344: same suggestion


L424: I think "field that has NO other fields" is confusing


L425:

u32 is slightly inaccurate, block heights are their own type:

-        // MinimumDepth is the only field with a `u32` output type.
+        // MinimumDepth is the only field with a `BlockHeight` output type.

@Chand-ra Chand-ra force-pushed the param_mutator_tests branch 2 times, most recently from e6b1af1 to 443ddee Compare May 7, 2026 06:44
@Chand-ra Chand-ra requested a review from ekzyis May 12, 2026 08:00

@morehouse morehouse left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of these tests are already covered in the public API tests. I think we should drop all the tests except the shuffle_subrange ones that actually provide some benefit.

Comment thread smite-ir/src/mutators/operation_param.rs Outdated
Comment thread smite-ir/src/mutators/operation_param.rs Outdated
Comment thread smite-ir/src/mutators/operation_param.rs Outdated
Comment thread smite-ir/src/mutators/operation_param.rs Outdated
Comment thread smite-ir/src/mutators/operation_param.rs Outdated
Comment thread smite-ir/src/mutators/operation_param.rs Outdated
Comment thread smite-ir/src/mutators/operation_param.rs Outdated
@Chand-ra Chand-ra force-pushed the param_mutator_tests branch from 443ddee to 67c9079 Compare May 18, 2026 05:24
Comment on lines +393 to +402
let mut shuffled = sorted.clone();

for _ in 0..100 {
shuffle_subrange(&mut shuffled, &mut rng);
shuffled.sort_unstable();
assert_eq!(
shuffled, sorted,
"shuffle_subrange must preserve all original elements"
);
}

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we also should test that shuffle generally does something.

Suggested change
let mut shuffled = sorted.clone();
for _ in 0..100 {
shuffle_subrange(&mut shuffled, &mut rng);
shuffled.sort_unstable();
assert_eq!(
shuffled, sorted,
"shuffle_subrange must preserve all original elements"
);
}
let mut shuffled = sorted.clone();
let mut diff_count = 0;
for _ in 0..100 {
shuffle_subrange(&mut shuffled, &mut rng);
if shuffled != sorted {
diff_count += 1;
}
shuffled.sort_unstable();
assert_eq!(
shuffled, sorted,
"shuffle_subrange must preserve all original elements"
);
}
assert(diff_count >= 70, "shuffle_subrange frequently does nothing");

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't look quite right to me.

Since our RNG is deterministic, diff_count will evaluate to the exact same number every time for a given seed, at which point we're more so testing the statistical distribution of the rand crate rather than the code's logic.

I believe it's better to test that shuffle_subrange mutates the input at least once. This way we can be sure the function isn't broken and just returning the identical array all the time.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea was to test that there isn't some unexpected bias towards doing nothing (either from our mutator implementation or the RNG), while also reducing the test flakiness in case later code changes cause slightly different mutations to occur.

But either way this is a strict improvement in testing coverage.

@Chand-ra Chand-ra force-pushed the param_mutator_tests branch from 67c9079 to ac7fa5a Compare May 19, 2026 04:47
@Chand-ra Chand-ra force-pushed the param_mutator_tests branch from ac7fa5a to 982dd67 Compare May 19, 2026 04:49
@Chand-ra

Copy link
Copy Markdown
Author

Rebased on top of the latest master to resolve a merge conflict.

Comment on lines +393 to +402
let mut shuffled = sorted.clone();

for _ in 0..100 {
shuffle_subrange(&mut shuffled, &mut rng);
shuffled.sort_unstable();
assert_eq!(
shuffled, sorted,
"shuffle_subrange must preserve all original elements"
);
}

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea was to test that there isn't some unexpected bias towards doing nothing (either from our mutator implementation or the RNG), while also reducing the test flakiness in case later code changes cause slightly different mutations to occur.

But either way this is a strict improvement in testing coverage.

@morehouse morehouse merged commit 9550964 into morehouse:master May 19, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants