test(integration): improve integration test suite coverage

[Huzaifa696]

Summary

The operator sender, JIT mint init, and deferred-remint paths had no E2E coverage. Several helpers were unreachable from the test crate. Mixed escrow .so artifacts also poisoned the validator's Once and cascaded test failures.

This PR closes those gaps with new test binaries driven through a scriptable wire boundary. Escrow .so runs are now grouped and rebuilt between groups.

Changes

• indexer/src/operator/sender/mod.rs — feature-gated test_hooks module exposing existing pub(super) helpers. No prod-build surface change.
• test_utils/src/{mock_rpc,mock_yellowstone}.rs — new scriptable JSON-RPC and Yellowstone gRPC harnesses.
• integration/tests/indexer/ — ~20 new sender/JIT/remint binaries plus reconciliation, gap-detection, resync, bootstrap, and storage-failure coverage.
• integration/Makefile, root Makefile, rust.yml — group coverage runs by escrow feature with a mid-run rebuild; pins cargo-nextest@0.9.130.
• contra-escrow-program/Makefile — build-no-clients target for the mid-run rebuild.

Impact

• Combined integration line coverage 66.8% → 80%+
• No production behaviour change: ~95% of new LOC is in a separate test crate; prod-side changes are additive test_hooks wrappers.

Where to look first

1. indexer/src/operator/sender/mod.rs — only non-mechanical prod change; verify test_hooks is feature-gated and additive.
2. integration/Makefile :: integration-coverage-indexer — the load-bearing group-ordering fix.
3. integration/Cargo.toml — every new [[test]] entry corresponds to a file under integration/tests/.
4. test_utils/src/mock_rpc.rs — Reply::Dynamic and the Drop impl that keeps panicking tests from leaking the Tokio task / fd.

Coverage Report

Component	Lines Hit	Lines Total	Coverage	Artifact
Core	9,043	10,447	86.6%	rust-unit-coverage-reports
Indexer	14,428	16,858	85.6%	rust-unit-coverage-reports
Gateway	991	1,115	88.9%	rust-unit-coverage-reports
Auth	541	596	90.8%	rust-unit-coverage-reports
Withdraw Program	118	230	51.3%	unit-coverage-reports
Escrow Program	1,170	1,951	60.0%	unit-coverage-reports
E2E Integration	9,423	11,635	81.0%	e2e-coverage-reports
Total	35,714	42,832	83.4%

Last updated: 2026-04-27 10:51:08 UTC by E2E Integration

[greptile-apps]

Greptile Summary

This PR significantly expands E2E integration coverage (~30 new test binaries) for both the contra and indexer suites, reorganises all test runs into escrow-feature groups (prod / test-tree / prod) to prevent validator_helper Once-poisoning, introduces a MockRpcServer + OperatorMockHarness test infrastructure, and exposes Storage::Mock / sender::test_hooks behind a new test-mock-storage feature for use by downstream test crates.

• P0 — integration-coverage-report will fail in CI: integration/Makefile calls python3 scripts/lcov_filter.py and python3 scripts/test_lcov_filter.py, but neither integration/scripts/ nor those files exist anywhere in the repository. The GitHub Actions step Generate combined E2E coverage report invokes this target directly, so the coverage upload and PR-comment steps will be blocked on every run.
• P1 — COVERAGE_FLOOR default does not guard the combined run: the default is 45.0, matching the indexer-only denominator (~45%), yet CI runs the full combined suite (~79%). A regression from 79% down to 46% would silently pass the floor check.

Confidence Score: 2/5

Not safe to merge — the coverage report CI step will fail on every run due to a missing Python script, and the coverage floor provides inadequate regression protection for the combined suite.

P0 finding (missing scripts/lcov_filter.py) will break the CI coverage pipeline on merge; P1 finding (COVERAGE_FLOOR default too low for combined run) further weakens the ratchet. Both must be resolved before merging.

integration/Makefile (missing scripts reference, COVERAGE_FLOOR default) and .github/workflows/rust.yml (invokes the broken target directly).

Important Files Changed

Filename	Overview
integration/Makefile	Major overhaul grouping test runs by escrow .so feature; introduces scripts/lcov_filter.py reference that doesn't exist (P0 CI breakage) and a contradictory COVERAGE_FLOOR default (P1).
.github/workflows/rust.yml	Pins cargo-nextest to 0.9.130 and adds a Run Redis-backend coverage tests step; integration-coverage-report step will fail because the referenced scripts/lcov_filter.py does not exist.
integration/Cargo.toml	Adds ~30 new [[test]] binary entries and new dependencies (redis, jsonrpsee, solana-keychain, yellowstone-grpc-proto); enables test-mock-storage feature on contra-indexer.
indexer/src/operator/sender/mod.rs	Adds a test_hooks module gated by #[cfg(any(test, feature = "test-mock-storage"))] that re-exports pub(super) constructor and recovery paths for integration test use; no production behaviour change.
indexer/src/storage/common/storage.rs	Expands Mock variant and mock module from #[cfg(test)] to #[cfg(any(test, feature = "test-mock-storage"))] so downstream integration test crates can use it; intentional and correctly scoped.
indexer/src/storage/postgres/db.rs	Removes unused commit_transaction/rollback_transaction helpers; adds coverage-ignore rationale comment for quarantine_all_active_withdrawals_internal.
test_utils/src/mock_rpc.rs	New MockRpcServer implementation backed by hyper; supports scripted FIFO replies, dynamic handlers, call-count and timestamp tracking — well-designed test infrastructure.
integration/tests/setup.rs	Fixes a latent bug where mixed_transaction's _amount parameter was silently ignored (hardcoded 1000 used instead); now passes amount correctly.
integration/tests/indexer/helpers/test_types.rs	Converts WAIT_TIMEOUT_SECS from a constant to a LazyLock reading CONTRA_TEST_WAIT_TIMEOUT_SECS from the environment, enabling coverage builds to use a higher timeout.
integration/tests/indexer/db_migration_race.rs	New tests for schema idempotency and duplicate-key race safety; header comment incorrectly names the parent binary as indexer_integration when it's actually reconciliation_integration.
integration/tests/contra/rpc/test_parallel_svm_burst.rs	New test driving the parallel-SVM SnapshotCallback path by submitting 20 conflict-free transactions concurrently; setup and assertion logic look correct.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[CI: integration-coverage job] --> B[integration-coverage-contra\nprod escrow group]
    A --> C[integration-coverage-contra-redis\nRedis warm-cache group]
    A --> D[build-test escrow\ntest-tree]
    D --> E[integration-coverage-indexer\ntest-tree group]
    E --> F[Rebuild escrow prod]
    F --> G[integration-coverage-indexer\nprod-feature group]
    B & C & G --> H[integration-coverage-report]
    H --> I{scripts/lcov_filter.py\nexists?}
    I -- NO --> J[❌ CI FAILS\nNo such file]
    I -- YES --> K[check-coverage-floor\nfloor=45.0]
    K --> L{combined pct >= 45?}
    L -- YES even at 46% --> M[✅ passes\nbut ~79% expected]
    L -- NO --> N[❌ floor breach]

Loading

_{Reviews (1): Last reviewed commit: "test(integration): expand E2E coverage a..." | Re-trigger Greptile}