Skip to content

TESTING.md

Latest commit

 

History

History
629 lines (471 loc) · 20.2 KB

TESTING.md

File metadata and controls

629 lines (471 loc) · 20.2 KB
Raw

Testing Guide

This guide explains the testing infrastructure, philosophy, and how to write tests for genai-rs.

Table of Contents

Test Categories

Unit Tests

Inline tests in source files covering serialization, builders, and internal logic.

make test                                 # Run all unit tests (uses cargo-nextest)
cargo nextest run -E 'test(/test_name/)'  # Run specific test by pattern

Location: src/*_tests.rs files and #[cfg(test)] modules

What they test:

  • Serialization/deserialization roundtrips
  • Builder pattern validation
  • Helper method behavior
  • Error type formatting

Integration Tests

End-to-end tests that call the real Gemini API. Require GEMINI_API_KEY.

make test-all                                        # Run all tests including integration
cargo nextest run --test interactions_api_tests --run-ignored all  # Single file

Location: tests/*.rs

Key test files:

File Coverage
interactions_api_tests.rs Basic API interactions
multiturn_tests.rs Stateful conversations
streaming_multiturn_tests.rs SSE streaming
tools_and_config_tests.rs Built-in tools configuration
function_calling_tests.rs #[tool] macro, auto-execution, multi-turn
agents_tests.rs Agent and background task patterns
multimodal_tests.rs Images, audio, video, documents

Property-Based Tests (proptest)

Automatic generation of test cases for serialization roundtrips.

cargo test proptest                       # Run proptest tests

Location:

  • src/proptest_tests.rs - Strategy generators for all types
  • tests/proptest_roundtrip_tests.rs - Integration proptests

What they verify:

  • Any valid type serializes and deserializes to the same value
  • Unknown variants preserve data through roundtrips
  • Edge cases humans wouldn't think to test

UI/Compile-Time Tests (trybuild)

Verify that invalid code fails to compile with helpful error messages.

cargo test --test ui_tests

Location: tests/ui/*.rs

What they test:

  • Type-state pattern enforcement (can't call with_system_instruction() after chaining)
  • #[tool] macro error messages
  • Invalid builder configurations

Canary Tests

Early-warning tests that detect when the API returns new step or content types.

cargo nextest run --test api_canary_tests --run-ignored all

Purpose: When Google adds new step or content types, these tests fail to alert us to add support.

Note: Skipped when --features strict-unknown is enabled.

Wire Format Tests

Verify actual API wire formats match our expectations.

File Purpose
wire_format_verification_tests.rs Offline format verification
api_wire_format_live_tests.rs Live API format verification
unknown_variant_tests.rs Unknown variant handling

Strict Mode Tests

Test behavior with --features strict-unknown which makes unknown types error instead of gracefully degrade.

cargo test --features strict-unknown

Running Tests

Quick Development Cycle

make test                                 # Unit tests only (~5s)

Full Test Suite

make test-all                             # All tests (~2-5 min with API)

Specific Categories

# By file
cargo nextest run --test multiturn_tests --run-ignored all

# By name pattern
cargo nextest run -E 'test(/function_calling/)' --run-ignored all

# With output
cargo nextest run --run-ignored all --no-capture

Environment Variables

Variable Purpose
GEMINI_API_KEY Required for integration tests
RUST_LOG=genai_rs=debug Enable debug logging
LOUD_WIRE=1 Show raw HTTP request/response
TEST_TIMEOUT_SECS Override default test timeout (default: 60)
EXTENDED_TEST_TIMEOUT_SECS Override extended test timeout (default: 120)

CI Pipeline

The GitHub Actions workflow runs these jobs:

Job What it does
check cargo check --workspace --all-targets --all-features
test Unit tests without API key
test-strict-unknown Unit tests with --features strict-unknown
test-integration 5 matrix groups with API key (see below)
fmt Format check
clippy Lint check
doc Documentation build
security cargo audit (runs in separate audit.yml workflow)
msrv Minimum supported Rust version check
cross-platform macOS and Windows builds
coverage Code coverage with cargo llvm-cov
build-metrics Clean build time measurement
ci-flakiness-report Daily flakiness analysis (creates ci-health issues)

Integration Test Matrix

Tests are split into 5 groups to parallelize and isolate failures:

Group Tests
core interactions_api, multiturn, streaming_multiturn
tools tools_and_config, agents
functions function_calling
multimodal multimodal, api_canary, temp_file
files-and-wire api_wire_format_live, files_api

Test Utilities

The tests/common/mod.rs module provides shared utilities:

Client Setup

mod common;
use common::*;

let client = get_client().expect("GEMINI_API_KEY must be set");
let response = interaction_builder(&client)
    .with_text("Hello")
    .create()
    .await?;

Retry for Transient Errors

// Retry on known transient errors (Spanner UTF-8, etc.)
let response = retry_on_transient(3, || async {
    interaction_builder(&client)
        .with_text("Hello")
        .create()
        .await
}).await?;

// Or use the macro for cleaner syntax
let response = retry_request!([client] => {
    interaction_builder(&client).with_text("Hello").create().await
})?;

Timeouts

use common::{test_timeout, with_timeout};

with_timeout(test_timeout(), async {
    // Test logic that might hang
}).await;

Environment variables for timeout configuration:

  • TEST_TIMEOUT_SECS - Default test timeout (default: 60 seconds)
  • EXTENDED_TEST_TIMEOUT_SECS - Extended timeout for multi-turn tests (default: 120 seconds)

Stream Consumption

let stream = interaction_builder(&client)
    .with_text("Hello")
    .create_stream();

let result = consume_stream(stream).await;
assert!(result.has_output());
assert!(!result.collected_text.is_empty());

Semantic Validation

For behavioral tests where exact output varies, use the suite-wide helper (it retries the validator on transient errors and tolerates only what survives the retries — see the retry-and-tolerance policy note under Minimal Test Assets):

assert_response_semantic(
    &client,
    "User asked about weather in Tokyo",
    response.as_text().unwrap(),
    "Does the response mention Tokyo's weather?"
).await;

Function Execution Error Detection

When testing create_with_auto_functions(), always verify executions succeeded:

let result = client
    .interaction()
    .with_text("What's the weather?")
    .add_functions(vec![get_weather_function()])
    .create_with_auto_functions()
    .await?;

// ✓ Check function was called AND succeeded
assert!(result.all_executions_succeeded(),
    "Executions failed: {:?}", result.failed_executions());

// ✗ Don't just check if function was called (misses missing implementations)
assert!(result.executions.iter().any(|e| e.name == "get_weather"));

Why this matters: If you declare a FunctionDeclaration but forget to register the implementation via #[tool] or ToolService, the library sends an error to the model instead of failing. The old assertion pattern passes silently!

Available helpers:

Method Description
result.all_executions_succeeded() Returns true if no executions have errors
result.failed_executions() Returns vec of failed FunctionExecutionResults
execution.is_error() Returns true if this execution resulted in an error
execution.is_success() Returns true if this execution succeeded
execution.error_message() Returns the error message if any

Writing New Tests

Integration Test Template

//! Description of what this test file covers.

mod common;
use common::*;

#[tokio::test]
#[ignore = "Requires GEMINI_API_KEY"]
async fn test_feature_name() {
    let Some(client) = get_client() else {
        println!("Skipping: GEMINI_API_KEY not set");
        return;
    };

    with_timeout(test_timeout(), async {
        let response = interaction_builder(&client)
            .with_text("Test prompt")
            .create()
            .await
            .expect("Request should succeed");

        // Structural assertions (preferred)
        assert!(response.as_text().is_some());
        assert_eq!(response.status, InteractionStatus::Completed);
    }).await;
}

Property Test Template

use proptest::prelude::*;
use crate::proptest_tests::*;

proptest! {
    #[test]
    fn roundtrip_my_type(value in my_type_strategy()) {
        let json = serde_json::to_string(&value).unwrap();
        let parsed: MyType = serde_json::from_str(&json).unwrap();
        prop_assert_eq!(value, parsed);
    }
}

Assertion Strategies

Decision Flowchart

Use this to choose the right assertion type:

Is it checking LLM-generated text content?
├── NO → Use structural assertions
└── YES → Is the expected value deterministic?
          ├── YES (error message, code execution result) → .contains() is OK
          └── NO (natural language response) → Use semantic validation

Structural Assertions (Preferred)

Check API mechanics without depending on LLM output:

// Good - structural
assert!(response.as_text().is_some());
assert_eq!(response.status, InteractionStatus::Completed);
assert!(response.function_calls().len() > 0);

Semantic Assertions (For LLM Output)

For behavioral tests where the LLM's response content matters:

assert_response_semantic(
    &client,
    "Context: user asked X, function returned Y",
    response.as_text().unwrap(),
    "Does the response incorporate Y?"
).await;

Use assert_response_semantic rather than calling validate_response_semantically directly — the helper retries the validator call on transient errors and asserts on the verdict, so a 503 on the validation round-trip doesn't hard-fail the test. Reach for the lower-level validate_response_semantically only when you need the verdict as a Result (e.g. inside a retry closure).

When to use semantic validation:

  • Multi-turn context preservation ("Does this recall the user's name?")
  • Function result incorporation ("Does this use the weather data?")
  • Factual correctness ("Does this identify Paris as the capital?")
  • Content understanding ("Does this describe the image colors?")

When NOT to use:

  • Status code verification
  • Field presence checks
  • Error message validation (deterministic strings)

Anti-Patterns to Avoid

These patterns cause flaky tests because LLM output varies:

// BAD - Single keyword that may be rephrased
assert!(text.contains("paris"));
// Model might say "The capital is Paris", "Paris, France", or "It's Paris"

// BAD - OR chains trying to handle variability
assert!(text.contains("red") || text.contains("crimson") || text.contains("scarlet"));
// Still misses "reddish", "ruby", "a shade of red", etc.

// BAD - Partial match that's too specific
assert!(text.contains("hik"));  // Trying to catch "hiking"
// Misses "outdoor activities", "trekking", "walks"

Correct approach:

// GOOD - Semantic validation handles natural language variability
assert_response_semantic(
    &client,
    "Asked about the capital of France",
    text,
    "Does this response correctly identify Paris as the capital of France?"
).await;

// GOOD - For color identification
assert_response_semantic(
    &client,
    "Showed a red image",
    text,
    "Does this response identify the color as red or a shade of red?"
).await;

Acceptable .contains() Usage

These patterns ARE appropriate because the values are deterministic:

// OK - Error messages from the library (deterministic strings)
assert!(error.to_string().contains("invalid API key"));

// OK - Code execution results (exact computed values)
assert!(text.contains("3628800"));  // factorial(10)
assert!(text.contains("24133"));    // sum of primes

// OK - JSON/schema structure checks
assert!(schema.contains("\"type\": \"string\""));

// OK - Format validation
assert!(email.contains("@"));

Unknown Step Checks

For forward-compatibility testing:

// Verify no unknown steps or content (canary test)
assert!(!response.has_unknown(),
    "API returned unknown types: {:?}",
    response.step_summary().unknown_types);

// Or handle gracefully
if response.has_unknown() {
    for (type_name, data) in response.unknown_steps() {
        log::warn!("Unknown step type: {} = {:?}", type_name, data);
    }
}

Test Data

Minimal Test Assets

tests/common/mod.rs provides minimal valid test data:

Constant Description
TINY_RED_PNG_BASE64 1x1 red PNG
TINY_BLUE_PNG_BASE64 1x1 blue PNG
TINY_WAV_BASE64 100 frames of 16-bit mono silence (valid WAV)
TINY_MP4_BASE64 One-frame 64x64 H.264 clip (valid MP4)
TINY_PDF_BASE64 "Hello World" PDF

All fixtures are complete, well-formed files the API accepts. Tests exercising them on the non-streaming create() path retry transient transport errors on the primary call (retry_request!, keyed on GenaiError::is_retryable plus the module's is_transient_error model-side-flake cases) and then assert strictly — a validation rejection (e.g. 400 invalid_request) fails loudly on the first attempt, while a 503 blip does not redden the suite. (Exceptions: the streaming path, where the primary call is not a single create(), and the canary tests, kept out of scope for this pass.) Semantic validation via assert_response_semantic (the suite-wide helper; one deliberate direct validate_response_semantically call remains, inside a retry closure that needs the verdict as a Result) applies the same policy: the validator call is itself retried on transient errors, then the helper asserts on the verdict, panics on non-transient validator errors, and tolerates transients that survive the retries with a SEMANTIC_VALIDATION_SKIPPED marker.

There are two such markers, and the distinction is why the second exists rather than reusing the first:

Marker Means
SEMANTIC_VALIDATION_SKIPPED The validator call itself failed transiently, so the verdict was never obtained.
LIVE_TOOL_EVIDENCE_SKIPPED The interaction succeeded, or failed in a way the test's triage guard did not recognise, but produced no evidence the tool actually ran.

Both are counted together. The CI integration step captures passing-test output via --success-output=final and greps for either: a ::warning:: annotation when any appear, and a failed step past a per-run threshold (see the marker-counting blocks in rust.yml and release.yml for the current numbers) — the escape hatch stays available for transient blips without being able to quietly become the normal path.

The rule for whether a skip gets a marker is distinguishability, not stability. A skip whose guard names the specific thing being skipped is left unmarked: no GEMINI_API_KEY, or a key not allowlisted for computer use — that guard requires the tool name and an unavailability phrase, so only a rejection about computer use reaches it, not an unrelated 4xx. Marking those would annotate every run forever, which is noise.

That scoping is deliberately not airtight, and the test says so: a model-level "computer use is not supported for this model" satisfies both halves too, so a regression of that kind would land in the unmarked branch. Leaving it unmarked is a recorded call, not an oversight — the alternative annotates every run on an un-allowlisted key indefinitely.

A skip that cannot tell a benign cause from a regression is marked, however stable its cause may turn out to be. The two MCP skips are the examples, and they get there differently. The Err-arm one is a catch-all: it fires on any error the triage guard did not recognise, so it cannot separate a third-party outage from a rejection phrased in words the guard misses. The Ok-path one fires on a fully-recognised condition — the interaction completed with no tool evidence — but a model that simply chose not to call the tool and a silently broken tool produce the identical response, so it cannot separate those either.

A permanently unreachable server is every bit as stable as an un-allowlisted key. What separates them is not stability but whether the branch could be hiding a real regression.

Test Fixtures

use common::{interaction_builder, stateful_builder};

// Pre-configured with default model
let builder = interaction_builder(&client);

// Pre-configured for stateful conversations
let builder = stateful_builder(&client);

Debugging Tests

Enable Logging

RUST_LOG=genai_rs=debug cargo nextest run -E 'test(/test_name/)' --no-capture

See Wire Traffic

LOUD_WIRE=1 cargo nextest run -E 'test(/test_name/)' --run-ignored all --no-capture

Run Single Test with Full Output

cargo nextest run -E 'test(/test_specific_feature/)' --run-ignored all --no-capture -j 1

Test Organization Philosophy

This section documents intentional design decisions about test organization.

Tests Organized by Feature, Not Pattern

Multi-turn conversation patterns appear in multiple test files:

  • multiturn_tests.rs - Core conversation mechanics (branching, long conversations, explicit turns)
  • streaming_multiturn_tests.rs - Streaming behavior in multi-turn contexts
  • function_calling_tests.rs - Function calling behavior (some tests use multi-turn)
  • interactions_api_tests.rs - Interaction features like thinking mode

This is intentional. Tests are organized by what they primarily test, not by whether they happen to use multi-turn patterns. A function calling test that uses multi-turn is testing function calling, not conversation mechanics. This organization makes it easy to find all tests for a specific feature.

Don't consolidate tests just because they share a pattern like multi-turn. Ask: "What is this test primarily verifying?"

Dual-Layer Serialization Testing

Serialization is tested at two layers:

Layer Location Purpose
Proptest src/proptest_tests.rs, tests/proptest_roundtrip_tests.rs Fuzzing with random inputs to find edge cases
Manual *_tests.rs files Document expected behavior, verify specific scenarios

Both are valuable:

  • Proptest finds unexpected edge cases automatically
  • Manual tests serve as documentation and catch regressions quickly
  • Manual tests run faster (no property generation overhead)

Don't remove manual roundtrip tests just because proptest exists. They're complementary, not redundant. For serialization (which is critical for API compatibility), belt-and-suspenders testing is appropriate.

Integration Test Matrix Groups

Integration tests are split into 5 CI matrix groups for parallelization:

Group Tests Rationale
core interactions_api, multiturn, streaming_multiturn Core API functionality
tools tools_and_config, agents Tool/agent patterns
functions function_calling Function calling (isolated for flakiness)
multimodal multimodal, api_canary, temp_file Media handling
files-and-wire api_wire_format_live, files_api File API and wire format

Tests are grouped by feature similarity and failure correlation. If one test in a group fails, related tests likely fail too, so grouping them reduces redundant CI runs.