Skip to content

Conversation

@irjudson
Copy link
Member

Closes #16

Adds a comprehensive 'Why Maritime Data?' section to the README explaining the rationale for using maritime vessel tracking data as the test environment.

What's Covered

Characteristics that match real-world sync challenges:

  • High volume and continuous flow (144K+ records/day)
  • Temporal ordering constraints (chronological requirement)
  • Geospatial and complex schema (lat/lon, metadata, multi-table)
  • Velocity flexibility (slow to fast data streams)

Production use case representation:

  • IoT & Telemetry (sensor streams)
  • Event Tracking (user activity, logs)
  • Time-Series Data (financial, monitoring)
  • Geospatial Analysis (fleet management, delivery)

Benefits for plugin testing:

  • Realistic scale, reproducible patterns
  • Multi-table testing capabilities
  • Global distribution patterns
  • Physics-based validation

When to use maritime data vs custom data:

  • Good fit for time-series, distributed workloads, multi-table testing
  • Consider custom data for domain-specific constraints

Why This Matters

Users evaluating this plugin can now understand:

  • Why we chose maritime data (not arbitrary)
  • How it represents their use case (IoT, events, time-series)
  • When to use it vs their own test data
  • What characteristics make good test data for sync plugins

Closes #16

Explains the rationale for using maritime vessel tracking data as the
test data generator for the BigQuery sync plugin.

Key points covered:
- Characteristics that match real-world sync challenges (high volume,
  temporal ordering, complex schema)
- How maritime data represents common BigQuery workloads (IoT,
  telemetry, event tracking, time-series)
- Benefits for plugin testing (realistic scale, reproducible,
  multi-table, global distribution, velocity flexibility)
- When maritime data is a good fit vs when to use custom data

Highlights that the synthesizer supports both slow and fast data
streams with adjustable generation rates.
@irjudson irjudson merged commit f392ef9 into main Nov 14, 2025
4 checks passed
@irjudson irjudson deleted the docs/maritime-synthesizer-rationale branch November 14, 2025 18:00
@irjudson irjudson restored the docs/maritime-synthesizer-rationale branch December 15, 2025 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document rationale for maritime data synthesizer

2 participants