Dynamic table creation via Harper Operations API

[irjudson]

Summary

Implements automatic Harper table creation via Operations API, eliminating manual schema.graphql definitions for data tables. Tables are created dynamically based on BigQuery schema introspection.

Closes #7

Key Discovery

Harper Operations API is fully schemaless - tables automatically index ALL fields without pre-definition. This dramatically simplified the implementation compared to the original design.

What's Included

Core Components

• OperationsClient - HTTP client for Harper Operations API (describe_table, create_table)
• SchemaManager - Orchestrates table creation and schema introspection
• TypeMapper - Maps BigQuery types to Harper types (for documentation)
• IndexStrategy - Documents indexing strategy (Harper auto-indexes everything)
• Integer ID generation - Deterministic SHA256-based IDs for fast indexing

Integration

• Modified src/index.js to initialize SchemaManager before SyncEngines
• Ensures tables exist before syncing begins
• Graceful degradation if Operations API unavailable (falls back to schema.graphql)

Configuration

operations:
  host: localhost
  port: 9925
  username: admin
  password: password

How It Works

1. On startup, SchemaManager.ensureTable() checks if Harper table exists
2. If not, creates table with create_table (only requires hash_attribute)
3. SyncEngine inserts BigQuery data with deterministic integer IDs
4. Harper automatically indexes ALL inserted fields (no schema pre-definition)
5. New fields in BigQuery are automatically handled on next insert

Performance Optimization

Integer Primary Keys:

• Deterministic SHA256-based integer IDs from record data
• Ensures fast indexing (no strings/GUIDs/objects as PKs)
• IDs are positive 53-bit safe integers
• Same input always produces same ID (idempotent)

Benefits

✅ Zero manual schema definitions for data tables
✅ Automatic schema evolution (Harper handles new fields)
✅ Fast integer indexing for optimal performance
✅ Thread-safe and idempotent table creation
✅ Simple implementation (no polling, migrations, or distributed locking)
✅ Graceful fallback to schema.graphql if API unavailable

Testing

• 97 tests passing (91 existing + 6 new integer ID tests)
• New test files:
  • test/schema-manager.test.js
  • test/integer-id-generation.test.js
  • test/operations-client.test.js
  • test/type-mapper.test.js
  • test/index-strategy.test.js
• Integration scripts for live testing in examples/

Changes

18 files changed, 1,734 insertions:

• New: src/operations-client.js, src/schema-manager.js, src/type-mapper.js, src/index-strategy.js
• Modified: src/index.js, src/sync-engine.js, src/config-loader.js, config.yaml
• Tests: 6 new test files with comprehensive coverage
• Docs: Updated design document and added implementation summary

Documentation

• Updated docs/plans/2025-11-13-dynamic-table-creation-design.md with implementation notes
• Added docs/internal/dynamic-table-creation-summary.md with complete feature summary
• Documented the schemaless discovery and its implications

Migration Path

Existing deployments:

• System tables remain in schema.graphql (SyncCheckpoint, SyncAudit)
• Data tables can be removed from schema.graphql
• Add Operations API credentials to config.yaml
• Restart - tables created automatically

New deployments:

• Only system tables in schema.graphql
• Configure Operations API credentials
• All data tables created dynamically

What We Didn't Build (Not Needed)

Original design included these components, but they're unnecessary due to Harper's schemaless nature:

• ❌ SchemaLeaderElection polling (no periodic checks needed)
• ❌ Schema migration logic (just insert, Harper handles it)
• ❌ addColumns() operation (not supported by API)
• ❌ Distributed locking for schema checks (no concurrent operations)
• ❌ Complex type change handling (everything schemaless)

---

Ready to merge! All tests passing, feature complete, fully documented.