Skip to content

feat: Project portability — UUID fields, template cloning, and cross-environment export/import #1208

@mihow

Description

@mihow

Summary

Add project portability features to enable two primary use cases:

  1. Template project cloning — Copy a project's structure (deployments, sites, devices, pipeline configs, taxa, filters) into a new project on the same instance, without copying processed data (images, occurrences, detections)
  2. Cross-environment export/import — Serialize a complete project graph to JSON and import it into a different Antenna instance (e.g., self-hosted → managed, prod → staging)

Background

The move_project_data management command (#1192) handles moving deployments between projects on the same instance. This issue covers the complementary operations: cloning and cross-environment portability.

A detailed design spec exists on the move-project-data branch: docs/claude/planning/project-portability-spec.md

Use Case 1: Template Project Cloning

Copy a project's configuration to create a new project with the same setup but no data.

What gets copied:

  • Project settings (default filters, score thresholds, default pipeline)
  • Deployment definitions (name, coordinates, S3 sources)
  • Devices and Sites
  • Pipeline configs and ProcessingService links
  • Taxa lists and taxa M2M links
  • Member list and roles
  • Tags

What does NOT get copied:

  • Source images, events, occurrences, detections, classifications, identifications
  • Jobs
  • Data exports

Proposed interface:

python manage.py clone_project --source-project 23 --name "New Project from Template"

Use Case 2: Cross-Environment Export/Import

Serialize a full project (including all data) to a JSON interchange format and import into another Antenna instance.

Prerequisites (from the portability spec):

  • UUID fields on all project-scoped models (18+ models)
  • Natural key methods for shared references (Taxon, User, Pipeline, Algorithm)
  • FK serialization using UUIDs for project-scoped and natural keys for global references

Key design decisions documented in the spec:

  • Integer PKs remain internal; UUIDs are external identity
  • Conflict handling modes: error, skip, update
  • Export ordering follows FK dependency graph
  • Source image files are NOT exported (S3 paths preserved; target instance needs access to same or copied bucket)

Proposed interface:

# Export
python manage.py export_project --project 23 --output nunavik-export.json

# Import
python manage.py import_project nunavik-export.json --on-conflict=skip --dry-run

Implementation phases

Phase 1: Template cloning (simpler, no UUIDs needed)

  • clone_project management command
  • Tests

Phase 2: UUID infrastructure

  • Add UUID fields to all project-scoped models (backfill migration)
  • Natural key methods on models
  • Add missing unique constraints (Deployment, Device, Site names within project)

Phase 3: Export/Import

  • export_project management command (JSON serialization with UUID-based FKs)
  • import_project management command (conflict resolution, FK remapping)
  • Tests

Open research areas

Documented in the spec (docs/claude/planning/project-portability-spec.md):

  • UUID backfill migration performance on large tables (SourceImage ~10M rows, Detection ~1M)
  • Detection/Classification uniqueness constraints
  • Taxonomy divergence between instances
  • Darwin Core / GBIF identifier conventions
  • Schema evolution between Antenna versions

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions