Metadata Collection Prototype #1062
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Design doc for a new generic collection type that uses lightweight metadata fetches (id + modified_gmt) to define list structure, then selectively fetches only missing or stale entities. Key design decisions: - Metadata defines list structure (enables loading placeholders) - KV store for metadata persistence (memory or disk-backed) - Generic over entity type (posts, media, etc.) - Batch fetch via `include` param for efficiency
Foundation types for metadata-based sync: - `SyncableEntity` trait: entities with `id` and `modified_gmt` fields - `EntityMetadata<Id>`: lightweight struct holding just id + modified_gmt Also adds `Clone` and `Copy` derives to `WpGmtDateTime` since the inner `DateTime<Utc>` is `Copy`.
Represents items in a metadata-backed list where entities may be: - `Loaded`: full entity available from cache - `Loading`: fetch in progress, shows placeholder with metadata - `Failed`: fetch failed, includes error message for retry UI Includes `HasId` helper trait for extracting IDs from loaded entities.
Abstracts metadata persistence so we can swap between in-memory and disk-backed storage. The in-memory implementation is useful for prototyping; can be replaced with SQLite/file-based later. Includes unit tests for all KvStore operations.
Lightweight fetch that returns only id + modified_gmt for posts, enabling the metadata-first sync strategy. Unlike `fetch_posts_page`, this does NOT upsert to the database - the metadata is used transiently to determine which posts need full fetching. Also adds `Hash` derive to `wp_content_i64_id` and `wp_content_u64_id` macros so ID types can be used as HashMap keys.
Batch fetch full post data for specific IDs using the `include` parameter. Used for selective sync - fetching only posts that are missing or stale in the cache. Returns early with empty Vec if no IDs provided.
Core collection type that: - Uses KV store to persist list metadata (id + modified_gmt) - Builds list with Loaded/Loading states based on cache status - Provides methods to find missing/stale entities for selective fetch - Supports pagination with append for subsequent pages Includes comprehensive unit tests for all operations.
- Format derive macros across multiple lines - Use or_default() instead of or_insert_with(Vec::new) - Add type aliases for complex closure types - Collapse nested if-let into single condition - Simplify test entity to unit struct
Documents what was built, where each component lives, test coverage, and differences from the original sketch. Also lists next steps.
Consolidated design after discussion covering: - Service-owned stores (`EntityStateStore`, `ListMetadataStore`) - Read-only traits for collection access - `MetadataCollection<F>` generic only over fetcher - Cross-collection state consistency - State transitions (Missing/Fetching/Cached/Stale/Failed) Also updated v1 doc with intermediate v2 notes.
Implements Phase 1 of the v3 MetadataCollection design: - `EntityMetadata` - Non-generic struct with `i64` id + `Option<WpGmtDateTime>` (optional modified_gmt for entities like Comments that lack this field) - `EntityState` - Enum tracking fetch lifecycle (Missing, Fetching, Cached, Stale, Failed) - `CollectionItem` - Combines metadata with state for list items - `SyncResult` - Result of sync operations with counts and pagination info - `MetadataFetchResult` - Updated to non-generic version Removes superseded prototype code: - Old generic `EntityMetadata<Id>` - `KvStore<Id>` trait and `InMemoryKvStore<Id>` - `ListItem<T, Id>` enum - `MetadataCollection<T, Id>` (old version) - `SyncableEntity` trait Updates `PostService::fetch_posts_metadata` to use new non-generic types.
Implements Phase 2 of the v3 MetadataCollection design: - `EntityStateStore` - Memory-backed store for entity fetch states - Maps `i64` ID to `EntityState` (Missing, Fetching, Cached, etc.) - Thread-safe via `RwLock<HashMap>` - `filter_fetchable()` excludes currently-fetching IDs to prevent duplicates - `EntityStateReader` trait - Read-only access for collections - `ListMetadataStore` - Memory-backed store for list structure - Maps filter key (String) to `Vec<EntityMetadata>` - Supports `set` (replace) and `append` (pagination) - `ListMetadataReader` trait - Read-only access for collections Both stores are memory-only; state resets on app restart.
Implements Phase 3 of the v3 MetadataCollection design: - `MetadataFetcher` trait - Async trait for fetching metadata and entities - `fetch_metadata(page, per_page, is_first_page)` - Fetch list structure - `ensure_fetched(ids)` - Fetch full entities by ID - `MetadataCollection<F>` - Generic collection over fetcher type - `refresh()` - Fetch page 1, replace metadata, sync missing - `load_next_page()` - Fetch next page, append, sync missing - `items()` - Get `CollectionItem` list with states - `is_relevant_update()` - Check DB updates for relevance - Batches large fetches into 100-item chunks (API limit) Also adds `tokio` as dev-dependency for async tests.
Add metadata sync infrastructure to PostService for efficient list syncing: - Add `state_store_with_edit_context` field for tracking per-entity fetch state (Missing, Fetching, Cached, Stale, Failed). Each context needs its own state store since the same entity can have different states across contexts. - Add `metadata_store` field for list structure per filter key. Shared across all contexts - callers include context in the key string (e.g., "site_1:edit:posts:publish"). - Add `fetch_and_store_metadata()` method that fetches lightweight metadata (id + modified_gmt) and stores it in the metadata store. - Update `fetch_posts_by_ids()` to track entity state: - Filters out already-fetching IDs to prevent duplicate requests - Sets Fetching state before API call - Sets Cached on success, Failed on error or missing posts - Add `PostMetadataFetcherWithEditContext` implementing `MetadataFetcher` trait, delegating to PostService methods. - Add reader accessor methods for collections to get read-only access: `state_reader_with_edit_context()`, `metadata_reader()`, `get_entity_state_with_edit_context()`.
Create the concrete type that wraps MetadataCollection for UniFFI:
- Add `PostMetadataCollectionWithEditContext` struct combining:
- `MetadataCollection<PostMetadataFetcherWithEditContext>` for sync logic
- Service reference for loading full entity data
- Filter for this collection
- Add `PostMetadataCollectionItem` record type with:
- `id`: Post ID
- `state`: EntityState (Missing, Fetching, Cached, Stale, Failed)
- `data`: Optional FullEntityAnyPostWithEditContext
- Add `create_post_metadata_collection_with_edit_context` to PostService
- Make types UniFFI-compatible:
- Add `uniffi::Enum` to EntityState
- Add `uniffi::Record` to SyncResult (change usize to u64)
- Use interior mutability (RwLock<PaginationState>) in MetadataCollection
for compatibility with UniFFI's Arc-wrapped objects
- Add `read_posts_by_ids_from_db` helper to PostService for bulk loading
- Document state representation approaches in design doc:
- Two-dimensional (DataState + FetchStatus)
- Flattened explicit states enum
Add Kotlin wrapper for PostMetadataCollectionWithEditContext to enable reactive UI updates when database changes occur. Changes: - Add `ObservableMetadataCollection` class with observer pattern - Update `DatabaseChangeNotifier` to support metadata collections - Add `getObservablePostMetadataCollectionWithEditContext` extension on `PostService` - Update implementation plan with Phase 7 completion
Add example screen demonstrating the metadata-first sync strategy with visual state indicators for each item (Missing, Fetching, Cached, Stale, Failed). Changes: - Add `PostMetadataCollectionViewModel` with manual refresh/loadNextPage controls - Add `PostMetadataCollectionScreen` with filter controls and state indicators - Wire up navigation and DI for the new screen - Update implementation plan marking Phase 8 complete
WordPress REST API defaults to filtering by 'publish' status, which caused drafts, pending, and other non-published posts to return "Not found" when fetching by ID. Changes: - Add explicit status filter including all standard post statuses (publish, draft, pending, private, future)
When fetching metadata, compare the API's `modified_gmt` against cached posts in the database. Posts with different timestamps are marked as `Stale`, triggering a re-fetch on the next sync. Changes: - Add `select_modified_gmt_by_ids` to PostRepository for efficient batch lookup - Add `detect_and_mark_stale_posts` to PostService for staleness comparison - Call staleness detection in `fetch_and_store_metadata` after storing metadata
Design for moving list metadata from in-memory KV store to database: - Three DB tables: list_metadata, list_metadata_items, list_metadata_state - MetadataService owns list storage, state transitions, version management - PostService orchestrates sync, owns entity state, does staleness detection - Split observers for data vs state changes in Kotlin wrapper - Version-based concurrency control for handling concurrent refreshes
Detailed plan with 5 phases, 17 commits, ordered from low-level to high-level: - Phase 1: Database foundation (DbTable, migration, types, repository) - Phase 2: MetadataService in wp_mobile - Phase 3: Integration with PostService, collection refactor - Phase 4: Observer split (data vs state) - Phase 5: Testing and cleanup Includes dependency order, risk areas, and verification checkpoints.
Implement the database layer for list metadata storage, replacing the in-memory KV store. This enables proper observer patterns and persistence between app launches. Database schema (3 tables): - list_metadata: headers with pagination, version for concurrency control - list_metadata_items: ordered entity IDs (rowid = display order) - list_metadata_state: sync state (idle, fetching_first_page, fetching_next_page, error) Changes: - Add DbTable variants: ListMetadata, ListMetadataItems, ListMetadataState - Add migration 0007-create-list-metadata-tables.sql - Add list_metadata module with DbListMetadata, DbListMetadataItem, DbListMetadataState structs and ListState enum - Add db_types/db_list_metadata.rs with column enums and from_row impls - Add repository/list_metadata.rs with read and write operations
Add helper methods for atomic state transitions during sync operations: - `begin_refresh()`: Atomically increment version, set state to FetchingFirstPage, and return info needed for the fetch - `begin_fetch_next_page()`: Check pagination state, set state to FetchingNextPage, and return page number and version for stale check - `complete_sync()`: Set state to Idle on success - `complete_sync_with_error()`: Set state to Error with message These helpers ensure correct state transitions and enable version-based concurrency control to detect when a refresh invalidates an in-flight load-more operation. Also adds `RefreshInfo` and `FetchNextPageInfo` structs to encapsulate the data returned from begin operations.
Implement MetadataService in wp_mobile to provide persistence for list metadata (ordered entity IDs, pagination, sync state). This enables data to survive app restarts unlike the in-memory ListMetadataStore. The service wraps ListMetadataRepository and implements ListMetadataReader trait, allowing MetadataCollection to use either memory or database storage through the same interface. Features: - Read operations: get_entity_ids, get_metadata, get_state, get_pagination - Write operations: set_items, append_items, update_pagination, delete_list - State management: set_state, complete_sync, complete_sync_with_error - Concurrency helpers: begin_refresh, begin_fetch_next_page Also fixes pre-existing clippy warnings in posts.rs (collapsible_if).
Add MetadataService as a field in PostService to provide database-backed list metadata storage. This enables list structure and pagination to persist across app restarts. Changes: - Add `metadata_service` field to PostService - Add `persistent_metadata_reader()` and `metadata_service()` accessors - Add `sync_post_list()` method that orchestrates full sync flow using MetadataService for persistence - Extend SyncResult with `current_page` and `total_pages` fields for pagination tracking The existing in-memory `metadata_store` is preserved for backwards compatibility with existing code paths. Future work will migrate callers to use the persistent service.
Mark completed phases and update with actual commit hashes: - Phase 1 (Database Foundation): Complete - Phase 2 (MetadataService): Complete - Phase 3 (Integration): Partial (3.2 done, 3.1 deferred) - Phase 5 (Testing): Partial (tests inline with implementation) Add status summary table and update dependency diagram with completion markers.
oguzkocer
force-pushed
the
prototype/metadata-collection
branch
from
08e6460 to
5880f40
Compare
Introduces database infrastructure for tracking list metadata, enabling efficient cache invalidation and stale detection for paginated lists. Changes: - Add migration for list metadata tables - Add DbListMetadata types for database operations - Add ListMetadata domain types - Add ListMetadataRepository with full CRUD operations - Update PostsRepository with stale detection helpers Extracted from prototype/metadata-collection branch. Original commits: - 14b01d80 (Implement stale detection by comparing modified_gmt timestamps) - e47cec89 (Add database foundation for MetadataService) - 2440a13f (Add list metadata repository concurrency helpers) - cc0c8a58 (Reset stale fetching states on app launch) - d64142fb (Split collection observers for data vs state updates) - fe7435c9 (make fmt-rust) - 2918b339 (Add parent and menu_order fields to list metadata items) - 25f88a49 (Rename last_updated_at to last_fetched_at in list metadata) - 1d709e70 (Simplify ListMetadataReader trait with combined ListInfo query) Note: Since we use a rebase/squash merge strategy, these commits may show "does not belong to any branch" on GitHub but remain accessible via URL.
Changes the database storage for `ListState` from TEXT strings to INTEGER values for better performance and type safety. Changes: - Update migration to use `INTEGER NOT NULL DEFAULT 0` for state column - Add `#[repr(i32)]` to `ListState` enum with explicit discriminant values - Implement `ToSql` and `FromSql` traits for direct rusqlite integration - Remove string-based `as_db_str()` and `From<&str>` implementations - Update all callers to use the enum directly with rusqlite params
The `FromSql` implementation now returns a proper error when encountering an unknown integer value, rather than silently defaulting to `Idle`. This makes data corruption issues visible instead of hiding them.
The repository struct has no state, so methods are converted from instance methods (&self) to associated functions. This removes the need to instantiate the struct before calling methods. Before: ListMetadataRepository.get_header(&conn, &site, key) After: ListMetadataRepository::get_header(&conn, &site, key)
Replaces individual INSERT statements with a single batch INSERT for better performance when inserting multiple items. Uses functional style with try_for_each and flat_map. Items are chunked to stay under SQLite's variable limit (999).
Replaces two separate queries (get_header + get_state) with a single JOIN query via get_header_with_state for better efficiency.
Replaces raw `&str` parameters with `&ListKey` throughout the repository API. This prevents accidental misuse of arbitrary strings as list keys and makes the API more self-documenting. The ListKey type provides: - Type safety at compile time - Conversion from &str and String via From trait - as_str() for SQL parameter usage - Display impl for debug output
- Condense doc comment, remove references to non-existent MetadataService - Return Result<usize, SqliteDbError> instead of logging internally - Ignore errors at call site with explanatory comment
Normalizes the schema by replacing (db_site_id, key) in items table with a foreign key to list_metadata. This: - Ensures referential integrity - Enables cascade delete (simplifies delete_list) - Reduces storage per item
Adds the `log` facade crate to enable proper logging. Debug logs added to key list metadata operations: - begin_refresh, begin_fetch_next_page - complete_sync, complete_sync_with_error - set_items, append_items, delete_list Replaces eprintln! with log::warn! for unknown table warnings.
Implement ToSql/FromSql traits for ListKey to simplify repository code by using the type directly in SQL params instead of .as_str(). Also add logging for reset_stale_fetching_states failures instead of silently ignoring errors. Changes: - Add ToSql/FromSql implementations for ListKey - Replace key.as_str() with key in all SQL params - Log warning on reset_stale_fetching_states failure
…st_key variants Allows callers who already have the list_metadata_id to skip an extra header lookup query.
Split functions that take site+key to allow callers with an existing list_metadata_id to skip the header lookup query. Renamed functions for consistency: - set_items → set_items_by_list_metadata_id / set_items_by_list_key - append_items → append_items_by_list_metadata_id / append_items_by_list_key - update_header → update_header_by_list_metadata_id / update_header_by_list_key - increment_version → increment_version_by_list_metadata_id / increment_version_by_list_key - get_state → get_state_by_list_metadata_id - get_state_by_key → get_state_by_list_key - update_state → update_state_by_list_metadata_id - update_state_by_key → update_state_by_list_key
Add `ConstraintViolation` variant to `SqliteDbError` to distinguish UNIQUE constraint violations from other SQLite errors. Use this in `ListMetadataRepository::get_or_create` to handle the rare case where another thread creates the same header between our SELECT and INSERT. Changes: - Add `SqliteDbError::ConstraintViolation` variant - Update `From<rusqlite::Error>` to detect constraint violations - Update `get_or_create` to catch constraint violations and re-fetch
Add `get_or_create_and_increment_version` that uses a single `INSERT ... ON CONFLICT DO UPDATE ... RETURNING` query to atomically create or update a header while incrementing its version. This reduces `begin_refresh` from 5-6 queries to 2 queries: - Before: get_or_create (1-2) + increment_version (2) + update_state (1) + get_header (1) - After: get_or_create_and_increment_version (1) + update_state (1) Changes: - Add `HeaderVersionInfo` struct for the return type - Add `get_or_create_and_increment_version` method using RETURNING clause - Update `begin_refresh` to use the new optimized method - Add tests for the new method
Remove `increment_version_by_list_metadata_id` and `increment_version_by_list_key` since version increment should only happen through `get_or_create_and_increment_version` as part of a proper refresh flow, not called arbitrarily.
Move concurrency orchestration logic out of the repository layer. These helpers (`begin_refresh`, `begin_fetch_next_page`, `complete_sync`, `complete_sync_with_error`) belong in a service layer since they: - Orchestrate multiple repository operations - Apply business rules (checking page counts, deciding state transitions) - Return domain-specific result types The repository now provides clean primitives that a service layer can compose: - `get_or_create_and_increment_version` for atomic create/increment - `update_state_by_list_metadata_id` for state changes - `get_header`, `get_items`, etc. for reads
Move `reset_stale_fetching_states_internal` from `WpApiCache` to `ListMetadataRepository` where it belongs. The repository owns the state table and should provide all operations on it. `WpApiCache::perform_migrations` still calls this method after migrations complete, but now delegates to the repository.
The repository layer should not dictate per_page values - this must be set by the service layer to match networking configuration. Changes: - Add PerPageMismatch error variant to SqliteDbError - Make per_page a required parameter in get_or_create - Make per_page required in get_or_create_and_increment_version - Update set_items_by_list_key to require per_page - Update append_items_by_list_key to require per_page - Update update_state_by_list_key to require per_page - Return error when existing header has different per_page
Update schema validation tests to use `get_table_column_names` helper, which verifies that column enum indices match actual database schema positions via PRAGMA table_info. This matches the pattern used by posts repository tests and provides stronger guarantees that the column enums won't break if migrations reorder columns.
…tion Merge the polished ListMetadataRepository from feature/list-metadata-repository while preserving concurrency helpers that were intentionally kept on this branch. Key changes from feature/list-metadata-repository: - Convert to associated functions (no &self parameters) - Add ListKey newtype for type-safe list key handling - Add per_page as a required parameter with validation - Use batch insert for list metadata items - Add FK from list_metadata_items to list_metadata - Store ListState as INTEGER instead of TEXT - Add get_or_create_and_increment_version for atomic operations - Add reset_stale_fetching_states method - Split get_items and get_state into by_list_metadata_id and by_list_key variants Preserved from this branch: - Concurrency helpers: begin_refresh, begin_fetch_next_page, complete_sync, complete_sync_with_error (for service layer orchestration) - posts.rs: select_modified_gmt_by_ids method (for staleness detection) Note: wp_mobile code needs to be updated to use the new associated function API.
Update all usages to work with the new associated function style and `ListKey` newtype from the merged feature/list-metadata-repository. Key API changes: - Use `ListKey` newtype instead of `&str` for type-safe key handling - Convert instance method calls to associated function calls (e.g., `self.repo.get_items(...)` → `ListMetadataRepository::get_items_by_list_key(...)`) - Add `per_page` parameter where required by the new API - Use `list_metadata_id: RowId` for `complete_sync`/`complete_sync_with_error` New convenience methods added to MetadataService: - `complete_sync_by_key`: Lookup list by key and complete sync - `complete_sync_with_error_by_key`: Lookup list by key and set error state Changes: - Update `ListMetadataReader` trait to use `&ListKey` - Update `MetadataCollection` to store `ListKey` instead of `String` - Update `MetadataService` to use associated functions and `ListKey` - Update `PersistentPostMetadataFetcherWithEditContext` to use `ListKey` - Update `PostService` methods to use `&ListKey` and new API - Update all tests to use `ListKey::from()` for key creation
Resolve conflicts in list_metadata.rs by preserving concurrency helpers (begin_refresh, begin_fetch_next_page, complete_sync, complete_sync_with_error) that were removed in the feature branch polish but are needed for the prototype.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please ignore this PR for now, I am just opening a draft PR so I can access the built artifacts from WPAndroid.