-
Notifications
You must be signed in to change notification settings - Fork 3
Add ListMetadataRepository for paginated list state tracking
#1069
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Introduces database infrastructure for tracking list metadata, enabling efficient cache invalidation and stale detection for paginated lists. Changes: - Add migration for list metadata tables - Add DbListMetadata types for database operations - Add ListMetadata domain types - Add ListMetadataRepository with full CRUD operations - Update PostsRepository with stale detection helpers Extracted from prototype/metadata-collection branch. Original commits: - 14b01d80 (Implement stale detection by comparing modified_gmt timestamps) - e47cec89 (Add database foundation for MetadataService) - 2440a13f (Add list metadata repository concurrency helpers) - cc0c8a58 (Reset stale fetching states on app launch) - d64142fb (Split collection observers for data vs state updates) - fe7435c9 (make fmt-rust) - 2918b339 (Add parent and menu_order fields to list metadata items) - 25f88a49 (Rename last_updated_at to last_fetched_at in list metadata) - 1d709e70 (Simplify ListMetadataReader trait with combined ListInfo query) Note: Since we use a rebase/squash merge strategy, these commits may show "does not belong to any branch" on GitHub but remain accessible via URL.
Changes the database storage for `ListState` from TEXT strings to INTEGER values for better performance and type safety. Changes: - Update migration to use `INTEGER NOT NULL DEFAULT 0` for state column - Add `#[repr(i32)]` to `ListState` enum with explicit discriminant values - Implement `ToSql` and `FromSql` traits for direct rusqlite integration - Remove string-based `as_db_str()` and `From<&str>` implementations - Update all callers to use the enum directly with rusqlite params
The `FromSql` implementation now returns a proper error when encountering an unknown integer value, rather than silently defaulting to `Idle`. This makes data corruption issues visible instead of hiding them.
The repository struct has no state, so methods are converted from instance methods (&self) to associated functions. This removes the need to instantiate the struct before calling methods. Before: ListMetadataRepository.get_header(&conn, &site, key) After: ListMetadataRepository::get_header(&conn, &site, key)
Replaces individual INSERT statements with a single batch INSERT for better performance when inserting multiple items. Uses functional style with try_for_each and flat_map. Items are chunked to stay under SQLite's variable limit (999).
Replaces two separate queries (get_header + get_state) with a single JOIN query via get_header_with_state for better efficiency.
Replaces raw `&str` parameters with `&ListKey` throughout the repository API. This prevents accidental misuse of arbitrary strings as list keys and makes the API more self-documenting. The ListKey type provides: - Type safety at compile time - Conversion from &str and String via From trait - as_str() for SQL parameter usage - Display impl for debug output
- Condense doc comment, remove references to non-existent MetadataService - Return Result<usize, SqliteDbError> instead of logging internally - Ignore errors at call site with explanatory comment
Normalizes the schema by replacing (db_site_id, key) in items table with a foreign key to list_metadata. This: - Ensures referential integrity - Enables cascade delete (simplifies delete_list) - Reduces storage per item
Adds the `log` facade crate to enable proper logging. Debug logs added to key list metadata operations: - begin_refresh, begin_fetch_next_page - complete_sync, complete_sync_with_error - set_items, append_items, delete_list Replaces eprintln! with log::warn! for unknown table warnings.
Implement ToSql/FromSql traits for ListKey to simplify repository code by using the type directly in SQL params instead of .as_str(). Also add logging for reset_stale_fetching_states failures instead of silently ignoring errors. Changes: - Add ToSql/FromSql implementations for ListKey - Replace key.as_str() with key in all SQL params - Log warning on reset_stale_fetching_states failure
…st_key variants Allows callers who already have the list_metadata_id to skip an extra header lookup query.
Split functions that take site+key to allow callers with an existing list_metadata_id to skip the header lookup query. Renamed functions for consistency: - set_items → set_items_by_list_metadata_id / set_items_by_list_key - append_items → append_items_by_list_metadata_id / append_items_by_list_key - update_header → update_header_by_list_metadata_id / update_header_by_list_key - increment_version → increment_version_by_list_metadata_id / increment_version_by_list_key - get_state → get_state_by_list_metadata_id - get_state_by_key → get_state_by_list_key - update_state → update_state_by_list_metadata_id - update_state_by_key → update_state_by_list_key
Add `ConstraintViolation` variant to `SqliteDbError` to distinguish UNIQUE constraint violations from other SQLite errors. Use this in `ListMetadataRepository::get_or_create` to handle the rare case where another thread creates the same header between our SELECT and INSERT. Changes: - Add `SqliteDbError::ConstraintViolation` variant - Update `From<rusqlite::Error>` to detect constraint violations - Update `get_or_create` to catch constraint violations and re-fetch
Add `get_or_create_and_increment_version` that uses a single `INSERT ... ON CONFLICT DO UPDATE ... RETURNING` query to atomically create or update a header while incrementing its version. This reduces `begin_refresh` from 5-6 queries to 2 queries: - Before: get_or_create (1-2) + increment_version (2) + update_state (1) + get_header (1) - After: get_or_create_and_increment_version (1) + update_state (1) Changes: - Add `HeaderVersionInfo` struct for the return type - Add `get_or_create_and_increment_version` method using RETURNING clause - Update `begin_refresh` to use the new optimized method - Add tests for the new method
Remove `increment_version_by_list_metadata_id` and `increment_version_by_list_key` since version increment should only happen through `get_or_create_and_increment_version` as part of a proper refresh flow, not called arbitrarily.
Move concurrency orchestration logic out of the repository layer. These helpers (`begin_refresh`, `begin_fetch_next_page`, `complete_sync`, `complete_sync_with_error`) belong in a service layer since they: - Orchestrate multiple repository operations - Apply business rules (checking page counts, deciding state transitions) - Return domain-specific result types The repository now provides clean primitives that a service layer can compose: - `get_or_create_and_increment_version` for atomic create/increment - `update_state_by_list_metadata_id` for state changes - `get_header`, `get_items`, etc. for reads
Move `reset_stale_fetching_states_internal` from `WpApiCache` to `ListMetadataRepository` where it belongs. The repository owns the state table and should provide all operations on it. `WpApiCache::perform_migrations` still calls this method after migrations complete, but now delegates to the repository.
ac33332 to
811eb46
Compare
The repository layer should not dictate per_page values - this must be set by the service layer to match networking configuration. Changes: - Add PerPageMismatch error variant to SqliteDbError - Make per_page a required parameter in get_or_create - Make per_page required in get_or_create_and_increment_version - Update set_items_by_list_key to require per_page - Update append_items_by_list_key to require per_page - Update update_state_by_list_key to require per_page - Return error when existing header has different per_page
Update schema validation tests to use `get_table_column_names` helper, which verifies that column enum indices match actual database schema positions via PRAGMA table_info. This matches the pattern used by posts repository tests and provides stronger guarantees that the column enums won't break if migrations reorder columns.
ListMetadataRepository for paginated list state tracking
jkmassel
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left one thought about potential enhancement, but not critical.
| /// Error message if state is error | ||
| pub error_message: Option<String>, | ||
| /// ISO 8601 timestamp of last state change | ||
| pub updated_at: String, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if it'd make sense to store this as an integer (in the DB too)? Seems faster than string parsing, and more space-efficient?
Not 100% convinced we need that particular performance win, just noting it because it came to mind.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense. We need these to be consistent across the board too, let me add a personal TODO for this and handle it on its own PR in the next 2 weeks.
Summary
Adds
ListMetadataRepositorytowp_mobile_cachefor tracking paginated list state, enabling efficient cache invalidation and stale detection.Database Schema
Type Hierarchy
Key Features
ListKeynewtype prevents misuse of arbitrary strings;ListStatestored as INTEGER withToSql/FromSqlget_or_createcatches constraint violations and retriesget_or_create_and_increment_versionusesINSERT ... ON CONFLICT DO UPDATE ... RETURNINGRepository Operations
Error Handling
ConstraintViolation- detected fromrusqlite::Errorfor race condition handlingPerPageMismatch { expected, actual }- when existing header has differentper_pagethan requestedDesign Decisions
per_pageis required: Must be passed by caller to match networking config; returns error on mismatch rather than silently using existing value_by_list_metadata_idfor callers with cached rowid,_by_list_keyfor convenienceChanges
0007-create-list-metadata-tables.sqlListKey,ListState, and DB types inlist_metadata.rsdb_types/db_list_metadata.rsListMetadataRepositorywith associated functionsConstraintViolationandPerPageMismatcherror variants toSqliteDbErrorlogcrate dependency for structured loggingreset_stale_fetching_statesafter migrations complete