Add ListMetadataRepository for paginated list state tracking (#1069)

* Add list metadata repository to wp_mobile_cache

Introduces database infrastructure for tracking list metadata, enabling
efficient cache invalidation and stale detection for paginated lists.

Changes:
- Add migration for list metadata tables
- Add DbListMetadata types for database operations
- Add ListMetadata domain types
- Add ListMetadataRepository with full CRUD operations
- Update PostsRepository with stale detection helpers

Extracted from prototype/metadata-collection branch. Original commits:
- 14b01d80 (Implement stale detection by comparing modified_gmt timestamps)
- e47cec89 (Add database foundation for MetadataService)
- 2440a13f (Add list metadata repository concurrency helpers)
- cc0c8a58 (Reset stale fetching states on app launch)
- d64142fb (Split collection observers for data vs state updates)
- fe7435c9 (make fmt-rust)
- 2918b339 (Add parent and menu_order fields to list metadata items)
- 25f88a49 (Rename last_updated_at to last_fetched_at in list metadata)
- 1d709e70 (Simplify ListMetadataReader trait with combined ListInfo query)

Note: Since we use a rebase/squash merge strategy, these commits may show
"does not belong to any branch" on GitHub but remain accessible via URL.

* Store ListState as INTEGER instead of TEXT

Changes the database storage for `ListState` from TEXT strings to INTEGER
values for better performance and type safety.

Changes:
- Update migration to use `INTEGER NOT NULL DEFAULT 0` for state column
- Add `#[repr(i32)]` to `ListState` enum with explicit discriminant values
- Implement `ToSql` and `FromSql` traits for direct rusqlite integration
- Remove string-based `as_db_str()` and `From<&str>` implementations
- Update all callers to use the enum directly with rusqlite params

* Return error for invalid ListState values instead of silent fallback

The `FromSql` implementation now returns a proper error when encountering
an unknown integer value, rather than silently defaulting to `Idle`.
This makes data corruption issues visible instead of hiding them.

* Convert ListMetadataRepository methods to associated functions

The repository struct has no state, so methods are converted from
instance methods (&self) to associated functions. This removes the
need to instantiate the struct before calling methods.

Before: ListMetadataRepository.get_header(&conn, &site, key)
After:  ListMetadataRepository::get_header(&conn, &site, key)

* Use batch insert for list metadata items

Replaces individual INSERT statements with a single batch INSERT for
better performance when inserting multiple items. Uses functional style
with try_for_each and flat_map.

Items are chunked to stay under SQLite's variable limit (999).

* Use JOIN query internally in get_state_by_key

Replaces two separate queries (get_header + get_state) with a single
JOIN query via get_header_with_state for better efficiency.

* Add ListKey newtype for type-safe list key handling

Replaces raw `&str` parameters with `&ListKey` throughout the repository
API. This prevents accidental misuse of arbitrary strings as list keys
and makes the API more self-documenting.

The ListKey type provides:
- Type safety at compile time
- Conversion from &str and String via From trait
- as_str() for SQL parameter usage
- Display impl for debug output

* Simplify reset_stale_fetching_states and return Result

- Condense doc comment, remove references to non-existent MetadataService
- Return Result<usize, SqliteDbError> instead of logging internally
- Ignore errors at call site with explanatory comment

* Add FK from list_metadata_items to list_metadata

Normalizes the schema by replacing (db_site_id, key) in items table
with a foreign key to list_metadata. This:
- Ensures referential integrity
- Enables cascade delete (simplifies delete_list)
- Reduces storage per item

* Add log crate for structured logging

Adds the `log` facade crate to enable proper logging. Debug logs added
to key list metadata operations:
- begin_refresh, begin_fetch_next_page
- complete_sync, complete_sync_with_error
- set_items, append_items, delete_list

Replaces eprintln! with log::warn! for unknown table warnings.

* Add ToSql/FromSql for ListKey and log stale state reset errors

Implement ToSql/FromSql traits for ListKey to simplify repository
code by using the type directly in SQL params instead of .as_str().

Also add logging for reset_stale_fetching_states failures instead
of silently ignoring errors.

Changes:
- Add ToSql/FromSql implementations for ListKey
- Replace key.as_str() with key in all SQL params
- Log warning on reset_stale_fetching_states failure

* Remove unused update hook helper functions

* Remove select_modified_gmt_by_ids from this PR

* Split get_items and get_item_count into by_list_metadata_id and by_list_key variants

Allows callers who already have the list_metadata_id to skip an extra
header lookup query.

* Remove redundant check_version function

* Add by_list_metadata_id variants for write operations

Split functions that take site+key to allow callers with an existing
list_metadata_id to skip the header lookup query.

Renamed functions for consistency:
- set_items → set_items_by_list_metadata_id / set_items_by_list_key
- append_items → append_items_by_list_metadata_id / append_items_by_list_key
- update_header → update_header_by_list_metadata_id / update_header_by_list_key
- increment_version → increment_version_by_list_metadata_id / increment_version_by_list_key
- get_state → get_state_by_list_metadata_id
- get_state_by_key → get_state_by_list_key
- update_state → update_state_by_list_metadata_id
- update_state_by_key → update_state_by_list_key

* Replace unwrap with expect for better panic messages

* Update migration count in Kotlin and Swift tests

* Handle race condition in get_or_create

Add `ConstraintViolation` variant to `SqliteDbError` to distinguish
UNIQUE constraint violations from other SQLite errors. Use this in
`ListMetadataRepository::get_or_create` to handle the rare case where
another thread creates the same header between our SELECT and INSERT.

Changes:
- Add `SqliteDbError::ConstraintViolation` variant
- Update `From<rusqlite::Error>` to detect constraint violations
- Update `get_or_create` to catch constraint violations and re-fetch

* Add optimized get_or_create_and_increment_version method

Add `get_or_create_and_increment_version` that uses a single
`INSERT ... ON CONFLICT DO UPDATE ... RETURNING` query to atomically
create or update a header while incrementing its version.

This reduces `begin_refresh` from 5-6 queries to 2 queries:
- Before: get_or_create (1-2) + increment_version (2) + update_state (1) + get_header (1)
- After: get_or_create_and_increment_version (1) + update_state (1)

Changes:
- Add `HeaderVersionInfo` struct for the return type
- Add `get_or_create_and_increment_version` method using RETURNING clause
- Update `begin_refresh` to use the new optimized method
- Add tests for the new method

* Remove standalone increment_version methods

Remove `increment_version_by_list_metadata_id` and `increment_version_by_list_key`
since version increment should only happen through `get_or_create_and_increment_version`
as part of a proper refresh flow, not called arbitrarily.

* Remove concurrency helpers from repository layer

Move concurrency orchestration logic out of the repository layer.
These helpers (`begin_refresh`, `begin_fetch_next_page`, `complete_sync`,
`complete_sync_with_error`) belong in a service layer since they:
- Orchestrate multiple repository operations
- Apply business rules (checking page counts, deciding state transitions)
- Return domain-specific result types

The repository now provides clean primitives that a service layer can compose:
- `get_or_create_and_increment_version` for atomic create/increment
- `update_state_by_list_metadata_id` for state changes
- `get_header`, `get_items`, etc. for reads

* Move reset_stale_fetching_states to ListMetadataRepository

Move `reset_stale_fetching_states_internal` from `WpApiCache` to
`ListMetadataRepository` where it belongs. The repository owns
the state table and should provide all operations on it.

`WpApiCache::perform_migrations` still calls this method after
migrations complete, but now delegates to the repository.

* Make per_page a required parameter with validation

The repository layer should not dictate per_page values - this
must be set by the service layer to match networking configuration.

Changes:
- Add PerPageMismatch error variant to SqliteDbError
- Make per_page a required parameter in get_or_create
- Make per_page required in get_or_create_and_increment_version
- Update set_items_by_list_key to require per_page
- Update append_items_by_list_key to require per_page
- Update update_state_by_list_key to require per_page
- Return error when existing header has different per_page

* Use PRAGMA-based column enum tests for list_metadata

Update schema validation tests to use `get_table_column_names` helper,
which verifies that column enum indices match actual database schema
positions via PRAGMA table_info.

This matches the pattern used by posts repository tests and provides
stronger guarantees that the column enums won't break if migrations
reorder columns.

Author: oguzkocer

Cargo.lock

@@ -45,6 +45,7 @@ hyper-util = "0.1.19"
 indoc = "2.0"
 itertools = "0.14.0"
 linkify = "0.10.0"
+log = "0.4"
 parse_link_header = "0.4"
 paste = "1.0"
 proc-macro-crate = "3.4.0"

Cargo.toml

@@ -10,6 +10,6 @@ class WordPressApiCacheTest {
 
     @Test
     fun testThatMigrationsWork() = runTest {
-        assertEquals(6, WordPressApiCache().performMigrations())
+        assertEquals(7, WordPressApiCache().performMigrations())
     }
 }

native/kotlin/api/kotlin/src/integrationTest/kotlin/WordPressApiCacheTest.kt

@@ -16,7 +16,7 @@ actor Test {
 
     @Test func testMigrationsWork() async throws {
         let migrationsPerformed = try await self.cache.performMigrations()
-        #expect(migrationsPerformed == 6)
+        #expect(migrationsPerformed == 7)
     }
 
     #if !os(Linux)

native/swift/Tests/wordpress-api-cache/WordPressApiCacheTests.swift

@@ -8,6 +8,7 @@ default = ["rusqlite/bundled"]
 test-helpers = ["dep:chrono", "dep:integration_test_credentials", "dep:rstest"]
 
 [dependencies]
+log = { workspace = true }
 rusqlite = { workspace = true, features = ["hooks"] }
 serde = { workspace = true }
 serde_json = { workspace = true }

wp_mobile_cache/Cargo.toml

@@ -0,0 +1,45 @@
+-- Table 1: List header/pagination info
+CREATE TABLE `list_metadata` (
+  `rowid` INTEGER PRIMARY KEY AUTOINCREMENT,
+  `db_site_id` INTEGER NOT NULL,
+  `key` TEXT NOT NULL,              -- e.g., "edit:posts:publish"
+  `total_pages` INTEGER,
+  `total_items` INTEGER,
+  `current_page` INTEGER NOT NULL DEFAULT 0,
+  `per_page` INTEGER NOT NULL DEFAULT 20,
+  `last_first_page_fetched_at` TEXT,
+  `last_fetched_at` TEXT,
+  `version` INTEGER NOT NULL DEFAULT 0,
+
+  FOREIGN KEY (db_site_id) REFERENCES db_sites(id) ON DELETE CASCADE
+) STRICT;
+
+CREATE UNIQUE INDEX idx_list_metadata_unique_key ON list_metadata(db_site_id, key);
+
+-- Table 2: List items (rowid = insertion order = display order)
+CREATE TABLE `list_metadata_items` (
+  `rowid` INTEGER PRIMARY KEY AUTOINCREMENT,
+  `list_metadata_id` INTEGER NOT NULL,
+  `entity_id` INTEGER NOT NULL,     -- post/comment/etc ID
+  `modified_gmt` TEXT,              -- nullable for entities without it
+  `parent` INTEGER,                 -- parent post ID (for hierarchical post types like pages)
+  `menu_order` INTEGER,             -- menu order (for hierarchical post types)
+
+  FOREIGN KEY (list_metadata_id) REFERENCES list_metadata(rowid) ON DELETE CASCADE
+) STRICT;
+
+CREATE INDEX idx_list_metadata_items_list ON list_metadata_items(list_metadata_id);
+CREATE INDEX idx_list_metadata_items_entity ON list_metadata_items(list_metadata_id, entity_id);
+
+-- Table 3: Sync state (FK to list_metadata, not duplicating key)
+CREATE TABLE `list_metadata_state` (
+  `rowid` INTEGER PRIMARY KEY AUTOINCREMENT,
+  `list_metadata_id` INTEGER NOT NULL,
+  `state` INTEGER NOT NULL DEFAULT 0,  -- 0=idle, 1=fetching_first_page, 2=fetching_next_page, 3=error
+  `error_message` TEXT,
+  `updated_at` TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
+
+  FOREIGN KEY (list_metadata_id) REFERENCES list_metadata(rowid) ON DELETE CASCADE
+) STRICT;
+
+CREATE UNIQUE INDEX idx_list_metadata_state_unique ON list_metadata_state(list_metadata_id);

wp_mobile_cache/migrations/0007-create-list-metadata-tables.sql

@@ -1,3 +1,4 @@
+pub mod db_list_metadata;
 pub mod db_site;
 pub mod db_term_relationship;
 pub mod helpers;

wp_mobile_cache/src/db_types.rs

@@ -0,0 +1,161 @@
+use crate::{
+    SqliteDbError,
+    db_types::row_ext::{ColumnIndex, RowExt},
+    list_metadata::{
+        DbListHeaderWithState, DbListMetadata, DbListMetadataItem, DbListMetadataState, ListState,
+    },
+};
+use rusqlite::Row;
+
+/// Column indexes for list_metadata table.
+/// These must match the order of columns in the CREATE TABLE statement.
+#[repr(usize)]
+#[derive(Debug, Clone, Copy)]
+pub enum ListMetadataColumn {
+    Rowid = 0,
+    DbSiteId = 1,
+    Key = 2,
+    TotalPages = 3,
+    TotalItems = 4,
+    CurrentPage = 5,
+    PerPage = 6,
+    LastFirstPageFetchedAt = 7,
+    LastFetchedAt = 8,
+    Version = 9,
+}
+
+impl ColumnIndex for ListMetadataColumn {
+    fn as_index(&self) -> usize {
+        *self as usize
+    }
+}
+
+impl DbListMetadata {
+    /// Construct a list metadata entity from a database row.
+    pub fn from_row(row: &Row) -> Result<Self, SqliteDbError> {
+        use ListMetadataColumn as Col;
+
+        Ok(Self {
+            row_id: row.get_column(Col::Rowid)?,
+            db_site_id: row.get_column(Col::DbSiteId)?,
+            key: row.get_column(Col::Key)?,
+            total_pages: row.get_column(Col::TotalPages)?,
+            total_items: row.get_column(Col::TotalItems)?,
+            current_page: row.get_column(Col::CurrentPage)?,
+            per_page: row.get_column(Col::PerPage)?,
+            last_first_page_fetched_at: row.get_column(Col::LastFirstPageFetchedAt)?,
+            last_fetched_at: row.get_column(Col::LastFetchedAt)?,
+            version: row.get_column(Col::Version)?,
+        })
+    }
+}
+
+/// Column indexes for list_metadata_items table.
+/// These must match the order of columns in the CREATE TABLE statement.
+#[repr(usize)]
+#[derive(Debug, Clone, Copy)]
+pub enum ListMetadataItemColumn {
+    Rowid = 0,
+    ListMetadataId = 1,
+    EntityId = 2,
+    ModifiedGmt = 3,
+    Parent = 4,
+    MenuOrder = 5,
+}
+
+impl ColumnIndex for ListMetadataItemColumn {
+    fn as_index(&self) -> usize {
+        *self as usize
+    }
+}
+
+impl DbListMetadataItem {
+    /// Construct a list metadata item from a database row.
+    pub fn from_row(row: &Row) -> Result<Self, SqliteDbError> {
+        use ListMetadataItemColumn as Col;
+
+        Ok(Self {
+            row_id: row.get_column(Col::Rowid)?,
+            list_metadata_id: row.get_column(Col::ListMetadataId)?,
+            entity_id: row.get_column(Col::EntityId)?,
+            modified_gmt: row.get_column(Col::ModifiedGmt)?,
+            parent: row.get_column(Col::Parent)?,
+            menu_order: row.get_column(Col::MenuOrder)?,
+        })
+    }
+}
+
+/// Column indexes for list_metadata_state table.
+/// These must match the order of columns in the CREATE TABLE statement.
+#[repr(usize)]
+#[derive(Debug, Clone, Copy)]
+pub enum ListMetadataStateColumn {
+    Rowid = 0,
+    ListMetadataId = 1,
+    State = 2,
+    ErrorMessage = 3,
+    UpdatedAt = 4,
+}
+
+impl ColumnIndex for ListMetadataStateColumn {
+    fn as_index(&self) -> usize {
+        *self as usize
+    }
+}
+
+impl DbListMetadataState {
+    /// Construct a list metadata state from a database row.
+    pub fn from_row(row: &Row) -> Result<Self, SqliteDbError> {
+        use ListMetadataStateColumn as Col;
+
+        Ok(Self {
+            row_id: row.get_column(Col::Rowid)?,
+            list_metadata_id: row.get_column(Col::ListMetadataId)?,
+            state: row.get_column(Col::State)?,
+            error_message: row.get_column(Col::ErrorMessage)?,
+            updated_at: row.get_column(Col::UpdatedAt)?,
+        })
+    }
+}
+
+/// Column indexes for the header + state JOIN query.
+///
+/// Query: SELECT m.total_pages, m.total_items, m.current_page, m.per_page, s.state, s.error_message
+///        FROM list_metadata m LEFT JOIN list_metadata_state s ON ...
+#[repr(usize)]
+#[derive(Debug, Clone, Copy)]
+pub enum ListHeaderWithStateColumn {
+    TotalPages = 0,
+    TotalItems = 1,
+    CurrentPage = 2,
+    PerPage = 3,
+    State = 4,
+    ErrorMessage = 5,
+}
+
+impl ColumnIndex for ListHeaderWithStateColumn {
+    fn as_index(&self) -> usize {
+        *self as usize
+    }
+}
+
+impl DbListHeaderWithState {
+    /// Construct from a JOIN query row.
+    ///
+    /// Expects columns in order: total_pages, total_items, current_page, per_page, state, error_message
+    pub fn from_row(row: &Row) -> Result<Self, SqliteDbError> {
+        use ListHeaderWithStateColumn as Col;
+
+        // state is nullable due to LEFT JOIN - default to Idle
+        let state: Option<ListState> = row.get_column(Col::State)?;
+
+        Ok(Self {
+            state: state.unwrap_or(ListState::Idle),
+            error_message: row.get_column(Col::ErrorMessage)?,
+            current_page: row.get_column(Col::CurrentPage)?,
+            total_pages: row.get_column(Col::TotalPages)?,
+            total_items: row.get_column(Col::TotalItems)?,
+            per_page: row.get_column(Col::PerPage)?,
+        })
+    }
+}

wp_mobile_cache/src/db_types/db_list_metadata.rs

@@ -3,9 +3,12 @@ use rusqlite::types::{FromSql, FromSqlResult, ToSql, ToSqlOutput};
 use rusqlite::{Connection, Result as SqliteResult, params};
 use std::sync::Mutex;
 
+use crate::repository::list_metadata::ListMetadataRepository;
+
 pub mod context;
 pub mod db_types;
 pub mod entity;
+pub mod list_metadata;
 pub mod repository;
 pub mod term_relationships;
 
@@ -15,13 +18,18 @@ pub mod test_fixtures;
 #[derive(Debug, Clone, PartialEq, Eq, thiserror::Error, uniffi::Error)]
 pub enum SqliteDbError {
     SqliteError(String),
+    ConstraintViolation(String),
     TableNameMismatch { expected: DbTable, actual: DbTable },
+    PerPageMismatch { expected: i64, actual: i64 },
 }
 
 impl std::fmt::Display for SqliteDbError {
     fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         match self {
             SqliteDbError::SqliteError(message) => write!(f, "SqliteDbError: message={}", message),
+            SqliteDbError::ConstraintViolation(message) => {
+                write!(f, "Constraint violation: {}", message)
+            }
             SqliteDbError::TableNameMismatch { expected, actual } => {
                 write!(
                     f,
@@ -30,12 +38,24 @@ impl std::fmt::Display for SqliteDbError {
                     actual.table_name()
                 )
             }
+            SqliteDbError::PerPageMismatch { expected, actual } => {
+                write!(
+                    f,
+                    "per_page mismatch: expected {}, but list has {}",
+                    expected, actual
+                )
+            }
         }
     }
 }
 
 impl From<rusqlite::Error> for SqliteDbError {
     fn from(err: rusqlite::Error) -> Self {
+        if let rusqlite::Error::SqliteFailure(sqlite_err, _) = &err
+            && sqlite_err.code == rusqlite::ErrorCode::ConstraintViolation
+        {
+            return SqliteDbError::ConstraintViolation(err.to_string());
+        }
         SqliteDbError::SqliteError(err.to_string())
     }
 }
@@ -64,6 +84,12 @@ pub enum DbTable {
     DbSites,
     /// Term relationships (post-category, post-tag associations)
     TermRelationships,
+    /// List metadata headers (pagination, version)
+    ListMetadata,
+    /// List metadata items (entity IDs with ordering)
+    ListMetadataItems,
+    /// List metadata sync state (idle, fetching, error)
+    ListMetadataState,
 }
 
 impl DbTable {
@@ -79,6 +105,9 @@ impl DbTable {
             DbTable::SelfHostedSites => "self_hosted_sites",
             DbTable::DbSites => "db_sites",
             DbTable::TermRelationships => "term_relationships",
+            DbTable::ListMetadata => "list_metadata",
+            DbTable::ListMetadataItems => "list_metadata_items",
+            DbTable::ListMetadataState => "list_metadata_state",
         }
     }
 }
@@ -107,6 +136,9 @@ impl TryFrom<&str> for DbTable {
             "self_hosted_sites" => Ok(DbTable::SelfHostedSites),
             "db_sites" => Ok(DbTable::DbSites),
             "term_relationships" => Ok(DbTable::TermRelationships),
+            "list_metadata" => Ok(DbTable::ListMetadata),
+            "list_metadata_items" => Ok(DbTable::ListMetadataItems),
+            "list_metadata_state" => Ok(DbTable::ListMetadataState),
             _ => Err(DbTableError::UnknownTable(table_name.to_string())),
         }
     }
@@ -249,7 +281,17 @@ impl WpApiCache {
     pub fn perform_migrations(&self) -> Result<i64, SqliteDbError> {
         self.execute(|connection| {
             let mut mgr = MigrationManager::new(connection)?;
-            mgr.perform_migrations().map_err(SqliteDbError::from)
+            let version = mgr.perform_migrations().map_err(SqliteDbError::from)?;
+
+            // Reset stale fetching states after migrations complete.
+            // Errors are logged but not propagated: this is a best-effort cleanup,
+            // and failure doesn't affect core functionality (worst case: UI shows
+            // stale loading state).
+            if let Err(e) = ListMetadataRepository::reset_stale_fetching_states(connection) {
+                log::warn!("Failed to reset stale fetching states: {}", e);
+            }
+
+            Ok(version)
         })
     }
 
@@ -271,7 +313,7 @@ impl WpApiCache {
                             // Ignore SQLite system tables (sqlite_sequence, sqlite_master, etc.)
                             // and migration tracking table (_migrations)
                             if !table_name.starts_with("sqlite_") && table_name != "_migrations" {
-                                eprintln!("Warning: Unknown table in update hook: {}", table_name);
+                                log::warn!("Unknown table in update hook: {}", table_name);
                             }
                         }
                     }
@@ -350,13 +392,14 @@ impl From<Connection> for WpApiCache {
     }
 }
 
-static MIGRATION_QUERIES: [&str; 6] = [
+static MIGRATION_QUERIES: [&str; 7] = [
     include_str!("../migrations/0001-create-sites-table.sql"),
     include_str!("../migrations/0002-create-posts-table.sql"),
     include_str!("../migrations/0003-create-term-relationships.sql"),
     include_str!("../migrations/0004-create-posts-view-context-table.sql"),
     include_str!("../migrations/0005-create-posts-embed-context-table.sql"),
     include_str!("../migrations/0006-create-self-hosted-sites-table.sql"),
+    include_str!("../migrations/0007-create-list-metadata-tables.sql"),
 ];
 
 pub struct MigrationManager<'a> {