Full cache outside of Zustand

[steaks]

Summary

Currently we only cache the most recently loaded value for a query (i.e. the value in the Zustand store). Users may want to access values they previously loaded quickly. We should explore caching “inactive” values for fast loads. To do this we will need to implement a proper cache.

As an example consider a dog store with dogDetails. When a user pulls up the details for a dog a second time we may want to have the value cached so it loads quickly. For the example below we could cache with the key currentDogId.

Implementing this feature may be a double-edged sword. On the positive side it can help developers make faster applications. On the negative side it can introduce a lot more complexity for the dev experience. With this feature some complications that come are 1) differentiate cache keys from invalidation events, 2) stale time vs cache time, 3) mark state vs evict from cache. If we implement this enhancement we should be very careful to not complicate most setups.

If we implement this we'll also need to consider the cache retrieval strategies. Some to consider are stale-while-revalidate, cache-first (hit the cache, refresh data if stale), network-first (refresh data first, fallback to cache if slow), network-only (ignore the cache), and cache-only (serve stale data, don't refresh data).

interface DogState {
  dogs: Query<DogState, number>;
  increasePopulation: Effect<DogState>;
  removeAllDogs: Effect<DogState>;
  currentDogId: number;
  dogDetails: Query<DogState, DogDetails>;
}

const fetchDogDetails = (get: () => DogState) => async () => {
  const currentDogId = get().currentDogId;
  const response = await fetch(`https://good.dog.com/dog?dogId=${currentDogId}`);
  return response.json();
};

const useDogStore = create<DogState>((set, get) => ({
  increasePopulation: effect(increasePopulation),
  removeAllDogs: effect(removeAllDogs),
  dogs: query(fetchDogs, s => [s.increasePopulation, s.removeAllDogs]),
  currentDogId: 0,
  dogDetails: query(fetchDogDetails(get), s => [s.currentDogId, s.removeAllDogs]),
}));

Acceptance Criteria

• Inactive values are cached
• Inactive values within cache time are accessible
• Inactive values outside of cache time are inaccessible
• Inactive values outside of cache time are removed from cache (garbage collection)
• Inactive values inside of cache time are removed if cache becomes too large (garbage collection)
• Getting started with Leo Query is still simple
• Leveraging inactive values caching is simple

Possible solutions

Separate dependencies into cache keys and invalidateOn events

Separate dependencies into keys and invalidateOn. When key(s) change then the data becomes stale but not necessarily removed from the cache. When values in the invalidateOn property change then the value becomes stale and is removed from the cache. The benefits to this solution are that it gives the developer more control over performance. The downsides are that it's more complicated.

const useDogStore = create<DogState>((set, get) => ({
  increasePopulation: effect(increasePopulation),
  removeAllDogs: effect(removeAllDogs),
  dogs: query(fetchDogs, s => [s.increasePopulation, s.removeAllDogs]),
  currentDogId: 0,
  dogDetails: query(fetchDogDetails(get), s => ({keys: [s.currentDogId], invalidateOn: [s.removeAllDogs]})),
}));

Use object notation as API

Rather than passing in a function, dependencies, and query options we can simply have users pass in a object. This allows us to make the API more explicit. Here's how it could look.

const useDogStore = create<DogState>((set, get) => ({
 increasePopulation: effect(increasePopulation),
 removeAllDogs: effect(removeAllDogs),
 dogs: query(fetchDogs, s => [s.increasePopulation, s.removeAllDogs]),
 currentDogId: 0,
 dogDetails: query({
   fn: fetchDogDetails(get),
   cacheKeys: s => [s.currentDogId],
   invalidateOn: s => [s.removeAllDogs]
 })
}));

[steaks]

Possible cache implementation:

import { Primitive } from "./types";

interface CacheEntry<T> {
  value: T;
  timestamp: number;
  staleTime: number | undefined;
  gcTime: number | undefined;
  gcTimeout: number | undefined;
  lastAccessTimestamp: number;
}

const MAX_CACHE_SIZE = 1000;

const cache = new Map<string, CacheEntry<unknown>>();

const evictLRU = () => {
  if (cache.size <= MAX_CACHE_SIZE) {
    return;
  }
  Array
    .from(cache.entries())
    .sort((a, b) => b[1].lastAccessTimestamp - a[1].lastAccessTimestamp)
    .slice(0, cache.size - MAX_CACHE_SIZE)
    .forEach(([key, value]) => {
      clearTimeout(value.gcTimeout);
      cache.delete(key);
    });
};

const key = (id: string, cacheKeys: Primitive[]): string => {
  return JSON.stringify([id, ...cacheKeys]);
};

const get = <T>(id: string, cacheKeys: Primitive[]): T | undefined => {
  const k = key(id, cacheKeys);
  const entry = cache.get(k) as CacheEntry<T> | undefined;
  if (!entry) {
    return undefined;
  }
  if (entry.gcTime && entry.timestamp < Date.now() - entry.gcTime) {
    clearTimeout(entry.gcTimeout);
    cache.delete(k);
    return undefined;
  }
  if (entry.staleTime && entry.timestamp < Date.now() - entry.staleTime) {
    return undefined;
  }
  entry.lastAccessTimestamp = Date.now();
  return entry.value;
};

const set = <T>(id: string, cacheKeys: Primitive[], value: T, staleTime: number | undefined, gcTime: number | undefined) => {
  const k = key(id, cacheKeys);
  const existing = cache.get(k) as CacheEntry<T> | undefined;
  if (existing) {
    clearTimeout(existing.gcTimeout);
  }
  let gcTimeout: number | undefined = undefined;
  if (gcTime) {
    gcTimeout = setTimeout(() => { cache.delete(k); }, gcTime) as unknown as number;
  }
  cache.set(k, {value, timestamp: Date.now(), staleTime, gcTime, gcTimeout, lastAccessTimestamp: Date.now()});
  if (cache.size >= MAX_CACHE_SIZE) {
    evictLRU();
  }
};

const remove = (id: string, cacheKeys?: Primitive[]) => {
  if (cacheKeys) {
    const k = key(id, cacheKeys);
    const value = cache.get(k);
    if (value) {
      clearTimeout(value.gcTimeout);
      cache.delete(k); 
    }
  } else {
    const k = key(id, []).slice(0, -1);
    cache.forEach((entry, cacheKey) => {
      if (cacheKey.startsWith(k)) {
        clearTimeout(entry.gcTimeout);
        cache.delete(cacheKey);
      }
    });
  }
};

const clear = () => {
  cache.forEach(entry => { clearTimeout(entry.gcTimeout); });
  cache.clear();
};

export default {
  key,
  get,
  set,
  remove,
  clear,
};