Performance considerations

Always measure performance in the context of your application. Measuring gives you a leg up over people who are too smart to measure.

Cache performance is determined by three key factors:

• Hit Rate
• Throughput
• Latency

As BenchmarkDotNet gains popularity, a lot of emphasis is placed on microbenchmark latency. Microbenchmarks can be a good predictor of performance if a cache is invoked in at very high frequency. However, in most cases hit rate is the critical metric to optimize. In a multithreaded application throughput is the next most important concern.

An analysis of hit rate, throughput and latency is provided here.

Hit Rate

Caches trade an increase in memory for a decrease in computation or latency. The larger the cache, the more items can be stored. When the cache is full, the cache replacement policy must decide which entry to discard when admitting a new entry. The two factors that determine hit rate are cache size and replacement policy.

Both LRU and LFU policies both provide excellent hit rate across different workloads. Choose a policy based on the characteristics of your workload and then tune cache size to get the best tradeoff between memory consumption and hit rate.

Throughput

Latency

Below are a few simple details that can affect cache lookup latency.

Key comparer

The type of the cache key, and its associated Equals and GetHashCode methods can influence lookup cache speed, and the distribution of the GetHashCode method will influence the number of collisions in the underlying hash table. You can specify an implementation of IEqualityComparer<K> at cache construction.

For example, if the cache key is a string, prefer ordinal to culture sensitive string comparison. The HashCode.Combine method used when autogenerating equality methods can be slower than handwriting the equivalent logic.

Feature usage

All cache features cost performance. This library has been designed such that disabling features eliminates cost. Therefore, try to use only the required cache features via the cache builder methods. Time-based expiry, atomic and scoped values incur a slight penalty for lookup latency. LRU metrics slightly reduce concurrent throughput.

Events incur an event args heap allocation when an event handler is registered.

IExpiryCalculator

IExpiryCalculator methods can be called at very high frequency. To get the lowest latency and highest throughput, avoid heap allocations and minimize computation within the GetExpireAfter* methods.

An efficient expire after write calculator is show below. Since expiry time is fixed it can be calculated once up front at initialization instead of per create/read/update call, avoiding floating point multiplication on the hot path.

public class ExpireAfterWrite : IExpiryCalculator<string, int>
{
    private readonly Duration timeToExpire = Duration.FromMinutes(5);

    public Duration GetExpireAfterCreate(string key, int value) => timeToExpire;
    public Duration GetExpireAfterRead(string key, int value, Duration current) => current;
    public Duration GetExpireAfterUpdate(string key, int value, Duration current) => timeToExpire;
}