DEVBOOK

Caching

17 min read

Edit this pageReport / suggest

Caching stores a copy of data closer to where it is consumed — in process memory, in a shared out-of-process store like Redis, or both — so that repeated reads skip the slower origin. The mechanism is simple: check the cache first; on a miss, fetch from the source, store the result, and return it. On a hit, return the stored copy without touching the source at all.

Some systems deliberately layer an in-process cache (L1) over a shared out-of-process cache (L2). L1 avoids serialization and a network hop but is duplicated per process and disappears with that process. L2 can be shared across instances and survive an application restart, but every lookup crosses a client, network, and backend boundary. Whether L2 survives a cache-node failure depends on the selected product’s persistence and replication contract. Use both tiers only when measurements show that the extra invalidation and capacity boundaries are worth the saved origin work.

The cache read path is simple; its correctness is not. Invalidation is a common risk, but incomplete keys, serialization drift, and security boundaries can be worse: omitting the tenant from a key can leak data across accounts. A complete design therefore covers the request pattern, freshness contract, outage behavior, capacity policy, and the selected product’s loss boundary together.

flowchart TD
  A[Request] --> B{Cache hit}
  B -->|Yes| C[Return cached]
  B -->|No| D[Fetch from source]
  D --> E[Store in cache]
  E --> F[Return]

Measure the Actual Path

CPU cache, process memory, storage I/O, and an application request are different measurement layers. A CPU L1-cache latency is not the latency of an IMemoryCache lookup, and a network round trip does not include Redis command execution, queueing, serialization, TLS, retries, or client-pool waits. Hardware generation, topology, payload size, and contention also change the ratios, so a fixed ladder is not a design contract.

Measure end-to-end hit and miss latency for the deployed path, plus origin load and hit ratio by key class. A cache is justified when avoided origin work improves a named latency or capacity target after accounting for miss cost, invalidation, and failure behavior. An L1 tier is justified separately when its measured gain exceeds the cost of per-process duplication and another freshness boundary.

Cache, retained log, or index?

“Faster copy” is the useful boundary. A cache entry is derivable or replaceable from an authority, so losing it should cost latency and origin load—not correctness or permanent data. Nearby systems may accelerate reads without being caches:

SystemWhat it ownsSafe response to total loss
Browser, CDN, database buffer pool, or materialized-result cacheA reusable copy or precomputed resultRe-fetch or recompute from the authority
Kafka topicRetained source records under a delivery and retention contractRestore from replicated log or backup; silently treating loss as a cache miss loses events
Search indexA derived query structure with mappings, analyzers, refresh, and query semanticsRebuild from the authority; reads are incomplete or unavailable until the rebuild catches up
Session or rate-limit storeTime-bounded operational stateApply an explicit fail-open or fail-closed policy; a “miss” can change security or user behavior

Before adopting a cache product, write down the authoritative source and the rebuild path. If neither exists, the component is acting as a system of record regardless of its name.

Cache Patterns

The five common strategies differ in who owns an origin request and what an acknowledged write means:

StrategyRead hit / missWrite pathFailure and retry boundaryFreshness
Cache-asideApp returns the hit; on miss it loads the origin and populates the cacheApp writes the origin, then invalidates or refreshes the keyA cache failure can bypass to the origin; coalesce repeated misses. Retry writes only when the origin operation is idempotentTTL plus explicit invalidation bounds staleness
Read-throughCache returns the hit; on miss the cache’s loader fetches and stores the origin valueUsually paired with a separate write strategyLoader failures reach the caller; bound retries and coalesce misses so one outage does not multiply origin loadLoader policy, TTL, and invalidation decide freshness
Write-throughReads use the cache; misses are loaded from the originCache synchronously writes the origin before acknowledgingOrigin failure fails the write. Retrying needs an idempotent operation or idempotency keyAcknowledged writes are current in the cache; out-of-band origin writes still need invalidation
Write-behindReads use the cache; misses are loaded from the originCache acknowledges, then queues an asynchronous origin writeCache or queue loss can lose acknowledged data; retries can duplicate a non-idempotent writeCache is freshest while the origin intentionally lags
Write-aroundExisting hits are served; a miss loads and populates as in cache-asideApp writes the origin and bypasses the cacheOrigin failure leaves the cache unchanged; retry under the origin’s idempotency contractInvalidate an old cached value on write or let its TTL expire

Write-around fits write-heavy data that is rarely read back: it avoids filling the cache with entries that may never be read, at the cost of making the first later read a miss.

Cache-aside with IDistributedCache:

public static async Task<string> GetUserName(
    string userId,
    IDistributedCache cache,
    Func<string, Task<string>> loadFromDb,
    CancellationToken ct)
{
    var key = $"user-name:{userId}";
    var cached = await cache.GetStringAsync(key, ct);
    if (cached is not null)
        return cached;
 
    var value = await loadFromDb(userId);
    await cache.SetStringAsync(
        key,
        value,
        new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(5) },
        ct);
    return value;
}

The same operation with HybridCache (.NET 9+) — stampede protection and L1/L2 layering are built in:

public class UserService(HybridCache cache)
{
    public async Task<string> GetUserNameAsync(string userId, CancellationToken ct)
    {
        return await cache.GetOrCreateAsync(
            $"user-name:{userId}",
            async cancel => await LoadFromDbAsync(userId, cancel),
            token: ct);
    }
}

Operating contract

Before selecting a client library, decide whether the workload has reuse, how stale each data class may be, who owns a miss, what an acknowledged write means, what happens at capacity, and how each request class degrades when the cache is unavailable. Measure hit ratio by route and key class rather than relying on one fleet-wide percentage: a 95% aggregate hit ratio can still hide an endpoint whose misses dominate database load.

QuestionConcrete decisionSignal that the decision is wrong
Does the workload have reuse?Measure request-key frequency and working-set sizeHit ratio remains low after warm-up
How stale may a value be?Assign a freshness budget per data type and derive TTL or invalidation latencyStale-read incidents or constant revalidation
Who owns misses?Choose an application loader, read-through loader, or background publisher; coalesce concurrent loadsOrigin requests per miss rise above one for a hot key
What does an acknowledged write mean?Name whether the origin, cache, queue, or replicas have accepted itDuplicate effects or acknowledged data loss on failure
What happens at capacity?Set a memory limit, admission rule, and eviction policy separately from TTLEvictions spike, hit ratio collapses, or writes fail under noeviction
What happens when the cache is down?Choose bypass, stale serve, partial degradation, or fail closed per data classA cache outage becomes an uncontrolled origin outage

Track hit ratio by route and key class, miss latency, origin requests caused by misses, eviction and expiration rates, invalidation lag, timeouts, and stale-read or version-mismatch counts.

Invalidation Strategies

Invalidation strategy is a correctness decision, not an optimization detail. Start by writing down your staleness contract, then pick the simplest strategy that meets it.

  • Explicit delete on write — on successful write, delete key or write the new value. If deletes can be lost, you still need a TTL as a safety net. Best when all writes go through one path that can delete or update cache.
  • TTL only — choose TTL from a staleness budget, not from guesswork. Add jitter and stampede protection for hot keys. Best when stale reads are acceptable and updates are infrequent or hard to observe.
  • Event-driven — publish invalidation events on writes, consume them in all app instances. Typical transports: message broker pub/sub, database change data capture, outbox pattern. If you cannot guarantee delivery, treat events as best-effort and keep TTL. Best when correctness matters and you can reliably emit change events.
  • Versioned keys — key includes a version, for example user-name:{userId}:v{version}. Version comes from row version, updated_at, etag, or a separate version store. Old keys naturally age out by TTL, no delete required. Best when deletes are expensive or unreliable, and you can carry a version token.

Decision rule of thumb:

flowchart TD
  A[Need cached reads] --> B{Max staleness is small}
  B -->|Yes| C{Can emit change events}
  B -->|No| D[TTL only]
  C -->|Yes| E[Event-driven plus TTL]
  C -->|No| F[Versioned keys plus TTL]
  D --> G{Hot keys exist}
  G -->|Yes| H[Add jitter and coalescing]
  G -->|No| I[Simple cache-aside]

Correctness and Staleness

Treat cached data as a replica with its own consistency model.

  • Staleness budget — the maximum age or divergence your product can tolerate, per data type. Example: prices might need seconds, user avatars can tolerate hours.
  • Eventual vs strong consistency — TTL-only and best-effort invalidation are eventual consistency. Strong consistency usually means bypassing cache or coupling cache and source writes in the same correctness boundary.
  • Read-your-writes — for user-facing writes, ensure the writer reads fresh data immediately after writing. Common patterns: write-through cache, delete on write, versioned key using row version, per-request bypass for the writer.
  • Stale-while-revalidate — serve slightly stale data fast while refreshing in the background. Trades bounded staleness for predictable latency and load. The pattern uses two TTLs: a soft TTL (freshness window) and a hard TTL (safety expiration). On a soft miss, the stale value is returned immediately while a background task refreshes the cache. On a hard miss, the caller blocks on a fresh fetch.

Stale-while-revalidate sketch inside a generic cache service — dual TTL with one in-process refresh owner per key:

private sealed record RefreshResult(T? Value, Exception? Error);
 
private readonly ConcurrentDictionary<string, Lazy<Task<RefreshResult>>> refreshes = new();
 
public async Task<T> GetAsync(string key, CancellationToken ct)
{
    var json = await cache.GetStringAsync(key, ct);
    var envelope = json is null ? null : JsonSerializer.Deserialize<Envelope<T>>(json);
 
    if (envelope is not null && DateTimeOffset.UtcNow <= envelope.FreshUntilUtc)
        return envelope.Value;
 
    var refresh = refreshes.GetOrAdd(
        key,
        cacheKey => new Lazy<Task<RefreshResult>>(
            () => RefreshAndReleaseAsync(cacheKey),
            LazyThreadSafetyMode.ExecutionAndPublication));
 
    if (envelope is not null)
    {
        _ = refresh.Value;
        return envelope.Value;
    }
 
    var result = await refresh.Value.WaitAsync(ct);
    if (result.Error is not null)
        throw new InvalidOperationException("Cache refresh failed.", result.Error);
 
    return result.Value!;
}
 
private async Task<RefreshResult> RefreshAndReleaseAsync(string key)
{
    try
    {
        var value = await LoadFromSourceAsync(key, CancellationToken.None);
        await WriteCacheAsync(key, value, softTtl, hardTtl, CancellationToken.None);
        return new RefreshResult(value, null);
    }
    catch (Exception ex)
    {
        logger.LogWarning(ex, "Cache refresh failed for {CacheKey}", key);
        return new RefreshResult(default, ex);
    }
    finally
    {
        refreshes.TryRemove(key, out _);
    }
}

refreshes is a service field, not request-local state. GetOrAdd may construct unused Lazy wrappers during a race, but only the stored wrapper’s Value starts a refresh. Soft-expired callers return stale data while sharing that task; hard misses await the same owner. The refresh logs failures and returns them as data, so the stale path cannot leave a faulted task unobserved. Request cancellation stops only that caller’s wait, not the shared refresh.

Notes:

  • Soft TTL is a latency contract. Hard TTL is a safety contract.
  • This dictionary and HybridCache coalesce only within one process. IDistributedCache does not provide atomic singleflight across instances; fleet-wide coordination needs a backend-aware lease or another distributed ownership protocol.

Stampede and failure modes

A stampede starts when many callers miss the same expensive key and independently load the origin. Coalesce refreshes, jitter expirations, and decide per data class whether an outage may serve stale, bypass under a rate limit, or must fail closed.

FailureRequest traceSafe boundary
Synchronized expiryMany keys expire together, producing a fleet-wide origin burstTTL jitter, staged warm-up, coalescing, and origin rate limits
Hot-key expiryOne popular key expires and every caller recomputes itOne refresh owner, soft/hard TTL, proactive refresh, and bounded stale serve
PenetrationRandom absent keys repeatedly miss and reach the originKey-space validation, short negative TTL, membership filter when applicable, and per-principal limits
Cache outageCache timeouts make every request bypassShort cache timeouts, circuit breaker, rate-limited bypass, stale serve for eligible data, and fail closed for authorization or quota state

Retries remain bounded and apply only to idempotent operations. An unbounded origin retry after a cache timeout multiplies load exactly when the dependency is weakest.

Redis and Memcached

Redis, Memcached, and EVCache are not interchangeable merely because they can serve values from memory. Decide whether values are disposable, which acknowledgement and persistence settings define loss, and how capacity eviction affects the origin.

DimensionMemcachedRedis
Core modelEphemeral opaque values distributed in memoryTyped in-memory structures with atomic commands
Capacity behaviorSlab classes and segmented LRU reclaim expired or unexpired itemsConfigurable maxmemory-policy; noeviction rejects memory-growing writes at the limit
Persistence and failoverLoss is a miss and the application repopulatesOptional RDB/AOF plus asynchronous replication; the loss window depends on configuration
Coordination featuresCAS for compare-and-set of a cached valueAtomic commands, scripts, streams, Pub/Sub, replication, Sentinel, and Cluster

Use Memcached for a plain disposable object cache. Use Redis when server-side structures or atomic operations justify its persistence, replication, and cluster choices. Redis command atomicity is not a relational transaction across arbitrary operations.

Netflix EVCache: four data contracts

Netflix’s EVCache illustrates four distinct contracts that can share an in-memory implementation:

RoleAuthority and rebuild pathLoss and freshness contract
Lookaside cacheDatabase or backend service remains authoritativeEviction is expected; TTL and invalidation bound staleness
Transient session stateThe cache may hold the only short-lived coordination valueLoss changes session behavior, so clients need an explicit recovery path
Precomputed primary read storeAn offline job publishes a generated data versionRetain or regenerate the last good generation before replacement
Distribution planeA publisher derives UI strings or translations from an upstream authorityReaders need versioned publication, rollback, and partial-generation handling

Redis as a cache or system of record

For authoritative Redis data, eviction must not discard keys and the acknowledgement boundary must name its loss window. RDB can lose writes since the last snapshot; AOF durability depends on appendfsync; replication is asynchronous by default; WAIT narrows but does not eliminate failover risk. Pub/Sub has no replay, while Streams retains entries and consumer-group state under the same persistence and replication configuration. Test process loss, host loss, and replica promotion against the exact deployment rather than inferring durability from an enabled setting.

Why Redis is fast—and when it stalls

Redis is fast because the working set stays in memory, clients are multiplexed, and most commands execute through a mostly serialized path. That same path makes large-key traversal, O(N) commands, Lua or module work, persistence fork pressure, AOF flushing, swapping, and network queues visible as tail latency. Measure the actual command and payload mix; watch slow commands, big keys, memory, persistence, and replication lag.

Tradeoffs

DimensionIMemoryCache (L1)IDistributedCache (L2)HybridCache (.NET 9+)
Request pathProcess-local lookup; no serialization or network hopClient pool, serialization, network, and backend executionProcess-local L1 with L2/origin fallback
CapacityBounded by app process memoryBounded by cache cluster (Redis, SQL)L1 bounded by process, L2 by cluster
SharingPer-instance, no sharing across podsShared across all instancesShared L2, per-instance L1
Stampede protectionManual (singleflight pattern)Manual (distributed lock)Built-in
SurvivabilityLost with the processSurvives app restarts; node-loss behavior is backend-specificL1 is process-local; L2 behavior is backend-specific
Tag-based invalidationNot supportedNot supportedBuilt-in
Best forSingle-instance apps, hot-path dataMulti-instance apps, shared stateDefault choice for new .NET 9+ apps

Decision rule: start with HybridCache for new .NET 9+ projects — it handles L1/L2 layering, stampede protection, and serialization out of the box. Fall back to IDistributedCache when you need explicit control over cache writes, or IMemoryCache for single-instance scenarios where distributed state is unnecessary.

Eviction under memory pressure

Expiration, admission, and capacity eviction answer different questions: whether an entry is too old, whether a candidate should enter, and which resident value leaves when memory is full.

Capacity policyFitsFails when
LRURecent access predicts reuseA sequential scan displaces the established hot set
LFUA stable popularity skew should survive burstsOld hot keys never age out
SLRUOne-hit entries should prove reuse before entering a protected segmentSegment sizes do not match the workload
FIFOInsertion order is a useful proxy and simplicity mattersOld frequently used entries are evicted
RandomMetadata must be minimal and reuse is roughly uniformPopularity is highly skewed

IMemoryCache is not bounded unless entries provide sizes and the cache has a SizeLimit. Redis uses maxmemory-policy; a pure cache commonly starts with an all-keys LRU or LFU policy, while noeviction rejects memory-growing writes. Choose from measured reuse distance, skew, object size, and miss cost, then watch eviction rate with hit ratio and origin load.

Other pitfalls

Unbounded high-cardinality keys, missing tenant or authorization dimensions, large serialized payloads, format drift, and cold deploys can erase the latency benefit or serve incorrect data. Bound the key space, version cached envelopes, measure serialization cost, and roll out gradually enough that origin load remains inside its capacity budget.

Cache penetration (missing-key floods)

Absent-key traffic follows request → cache miss → origin not found. Validate impossible keys, use a short negative TTL for repeated misses, consider a Bloom Filter when membership is known, and rate-limit by principal. Negative entries must be invalidated on creation; Bloom false positives still reach the origin; versioned keys still need TTLs so obsolete versions do not grow without bound.

Questions

References

Graph View

Backlinks