Surviving a Redis Crash
Walk through a Redis crash from healthy reads to degraded fallback, DB protection, circuit breaking, L1 hot keys, replica failover, gradual warm-up, and cache-optional correctness.
Continue with the guided workspace
Use the paired workspace to connect this guide's mental model with live component state, visible signals, and the outcome each step or scenario is meant to prove.
Redis is a helper, not a lifeline. It makes reads fast, but it can disappear in an instant — a crash, a restart, a network blip, or an eviction wave. If the application treats Redis as required, that crash becomes a full outage. If it treats Redis as optional, the crash becomes a slowdown the system can absorb.
This guided hybrid visualization models a read-heavy service (product page views) that caches in Redis in front of a database. Step through the crash, degraded fallback, database protection, circuit breaking, L1 local cache, replica failover, and gradual warm-up. The metric that matters is not “is Redis up” — it is “does the database survive the loss of Redis, and does the user still get a response.”
Components and Why They Exist
Clients
Clients generate the read traffic — product page views — at a configurable rate. This is the load the whole system has to serve, and the request rate is the primary knob for stress-testing. When Redis crashes, these requests do not stop; they pile onto whatever path the resilience strategies leave open, which is why protecting the database matters more than protecting Redis.
App Server
The application server is where every resilience strategy lives. It checks the L1 local cache first (if enabled), then the L2 Redis cache (if the circuit breaker allows), then falls back to the database. The circuit breaker wraps every Redis call so repeated failures stop costing a full timeout each. The fallback decision — “Redis is gone, go to the database” — is made here, and so is the rate-limiting gate that decides how much of that fallback traffic actually reaches the database.
Redis Cache (L2)
Redis is the shared, in-memory cache that makes reads fast. It holds the warm keys that the application reuses across requests and across instances. In this simulator it is the component that crashes, and with replicas enabled it can fail over to a healthy node after a detection and promotion delay. On recovery, the gradual warm-up strategy refills the top queries first instead of letting the cold cache dump the full load onto the database.
Database
The database is the source of truth — correct but slow. Correctness must never depend on Redis; only speed should. When the cache is gone, the database receives the fallback load, and the rate limiter (a token bucket) caps how many queries per second actually reach it, rejecting the excess instead of overloading it. Watch the database load gauge turn red when the cache disappears with no protection, and stay green when rate limiting and L1 absorb the spike.
Edge Cases and Misconceptions
- Redis is not the source of truth. If application correctness depends on Redis, a crash corrupts it. Writes always go to the database first; Redis only mirrors for read speed.
- A circuit breaker does not fix Redis — it protects the application. It stops the app from paying a timeout on every call while Redis is down. The actual recovery still has to happen (restart, failover, or warm-up).
- Replicas do not make cache loss impossible. A replica promotion takes time (Sentinel detection + election + promotion, typically 30–60 seconds). During that window the app still needs the fallback, circuit breaker, and rate-limiting strategies to survive.
- L1 local cache trades memory for resilience. It keeps the hottest keys in each app process so they survive a Redis crash, but it costs memory per instance and can serve stale data across instances until the TTL expires.
- Warm-up is about the database, not Redis. A cold cache after recovery is fine for Redis; the danger is the thundering herd of misses that hits the database all at once. Gradual warm-up spreads that load over time.
- Rate limiting rejects some requests on purpose. During a cache outage, rejecting the excess is better than letting the database overload and fail every request. A 429 or stale response is a controlled degradation; a dead database is an uncontrolled outage.
Use the Surviving a Redis Crash guided demo to walk the system from normal reads through failure, protection, failover, and recovery.
Common Questions
Why should the database be the source of truth instead of Redis?
Redis is an in-memory cache that can disappear instantly on crash, restart, or eviction. If correctness depends on it, a crash corrupts the application. The safe contract is: the database holds the authoritative state and Redis only holds a copy for speed. Reads fall back to the database when Redis is gone, and writes always persist to the database first.
How does a circuit breaker protect the application when Redis fails?
Without a circuit breaker, every request tries Redis, waits for a timeout, then falls back to the database — so the application pays the full Redis timeout latency on every call while Redis is down. A circuit breaker opens after a configurable number of consecutive failures (typically 5), after which requests skip Redis entirely and go straight to the database, avoiding the timeout penalty. It periodically enters a half-open state to probe whether Redis has recovered.
What is the difference between a token-bucket rate limiter and simply capping database connections?
A token bucket enforces a sustained request rate (refill rate) while allowing short bursts up to the bucket capacity, which matches how databases handle transient spikes. A hard connection cap rejects everything above the limit immediately and can cause head-of-line blocking. The token bucket smooths the load so the database stays productive instead of oscillating between idle and overloaded.
How does Redis Sentinel or Cluster failover actually work?
Sentinel monitors the master with periodic pings. After a configurable down-after threshold (commonly 30 seconds), it marks the master subjectively down. Once a quorum of Sentinels agrees, the master is objectively down and a leader Sentinel is elected. The leader promotes the healthiest replica to master, reconfigures the other replicas to follow it, and notifies clients of the new address. Total failover time is typically 30–60 seconds.
Why warm up a cache gradually after recovery instead of letting it fill on demand?
A cold cache after recovery means every read misses and falls through to the database. If full traffic resumes instantly, the database receives the entire load at once — a thundering herd that can overload it and cause a second outage. Gradual warm-up preloads the top queries first (the small set of keys that drive most traffic) and background-fills the rest, so the database load ramps up smoothly instead of spiking.
What is a two-tier (L1 + L2) cache and when does it help?
L1 is a small in-process cache inside each application server (fastest, no network), and L2 is Redis (larger, shared across instances, slightly slower). L1 absorbs the hottest keys so they never hit the network. When Redis crashes, L1 keeps serving those hot keys from memory, shielding the database from the portion of traffic that the L1 hit rate covers. The tradeoff is memory per app instance and cache coherence across instances.
What happens if Redis crashes and no resilience strategy is enabled?
Every read tries Redis, times out, and either errors out or falls through to the database unprotected. The database receives the full request load at once, its connection pool saturates, latency spikes, and queries start timing out or being rejected. Success rate collapses and the service becomes effectively unavailable — the exact failure the seven strategies exist to prevent.