Cascading Timeout & Latency Injection

Mental Model

Cascading timeout propagation occurs when an upstream service blocks waiting for a slow downstream dependency. In distributed microservice architectures, this blocking behavior can deplete thread pools across multiple service boundaries, propagating a localized failure all the way back to the API gateway and clients.

The core mental model is a side-by-side comparison of two identical service chains: a Vulnerable Lane (with no circuit breaker protection) and a Protected Lane (with automatic circuit breaker protection). Injecting latency or errors into the deep downstream dependency shows how thread pool depletion spreads upstream, how retries amplify traffic under failure (retry storms), and how circuit breakers dynamically open, fail-fast, and safely probe for recovery.

Components and Why They Exist

Browser Clients

Represents the end-users or client applications generating requests. Dispatches requests at a configurable rate (RPS) and tracks metrics like success rate and p95 latency. In the demo, observe client traffic in both lanes and compare the real-time success rate and p95 latency.

Gateway

The single entry point for all client requests in each microservice lane. Routes incoming requests to Service A. It operates with a finite worker thread pool. In the demo, watch Gateway thread pool saturation (threads used) in the Vulnerable lane when Service A slows down, leading to dropped requests.

Service A

An intermediate upstream microservice. Handles traffic from the Gateway and makes downstream network calls to Service B. In the Resilient lane, it wraps these calls with a circuit breaker. In the demo, observe the circuit breaker state badge (CLOSED, OPEN, HALF-OPEN) and the worker threads usage in the Resilient lane.

Service B

A downstream dependency that interacts with the Database. Processes requests from Service A and executes queries against the database. In the demo, watch for injected latency spikes (e.g., 2500ms) or error rates on Service B.

Database

The persistent storage layer queried by Service B. Processes SQL queries or data fetches. In the demo, notice that monitor the database load (dbRps) under a retry storm in both lanes.

Edge Cases and Misconceptions

• Upstream Thread Pool Exhaustion: When a downstream service slows down, every upstream server calling it must hold its connection open. In thread-per-request architectures, this holds worker threads hostage. Once all threads are used, the server cannot accept any new traffic, propagating the outage upstream.
• Retry Storm Amplification: Retries are designed to handle transient network hiccups. However, if a downstream dependency suffers a persistent outage or overload, retries amplify the failure. If every client retries 3 times, traffic volume increases by up to 300%, turning a minor slowdown into a complete outage.
• Circuit Breaker Flapping: A circuit breaker that continually trips open, enters half-open, and immediately trips open again is “flapping.” This indicates that the downstream service has not recovered, or the probe traffic is too high. Production systems use exponential backoff and jitter for half-open probes to avoid overloading recovering nodes.
• Timeouts Alone Do Not Prevent Exhaustion: Setting a 1-second timeout does not protect thread pools under heavy traffic. If requests arrive faster than 1 per second per thread, the threads will still saturate and exhaust the pool. Circuit breakers are required to fail fast immediately.