Load Balancer Crash Simulator

Load balancers are critical components in modern architecture, serving as the single point of entry for user traffic.

Mental Model

When a load balancer crashes, the entire system is at risk of an outage unless resilient architectures are in place. In a High Availability (HA) active-passive setup, the standby load balancer monitors the active primary node via heartbeat keepalives. If heartbeats stop, the standby node promotes itself to active, taking over the virtual IP. Clients query a DNS server to resolve the load balancer’s IP; if the IP changes during failover, clients may continue targeting the dead node due to stale DNS caches (ignoring DNS TTLs). The load balancer routes traffic to a pool of backend servers, while a health checker monitors backend responsiveness, dynamically adding or removing nodes to prevent routing traffic to broken application instances.

Components and Why They Exist

Clients

The source of the request traffic consuming the application. Sends HTTP requests, caches DNS results, and handles connection failures by retrying or displaying errors. In the demo, watch the client panel send request signals and show successful/failed connections.

Active Load Balancer (LB A / LB B)

The primary entry point responsible for distributing traffic across healthy backend servers. Receives incoming client requests and routes them to backends while checking active session limits. In the demo, watch it route traffic and show an “active” status.

Standby Load Balancer (LB A / LB B)

Provides failover redundancy to prevent a single point of failure. Monitors the active node via heartbeat pings and promotes itself to active if the primary node goes down. In the demo, watch its status transition to “active” during failover.

DNS Server / Cache

Translates user-facing hostnames into the IP addresses of the active load balancer. Serves IP records with a set Time-to-Live (TTL) and manages mapping updates when a load balancer fails. In the demo, watch the TTL counter countdown before clients query the new IP.

Backend Nodes / Servers

The application pool that processes the request logic and queries databases. Accepts connections from the load balancer, queues requests, and returns responses within capacity constraints. In the demo, watch request dots queue up, and nodes light up red when overloaded.

Health Checker

Ensures the load balancer has an accurate view of backend node viability. Sends periodic probes to backend endpoints, removing unresponsive nodes and slowly warm-booting recovered ones. In the demo, watch health check success/failure status dots.

Edge Cases and Misconceptions

• Misconception: High Availability (HA) failover is instantaneous and zero-loss: HA failover always introduces a small detection window (e.g., keepalive timeout) where the standby node determines the primary is dead. During this transition, client requests will fail.
• Misconception: Reducing DNS TTL to 0 solves client cache issues: Many clients (browsers, JVM runtimes, proxy servers) ignore DNS TTL settings or enforce a minimum cache duration (e.g., 60 seconds). Even if DNS is updated, traffic will still flow to the dead node’s IP.
• Edge Case: Fail-Open Routing vs. Cascading Failures: When health checks mark all backends unhealthy (e.g., due to a temporary network partition between the LB and backends), a “fail-open” policy routes traffic anyway to prevent a total outage. If the backends were actually overloaded, failing open might permanently crash them.