Kubernetes Autoscaling Under Load

Autoscaling in Kubernetes is a multi-layered coordination system that operates on two levels: Horizontal Pod Autoscaling (HPA) scales the number of pod replicas based on resource demand, while the Cluster Autoscaler (CA) scales the underlying node infrastructure when resource limits are reached.

Mental Model

When user traffic surges, pod CPU/memory utilization rises, but autoscaling is delayed by metric collection, reconciliation, scheduling, image startup, and readiness. The cAdvisor agent on each node collects usage metrics, which are polled periodically by the Metrics Server. The Horizontal Pod Autoscaler (HPA) queries the Metrics Server and calculates the required pod replica count based on a target utilization threshold. If existing nodes lack capacity to host the new replicas, the pods are scheduled but remain in a Pending state. The Cluster Autoscaler detects these pending pods and commands the cloud provider to provision new virtual machine nodes. Once nodes join the cluster, pending pods can schedule, pull their images, boot, pass readiness checks, and join the Ingress routing pool to handle the traffic.

Components and Why They Exist

Traffic Generator

Simulates external user requests hitting the cluster at different rates (RPS). Directs traffic spikes or baselines to Ingress, causing requests to be dropped (yielding 503 errors) if pods are overloaded. In the demo, watch the RPS volume and the percentage of successful vs dropped requests on the metrics charts.

Ingress Controller

The entryway for traffic into the cluster. Performs load-balancing, routing traffic round-robin to pods currently in the Ready state. In the demo, watch how traffic particles are distributed across active, healthy pods.

Horizontal Pod Autoscaler (HPA)

Automates pod scaling to handle varying load dynamically. Regularly polls the Metrics Server for CPU utilization and calculates the desired replica count based on target thresholds. In the demo, watch the HPA scale up the desired replica count after CPU usage exceeds the target threshold.

Cluster Autoscaler

Resolves node-level resource exhaustion by adjusting node counts. Detects pods stuck in a Pending state due to lack of CPU/Memory on existing nodes, and triggers the provisioning of new virtual machine nodes. In the demo, watch it add nodes when pods are pending, or terminate empty, unused nodes.

VM Nodes

The host machines (virtual or physical) that provide CPU and memory resources to run containers. Hosts pod replicas up to its maximum pod capacity. In the demo, observe nodes being provisioned (indicated by spinning loaders) and capacity usage.

Pods

The smallest deployable units in Kubernetes running the application containers. Progresses through lifecycles: Pending (scheduling), ImagePulling (downloading), Initializing (booting + readiness checks), Ready (serving), and Terminating. In the demo, watch pods change colors as they transition through lifecycle states.

Edge Cases and Misconceptions

• Misconception: Autoscaling is instantaneous: There is a cumulative lag consisting of metrics scrape delays (cAdvisor/Metrics Server), HPA calculation loops, cloud VM provisioning delays (which can take minutes), and pod boot times.
• Misconception: Setting CPU limits high prevents pod crashes: If a pod runs at 100% CPU capacity for too long under load, it can become unresponsive; CPU limits do not create capacity or guarantee prompt replacement. Runtimes might crash or experience cgroups OOM kills, leading to a cascading crash loop as traffic redirects to surviving pods.
• Edge Case: Scale-Down Cooldown (Thrashing): If the scale-down stabilization window is too short, brief fluctuations in traffic will cause the cluster to repeatedly add and delete pods (thrashing), wearing out resources and degrading performance.
• Edge Case: Karpenter vs Cluster Autoscaler: Unlike CA which waits for pods to be Pending and spins up nodes in minutes, advanced schedulers like Karpenter bypass the scheduler queues to provision right-sized nodes in seconds, reducing outage duration.

Open the Kubernetes Autoscaling simulator sandbox after the walkthrough when you want to experiment with load rates, probe settings, node capacity constraints, and cooldown policies.