DEVBOOK

Horizontal Scaling

In progress

5 min read

EditReport

Horizontal scaling (scale-out) adds service instances and divides work among them. Software Architecture/Distributed Systems/Scalability Patterns/Vertical Scaling instead gives one instance more resources. Scale-out can increase capacity and reduce the effect of one instance failing, but only while another component does not become the bottleneck.

Stateless request handlers are the simplest case because any replica can accept the next request. Shared sessions and files move to an external store, and a load balancer routes only to ready instances. Stateful services can also scale horizontally. They need explicit partition ownership, replication, and a routing or coordination rule when ownership moves.

How Autoscaling Adds Capacity

An autoscaler observes a signal, computes a desired replica count, starts or stops instances, and waits for the new capacity to become usable. The signal must track the constrained resource. CPU is useful for CPU-bound request handlers. Queue age or backlog is often better for workers. More application replicas cannot repair a saturated database, connection limit, or external API quota.

This ASP.NET Core example externalizes session storage through the registered distributed cache. It covers incidental session state only. Business state still needs a partition owner and a replication plan.

// Program.cs
builder.Services.AddStackExchangeRedisCache(options =>
{
    options.Configuration = builder.Configuration["Redis:ConnectionString"];
    options.InstanceName = "myapp:";
});
 
builder.Services.AddSession(options =>
{
    options.IdleTimeout = TimeSpan.FromMinutes(20);
    options.Cookie.HttpOnly = true;
    options.Cookie.IsEssential = true;
});
 
// No in-memory session provider — all session data goes to Redis

This Kubernetes HPA targets average CPU utilization and limits how quickly capacity is removed:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300   # prevent oscillation
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60

The five-minute stabilization window does not pause every scale-in for five minutes. The controller considers recent recommendations and uses the highest value in the window, which smooths flapping metrics. New instances still need startup and readiness time before they contribute. Scale-out therefore reacts after load is observed unless spare capacity or an earlier signal covers that delay.

What Limits Scale-Out

  • Local state: in-memory sessions, local files, and per-instance locks break when the next request reaches another replica. Incidental state moves to an external store, while intrinsic state is routed by a stable owner.
  • Downstream pressure: each replica can add database connections and concurrent calls. Twenty replicas with a pool size of 100 expose up to 2,000 database connections. Pool sizes and total concurrency need bounds set against the dependency, not per-instance convenience.
  • Uneven work: sticky sessions and hot partition keys can leave one instance saturated while fleet averages look healthy.
  • Scale-in: an instance stops accepting new work, drains connections or leases, and allows its handlers enough shutdown time. Removing an owner before its work moves can lose or duplicate effects.

Software Architecture/Distributed Systems/Load Balancing explains how routing excludes unready instances and distributes work among the rest.

Horizontal scaling earns its complexity when measured capacity grows enough to meet the target. If doubling replicas produces little throughput improvement, the bottleneck is elsewhere. Scalability Patterns explains how to identify that bottleneck before choosing a scaling method.

Questions

References

Graph View

Backlinks