Intro
Scalability is a system’s ability to keep serving requests as load grows by adding resources, without a proportional drop in reliability or latency. In interviews, this matters because most “works at 1k RPS” designs fail when asked “how does this reach 10x?” The goal is not just to survive spikes, but to scale in a way that is cost-efficient and operationally predictable. You start thinking about scalability as soon as you can identify request volume, traffic shape, data growth, and the first likely bottleneck.
Concrete interview lens: if checkout traffic grows from 1,000 RPS to 10,000 RPS, a good answer is not “add more servers” but “measure where saturation appears first, then apply the right pattern for that bottleneck.”
Core Patterns
| Pattern | Primary bottleneck addressed | How it helps | Tradeoff and interview caveat |
|---|---|---|---|
| Horizontal scaling (stateless services behind LB, see Load Balancing) | App CPU and request concurrency | Add service instances behind a load balancer to increase throughput and availability | Requires stateless handlers; sticky sessions can hurt elasticity |
| Database read replicas | Read-heavy relational load | Offload read queries from primary to replicas | Replica lag can break read-after-write expectations |
| Database sharding | Write throughput and dataset size | Partition data by key so writes and storage spread across shards | Rebalancing, cross-shard queries, and hotspot keys add major complexity |
| CQRS (see CQRS) | Read/write contention with different query needs | Separate write model from read model to optimize each independently | Eventual consistency and projection maintenance must be explicit |
| Caching (see Caching) | Repeated expensive reads | Serve hot data from in-memory cache to reduce DB/API pressure | Cache invalidation and staleness policy drive correctness risk |
| CDN | Static asset latency and origin egress | Move static content to edge locations close to users | Cache-control mistakes can serve stale or private content |
| Async processing and message queues (see Message Queues) | Synchronous dependency latency and burst traffic | Buffer work, decouple producers/consumers, smooth spikes | Requires idempotency, retry policy, and dead-letter handling |
| Connection pooling | Expensive connection setup and DB connection limits | Reuse open connections to reduce handshake cost and limit churn | Pool exhaustion often appears as latency spikes before hard failures |
| Event-Driven Architecture (see Event-Driven Architecture) | Tight coupling between services | Publish events so services scale and evolve independently | Ordering, duplication, and schema evolution must be designed upfront |
| Load shedding and rate limiting | Overload collapse during spikes | Reject or defer excess traffic early to protect critical paths | Requires clear priority rules and client retry behavior |
Pattern Walkthrough (Quick Explanations)
- Horizontal scale works best when each request can be handled by any instance, so session and cache state must be externalized.
- Read replicas are usually your first database scale step for read-heavy APIs, but you must call out replication lag.
- Sharding is usually late-stage because operational and data-model complexity is high.
- CQRS helps when read models and write invariants conflict; it is not mandatory for every CRUD app.
- Caching is often the highest ROI pattern when read repetition is high and staleness tolerance exists.
- CDN is often a high-ROI optimization for cacheable static assets and can reduce origin cost.
- Queues protect upstream systems from spikes and third-party slowness.
- Connection pooling is usually low-effort, high-impact hygiene before more dramatic architecture changes.
- Event-driven design scales team autonomy and workload isolation, but consistency guarantees must be explicit.
Scaling Decision Framework
Start with telemetry and saturation, not architecture fashion.
flowchart TD A[What is the bottleneck] --> B{Bottleneck type} B -->|CPU or compute| C[Scale stateless services horizontally] B -->|Read-heavy DB| D[Add read replicas and cache] B -->|Write-heavy DB| E[Use partitioning or sharding plus write-path optimization] B -->|Read and write model contention| H[Consider CQRS] B -->|External API latency limit| F[Queue plus rate limit plus retry policy] C --> G[Re-measure p95 latency and saturation] D --> G E --> G H --> G F --> G
.NET and Azure Context
- Azure App Service scale-out adds instances quickly for stateless ASP.NET Core apps; pair it with health checks and external session state.
- Kubernetes Horizontal Pod Autoscaler (HPA) scales pods by CPU, memory, or custom metrics; this is useful when queue depth is the real trigger.
- For PostgreSQL-heavy workloads, PgBouncer helps with connection pooling so app scale-out does not overwhelm DB connection limits.
- Redis is a common distributed cache choice in .NET systems (session state, response caching, hot lookups, rate limiting counters).
Minimal ASP.NET Core example with Redis distributed cache:
builder.Services.AddStackExchangeRedisCache(options =>
{
options.Configuration = builder.Configuration.GetConnectionString("Redis");
options.InstanceName = "scalability-patterns:";
});Simple production pattern:
- Keep APIs stateless.
- Put Redis between API and database for hot reads.
- Add read replicas before discussing sharding.
- Introduce queues where downstream latency is unpredictable.
Tradeoffs
| Choice | Better when | Worse when |
|---|---|---|
| Vertical vs horizontal app scaling | You need immediate capacity and low migration risk | Single-node ceiling and blast radius become dominant |
| Read replicas vs caching for reads | Queries are complex and freshness matters more than latency | Cache hit ratio is high and stale-tolerant reads dominate |
| Sharding vs larger primary DB | Write throughput and data size exceed one node limits | Team is small and cross-shard operations are frequent |
| Sync calls vs queue-based async | User needs immediate result and latency budget allows it | Dependency is slow or rate-limited and bursty traffic is expected |
Pitfalls
-
Scaling before finding the real bottleneck
What goes wrong: teams add app instances while p95 remains high.
Why: the bottleneck is often DB lock contention, external API latency, or connection saturation.
Mitigation: baseline telemetry first, then scale the saturated component. -
Premature sharding
What goes wrong: delivery speed drops and incident complexity rises.
Why: shard routing, cross-shard queries, and resharding become permanent operational overhead.
Mitigation: exhaust simpler options first (indexes, read replicas, caching, partitioning, queueing). -
Stateful services that cannot scale horizontally
What goes wrong: sticky sessions and per-node memory state cause uneven load and failover pain.
Why: user session or cache state is stored in-process.
Mitigation: externalize session to Redis and keep handlers stateless. -
Ignoring database bottlenecks while scaling app tier
What goes wrong: more app instances generate more DB pressure and failures happen faster.
Why: DB CPU, locks, or connection limits were already near saturation.
Mitigation: profile queries, add indexes, tune pools, use read replicas, then scale app tier.
Questions
When would you choose read replicas instead of CQRS for a scaling problem?
Expected answer:
- Choose read replicas when main pressure is read throughput on an existing relational model.
- Choose CQRS when read/write models diverge and read projections need different shape or storage.
- Mention consistency behavior: replicas have lag; CQRS read models are eventually consistent by design.
- Mention complexity: replicas are simpler operationally than full CQRS/event projection pipelines. Why this is strong: It balances architecture fit, consistency, and operational cost.