Distributed systems are hard because the network is unreliable and time is messy: partial failures, latency, and inconsistent views of the world. These notes focus on the core concepts that show up in production: consistency tradeoffs, messaging, coordination, and failure handling. Example: CAP is not a slogan; it explains why a partition forces you to pick between availability and strong consistency.
Quality attributes and measurable targets
The four labels are an orientation mnemonic, not independent boxes. Logging does not create reliability, and load balancing alone does not create availability. Turn each attribute into a workload, target, failure model, and measurement window:
| Attribute | Concrete target | Mechanism question |
|---|---|---|
| Scalability | Sustain 10,000 checkout RPS with p99 below 400 ms, errors below 0.1%, and cost below $0.002/request | Which resource saturates first, and how much capacity does one added unit buy? |
| Availability | 99.95% successful eligible checkout requests over 30 days | Which zone, region, dependency, or control-plane failures remain in the request path? |
| Reliability | No duplicate charges; committed ledger entries survive a zone loss; RPO 0 and RTO 15 minutes for payment writes | What correctness, durability, detection, and recovery evidence proves this? |
| Performance | p50/p95/p99 latency plus completed throughput and CPU, memory, I/O, and network per unit of work | Which stage consumes the latency and resource budget under the declared traffic mix? |
Targets conflict. Synchronous multi-region replication may improve durability and recovery while raising write latency and reducing partition-time availability. Caching may improve latency and origin efficiency while weakening freshness. State the business invariant that wins, then test the counter-cost.
Map symptoms to mechanisms, then test the tradeoff
Treat the visual as diagnostic prompts, not prescriptions:
| Symptom | Candidate mechanism | Condition that must hold | Counter-cost and proof |
|---|---|---|---|
| Read-heavy origin | Cache, read replica, or index | Reads repeat, tolerate a freshness budget, or scan avoidable data | Invalidation/lag/write amplification; compare hit ratio, query plan, and p99 |
| High write load | Batch, partition, append log, or queue | Writes can be grouped, distributed by a stable key, or acknowledged asynchronously | Hot partitions, delayed visibility, replay; load-test the real key distribution |
| Single point of failure | Replication plus Failure Detection and tested failover | Replicas do not share the same failure domain | Coordination, lag, split brain; inject the failure and measure RTO/RPO |
| High request latency | Trace the critical path, then cache, index, colocate, or defer work | The measured slow stage matches the chosen mechanism | Staleness, coupling, or async complexity; compare end-to-end percentiles |
| Large immutable payloads | Object storage plus a claim-check reference | Consumers can fetch by durable ID and checksum | Extra fetch, authorization, lifecycle; test loss and expired credentials |
| Poor diagnosis | Correlated traces, metrics, and logs | Context propagates through sync and async boundaries | Telemetry cost/cardinality; reconstruct one failed request from evidence |
Decision table for recurring system tradeoffs
Ask the same four questions for every choice: what is the workload, what fails, what consistency is required, and who operates the added machinery?
| Choice | Prefer first side when | Prefer second side when | Follow-up note |
|---|---|---|---|
| Scale up / scale out | Immediate headroom and one-node simplicity matter | Replica interchangeability and failure isolation justify distribution | Scalability Patterns |
| Synchronous / asynchronous | The caller needs the result inside its latency budget | Work can complete later and bursts need buffering | Message Queues |
| Strong / eventual consistency | A stale or conflicting result breaks an invariant | Temporary divergence has a repair rule and UX budget | Consistency Models and CAP theorem |
| Normalized / denormalized reads | Write integrity and flexible queries dominate | A known read shape needs predictable low latency | CQRS |
| Stateful / stateless service | Local state is intrinsic and partitioned deliberately | Any healthy replica should serve the next request | Load Balancing |
| Central orchestration / event choreography | Ordered workflow state and compensation must be explicit | Reactions are independent and ownership can stay distributed | Event-Driven Architecture |
Do not select SQL versus key-value, REST versus GraphQL, or batch versus stream from a label alone. Start from access patterns, ordering/freshness, replay window, failure recovery, and the team’s operating skills.
Distributed identifier requirements and failure modes
Design IDs from the contract:
- Collision domain: unique within a table, tenant, region, or the whole system?
- Ordering scope: no order, rough creation order, or monotonic order per generator?
- Clock behavior: what happens when time moves backward or two nodes share a timestamp?
- Bit budget: how many years, nodes, and IDs per time unit fit before exhaustion?
- Node allocation: static worker IDs, leased IDs, coordination service, or random space?
- Storage locality: random IDs can fragment B-tree indexes; time-ordered IDs improve locality but create hot ranges.
- Information leakage: timestamps, node IDs, and sequence rates can reveal creation time or volume.
- Recovery: generator restarts must not reuse sequence state or an expired node lease.
A Snowflake-like 63-bit payload might allocate 41 bits to milliseconds, 10 to a worker ID, and 12 to a per-millisecond sequence: about 69 years from a custom epoch, 1,024 workers, and 4,096 IDs/ms/worker. Those numbers are a contract, not defaults. Clock rollback must block, switch to a safe logical time, or fail closed; silently reusing a prior timestamp and sequence can collide.
Use Unique ID Generation for the full decision. UUIDv4 buys coordination-free randomness; UUIDv7 adds time ordering with a standardized 128-bit layout; database sequences buy compact monotonic IDs inside one coordination domain. None should be judged by whether it is “numeric” or whether creation order can always be reconstructed from the identifier.
Pattern Map
The visual mixes patterns from different layers. Use this linked map to keep the problem and cost visible:
| Group | Pattern note | Problem solved | Cost introduced |
|---|---|---|---|
| Communication | API Gateway | Stable policy and routing boundary for many APIs | Central latency, configuration, and blast radius |
| Communication | Webhooks | Push events across organizational boundaries | Signature, retry, replay, and reconciliation contract |
| Resilience | Circuit Breaker | Stop repeated calls to a failing dependency | Threshold tuning and open-state degraded behavior |
| Resilience | High Availability and Failure Detection | Maintain service through component failure | Redundancy, detection delay, failover ambiguity |
| Data modeling | CQRS and Event Sourcing | Separate read/write models or retain authoritative event history | Projection lag, schema evolution, replay operations |
| Coordination | Distributed Locks | Serialize one lease-scoped critical action | Fencing, expiry, quorum availability, deadlock risk |
| Messaging | Message Queues and Idempotency | Buffer, fan out, and safely retry work | Duplicates, ordering scope, backlog, DLQ ownership |
| Partitioning | Consistent Hashing | Limit key remapping as nodes change | Virtual-node tuning, hotspots, membership changes |
Patterns compose but are not substitutes. A leader election can choose one coordinator; a circuit breaker protects calls to it; pub/sub distributes its events; sharding partitions data. Each operates at a different boundary and needs its own failure test.
References
- Distributed computing (Wikipedia) — overview of distributed-computing definitions, history, and problem classes.
- Google SRE: Service Level Objectives — primary operational guidance for measurable indicators, targets, and evaluation windows.
- UUIDs (RFC 9562) — current IETF UUID layouts and requirements, including time-ordered UUID version 7.
- Azure Architecture Center design patterns — official catalog for distributed communication, resilience, data, and coordination patterns with consequences.
ByteByteGo provenance
- Fantastic four of system design — provenance for the four-attribute mnemonic, converted into measurable targets.
- Common system-design problems and solutions — provenance for the symptom map, qualified with conditions and counter-costs.
- System-design tradeoffs — editorial lead for the decision table; its false CAP/CA visual was rejected.
- Unique ID generator — provenance for identifier requirements; its inaccurate UUID and Redis allocation visual was rejected.
- Distributed-system patterns — provenance for the pattern inventory, regrouped by layer and linked to detailed notes.