Circuit Breakers & Bulkheads

The problem

In distributed systems, failures don’t stay local.

One slow dependency → thread pools fill
→ retries amplify traffic
→ upstream services stall
→ the outage spreads.

This is called a cascading failure.

Large-scale platforms at companies like Netflix and Amazon design their systems assuming:
dependencies will fail.

---

One-line definition

Circuit breaker prevents calling a broken dependency.
Bulkhead prevents a broken dependency from consuming all your resources.

They solve two different failure modes.

---

The mental model

When a dependency becomes unhealthy:

• stop sending traffic to it

• isolate its resource usage

• allow the rest of the system to survive

---

Single system view

Clients
   |
   v
+----------------------+
|     API Service      |
+----------------------+
   |             |
   |             |
   v             v
+---------+   +---------+
| Pool A  |   | Pool B  |   <-- Bulkheads (isolated resources)
+---------+   +---------+
    |             |
    v             v
+-----------+  +-----------+
| Service X |  | Service Y |
+-----------+  +-----------+
      |
      v
[CIRCUIT BREAKER]
      |
      v
 Downstream Dependency

---

Circuit breaker – what actually happens

A circuit breaker has three states:

Closed

→ traffic flows normally

Open

→ requests are rejected immediately

→ no network calls

→ no threads blocked

Half-open

→ a small number of test requests

→ decide if recovery is real

Why this matters:

Fail fast is better than fail slowly.

Slow failures:

• keep threads busy

• hold sockets

• amplify retries

---

Bulkheads – the other half of the story

Bulkhead means:

One dependency must not consume resources needed by others.

Typical bulkheads:

• separate thread pools

• separate connection pools

• separate queues

Without bulkheads:

A single slow downstream can exhaust:

• worker threads

• database connections

• event loops

And your healthy features die with it.

---

Circuit breaker vs timeout vs retry

Timeout:

→ limits how long you wait.

Retry:

→ increases pressure during failure.

Circuit breaker:

→ stops useless traffic entirely.

If you retry without a breaker,

you are helping the outage spread.

---

Real-world example

Think about a typical media or commerce backend:

API

→ personalization service

→ recommendations service

→ payments service

If recommendations become slow:

You do NOT want:

checkout threads waiting on recommendations.

With bulkheads:

• checkout has its own pool

• recommendations can fail independently

With a circuit breaker:

• the system quickly skips recommendations

• a fallback is served

This failure isolation pattern is widely used in large production systems, including those at Netflix scale.

---

The most common mistake

Only adding circuit breakers.

Without bulkheads:

Even when the breaker opens,

the damage is already done:

• thread pools are exhausted

• request queues are full

• memory pressure is high

Circuit breakers protect dependencies.

Bulkheads protect your own service.

You need both.

---

Advanced consideration: shared pools are dangerous

A single global thread pool is a hidden coupling.

If:

• dependency A stalls

• it occupies most threads

Then:

• dependency B becomes slow

• even if B is perfectly healthy

Isolation is more important than efficiency under failure.

---

Design checklist

Before shipping any microservice:

• Does every remote dependency have a circuit breaker?

• Does each major dependency have its own thread / connection pool?

• Do you have fallbacks when the breaker is open?

• Are retries disabled or heavily limited when a breaker is open?

• Are critical paths isolated from best-effort features?

---

Key takeaways

• Cascading failures are the default in distributed systems

• Circuit breakers stop useless traffic to broken dependencies

• Bulkheads prevent one dependency from consuming all resources

• Isolation is more important than peak efficiency