← Back to projects

Architecture deep dive

When a ferry operator goes down,
the booking keeps going.

Search answers in 0.14 seconds when the operator is down. Cached sailings instead of empty results. A cart that keeps customers moving.

57% shorterSlowest 5% of operator search calls: 7.4s → 3.2s
0.14 secondsAnswer time with the circuit open
Keep bookingCached sailings, default accommodation and skipped extras

The problem

One operator. Repeated dead ends.

We sell tickets from ferry operators. In three weeks in April 2026, one large operator's system was unavailable on 17 of 21 days, about 18 hours in total. Customers waited for searches, got empty results and became stuck in the cart.

The operator failed at roughly twice its normal hourly traffic. We ruled out our own retries overloading it. The fix was in how we handled the outage: stop waiting for answers that won't arrive, then use the last good data to keep the journey moving.

The existing foundation

Make the breaker useful downstream

The operator already sat behind an external service gateway (ESG): a service that translates between our platform and the operator's API. It already had a circuit breaker, a switch that stops calling an unhealthy system and later tests its recovery.

Request records enter a DynamoDB stream, a feed of database changes. The stream invokes a Lambda, a small piece of code AWS runs on demand. A CloudWatch alarm watches its error rate; a circuit-switch Lambda changes which Lambda consumes the stream.

The breaker pre-dated this work. The change was to make it faster, visible and actionable for search and cart.

Stream consumer routing

Which Lambda the stream invokes

DynamoDB streamRequest records
Real LambdaFull batches
Fallback LambdaPublishes failure event
No operator API call
Operator systemOutbound API call

Solid lines: stream invokes a Lambda. Dashed line: Lambda calls the operator API.

Search

Wait less. Return sailings.

Set timeouts from real answers

Almost no useful answers arrived after the operator's second response-time cluster, around 2.3 seconds. We cut the search timeout 7s → 3s and the price timeout 5s → 2.75s, just beyond the useful responses.

Make state changes visible

Structured logs, a CloudWatch metric and a Grafana timeline exposed the breaker's state. Alarm history showed about 100 seconds to open. We reduced the price-call alarm from three evaluation periods to two. Expected detection time: around 40 seconds.

Use the last good answer

Search checks a recent cache first, then the operator, then the last known good answer. A cache is a saved response. During an outage, customers see previously returned sailings instead of an empty search.

Search during an outage

From a cache miss to saved sailings

Search
Recent cacheFresh answers
Operator gatewayBreaker open
Last-good cacheSaved sailings
Read the sequence
  1. Search checks the recent cache.
  2. No fresh answer is available.
  3. Search asks the operator gateway.
  4. The open breaker returns failure immediately, without calling the operator.
  5. Search checks the last-good cache.
  6. Saved sailings return to search.

What it did to search

The slow tail got cut in half

These are the durations of the Lambda that calls the operator for sailings, four weeks either side of the timeout change.

The healthy median stayed at about 1.3 seconds. The average fell 20%, from 1.93s to 1.54s. On outage days, the median fell from 4.1s to 1.5s.

The median daily failure rate fell by about a quarter in the initial post-change period. Peak-season failures rose as traffic reached 2.4 times April's level, while p95 stayed at 3.2s.

CloudWatch Lambda durations and error counts; four weeks either side of the change. The post-change period had roughly twice as many calls.

The cart

One state. Three steps that carry on.

Customers who had already picked a sailing could still hit dead ends in accommodation, extras and price. We made those steps use default accommodation, skip extras and keep the cached price, including after a refresh, saved link or back-button visit.

Let the breaker drive off-sale

We initially mapped four meanings of unavailable: operator down, breaker open, breaker half-open and operator taken off sale. Rather than track all four in the cart, we connected the breaker to the legacy system's existing off-sale switch.

The ESG publishes state changes. A listener takes the operator off sale when the breaker opens, and puts it back on sale when half-open or closed. The cart can then ask one question: is this operator off sale?

Failed legacy writes retry with growing gaps, from 5 seconds to a minute, then raise a fault event for investigation.

State change sequence

One event connects the breaker to the cart.

ESGListenerLegacy systemCartState-change eventSet off saleRead off-sale state
ESGBreaker opens
↓ state-change event
Listener serviceReceives the event
↓ set operator off sale
Legacy booking systemStores the off-sale state
↓ cart reads that state
CartDefault accommodation. Skip extras.
Use the cached price.
Half-open or closed sends the reverse state change: put the operator back on sale. The listener retries failed writes with growing gaps, from 5 seconds to a minute, then raises a fault event if retries run out.

Cart impact

Customers could keep moving.

Accommodation, extras and price now carry customers through an off-sale period, including saved links, refreshes and back-button visits.

In the first outage after the change, orders for that operator per hour of downtime were roughly 15 times the rate in the previous week's outages. That comparison comes from a handful of orders on each side, without a formal baseline, and covers both this work and a colleague's cached-price fallback for payment and booking.

Next: compare breaker state with minute-by-minute cart progression.

Trade-offs and next steps

What a cached answer costs

Stale prices

Every fallback in this post trades freshness for availability. Cached sailings can have sold out, and a cached price can be wrong by the time the customer pays. When we showed customers a "the price has changed" message at payment, most of them left. So the cache keeps customers moving through the outage, but the price check at payment becomes the next place they drop off. Making that moment less of a shock is the next problem.

Two gateways doing one job

Search and price for this operator go through two separate ESGs, each with its own breaker and its own alarm settings. They should become one gateway with one breaker, so a single state describes whether the operator is up.

Validate faster detection

Check the new alarm history against the expected 40-second detection time.

Words used on this page
Cache
A saved copy of an earlier answer, used instead of asking again.
Circuit breaker
A switch in front of an outside system that stops sending it requests while it's failing, then tests whether it has recovered.
CloudWatch alarm
An AWS rule that watches a metric and changes state when it crosses a threshold.
DynamoDB stream
A feed of changes to a DynamoDB table that can trigger a Lambda.
External service gateway (ESG)
A service at the edge that talks to an outside system and translates for everything behind it.
Fault event
A message published when a step fails, so the failure can be collected and monitored.
Lambda
A small piece of code AWS runs on demand, with no server to manage.
Median
The middle value. Half of the measurements were faster, half were slower.
Off sale
An operator the marketplace has stopped offering for booking, for now.
p95
The value 95% of measurements were faster than. It shows the slow tail.
Ferry operator
The company that runs the ferries and whose tickets we sell.
Timeout
How long we wait for an answer before giving up.

The next deep dive, in your inbox.