Experiment deep dive

More sailings on screen,
more customers to the cart.

I built the search-results layout and worked on the experiment and tracking. The redesign showed more sailings; the tracking checks told us whether to trust the lift.

The result

+12.7%

Relative lift in customers reaching the cart on the English-language site, measured after the tracking fixes. +10.7% across the whole run.

Buying a ferry crossing means comparing sailings: times, operators and prices. When only a few results fit on screen, that means scrolling back and forth.

The idea was simple: show more sailings without taking useful information away. Then test whether more customers chose one.

The change

Denser, with nothing removed

The experiment's rules were set up front: return the same sailings at the same prices, keep the details customers need to compare, and fit more results on the first screen. On phones, nothing could overlap or need sideways scrolling.

The risk was clear too. A denser page might be harder to read. The experiment would measure whether customers still found a sailing worth choosing.

Before and after

Open either image to zoom
Before: rectangular rowsRoute, departure, duration, operator and price read across each row.
After: rounded cardsDeparture and arrival times read across each card, with deposit options on eligible sailings.

Desktop screenshots of the old and new layouts. Open either image, then tap it to zoom.

The experiment

Half the customers,
one week

Customers on the English-language site were split at random. Half saw the existing layout, the control. Half saw the new layout, the treatment. The measure was the share who went from search results to the cart.

The new layout was ahead from the third day and stayed ahead. Across the full run, the relative lift was +10.7%. The reported probability that it was better was above 99.9%.

A shorter French-language run pointed the same way, but its groups were uneven for reasons the fixes didn't explain. It was supporting evidence, not the main result.

Trusting the result

Two tracking problems worth separating

A lift is only useful if you can trust the record of who saw what. Two problems showed up in that record.

The groups were uneven for two days

Early in the run, too many customers were recorded in the control group for it to look like chance. Those were the same two days the view-tracking code was being fixed. One condition always returned true, recording views it shouldn't have.

After the fixes, the groups were even again. Excluding those two days made the relative lift larger: +12.7%, rather than +10.7% across the whole run.

Some customers appeared in both groups

Some customers were recorded seeing both layouts. That makes it unclear which layout influenced their decision, so the results here count only customers recorded in one group.

The cause of that overlap was still unresolved. Because the same assignment code is used across experiments, finding it matters beyond this redesign.

What changed

The layout became the default

More sailings on the first screen. No useful detail removed. More customers reaching the cart.

The important part was that the lift survived the tracking checks. Removing the bad days strengthened the result rather than explaining it away.

Words used on this page
Control
The group that sees the existing version.
Treatment
The group that sees the new version.
Relative lift
The proportional increase compared with the control. It is not the percentage of all customers who reached the cart.
Above the fold
The part of a page visible before scrolling.

The next deep dive, in your inbox.