Experiment deep dive
More sailings on screen,
more customers to the cart.
I built the search-results layout and worked on the experiment and tracking. The redesign showed more sailings; the tracking checks told us whether to trust the lift.
The result
Relative lift in customers reaching the cart on the English-language site, measured after the tracking fixes. +10.7% across the whole run.
Buying a ferry crossing means comparing sailings: times, operators and prices. When only a few results fit on screen, that means scrolling back and forth.
The idea was simple: show more sailings without taking useful information away. Then test whether more customers chose one.
The change
Denser, with nothing removed
The experiment's rules were set up front: return the same sailings at the same prices, keep the details customers need to compare, and fit more results on the first screen. On phones, nothing could overlap or need sideways scrolling.
The risk was clear too. A denser page might be harder to read. The experiment would measure whether customers still found a sailing worth choosing.
Before and after
Open either image to zoomDesktop screenshots of the old and new layouts. Open either image, then tap it to zoom.
The experiment
Half the customers,
one week
Customers on the English-language site were split at random. Half saw the existing layout, the control. Half saw the new layout, the treatment. The measure was the share who went from search results to the cart.
The new layout was ahead from the third day and stayed ahead. Across the full run, the relative lift was +10.7%. The reported probability that it was better was above 99.9%.
A shorter French-language run pointed the same way, but its groups were uneven for reasons the fixes didn't explain. It was supporting evidence, not the main result.
Trusting the result
Two tracking problems worth separating
A lift is only useful if you can trust the record of who saw what. Two problems showed up in that record.
The groups were uneven for two days
Early in the run, too many customers were recorded in the control group for it to look like chance. Those were the same two days the view-tracking code was being fixed. One condition always returned true, recording views it shouldn't have.
After the fixes, the groups were even again. Excluding those two days made the relative lift larger: +12.7%, rather than +10.7% across the whole run.
Some customers appeared in both groups
Some customers were recorded seeing both layouts. That makes it unclear which layout influenced their decision, so the results here count only customers recorded in one group.
The cause of that overlap was still unresolved. Because the same assignment code is used across experiments, finding it matters beyond this redesign.
What changed
The layout became the default
More sailings on the first screen. No useful detail removed. More customers reaching the cart.
The important part was that the lift survived the tracking checks. Removing the bad days strengthened the result rather than explaining it away.
Words used on this page
- Control
- The group that sees the existing version.
- Treatment
- The group that sees the new version.
- Relative lift
- The proportional increase compared with the control. It is not the percentage of all customers who reached the cart.
- Above the fold
- The part of a page visible before scrolling.