“I have a rule-of-thumb for picking A/B test winners: Whichever version doesn’t make any sense or seems like it would never work, bet on that. More often than I like to admit, the dumber or weirder version wins.
This is actually how I tell if a company is truly optimizing their funnel. From a branding or UX perspective, it should feel a little “off.” If it’s polished and everything makes sense, they haven’t pushed that hard on optimization.”
...but if you take this too far you end up with something like Amazon.com, where everything is “a little off”?
That's actually the part that I found a little bit ironic (though I very much enjoyed the post), considering that the author links to and praises a study that points out that many A/B tests result in illusory wins because growth teams end up testing a zillion different variations and stop tests early whenever one alternative seems promising.
In such an environment, you're bound to end up with all kinds of weird "winners" but there's no guarantee that (1) they're actually better than the alternatives you tested and (2) even if they are, that the advantage is stable and not just a temporary novelty effect.
That part also jumped out to me, but made me worry that the author isn't accounting for novelty effects. When you make something weird at first it does well, and then later as people get used to it the lift goes away or even turns negative.
With a lead gen funnel, where everyone is going through for the first time, this is much less of an issue than a site with long term users. In the latter case you want to measure learning effects, while in the former no individual is in a position to learn. Novelty could still wear off in lead gen, though, as companies copy each other and your weird new pattern starts to show up elsewhere and become familiar.
I always use Amazon and eBay as go-to examples of Optimization that works as a counter-force to those that want massive redesigns every 6 months because some other new app or competitor has some sexy slick UI.
"Paradoxically, Amazon's design may work well for Amazon itself. The company is simply so different from other ecommerce sites that what's good for Amazon is not good for normal sites."
“I have a rule-of-thumb for picking A/B test winners: Whichever version doesn’t make any sense or seems like it would never work, bet on that. More often than I like to admit, the dumber or weirder version wins.
This is actually how I tell if a company is truly optimizing their funnel. From a branding or UX perspective, it should feel a little “off.” If it’s polished and everything makes sense, they haven’t pushed that hard on optimization.”
...but if you take this too far you end up with something like Amazon.com, where everything is “a little off”?