This is really important, I think, and overlooked.
A/B testing micro-optimizes an outcome you're searching for. It doesn't tell you the long term trajectory of customers, because that takes way too long to get feedback. An example: "dark patterns". Facebook is deeply damaged, in part by making choices from well-run A/B trials.
Worse, A/B testing searches out local maxima. Maybe you get locked into an approach that precludes you from making the changes that would really drive the visit ultimately.
Performance indicators and properly using quantitative information are really important-- and many executives aren't versed at this. But, conversely, you can't use A/B testing to decide who you are as a company and define your relationship with the customer.
I dunno, I remember going to a talk at SXSW in 2011 about how A/B testing helps you reach local maxima but you need more to reach the next “peak”. I believe the story about Google testing 40 shades of blue was also in common currency back then.
I don’t think any of this stuff is new or undiscovered, really.
Well, I don't think anyone anticipated the "dark patterns" stuff at the time-- that we could be training technology to be kinda-evil in a way that would have massive consequences-- national political, reputation, etc.
A/B testing micro-optimizes an outcome you're searching for. It doesn't tell you the long term trajectory of customers, because that takes way too long to get feedback. An example: "dark patterns". Facebook is deeply damaged, in part by making choices from well-run A/B trials.
Worse, A/B testing searches out local maxima. Maybe you get locked into an approach that precludes you from making the changes that would really drive the visit ultimately.
Performance indicators and properly using quantitative information are really important-- and many executives aren't versed at this. But, conversely, you can't use A/B testing to decide who you are as a company and define your relationship with the customer.