He doesn't directly address that relentless A/B testing can be a pathology that removes the soul and flexibility from your product and can wall in future growth. This is because you have to optimize on one (or selected) KPIs but the true health of your business is more nuanced than that. Large analytics operations try to solve this by creating ever more complex KPI models but that is not attainable for most.
As a crude example if you have a community site like Reddit you could harm the community by over-optimizing your conversion funnel for Reddit Gold, while still substantially growing the Reddit Gold business. For a time.
He touches on this by admitting that the super-optimized KISSMetrics homepage was not the right strategy but I would have loved read more.
I believe there will be plenty more stories like this coming out and landing well. Some of the testing I've managed hasn't helped end-to-end conversion rate _at all_ and did nothing short-term results even when it looked great at first glance. I feel pretty jaded about "performance marketing" and "growth marketing" at this point. There are legitimate growth marketers out there but it's become a cop out for The Hard Things in marketing.
Essentially, it's become really easy to get leads for cheap and incredibly expensive to get them to convert. The question becomes, which master do you serve?
I think marketing leaders need to delicately balance the KPIs with the vision of the company or product. In High Output Management, Andy Grove talked about negative indicators for KPIs as well (forgive me, don't have book within reach). I don't think any time gets spent on that in my network. Granted, it is hard to do with small teams and you might say it's the wrong thing to focus on for small teams. Eventually, you reach the point where the numbers look too good to be true and they usually are--meaning you won't convert these people because they're not ready to buy what you're selling. They're just ready to get the free info you're offering them in two clicks of their time.
I appreciate the candidness about failure even if it wasn't detailed. Can't wait for more similar stories.
If one claims to have growth experience. Enough to create a blog post about it. And one can't figure out the appropriate strategy to make a return on your head count - then stop pretending . A growth team should be designed to meet the business requirements and make a return (on top of your head count). Because one can't create an effective strategy to produce a return, doesn't mean you toss out the entire idea of creating Growth Teams. Because you failed - it doesn't mean its bad. It wasn't bad for Facebook.
As a crude example if you have a community site like Reddit you could harm the community by over-optimizing your conversion funnel for Reddit Gold, while still substantially growing the Reddit Gold business. For a time.
He touches on this by admitting that the super-optimized KISSMetrics homepage was not the right strategy but I would have loved read more.