A/B testing has become somewhat of a cargo cult, with tools like optimizely making it insanely easy to run tests. IMO, simple copy tweaks or button color changes are usually waste of time tests.
I'm a big fan of running A-B tests on major product changes - which is basically just standard business benchmarking. Measuring the impact and understanding the results of a change are important whether you run a web app or a flower shop.
PPC is the one place for me where smaller AB tests are absolutely essential. In most other scenarios there are too many confounding variables to really understand much from small tweaks in a product.
100% agreed. I've worked for companies where A/B testing is so consuming that nobody see that they're shuffling deck chairs on the Titanic.
Like ANY other tool, A/B tests have their place, but the incremental gains you're likely to produce from a different color button pale in comparison to what else you might be able to do with that mental energy.
Also completely agreed that paid traffic is a completely different beast.
I'm a big fan of running A-B tests on major product changes - which is basically just standard business benchmarking. Measuring the impact and understanding the results of a change are important whether you run a web app or a flower shop.
PPC is the one place for me where smaller AB tests are absolutely essential. In most other scenarios there are too many confounding variables to really understand much from small tweaks in a product.