Sadly, there is an issue with the Novelty Effect[1]. If you push traffic to the current winner, it probably won't validate that its the actual winner. So you may trade more conversions now, for a higher churn than you can tolerate later.
For example, you run two campaigns:
1. Get my widgets, one year only 19.99!
2. Get my widgets, first year only 19.99!
The first one may win, but they all cancel at the second year because they thought it was only for one year. They all leave reviews complaining that you scammed them.
So, I would venture that this idea is a bad one, but sounds good on paper.
PS. A/B tests don't just provide you with evidence that one solution might be better than the other, they also provide some protection in that a number of participants will get the status-quo.
> So, I would venture that this idea is a bad one, but sounds good on paper.
It's a great idea, it's just vulnerable to non-stationary effects (novelty effect, seasonality, etc). But it's actually no worse than fixed time horizon testing for your example if you run the test less than a year. You A/B test that copy for a month, push everyone to A, and you're still not going to realize it's actually worse.
Yeah. If churn is part of the experiment, then even after you stop the a/b test for treatment, you may have to wait at least a year before you have the final results.
For example, you run two campaigns:
1. Get my widgets, one year only 19.99!
2. Get my widgets, first year only 19.99!
The first one may win, but they all cancel at the second year because they thought it was only for one year. They all leave reviews complaining that you scammed them.
So, I would venture that this idea is a bad one, but sounds good on paper.
[1]: https://medium.com/geekculture/the-novelty-effect-an-importa...
PS. A/B tests don't just provide you with evidence that one solution might be better than the other, they also provide some protection in that a number of participants will get the status-quo.