> they have no ability to measure the effect of any given design intervention over anything more than the duration of a test
It's actually pretty common to have a holdout group that's basically a long term control group for a series of experiments. This way companies are able to measure long-term impacts, including negative ones.
Are we talking about the same thing? Not only have I never encountered any business back-testing experiments in any way, but purposefully running "losing" variants on a percentage of traffic would be anathema.
It's actually pretty common to have a holdout group that's basically a long term control group for a series of experiments. This way companies are able to measure long-term impacts, including negative ones.