People often don’t determine sample sizes at all! And doing power calculations without an idea of effect size isn’t just hard but impossible. It’s one of the inputs to the formula. But at least it’s fast so you can sort of guess and check.
Anytime valid inference helps with this situation, but it doesn’t solve it. If you’re trying to detect a small effect, it would be nicer to figure out you need a million samples up front versus learning that because your test with 1,000 samples a day took three years.
Still, anytime is way better than fixed IMO. Fixed almost never really exists. Every A/B testing platform I’ve seen allows peeking.
I work with the author of the second paper you listed. The math looks advanced, but it’s very easy to implement.
Anytime valid inference helps with this situation, but it doesn’t solve it. If you’re trying to detect a small effect, it would be nicer to figure out you need a million samples up front versus learning that because your test with 1,000 samples a day took three years.
Still, anytime is way better than fixed IMO. Fixed almost never really exists. Every A/B testing platform I’ve seen allows peeking.
I work with the author of the second paper you listed. The math looks advanced, but it’s very easy to implement.