As kevinwang has pointed out in slightly different terms: the hypothesis that seems wooly to you seems sharply pointed to others (and vice versa) because explanationless hypotheses ("changing the colour of the button will help") are easily variable (as are the colour of the xkcd jelly beans), while hypotheses that are tied strongly to an explanation are not. You can test an explanationless hypothesis, but that doesn't get you very far, at least in understanding.
As usual here I'm channeling David Deutsch's language and ideas on this, I think mostly from The Beginning of Infinity, which he delightfully and memorably explains using a different context here: https://vid.puffyan.us/watch?v=folTvNDL08A (the yt link if you're impatient: https://youtu.be/watch?v=folTvNDL08A - the part I'm talking about starts at about 9:36, but it's a very tight talk and you should start from the beginning).
Incidentally, one of these TED talks of Deutsch - not sure if this or the earlier one - TED-head Chris Anderson said was his all-time favourite.
plagiarist:
> That doesn't test noticing the button, that tests clicking the button. If the color changes it is possible that fewer people notice it but are more likely to click in a way that increases total traffic.
"Critical rationalists" would first of all say: it does test noticing the button, but tests are a shot at refuting the theory, here by showing no effect. But also, and less commonly understood: even if there is no change in your A/B - an apparently successful refutation of the "people will click more because they'll notice the colour" theory - experimental tests are also fallible, just as everything else.
Will watch the TED talk, thanks for sharing. I come at this from a medical/epidemiological background prior to building software, and no doubt this shapes my view on the language we use around experimentation, so it is interesting to hear different reasoning.
Good to see an open mind! I think most critical rationalists would say that epidemiology is a den of weakly explanatory theories.
Even though I agree, I'm not sure that's 100% epidemiology's fault by any means: it's just a very difficult subject, at least without measurement technology, computational power, and probably (machine or human) learning and theory-building that even now we don't have. But, there must be opportunities here for people making better theories.
As usual here I'm channeling David Deutsch's language and ideas on this, I think mostly from The Beginning of Infinity, which he delightfully and memorably explains using a different context here: https://vid.puffyan.us/watch?v=folTvNDL08A (the yt link if you're impatient: https://youtu.be/watch?v=folTvNDL08A - the part I'm talking about starts at about 9:36, but it's a very tight talk and you should start from the beginning).
Incidentally, one of these TED talks of Deutsch - not sure if this or the earlier one - TED-head Chris Anderson said was his all-time favourite.
plagiarist:
> That doesn't test noticing the button, that tests clicking the button. If the color changes it is possible that fewer people notice it but are more likely to click in a way that increases total traffic.
"Critical rationalists" would first of all say: it does test noticing the button, but tests are a shot at refuting the theory, here by showing no effect. But also, and less commonly understood: even if there is no change in your A/B - an apparently successful refutation of the "people will click more because they'll notice the colour" theory - experimental tests are also fallible, just as everything else.