A quick aside: A/Bingo exists primarily because of HN, whose articles and discussion drove me onward to finishing it. Thanks!
If you have questions, I've got answers, at least for the next hour or so. (Tonight is date night. I'm dedicated to my work, but I'm not that dedicated.)
This looks like a terrific resource. I was just setting up an A/B test the other day with Google Website Optimizer ... I just wanted to test a single word change. To go through GWO, I had to set up a multivariate test, with multiple js snippets going in multiple places on multiple pages.
I haven't installed A/Bingo on any live sites, but I'm looking forward to kicking the tires. Thanks for releasing it.
Since it's open source, have you thought about setting it up at Github? Since that's becoming such a standard resource, it'd make looking through it easier.
To go through GWO, I had to set up a multivariate test, with multiple js snippets going in multiple places on multiple pages. That was EXACTLY my pain point with their multivariate option. It took me 15 minutes to set up a single test on a single call to action, and then 15 minutes more to tear it down when I was done.
Testing a call to action should look like:
<%= ab_test("call_to_action", ["Buy it today!", "Get it today!"]) %>
And now, it does.
]
Since it's open source, have you thought about setting it up at Github?
Yes, I did think about it. We're all businessmen here, right, so we can discuss the following in a mature fashion:
1) Github gives out free hosting to OSS projects in the hopes of encouraging people to host OSS projects of value on Github. This means Github collects links and mindshare, which they then monetize through selling hosting services.
2) I write OSS (and provide statistics, and write articles, etc) in the hopes that it helps other businesses. This means I collect links and mindshare, which I then monetize through selling software to elementary schoolteachers.
Given that I did the work, I think it is reasonable that I collect the majority of the benefits, rather than giving them away to a hosting provider.
If copy/pasting the git clone line is too much work to see the code, you may wish to reconsider using A/Bingo. I made it easy -- I can't make it THAT easy. But if you really want to see it in Github, it is MIT licensed, upload it to your own account and then you can view it in your browser to your heart's content.
I would disagree on one point, Github isn't just for hosting, it's for collaboration. That's the reason most people use it for OSS project.
I can, and do, host my own git repos on a private server, but when I open source something I like having it on Github where people can fork it and send me pull request for new features and bug fixes.
I'm a very active GWO user. I haven't tested A/Bingo yet, but based on the description, there's one other advantage you have: I don't have to rewrite erb as javascript in order to execute various tests.
If a lot of your view layer is written in erb (likely b/c it's not javascript-heavy), then to use GWO, you have to basically rewrite all of it in js, since GWO dynamically inserts client-side code into the spots you specify.
So if you have a display loop that uses a bunch of object properties in your erb, for example, you have to copy those objects/values to js variables and move the loop to js. It's all doable, it's just very non-DRY and just altogether unpleasant.
(The other option with GWO is to use the Javascript to load partials, which I do now, but which has some unusual properties, like lazy loading.)
Having the a/b decision be server-side is a nice thing. I'm looking forward to trying it out.
(Also, not having to pull out piles of GWO code every time is a nice thing too.)
This is great, thank you. I was literally going to start working on a Rails A/B framework after reading HN today. Funny how that works.
Edit: In the newest version of Rails (2.3.3) and IIRC a few versions before that as well, the timestamp field is called created_at, not created_on - I was getting an error when running the migration.
Thanks for the bug report. I can't address it right now as I'm at an Internet cafe, with my laptop spending the night at my designer's house, but I'll fix it in the morning. Off the top of my head I don't remember anything that actually uses the timestamps, so feel free just to comment out the line (was it assigning an index, maybe?) that caused the exception.
Do you mean "How positive does it have to be for you to not revert it when you end the test?"
I'd collect 2% improvements all day long if they looked statistically significant. That being said, my mental goal is to run one test a week, and to collect 5% improvements on half of my tests. The miracle of compound interest means that, if I sustained that for a year, I'd increase sales by 250%. You can see my stats and tell that I've never had a year over year improvement that good, but now since the cost of A/B testing just got muuuuuuuch cheaper for me, I'm optimistic.
Yes. Effect size is the difference in means in proportion to the pooled standard deviation to the two treatments, and is a great measure of how large the difference is. All groups are different given large enough n. Whether that difference means anything is dependent on the effect size.
When the effect size is large you know that whatever effect you are seeing is much less likely to be chance. It's also less susceptible to large differences in variation, which the tests you are using to test significance use as an assumption (variation is equal).
If you have questions, I've got answers, at least for the next hour or so. (Tonight is date night. I'm dedicated to my work, but I'm not that dedicated.)