The goal at first was to just see what numbers people gravitate toward. I've heard lots of conjecture about how people pick 7 or 3 or 4 more than others, and for a variety of reasons, but had a hard time finding actual demonstration of this. Then, while implementing a choosing system, the problem became: how do you present the information so as not to bias it? This is why there are four different ways of picking. There are also a couple other metrics being measured, including a difference in phrasing (Pick a number… vs Pick a random number…) which may be interesting.
Apologies for any bugs or general wonkiness. The whole thing was a ~2 hour impulse project.
PS: The data will absolutely be shared! Just need time to do a breakdown of all the different permutations.
Interesting idea, but the slider may not be a great interface. It stops the experiment being purely about numbers - I found I picked '5' because it was close to being nice and symmetrical.
I think that is a good datapoint, myself. If there is a predisposition for people to choose "5" under those circumstances, we will have learned something.
This would be a fantastic app to run A/B -like testing on to determine whether say a circular "picker" gives different results than a horizontal "picker". You could even tighten it down to see if a random distribution of numbers between 1 and 10 gives different results than a ordered distribution.
It fits the model perfectly and, at least to my perspective, very clearly.
I also took 5 because it was close. Also, since I know people tend to pick 7, I somewhat tend to don't pick 7 when I'm asked something :p You could maybe "randomize" the order of the numbers, or hack them up so, for instance, 7 is far left and 3 is far right.
If there are enough data points (which it seems like there is), and each UI is chosen a random 1/4 of the time, then it will be very easy to isolate and determine the effect of the slider UI. Problems would arise if the slider was chosen 60% of the time in Firefox, but only 30% of the time in Opera, and if Opera had a 20% higher population of hackers, who had a distribution centered around 6 as opposed to the aggregate population centered around 5; as is, it sounds like there's plenty of random sampling solving these complications.
No, the UI is different every time you refresh the page. There are 4 UIs, randomly presented.
I'll be fascinated to see how the different UIs skew the results. Of course, there are now so many variables in this "experiment" it'll be hard to make any concrete determinations.
Honestly I wasn't expecting to get so many number picks so fast. The plan was to start putting together an analysis page once I got enough data, which I thought would be a couple weeks. I'm going to try and get one together as soon as possible.
Edit: Just had to enable billing in AppEngine so it can keep going! Way more data than I ever expected.
An alternative idea would be to display a link to a page where the data will be displayed once it is ready. I agree that just getting a "Science thanks you" message feels a little like being cheated, but it didn't bother me because I knew I'd be coming back here and waiting for info about results.
Hey, so I submitted it to reddit thinking it might get you a few more responses. It kinda ended up being the number one link on the front page. Hope that didn't hurt too much.
Thanks for submitting! The number of responses is vastly more than I expected, pushing 100k uniques now. AppEngine has handled it wonderfully. It blew past the free quota, so I did have to enable billing. But, it's still only pennies so far, and a quota reset is coming up soon.
The amount of data has been a challenge, since I have to put together more efficient stats tracking. A good problem to have, of course.
I was expecting to have to spam the crap out of my Twitter feed and leave links everywhere to get even 100 participants. Glad that wasn't the case!
This is where Appengine shines - it scales painlessly, so there won't be any hurt (apart from being billed for the resources).
This scalability comes at the cost of a wee paradigm shift from relational database to datastore mindsets, but is well worth it. Developers can do what they do best instead of having to become system architects and admins.
I would imagine you would get a distribution favored to 3 and 7. We did similar research during a Cognitive Science class. We asked a number from 1 to 4, and got over 40% 3's.
We also did research to find favored Mastermind patterns. Bias was a large problem there too. When presented with colors, people would pick a single color more often or place the same colors next to each other. When presented with letters, people would try to spell out words.
Peculiar: In product pricing and conversion testing, prices with 7's and 9's seem to provide more favorable results. I believe this is akin to the favorite-color bias we happened upon (7 is my lucky number!), mixed with the slight confusing nature of calculating/rounding down a price ending in 7 or 9 (hey, it's still $2999, so just 2 grand and then some).
Does your university have a significant Asian population?
Three (homonym with "alive") is a lucky number in Chinese culture. Four (homonym with "death") is an unlucky number. Apartment buildings built for Chinese persons often skip all floor numbers with "4" in them as well as all apartment numbers with "4" in them.
Don't know about 3, but if a price includes lots of 8s, the seller is definitely aiming at Chinese buyers. Seeing this left and right here in Vancouver, and it looks corny at best. Like trying to lure Russians with a picture of vodka, or Americans with that of a cowboy hat.
"I’m staying at a hotel right now, there’s no 13th floor because of superstition. But come on man, the people on the 14th floor, you know what floor you’re really on. If you jump out of the 14th floor hoping to kill yourself, you will die earlier." ~Mitch Hedberg
He makes a fair point though--if you're so superstitious that you're uncomfortable with the thirteenth floor, shouldn't you be uncomfortable about it regardless of its nomenclature?
I would also guess that it's biased more toward 7 than toward 3. That's the conventional wisdom, isn't it? Below 4, 3; below 10, 7; below 20, 17; below 40, 37. Or has this bit of pop-math folklore been lost in translation?
Whenever I see these sorts of things I try to psych them out. In this case I deliberately thought of a number, discarded it, and repeated this process several times (rather than putting down 7, which was my first thought). So, unless you're trying to measure frequency of picked numbers when the pickers are trying to game the system, I don't think this will prove much. Nice interface though.
That's why one of the metrics is time, both from start of page to picking a number, then start to hitting choose. It's not perfect, but it'll hopefully be possible to differentiate between people who pick right away, and people who think about it.
This would drastically vary.
When A person tells B to pick a number there are various things going on in mind. Once its brought down to select from 1-10 the choices are narrowed down to a single digit number. Hardly anyone would choose 1 or 10. If someone's birthday comes in between those numbers, the choice becomes obvious, other wise its moves to the favorable number or a lucky number.
7 being the universal lucky number many believe in that. For numerlogy believers 3 comes to be a common lucky number amongst many. So the choice, not sure why 4.
Anyways the reasons vary alot.
Do keep us updated w/ the result want to know how mind works over numbers for all.
The only problem being, many would refresh and choose all numbers, why don't you add a simple cookie or ip restriction to allow to choose only once? This would reduce alot of fake entries.
I presume this is your site? Your design has a serious flaw: the number "1" is "selected" when you first view the page, which biases the results. You apparently tried to "fix" this by forcing the user to move the slider before the choice would be accepted, but that also biases the results.
The only way to really make it unbiased is with a text box. The next best thing would have been ten buttons in a row (but then you would have had to make sure that they didn't go off the right side of the screen).
I'd actually be interested in how different ways of presenting the choice affect the number chosen. It would be interesting of the author ran an A/B test with different methods of selection (text box, row of buttons 1-10, row of buttons in random order, slider, etc) to see what the differences would be.
I wonder if adjusting the the numbers would influence the choice. Instead of listing them 1-10, maybe try a run with the numbers not listed in sequential order.
There is actually a "mind trick" where you make someone choose a number between 1 and 4, guess it is 3, then make them choose a number between 1 and 10, and guess it is 7. It works, because there is a statistical bias toward these numbers, but the way you do the trick is also important (the way you gesture with your hands primes the person to answer with these numbers).
I've always loved such mind tricks, and I have done this one quite a few times. Before discovering HN, I used to go out a lot, and, when talking to cute girls, I've done this "trick" many times for giggles. Some girls really believed I was a psychic when I got it right :)
Funnily enough, knowing about this trick and the "statistical preference" for the number 7 made me choose 7 in this experiment, without thinking twice about it. I realized this afterward...
I wish this data could be analyzed- I was debating between three, five, and seven, and chose nine just to throw off Benford's Law. I really think it is the first time I have ever chosen nine.
An interesting extension of this could be to add a poll here on HN asking, "Which number did you pick?" and then comparing the poll results to the actual results.
Are you trying to estimate the bias depending on the type of selector used (slider, input, options, ...)?
Very interesting concept :)
EDIT, forgot to add: if this is the case, have you thought of storing the type of selector in a cookie, so that refreshing the page gives always the same type?
Yeah, I considered a more involved way of choosing the interfaces to present — the server attempts to even out the distribution of interfaces, use cookies to limit, etc — but I also just really wanted to get this made in a couple hours. The current implementation is about as simple as it gets.
The source shows he's also timing the pick, which I think is probably the more important dependent variable here. Doesn't matter what number you chose - it's how long it took you to find it, click it, and hit submit.
Indeed. The goal there is to see if people think about the choice, then second guess it before actually submitting.
Edit: Quickly skimming through the database, it also reveals "tab" loaders who open up a bunch of tabs and let them sit. (It couldn't possibly have taken someone 300s to make a decision!) Not the goal, and it does skew the data, but still interesting.
I hope he gathers data from different social groups. Would be interesting to prove that techie-types have a numerical bias different from marketing types.
On Refreshing, it changes the interface, every time you get a different input method. I guess its that way so as to avoid being biased over any number by default.
The other methods don't have a default. Sliders don't really present a good way of doing a "non-choice" default. I decided to leave it fixed at 1 to start to see if that particular method skews because of the default. (Not exactly precise and thorough, unfortunately. There could be more metrics, like starting position, but I wanted to keep it simple so I could get it up and running quickly.)
I am interested in finding out how many people decided not to choose a number at all, and if that fact could help mitigate the experiment's inherent bias.
As for people making multiple entries into the experiment, a simple IP filter could help reduce overall error.
I think it would be neat to offer an option to not choose a number, but rather a number to help offset selector's bias (in participating in the number choosing experiment knowingly). It's similar to the multiple choosing UIs.
Nonetheless, I can only appreciate simple things like this that lead to a spirited and educated read.
Cheers and can't wait to see "final" results and accompanying analysis.
Interesting project. I'm looking forward to see the results. This reminds me some of the mental algorithms I was thinking about: How do you generate a random number without electronic devices? How do you generate a random permutation without electronic devices? How do you generate a hash without electronic devices? How do you do the above most efficiently? How about without any pen, pencil or paper?
It will be cool if you can master mental cryptography.
A more in depth breakdown is coming. The number of responses has been far, far beyond what I expected — several orders of magnitude more. Once I get together a good way to efficiently track and display the graphs, I'll make them live.
Upon being shown a circle and asked to pick a random number, I thought for a bit and realized that no matter what I picked I wouldn't believe I had chosen it randomly. Then I wrote a one-line script to generate a random number from 1 to 10 and I put its output as my answer. Perhaps this defeated the purpose of the site, but I find it so hard to choose when I know my response is being measured.
Can't wait to hear how many people picked 10 (as I did). If it hadn't been for science, I would probably have picked 7 like everybody else.
Once I thought it would be good to play the numbers "1 2 3 4 5 6" in the lottery because they are as likely as other numbers, and I thought people would not pick them. Turns out lots of people play "1 2 3 4 5 6".
In fact the only advantage you can have in the lottery is of decreasing the chances of hitting a winning number that was also chosen by someone else and thus having to split the Jackpot. Many people let the computer choose, which helps limit duplicates, but if you want to choose numbers then be sure to select numbers over 31. Many people select important dates and so don't have any "lucky numbers" over 31.
That said, if you really want to increase your return for playing the lottery, sticking your money in a cookie jar is a far better investment, provided you already ate all the cookies in the jar.
I know - haven't played in ages, either, but sometimes just for fun it seems OK. In that case it should be considered more of an entertainment than an investment.
As for numbers above 31 - of course it only works as long as not many people realize they should pick numbers above 31. In the long run I suppose random numbers are the best bet.
Brilliant idea, but haven't you built in bias by unblinding the experiment? I followed the link, got one implementation and keyed in '5'. Then I read your write-up and realised that there are more implementations, reloaded the page and thought deeply about the number I would/should choose on the slider.
yesterday i was improving my excel skills about statistics, instead of use the random funcion i thought to ask a number from 1 to 10 to some friend and see the histogram. just after some question i was surprised about the results and so i asked more people. up to now i asked 24 people(46 number in total,the first 2 was just one question), i also asked a second number after recived the first. my result are: that 7 win with 31% of answer followed by 5 with 17% the last are 10 (zero answer), 1 with 1 answer and 2 and 6 with 2 answer each. you have to note that 83% say a odd number as a first one.noone told me a number that wasnt integer. whats going on? the majority of people was italian, some spanish and some german.Guido
After you've entered your unbiased data, make sure to refresh the page and see what else you could have been confronted with. You'll notice the different variables in the experiment - just make sure to do this after in order for it to not influence your decision.
I'd imagine 7 would be less popular or people that have previously heard of a bias towards 7, would be hard to have a test group though that has never heard anything at all about a bias. It's still the highest so far in this experiment though.
I would imagine that if you did this survey in a crowd of sports fans, you would get a different distribution based on town or player followed. Plus, if I remember right 7 and 4 and not terribly lucky numbers in China.
I'm curious to know if the typing prompt scores fewer 10's because of increased typing effort and if the hover-to-reveal circle gets more centrally-located numbers due to Fitt's law
I picked 7, because that's what I expected other people to gravitate to, and I wanted to see if you had some fun response if I picked the most common alternative. :P
As long as you rotate the starting positions of the numbers on the ring and track both the chosen number and the cardinal position of their chosen number on their board. Maybe people are biased to numbers falling at the "three o'clock" position, that would be interesting to find out.
I remember reading an article on this about year ago on HN. I think that the conclusion seemed to be that human prefer odd and prime numbers (at least in range 1 - 20).
I was just thinking it'll be 7. My own theory is most people don't want to be in the middle, so they go for the middle of the upper half. That's what I did anyway.
Yes? What do you mean by app? It's an app whose entire purpose is to collect data. I wouldn't think of it as a Trojan horse, since it doesn't pretend to be anything else. I guess it's Trojan in that it was made in Troy, NY.
Without something like facebook or twitter login, the system will be abused. There is no stopping someone who just reloads ( or even make an automated script heh ) and keeps on choosing, resulting in flawed data. Just saying.
Totally agreed. I'm relying on people not caring enough to abuse it — a bit risky, I know. I just wanted something simple that wouldn't require people to jump through hoops just to vote. Also, the IPs are now one of the metrics, for regional breakdown, so that could be used to spot obvious anomalies.
That's what I was thinking too. My choice was influenced at least slightly by where my mouse was at picking time. Would have been better to have a text entry field I think.
Reminds me of Benford's Law [1], which stipulates that in many lists of numbers from real-world data, the leading digit is 1 30% of the time, and larger digits occur as the leading digit with lower and lower frequency (logarithmically). This has been used to detect made-up numbers in accounting which later turned out to be fraudulent behaviour.
Its a cool idea but I would recommend A/B testing different GUI for picking numbers for removing some of the possible biases. I personally picked 7, most likely because the human eye moves from top left to bottom right so I scanned what the choices were then picked the number near where my eye last looked.
Also, I would love to see a heat map/click map of that page.
The goal at first was to just see what numbers people gravitate toward. I've heard lots of conjecture about how people pick 7 or 3 or 4 more than others, and for a variety of reasons, but had a hard time finding actual demonstration of this. Then, while implementing a choosing system, the problem became: how do you present the information so as not to bias it? This is why there are four different ways of picking. There are also a couple other metrics being measured, including a difference in phrasing (Pick a number… vs Pick a random number…) which may be interesting.
Apologies for any bugs or general wonkiness. The whole thing was a ~2 hour impulse project.
PS: The data will absolutely be shared! Just need time to do a breakdown of all the different permutations.