It's not clear from the article what kind of business model the Startup in question is planning to use. This is relevant, as the fact the 73% of random strangers said they would "use the service as described" doesn't give any indication that they'd be willing to pay for it.
Clearly, the author got his $27.50 worth; how much that's really worth, though, in the long term, remains to be seen.
Yeah, I've done this before. Of the 200 people I surveyed a much higher % (around 95%) said they liked and would use the service...and then did not. My experience with this stuff makes me think that people who do it generally have a perception that you are their "boss" and that they need to sugar coat everything they tell you.
TLDR: Be HIGHLY suspect of any results you get from mechanical turk (same goes for feedback army and others).
It applies to everyone. Not just mechanical turk folks. When someone says they 'will use' your product they are non-committal. (I have done this so many times for the fear of offending the person who is asking me). When its time to take out their credit cards then they backtrack. They are 2 different decision points. The later is what matters to startups.
When its time to take out their credit cards then they backtrack
And to be fair, even if I am trying to be super-honest, I find it very hard to decide whether I would be happy to whip out a credit card from a concept pitch. It's not just a question of "would I pay for A", it is a question of "would I pay for A it it really rang my bell", and I can't tell whether you are going to ring my bell or not until I have tried your product and your customer support philosophy.
When I look at my credit card statement, I see a bunch of companies not just whose product I like and use, but also a bunch of folks for whom I can say "I like the way they do business".
I didn't pay to cloud-store photos for over a decade, until I found smugmug. So what is the right answer to the question "would you pay to store photos"?
This isn't the kind of nuance you would get from an MT survey I don't think.
Exactly, this is very relevant. The type of person who stumbles upon your web app is isn't necessarily the type of person who decides a good use of their time is completing mechanical turk tasks. But as long as you're aware of this when considering your results, I suppose they can still be considerably valuable.
Clearly, the author got his $27.50 worth; how much that's really worth, though, in the long term, remains to be seen.