We tried this just yesterday at a couple Starbucks locations in Toronto for our new lines of 3D printed jewelry (http://www.hotpopfactory.com/). It was great, because we could hone in on specific demographics that interested us based on the location and time of day.
For the most part we found people weren't actually interested in taking any money for coffee (we didn't go the gift card route, we just approached people in line directly). When we offered to pay some people actually found it a bit off-putting. That said though, when you approach people candidly looking for feedback on a project you're personally invested in, people seemed pretty generous with their time.
We ended up getting great feedback on everything from the usability of our website, to the desirability of our product and pricing. We even ended up finding a couple new customers along the way. I'd highly recommend this experiment for any consumer facing product, it can be a bit daunting at first, but when used strategically it can be far more productive then having your head down coding in isolation all day.
I can't say exactly why it is, but I bet no one would have felt off-put if you had offered gift cards instead of cash. People in general will be much more comfortable taking a gift card than cash.
The person probably thinks "I can afford a cup of coffee so I don't need your money". A gift on the other hand is meant to be given away so seems alright.
I'll sidetrack for a minute and say I think the jewelry looks awesome.
Shooting-from-the-hip site feedback: it is a bit weird that some (maybe all?) of the product pages recommend–"you might also like"–the same product you are already looking at.
Love the product as well! My from-the-hip feedback:
* From the opening screen I couldn't figure out if you were Hot Pop factory or Platonix. After I got that you sell Platonix and Boreal I still don't know how you two are related.
* My #1 fear: How easily does this stuff break?
* Where's the story? You probably have a good one (3D-printing) but I don't see it here.
* Where's the gold? Annoying question I know, but people want metals so have you considered a printing moulds and smelting process?
Thanks very much for the feedback from both of you. Some of it mirrors what we heard during our experiment. We're gradually going back over the website to clarify the communication of some of these points more clearly and fix some usability niggles.
As for the "gold", its something we would love to do, and we plan on getting there eventually. For now one of the benefits of using 3D printing is that we can fulfill orders on demand and there is very little cost of carrying inventory, using metals ends up negating this advantage.
If possible, say that you're doing it as research for an agency.
People are nice and don't want to hurt your feelings. ("Yeah, I'd totally buy that! Oh, I didn't bring my credit card, sorry...")
Also, if you have competitors, try getting feedback on their apps as well. It can give you hints about what they are doing right and wrong, and how you can differentiate.
My team has done similar user testing in Starbucks quite a few times. Here's my $0.02: If you're willing to be bold, walk right up to people and ask, you don't need to offer free coffee or gift cards. We've found this is a better strategy for a few reasons: 1. It's cheaper. 2. It does not dramatically change the number of test users you can get in a given afternoon. And 3. You're more likely to get critical negative feedback if you dont put the users in the position of owing you for the coffee.
This. As much as it may work right now, because basically no one is doing it, as soon as it becomes a popular method, it seem to me it will stop working pretty quickly. Everyone doing it will become equivalent to a Jehova's Witness knocking on your door.
That said, I imagine I'd be more okay with it if every time I said yes I got a free coffee.
Aside: I wonder how different reactions would be in different countries/cultures.
In Mexico people try to trick you by asking something for something innocuous, like the time, and then ask you for money.
Also a supposedly common scam is to get personal info such as names and phone numbers in surveys and, later, to use this information to make fake kidnapping calls sound more plausible.
The upshot is that you learn to ignore people who try to get your attention. I don't think it would work too well here.
On that note, I'd imagine that "don't ask for ANY personal information except an email address, right at the end" is a pretty good rule for this sort of testing anywhere in the world. Good point!
The technique is somewhat self-limiting. Unlike, say, email spam or online annoyances, you've actually got to be there in person for it (though sufficiently large organizations could hire bodies for the purpose).
Still: the method has significant costs. That's a good think in the "preventing overuse" department.
To me, what made what he was doing appealing was that he wasn't directly soliciting people. He made the arrow and the sign. He wasn't harassing anyone in line. I think that makes all the difference.
This is kind of silly. Doing street level market research is not a new idea and approaching people directly isn't some huge revelation. There aren't so many entrepreneurs in the world testing ideas that you would ever be affected by this more than once.
Don't be afraid to approach people and ask for their help! Most people would love to know that someone cares about their opinion. There are huge online communities that run entirely off this principle, like Yelp or Quora. If someone's not interested, they'll tell you.
This is fine on the street, but not inside a Starbucks. As soon as you start approaching customers inside a business, they have the right (and obligation) to ask you to leave.
The OP had a table set up with the gift cards, and was only (I think) engaging people who came up to the table. Presumably he had the permission of the shop management to set up on the table. I'm happy with that provided he wasn't hogging the big table at peak times. If you are not interested, just don't go to the table.
I'd hate people walking up to me at random inside the coffee bar (UK reserve perhaps).
In any case offering someone something in advance will make them feel obligated in some way and for that matter even put them in a better frame of mind in general.
My feeling is that if you want honest feedback what I've always done is say "tell me what you don't like about this" rather than "what do you think". That way the person is freed to be critical.
Since you're getting a lot of hating here, I'll just say:
1) If someone does this in my favourite coffee shop, they're polite, and they take my first "no" for an answer, I'm completely OK with this.
2) Having done similar things in the past, it's my experience that - with the caveats above - most people are OK with this.
Obviously, some level of social skill, ability to tell if your presence will be seriously unwelcome, and general politeness and diffidence is rather necessary. Don't send the guy who never washes and keeps getting complaints from women at parties. But if you pass that hurdle, you'll be fine - and as someone said below, it's not like there are enough startups out there that this is ever going to become a major problem.
Along the same lines: I launched a social calendar app with my roommate for students at UMichigan and to initially gain users, we bought $55 worth of pizzas and to get a slice, all students had to do was download our app and register as a user--we setup in one of the busiest halls on campus and needless to say, the pizza ran out pretty quickly.
Looking back, it was great for the numbers and a was a good marketing strategy, but we didn't gain much else from it. I wish we had setup a demo or something of the sort so that we could have also received feedback. Asking vital questions about the product should be geared toward customers, not as much founders, and we learned that the hard way.
Some guys with an app paid for all food orders during a 4 hour block (10pm-2am) at a popular foodtruck at SXSW. All you had to do to get a free ~$7 burrito (some people got a bunch of stuff) was to download their app and create a photo album.
While this boosted downloads during that time period I have not looked at the app (I don't even remember what it was called) since. It would have been more memorable if they had discussed the app with me for the free burrito.
Exactly. Sating my apetite for a burrito at 1:50am will make me happy but there are better ways of generating more interest and in a way that people will remember for less money.
Interesting that sitting in a cafe offering people $5 cash for their opinion would be a terrible idea, but offering them a free coffee is brilliant.
To @vishl, I don't take food from people on the street. I don't trust it. "What if it's poisoned?" What you're thinking does happen - some people offer me free candies, Reese Peanut Butter cups etc as an incentive to talk to me on the street. I avoid them.
The coffee coming from Starbucks directly makes it safer.
I think psychologically the voucher is worth more because it's something they were already going to spend, it's a tangible saving. I'm not a social scientist or whatever, but logically I think that's how people may look at this offer.
I definitely agree that if you're not producing the consumable that's a benefit too (although of course they'd probably use $5 cash to buy a coffee).
'disgust' is interesting and not necessarily rational.
Give most people a glass of water and ask them to spit in it, and then ask them to drink it. Most people don't drink it. It's their spit, it's only just come out of their mouth, so why don't they drink the water?
From what I understand, market analysis can always be done on the cheap if you want to take some accuracy risk - there's a clear self selection bias here, along with a geographic one (there may a common socioeconomic background for the coffee shop patrons)
Also, if you care about kids, it may have worked better at a coffee shop in front of a toy store in a mall, or at the food court if it's near say a pet store, to get actual parents - there might be fewer parents at a typical starbuck. But that is important if you believe they are you client - ie the persons who'll make the purchase decision, but maybe it will be the kids themselves in this example?
Critic aside, it's a great idea for a generalist app, but it may be less than ideal for say a specialized app.
Also, you should consider the alternatives - how much feedback do you think the same $50 may have got you using the mechanical turk?
There is a sincerity risk, but if you get say 100x more feedback, with the law of large numbers, I think that at least the obvious flaws could be noticed.
>Also, if you care about kids, it may have worked better at a coffee shop in front of a toy store in a mall
Also if you're actually trying to get kids to interact with the app, you'll get a much better response if you have a woman with you. Parents are more likely to be comfortable letting their children interact with a woman.
> He never asked if I’d give him money today for it. He got feedback from someone he didn’t know would be a paying customer or not.
That's because verbal pricing feedback from random people tends to be terribly inaccurate. Most people actually have a pretty poor conscious idea of what they will buy and how much they would pay for a future product.
Many times I've seen people in focus groups say they would pay $hundreds for an electronics product, then reveal that (for instance) they just had to sell their stereo on Craigslist for rent money. (Translation: no way in hell they would actually pay $hundreds for the product.)
This sort of guerilla action is great for usability testing of a mass-market product because the utility of the feedback increases with the ignorance of the test subject. If your product can be understood and used by tech luddites, it will work fine for tech savvies too.
But pricing research needs much more rigorous look at valid data on potential markets and customers.
Too many early-stage companies focus on mobilizing people through — if you'll forgive the dreadful buzzword — social media.
If you can connect directly with your audience face-to-face, as Jim did, that's often a smarter strategy. I like the way Jim was able to segment his audience in a low-tech way (chatting with the parent-kid teams who are his core users/customers).
Yeah, you might be able to achieve a greater # of impressions online, but it's much harder to make those impressions meaningful and to make sure that they translate into action.
("Raising awareness" is complete bullshit. If people are aware but they don't take action, no one cares.)
I've considered standing outside on a hot day with a cooler of icecream sandwiches. At $1 each it's even more cost efficient, though people may not be as willing to stop and chat as they are in starbucks.
I imagine a future where instead of running the gauntlet of clipboard wielding public researchers, it'll be pseudo-entrepreneurs hawking their herp-derp OS app.
This sort of approach is a classic way of doing cheap usability testing.
If you've not done this sort of work before I'd recommend getting hold of the (short and entertaining) "Rocket Surgery Made Easy: The Do-it-yourself Guide to Finding and Fixing Usability Problems" by Steve Krug. It'll give you a bunch of useful tips on how to get the best out of our time.
On the other, other hand, you're giving some of his/her customers an interesting, novel experience where they get to feel that their opinion is valuable. Oh, and ensuring that they'll remember that Starbucks far more than most others.
Does this remind anyone else of the South Park episode "Free Hat", where Cartman writes "Free Hat" on the advertising poster in the belief that freebies are necessary to attract people?
Asking if someone will pay is probably as valuable as not asking. Most people aren't going to say something bad about your work, when you are right in front of them. And talk is cheap.
The main investment here is time, especially compared to online usability testing services (usertesting.com etc.).
That said, early in product development in-person feedback is invaluable and you can get so much more out of it than of a simple video of a person using your product.
Yeah, I like this. A founder I worked with did something similar but just gave out gift cards to talk to people about their needs in the space she was working in. Before having a product to show off.
It seems like a great way to get out of the echo-chamber a bit.
I've done this exact thing for images for a site I was building..
Most will say know but others are very willing and will even sign a release. It's much cheaper than purchasing photos from a stock photography site and much more fun.
For the most part we found people weren't actually interested in taking any money for coffee (we didn't go the gift card route, we just approached people in line directly). When we offered to pay some people actually found it a bit off-putting. That said though, when you approach people candidly looking for feedback on a project you're personally invested in, people seemed pretty generous with their time.
We ended up getting great feedback on everything from the usability of our website, to the desirability of our product and pricing. We even ended up finding a couple new customers along the way. I'd highly recommend this experiment for any consumer facing product, it can be a bit daunting at first, but when used strategically it can be far more productive then having your head down coding in isolation all day.