I've done this before. Not a mobile app but a unit of hardware that is at bars (think those digital jukeboxes). During the final stages of testing we mounted one at a friend's place and set it to debug mode (so anything with a magnetic strip passed as being a valid credit card).
After a few hours of drinking and watching friends we uncovered some interesting usability concerns. The only problem was we never designated someone as a sober-recorder so the notes were almost entirely useless.
That's hilarious. I should've mentioned in the section on testing drunks that the moderator should stay at least somewhat sober :) but I'm glad you got some insights at least.
Interesting article and very relevant since I've been conducting bi-weekly usability testing sessions. I've had a lot of success using IamExec.com to find participants for one-hour loosely scripted sessions.
Some feedback:
It's hard to tell what's a link as the purple really blends in with the black.
The dichotomy between "newbie's" and non-newbie's is a bit of a turn-off and could have been more fluidly discussed.
Wouldn't completing a list of goals contradict "not ask leading questions"? Why not just give the app to the user and tell him/her to do his job with it? Instead of telling him what to do.
In the end, the app is supposed to help him accomplish what they want, not what you want them to do.
No, it's not a contradiction to have an agenda in mind and to also try not to ask leading questions.
An example of a leading question in this context would be like asking them "do you see the button labeled 'learn more' to your right?", which really doesn't test anything other than the fact that they're literate and their eyes work.
Compare this to asking them "now what do you see on the page and what do you want to do?" Maybe they'll want more information and hit the learn more button, maybe they want info and don't see it, or maybe they want to do something else entirely.
And sure, you can just give the app to the user and see what they do, without writing up goals in advance. That's the unscripted method that Mark Hurst popularized and I linked to. Here's an even longer article by him on the same method:
But that's not appropriate for every situation. Unscripted stuff is great for getting the lay of the land and uncovering as many problems as you can, or finding out if they'd use your app at all even if you got rid of all the obstacles. Often times you already know about specific usability issues, however, or you want to make sure they cover all the bases in an app with a lot of moving pieces.
It depends on the method of testing. If I'm doing an in-person session, I'll record the session with Silverback, review the footage in there and make some notes along the way, then build a quick powerpoint deck with major findings and selected video clips.
If I'm using usertesting.com, then that stuff is all built into the app itself and you can just send the videos around, which'll be annotated with your feedback. Each note is a clickable timestamp that takes you straight to the time slice you want to highlight.
Either way, it's important to not only review the test afterwards but to also pull out clips that you can show the team, because video has such a motivational impact and cuts through debate. It helps if you can take a few minutes in imovie to make a "lowlights reel", where different people all run into the same obstacle.
And if I'm doing the drunken bar testing, I just bring a witness to watch and compare notes with, hopefully also somebody sober :)
I've done this before. Not a mobile app but a unit of hardware that is at bars (think those digital jukeboxes). During the final stages of testing we mounted one at a friend's place and set it to debug mode (so anything with a magnetic strip passed as being a valid credit card).
After a few hours of drinking and watching friends we uncovered some interesting usability concerns. The only problem was we never designated someone as a sober-recorder so the notes were almost entirely useless.