I think all you've discovered here is that users find it annoying to be taken through a bunch of signup screens before finding out a website is not free.
You've drawn a lot of conclusions out of a very small experiment with a lot of uncontrolled variables. I'm glad to hear you're thinking of testing again more carefully.
Your points are valid, although I fail to understand why you feel it's necessary to adopt such a condescending and superior tone both in your comment and in your article.
It's hard to tell someone they failed at something basic without coming off as condescending or superior. I'll try to do better next time, though. I don't like posting negative/attacking stuff, generally. :-/
I have had a hard time with the tone thing in the past as well. One thing that worked for me is trying to think about how my favorite educators in the past would try and reach out to me to "lift up" my level of skill instead of "put down" the work that I had done. I know that you are correcting the OP because you'd like to help, but your help is more effective if it is seen as a helping hand and not a disapproving glance.
A basic formulae could be a) acknowledge the positive attributes of the attempt (it is great to see you trying to optimize, and you are right to keep detailed numbers about what was going on, your funnel analysis was also spot-on.) b) Point out that he could do better. This emphasizes the growth potential instead of the shortcoming (one has an implicit expectation that the person can and will do better, the other is frequently taken as a rejection of the person.) An example might be: "you ran the test, but your test would be more helpful if you did it in a more controlled way..." c[optional]) suggest additional resources so the person can follow-up at their own pace, in the manner they prefer. Ex: (see this great post about having a control when testing your site...)
Now, if someone has ignored your previous feedback and it is your obligation to help them, things get a bit more nuanced.
Instead of "terrible analysis" you could talk about where it has "room for improvement" -- you know "Glad to see you making an effort to figure out how to improve the site. That's a great start. I know you already know (because you said it in the article) that it wasn't a very well designed test run. Some suggestions on how you can do it better the next time around: (blah de blah, all kinds of intelligent stuff here). Thanks for sharing."
Or something like that.
I'm fond of the quote "I'm too truthful to be good". But I also am fond of the remark about how folks who "love the brutal truth love the brutal part more than the truth part". So I've spent a lot of years working on trying to develop some desperately needed velvet gloves. If I can manage it (er, at least somewhat -- though this post might not exactly be a good example), surely you can. :-)
Haha, you responded before my post, saying roughly the same thing. Cheers.
The "too truthful to be good" has a bit of danger in it; it seems to juxtapose goodness and truth, and I don't think they're mutually exclusive. I could just be misinterpreting it, though.
It's a tongue-in-cheek movie quote (that reverses the usual "too good to be true" saying). I am infamous for being excessively honest. I also have a history of having folks tell me what a "good person" I am and such, something I call "accusations of sainthood", which I am not fond of. I guess I liked the quote for the idea of suggesting that these folks who knew me to be politically incorrect and excessively honest really should lighten up on the accusations of sainthood because "I'm too truthful to be (that) good". :-)
Says the guy who makes his argument with the phrase "I fail to understand." Let's face it, it's hard not to be condescending when we know we're right. For one thing, it has a lot of visceral appeal. For another, it's a strong rhetorical position for convincing people not familiar with the subject matter.
Would you have taken his comments as seriously if he would have adopted a style of communication where he would be more forthcoming? That other style would have included all kinds of 'nice' words, that distract from the message and provide all kinds of room for you to think that 'it wasn't as bad', 'I only need to polish/fix some details'. I think swombat has done you the favor of driving the seriousness of your mistakes home, making sure you don't forego thousands of dollars, just because he was trying to be nice.
In a face to face conversation, you can be nice all you want, because you have both time and non-verbal oppurtunities to make the point. The internet is different, for better and for worse.
I'll just chime in here and say "I agree 100%", since all replies to so far (excluding the OP's) are basically calling you a moron. :)
Yes, the test had some serious flaws, but I imagine that removing the free plan was by far the biggest factor in the results - especially because there's no demo and no trial. I can't imagine signing up for a pay-only service if I can't even try out a demo of it before entering in my credit card.
I found value from the post, even though, yes, it is flawed.
As others have pointed out, my post came across as overly harsh and condescending. To balance this out slightly, I'd like to point out that I found out about your article not via HN, but via my RSS reader, since I follow your blog! Keep up the good posts! (the ones for which I subscribed!)
look mate, I've never met you and I have no idea what you're really like, but in your comment here and linked post you come across as quite a dick; one of those people who feels the need to publicly humiliate others in order to demonstrate their superiority
Normally I don't have a problem with it so I think it's related to the current traffic. I deactivated the plugin until things slow down a bit. Thanks for pointing it out.
Rather than doing this, I opted for a resource-based trial for historious. Free users get 300 bookmarks, if that's enough for them, that's great, if not, they can pay to get more. This way they don't put very much strain on the service but can still use it.
Another good idea, if your app can work with it, is to not let free users create any data. For example, for a website screenshot service, you could let free users access screenshots that paid users have created, but not create any of their own. This makes free users almost free to host but doesn't work for many apps.
This is a good idea to try, but November and December are weird months and it might not be reasonable in your space to assume they are comparable. I work in comparison shopping where it's really not true. You mentioned A/B testing, and I think you should try this again as an A/B test so that you can eliminate seasonality.
Just curious, what is your reasoning behind requiring a username and an email? I've been experimenting lately with super minimal signups and the two seem more and more redundant for most web apps, but I want to hear your thoughts (and maybe it's a potential optimization point?).
We just did phone number + password only for our SMS app textweight.com and the sign up rate is pretty amazing. I need to do some A/B testing with requiring e-mail.
Crazy thought: phone number only, then after they are in, require they enter one? If they don't set one or forget, they just get a text and reset it? The psychology of this is interesting because on one hand they might feel less safe but on the other they might not care and just want to jump right in!
It was a part of the restful authentication plugin I originally used to set it up [1] :)
For this app its necessary because I allow users to share the timelines they create and I display their user names on the shared timelines. Your train of thought is good though.
Yea public facing stuff seems like the only time it's really needed. One way is maybe to defer that until they actually share something and then have to pick. I sometimes take their username from the email (before the '@') and use that if necessary but it's not a great solution.
I'm pleasantly surprised to see a good percentage of your free users upgrading to the paid version. Why don't you test a time-bound free trial before switching back to freemium?
I think your example timeline doesn't show off a lot of the cool features you have, and that is unfortunate. In your future A/B testing, you may want to explore a timeline that includes pictures... perhaps of something like the Homestead Act or such.
I noticed that you changed your colors to a lighter color.
From what I see, many logos can be modernized so easily by switching to "pastel colors", or colors with more white in them. These are duller, but in my opinion, much more modern.
You've drawn a lot of conclusions out of a very small experiment with a lot of uncontrolled variables. I'm glad to hear you're thinking of testing again more carefully.