While working there, when they were interviewing, I would always go over to the interviewee to see how they were doing, and maybe give them pointers.

My conclusion was that this task is too hard for a significant % of programmers. Here are two memorable cases:

Guy was a mathematician in his last year. I went over, and saw some odd looking code. Turns out he was trying to first rotate the entire problem so that the line was the x-axis, because then presumably it's easier to solve. Unfortunately, he couldn't get the rotation right.

This other guy was a CS major in this last year. I went over, and he was looking at the MSDN docs and it was obvious he was searching for polygon related functions. Turns out he thought there must be a function somewhere in the MSDN docs that solves this problem, there's no point in writing it!

Speaking of the need to ask any candidate to solve technical problems on the whiteboard:

The best people not only welcome a challenge, they require it. All the great people I know would never work for a company that doesn’t perform a tough, formal interview because they recognize the company will mostly end up with bad employees.

I'm all for vetting technical candidates, but I haven't experienced many places that perform this process intelligently.

There's a web shop around here. I had a great interview with the owner, he was excited to hire me, even making overtures about how I was going to have a job in the next couple of weeks, etc. I just needed to do this one little formality, which was a "real-world" code test that was about three-five days of work. This is pretty excessive imo, and the place definitely didn't warrant that kind of effort for the atmosphere or the pay, so I gave up on it.

There's another company around here that makes niche software. The first step of interviewing there is a written test more than thirty pages long, covering C, C++, Java, and C#. The only people you see the whole time are the secretaries, never meet anyone from HR or IT or anything else. After the test, if you get called back, there are several interviews before a final hire.

The problem isn't that they're attempting to vet for competency, it's that the ways in which they vet are silly and arrogant. They think that theirs is the only company out there, that we should feel so privileged to get a job there that we're willing to dedicate weeks to their "quality control" measures.

In my opinion, companies should focus on the "smart and gets things done" prescribed by Joel, not thirty pages of trivia about Cs and Java. Such tests generally test memorization abilities and not practical thinking abilities; one can sit in class and memorize syntax and data structure implementations in hundreds of programming languages, but still write crappy code in the real world.

They should also realize that most of these guys who are walking CS textbooks are completely dysfunctional socially. I don't mean that these people are awkward or have bad hygiene or any thing like that, I mean that they can't work together with the other engineers on their team, they're focused on proving that their solution is right, and generally just lead to lots of in-fighting and drama. They think that their intelligence or above-average competence is a license to ignore decorum and be cruelly rude and blunt to those around them. We see this attitude in open-source a lot; how much worse is it when the mailing list is private?

Honestly I think the key to a good hire is getting to know the candidate. Attitude is 90%. One should also control for competence, but should be reasonable about it. Again, Joel's couplet "smart and gets things done" is a great motto for the technical recruiter.

you wouldn't accept "attitude being 90%" of a doctor's competence. A lawyer with 90 % attitude and 10 % competence will lose your case for you. Why should programmers get away with "attitude"?

There is no excuse for not being able to answer basic data structure questions if you call yourself a programmer. The fact that some Google interviewers layer on arrogance and excess doesn't obviate the basic idea.

"They should also realize that most of these guys who are walking CS textbooks are completely dysfunctional socially. mean that they can't work together with the other engineers on their team, they're focused on proving that their solution is right, and generally just lead to lots of in-fighting and drama. They think that their intelligence or above-average competence is a license to ignore decorum and be cruelly rude and blunt to those around them."

Stereotype much? I don't have any kind of CS degree, but I have met and worked with many people with MS and PhD degrees in CS, who knew theri datastructures and algorithms cold, and the ratio of good folk to dumbasses is if anything greater than among than the population of "developers" who wrote a welcome web page in PHP and think algorithms are unimportant.

(see? I just used your rhetorical trick in the opposite direction. Making weak arguments based on anecdotes is easy)

These guys are working on an algorithms/data structure heavy project. "90% attitude" is a guaranteed fail.

I haven't interviewed in quite awhile. The last time I did, every time I saw a relatively young group in the conference room my heart just sank. I knew that I was going to be spending the next hour talking about red-black trees and low level language trivia.

There has been a point in my life where I knew the implementation details for various sorting algorithms and exotic data structures. That was called 'college'. More than a decade into my career, I simply don't remember most of it. I've been coding at a senior-level (meaning, primary designer and implementation) for about 8 years. In that time I've only had to build a data-structure of any complexity once. It was the internal structure for a word processor (Quickword). Everything else has been deferred to standard libraries of the various languages I've worked in.

It seems to me that we should be trying to identify engineers who can think at the abstract level great software requires. Identifying that is fantastically difficult, however. To the point that I'm not sure if I know how to do it even today. I do tend to ask more design related questions (not logic puzzles). I've found that 10-15 minutes of talking about a specific problem tell me everything I need to know about a candidate.

I know what you are talking about. Not too long ago, I was "that guy" - the guy whose heart sank at such interviews and couldn't get what the focus on "trivia" was all about. I know how to write software dammit why ask me all these stupid questions about the fine detail of Red Black Tree implementations and language edge cases?

"There has been a point in my life where I knew the implementation details for various sorting algorithms and exotic data structures. That was called 'college'. More than a decade into my career, I simply don't remember most of it. "

I had to learn CS from scratch (because I quit working on enterprise sw after 10 years and started working on projects where such knowledge was important), without a formal degree, mentors, or any guidance whatsoever. So in a sense I have traversed the opposite arc from most college educated developers.

I first became a "in the trenches" programmer and then get a CS and math education (entirely self taught). I was an "architect" at ThoughtWorks (At TW, "architects" code every day) ar at least they used to when I was there) , working on large enterprise systems, when I stopped doing outsourced enterprise sw and started working on Machine Learning/AI projects and found myself neck deep in hairy math with no clue what was going on. I have sat in 3 hour meetings not comprehending a single sentence because the other guys were speaking "math-y programming" and I knew only "hands on programming". It is a terrible, sinking feeling.

Which is all just away of saying, there are projects on which deep algorithmic/data structure knowledge (and other hard things) is important. And if a company's success or failure in such a project depended on finding people with that knowledge they would be foolish to ask "abstract" questions at interviews.

Whether this is important enough to be probed in an interview or not depends on what the company is doing. If you are building simple db backed websites you can get by with understanding lists and hashtables. If you are building databases (like rethinkdb) is, then being very thorough the nuances of RB trees, networking, OS internals etc becomes significant.

I agree that the interview should test for what the skills the job would need in real life, if that is what you are saying. I was just irritated at the stereotyping of programmers with deep CS knowledge as lacking team work and being too abstract and so on. I know from personal experience how hard that stuff is. And it is orthogonal to "in the trenches " knowledge and experience. You can be very good at both.

However, I would want to know if you have the general idea that the way to solve some problems is to have a sorted collection or a ring buffer or whatever, even if you add on "but i'd have to look up the implementation details."

And most people defending them pretend like the trivia and minutia never happen.

As with anything else, I think a formal development test or question is something that can be done well or done poorly.

You just can't generalize the applicability or usefulness of something practiced by a million different people in a million different ways.

Still, the analogue to doctors or lawyers is quite different. Most programming languages are pretty similar and once you get the basic concepts down you don't need to radically modify it to switch from one context to another. Information and manuals about programming languages, troubleshooting, etc., is available plentifully online, and someone with a basic grasp and an attitude of teachibility and eagerness to learn can make a lot of progress very quickly.

Things don't work the same way with lawyering or medicine, but probably the most important difference is that someone isn't going to be sent to the jail or the morgue because your programmer wrote a naive implementation of something (generally).

I agree with the basic idea of reasonable testing, which is why I said that it "sounds good on paper". It's just that most places don't do it right, so I don't get excited anymore about workplaces that claim to test for competence.

I think some of the jobs I've taken which I later looked on as regrets could have been avoided if I was paying more attention / placing more weight on the poor interview process.

According to your criteria, you're thinking more of 'potentially world class', which I guess is more in line with your price range. The world class guys work for Google.

Depends on your budget and appetite for risk. If you have the budget and want to reduce your risk, IMHO there's no better indicator of future performance than past performance.

I've got a pretty good track record. How do I get to work in your world? :-)

IME, for regular employees, the difference in pay between a respectable senior developer, technical lead or project architect and a junior developer straight out of university is perhaps a factor of 3-4x. Unless you go into management and rise to a senior level, it pretty much plateaus after that in most places.

On the other hand, the difference in productivity between the same types of people can be a lot more than 3-4x. By the time you factor in the improved communications and general reduction in overheads from having a smaller team of individually more productive people, top people are easily worth a disproportionate amount, if only you can find them and convince them to work for your company instead of someone else's (including their own).

Are you saying people with great track records are likely to be very unproductive? That seems unlikely ...

Presumably everybody inside the company knows what rank everybody else has, so to that extend the salaries are indeed open; but from the applicant's perspective, all you seem to have done is renamed "negotiate salary" to "negotiate rank".

In practice, when you have only three levels, most people have a good intuitive feel for where they fall, and it's usually in tune with our evaluation. I agree that this system has its problems, and we'll try to work out the kinks over time. It will be interesting to see how it evolves.

  Along the sames line, if there is a variation, then write it down as the norms or minimums, at a minimum, they will be able to do 3 out of this 6 things, etc.

Of course the downside is that if one employee gets a raise, everybody gets a raise (unless the employee gets promoted to a higher position). From the POV of the employees this is a good thing, because that means everyone is fairly compensated.

Or does a different scale apply to you? If so, fair enough.

At a place I worked before, I was pretty embarrassed when I found out I was earning 15% more than a fellow colleague on the team who was just as good if not better than me, purely because I happened to join when market rates were X, whereas he joined when they were X-15%.

I suspect this is a big reason companies don't want employees to talk about compensation.

Though where I am now, we have an "everyone gets the same raise" policy, combined with hiring practices that ensure substantial variances in skill of employees. Not a great situation either.

So I think the fixed scheme really depends on getting your hiring absolutely right, if you screw up, you have to be willing to face up and let the person you screwed up with go.

As founders, we get the lowest full-time salary possible on the scale (currently $65K).

if you screw up, you have to be willing to face up and let the person you screwed up with go

Absolutely. That's one of the benefits of the open system - it prevents the company from keeping bad apples.

1) You wouldn't recognize a Dennis-Ritchie- or Claude-Shannon-to-be if they would spit you in the face, because at the point in their lives where they would consider to work for you (if there ever was such a point), their achievements would not stand out nearly as much. Robin Milner died saturday and I just learned he didn't even have a Ph.D. Pretending you'd be even remotely capable of recognizing

2) What gives you the idea that people with such potential would apply at RethinkDB? You are describing a group of maybe a thousand people worldwide. Many have academic ambitions; others are interested in completely different subjects. There are more than a thousand companies looking for these people. If you'd manage to hire one, you'd be lucky.

3) Even if you were able to recognize the ability for such achievements and get them to apply, it is doubtful they would able to achieve those peaks in your corporate environment. Claude Shannon didn't have to satisfy any customers. He could spend years developing a theory, whose practical value was only clear after it was sufficiently developed.

4) A corollary of 2): the people you've hired so far are not the kind of people you describe.

It would be nice if, just for once, someone would try to describe the people they actually hire, instead of pretending they only hire potential Field medalists, Nobel laureates and other people of such fame that all of their actual developers know them.

  [..] I am often unable to solve complex problems during 
  the interview because of the nervous pressure. But I don’t
  want to hire people as good as me, I want to hire people
  much better than me.

6) And finally, about that Spolsky quote: I'm convinced he put it there to make sure no one else would hire the ones he would like to hire. If you don't have any doubts about a candidate, then you haven't pried enough to expose their weaknesses. That, or they are too glib. You should have doubts about every single one of your interviewees.

PS: this article, as well as the ones by Spolsky and Yegge referenced in the article, contain sound advice. However, all three of them go overboard at some point, which inspired the above. This comment might be considered as tearing apart a specific point, while ignoring the larger whole. This is not my intention: the article is worth reading.

Oddly, you said one thing that just didn't sit well with me. I re-read it several times trying to figure out why I was so uncomfortable with it:

Claude Shannon didn't have to satisfy any customers. He could spend years developing a theory, whose practical value was only clear after it was sufficiently developed.

This is exactly the opposite of my thinking. I've been in academia and in the trenches. I prefer the trenches.

Where I come from you don't have anything until you're able to satisfy customers. Building theories, tools, and new approaches is great, so great, in fact, that some of the best breakthroughs I've ever seen were because a customer demanded something people like us never imagined or thought possible.

I love satisfying customers with whatever I have in my back pocket. And lots of that stuff was developed by the giants whose shoulders I stand on. I always imagined that they were satisfying their own customers, but I guess quite a few of them were just doing "basic research".

You reminded us that there needs to be room for both the "trench warrior" and the "dreamer". When you're building a team, you need to decide who would be better for the role at hand and approach the recruiting accordingly.

Claude Shannon didn't have to satisfy any customers. He could spend years developing a theory, whose practical value was only clear after it was sufficiently developed

is historically wrong; certainly reading the wikipedia page on Shannon makes it sound like he developed his ideas while doing quite practical work during WW II.

Someone please correct me if my impression is incorrect.

I think the difference you're describing is in direction, not position. On the side you prefer, you figure out what someone wants, and heads toward that. On the other, you head towards something you want or feel like doing. I assume you've found the right project for you when the sides are the same.

You wouldn't recognize a Dennis-Ritchie- or Claude-Shannon-to-be if they would spit you in the face

Nine out of ten times, sure, nobody would recognize Dennis Ritchie in the rough. But if you're hiring the first five employees, you don't care about nine out of ten - you care about that one time you do recognize a promising candidate. Every selective process has a false negative rate, there is no way around that. Amazing people get turned away by all kinds of smart people in all kinds of places all the time. If you have ideas on how to lower the false negative rate without increasing the false positive rate, I'd love to hear how you'd do it.

What gives you the idea that people with such potential would apply at RethinkDB?

Because they already have. Smart and driven people apply for the same reasons smart and driven people do anything - they're interested in the problem, and have an innate desire to solve it. In any case, if you sit around and wait for great people to apply, the sourcing part of your recruiting process isn't done right. If you want great people, you have to go out, find them, contact them directly, and then do everything in your power to close them.

People that botch their interview because they are nervous can't be better than you?

Of course they can, but again, it's about increasing the false negative rate to lower the rate of false positives. I can afford to turn some great people away, but I can't afford to hire someone who isn't great. Any software startup that works on hard technology has the same constraints - if you hire people that are anything less than stellar, you significantly lower your chance of success. In a startup you (mostly) don't have geographical limits for your customers, but the corollary is that you also don't have geographical limits for your competition. When you're targeting every customer in the world, you're also competing with every company in the world. Going into that with anything but the top-notch team is suicide.

Because they already have. Smart and driven people apply for the same reasons smart and driven people do anything - they're interested in the problem, and have an innate desire to solve it. In any case, if you sit around and wait for great people to apply, the sourcing part of your recruiting process isn't done right. If you want great people, you have to go out, find them, contact them directly, and then do everything in your power to close them.

I'm going to hazard a pretty safe guess that one of the reasons you can blithely assert this is that you've never actually personally known someone with the abilities of a Dennis Ritchie or Claude Shannon. It is easy to overestimate the abilities of the best you've known if you've never known the truly best.

This is not a slight on the abilities of some of the developers you are lucky to have. It is a well deserved compliment to some of the giants of computing.

> Because they already have.

So how much of the company do they own? I ask because I have noticed that people with the combination of talent, experience and work ethic you are looking for are rarely content to be just someone else's employee. They'll either be founders of their own company, or hold a low-employee-number position in someone else's business with plenty of stock options, or work as independent consultants, or otherwise do something where what's in it for them is on a par with what they bring to the table.

I've always preferred formal questioning (i.e. asking questions, working with whiteboards etc.) BUT informal setting (so jeans/tshirt, sat on the sofa's with the whiteboard, small talk and chit chat etc.)

This is partly because that is how the company is - and so we interview like we work - but also because I used to hate interviews where you sit over a desk from someone in a suit and tie...

Other good advice I can think of off the top of my head: go out to lunch with your potential candidates. I know the OP advises against that (and agreed, you shouldn't get too personal/involved) but an informal lunch with all the candidates together can be a great ice breaker and a way to judge their personality/interests etc.

As an interviewer, I couldn't care less what the candidate is wearing or what he looks like (so long as he has basic respect for other people).

I'd much rather see (both as a candidate and employer) a group lunch with one candidate and a few members from the team they'll be working with - ideally after they've had at least one interview, and not including the person who'll be interviewing them right after lunch.

I doubt anyone can reliably figure out in a few hours if someone will be a productive addition over a few years.

A lot of my ideas run aground when the issue of matching the interviewee's expectations comes up. If good interviewees won't take a job without a broken interview process, perhaps all you can do is supplement it?

And the results are many times not those wanted ... those looking for superstars end-up with subpar performance, while those looking for grunts will end up with good people that will leave sooner or later.

I once did an interview were on one task I had to program a DFA for a Lego-mindstorm with the technical questions being pretty heavy for the kind of work they did ... like if I were at Google, how would I model an algorithm for finding similar links (with the discussion flowing to sparse matrices, distance functions, clustering)

... only to find out that they were a company that did online quizzes, with the developers involved building them with a semi-automated process involving an Office-plugin that generated PHP code and some fixes here and there to make it work where the generator failed. One day I'll submit the story to DailyWTF. That project manager interviewing me had to be on crack :)

Not saying it's the case for RethinkDB ... they look like a fun project. Just wanted to share my experience with the current state of affairs.

I put up with it because it was for my first job and I was desperate. Now, I don't do all day interviews, long programming assignments, recite the alphabet backwards while parachuting and coding a quick sort on a laptop, etc, at least not for the typical job. I might jump through some hoops for the dream job, but most jobs just aren't.

I especially like the idea of only hiring people that intimidate you ... so long as the only intimidating qualities are their skills and knowledge!

The idea to always "hire people smarter than you" has never really made sense to me. If you always hire people smarter than you, won't everyone eventually realise they're working for an idiot and leave?

The people I've enjoyed working the most for have been people around the same intelligence as me, IMO.

During an interview, put your candidate outside his comfort zone, if you can, and it will tell you much more than any formal bullshit.

I hope you're not trying to imply that coding a binary search is challenging for most of your interviewees. :-)

"Although the basic idea of binary search is comparatively straightforward, the details can be surprisingly tricky…" — Professor Donald Knuth

When Jon Bentley assigned it as a problem in a course for professional programmers, he found that an astounding ninety percent failed to code a binary search correctly after several hours of working on it, and another study shows that accurate code for it is only found in five out of twenty textbooks (Kruse, 1999). Furthermore, Bentley's own implementation of binary search, published in his 1986 book Programming Pearls, contains an error that remained undetected for over twenty years.

http://en.wikipedia.org/wiki/Binary_search_algorithm#The_alg...

I do want to see a candidate speak about its complexity (demonstrate it at least intuitively), and about the impact that the partitioning function has.

But these are things any CS college student in his second year should know about (otherwise they've wasted their time in college, and frankly I haven't seen many of those).

I'd really wish we'd stop with the "mine is bigger than yours attitude" ... there are lots of good developers out there. They are just more specialized in other domains, and finding the right match is indeed hard ... but the faulty interviews we are so accustomed with are to be blamed.

Also, the page provided by sanj contains a recount by a Google employee where he and most of his fellow CS Ph.D. students at CMU were unable to provide a correct implementation of binary search.

Sometimes smart people make dumb mistakes, especially when rushed or stressed.

In Java for example, an "int" is 32 bits, even on 64 bits machines/VMs ... where the address space can reach 16.3 million terabytes.

So this little bug can byte, but only if you're manipulating large datasets in memory. And then it's easily discoverable, on top of managed languages anyway (since the index used on an array can't be negative).

    if (a[i] < a[j])

http://googleresearch.blogspot.com/2006/06/extra-extra-read-...

I've been asked twice to solve out-of-interview puzzles, and once an in-interview puzzle. In each case, I asked for other people's (anonymized) solutions/times. In both cases, this was with Common Lisp using companies. Your experience may differ, but:

* Everyone else took shortcuts in out-of-interview puzzles. Most wrote almost no comments (a couple had overly chatty ones), nor was the code packaged in an easy-to-run way. Even most good solutions would cut a corner by (for instance) assuming that the input was magically pre-parsed from strings into symbols.

So, there'd be a couple of good-but-somewhat-unpolished answers, a few which were visually messy and over-simplified the problem, a couple where it wasn't really solved and the programmer maybe apologetically produced a best-attempt, and one which wasn't even in a known programming language. (The last one deserves the most sympathy, as he/she almost certainly had mental problems.)

In contrast, imagining that everyone else was a hyper-professional Programmer O' Doom, I packaged it all with ASDF (including instructions) and polished it as if, say, generations of island-stranded maintenance programmers would use my solutions to figure out where to sit in order to not be eaten by a cannibal.

One thing that probably goes over well is to point out a possible problem with a part of your code -- and a justification. Like a comment saying, "Note: Some Lisp implementations don't perform tail-call optimization. However, for real-world inputs, they would nevertheless suffice and be more maintainable than alternatives."

* People are human, and often have problems with in-interview puzzles. Part of my mind mysteriously blanked for a short while when asked what I felt was a trivial question -- permute a list of 3 elements (performance irrelevant, run the solution in Common Lisp). My imaginary Programmer O' Doom no doubt typed it from finger-memory in seconds, after whimsically sketching a solution where she randomly permuted the elements until enough unique ones were uncovered, or enumerated them all at read-time or something.

So, I stammered out an apology because I took a few minutes, jarred and ready to crawl into a hole for being unmasked as a fraud. Well, it turns out that the previously fastest solution took over an hour. (Of course I remained a fraud, albeit a still-unmasked one...)

Maybe it helps to think of such interview puzzles as a symptom of society's diseased way of working; then there is less stress. They're fundamentally mindless and people often derive an elitist thing from them; it's hard not to. I may welcome them, but it's also the case that each side whores out natural curiosity for the sake of filling some corporate job.

That's intended to mean that the the recursive solution is more maintainable than alternatives. (Or so I glibly claimed...)

And of course, I meant "masked fraud".

This made me smile as I'm actually at work reading this article on my employer's dime