Is this a major business problem for you? Are you trying to solve it using actual money and process or just winging it really hard every time? There are known programmers who are really good (USA Computing Olympiad) and some of them will be articulate and you could find someone who's a good fit and hire them as a consultant to help tell the difference... though that's only off the top of my head.

Every time I hear somebody describe a "big problem" they've been "trying very hard to solve" I wonder whether they've focused on it enough to (a) step back and think about possible tools to make the job easier (b) resort to professional specialization (c) make it the job responsibility of a particular person or (d) spend actual money.

So, maybe this will come as a shock, but "computing olympiad" success doesn't correlate with success as a professional programmer. There's so much more than raw intellectual horsepower to being a good team engineer that it can't be captured with any one test.

Which is to say, the parent is right. Identifying good programmers is a hard problem. Harder than identifying people who do well at coding contests.

IOI and similar programming competitions do not select for good programmers. They select for people who can solve small algorithmic problems efficiently in a short time period with throwaway code. There is some overlap, but also a lot of perverse incentives. In professional programming, the tortoise usually beats the hare.

(I have competed at a national level and won a national college-level programming competition myself, and don't really take it all that seriously.)

I'm not just trying to find talented programmers. I'm trying to address the more general problem of figuring out whether somebody is good at X. You're right that the easiest way to judge on a case-by-case basis is to ask known experts. But since this is/was a cognitive bias of mine, I'm trying to overcome it myself.

I have spent a lot of time, done plenty of reading, and spent my own money on this problem. Like you said, I actually have paid experts to give me advice on how to identify talented people their field. More frequently though, I'll network and ask in an informal setting. And as a result, I have managed to severely correct several biases I had, such as thinking that anyone who couldn't do basic Math must be not be very bright.

One thing that's relatively easy (i.e., only took me a few months of effort) is learning how to identify generally smart people who learn fast. (I would actually divide this into learning social systems quickly vs. learning technical systems quickly.) If two people are entering a field or starting a job at the same time, I can generally predict which one of them will learn faster after a 30 minute conversation. The biggest surprise was that very smart people can be catastrophically bad at things "normal people" would consider basic-like eating spaghetti or not offending their interviewer (or even realizing they'd offended someone.) Before I tended to assume that fast learners would be good at most things and exceptional at a few things.

A more challenging problem: say I meet a lawyer, or doctor, or anyone with a lot of experience in an area I don't know much about. How can I tell if they're genuinely good at what they do? So far, general intelligence + percentile-based accomplishments ("I boosted revenues by X which is better than 95% of marketers...") + asking them to walk through a real life situation has been my best predictor. I'll validate this by asking known experts afterwards or looking up the person's career track.

Even so, it's not a particularly good predictor, and asking a real expert works much better. But by looking at actual data, even if it's relatively weak and anecdotal, my ability to accurately judge competence has become much stronger over the last few years. I think this is something that most businesses could benefit from, but the more common response seems to be throwing up their hands and complaining about the lack of qualified people.

Edit: when I said "how we judge talent is actually one of the biggest problems", I should have specified "we as a society." It clearly came across as "we as a company", sorry about that.

People want a programmer who can only juggle 3 items, but does it so well they are juggling chainsaws. That's easier to identify - it's their production (portfolio = chainsaws) and the quality of their production (referrals = not missing any arms). If you can verify that people coded what they claim they coded, it should be fairly easy to tell who the good programmers are.

If you're having trouble telling who the good ones are, you haven't found any yet.

To extend the analogy, there are plenty of jugglers who are capable of juggling 3 chainsaws, but find that they can only get paid to juggle 2 plastic balls over and over again. Variation in their craft is punished, not rewarded. If you only look at whether they've juggled chainsaws before, you'll miss out on a lot of great talent.

I've seen many, many people who are much better than their portfolio or past accomplishments indicated-especially people in their 20s who've just happened to work at crappy companies and end up on bad projects.

Someone who's never juggled before may not be capable of juggling three chainsaws at first, but you can look at their athletic record in other areas, examine their flexibility, coordination, and willingness to practice, and use these to predict whether that person will be able to learn how to juggle chainsaws. How do you figure out which of these areas matters? By asking existing expert jugglers.

This hits too close to home. I am trying to figure out what to do about it.

In my case it came down to the realization that the standards people use to judge your skills are very arbitrary. Once I went out and got a few certificates I started getting job offers. Nobody cared about the fact that I was smart enough to learn SAS in two weeks or make my previous employers large sums of money, they only cared about the fact that I had the certificate. C'est la vie.

In every industry there's some sort of bar past which people start treating you like a real person. Getting past this bar often has little to do with merit-think having a degree from a "prestigious" college or having exceedingly specific previous experience. Try to figure out what that bar is in the field you want to work in and how you can show that you've cleared it. Programming is a diverse field but a few bars I've seen people use (not saying I agree with these) are "has experience in Ruby/Python/Functional Programming/my favorite language", "has contributed significantly to open source", "recommended by someone I know who is competent", and so on.

The last one is IMO the most valuable bar to clear. The more you can get out and know people, and show them that you're competent, the easier it is to get better jobs and better projects in the long run.

Hired to be a programmer. Showed up and they decided I was going to be a business analyst. I've got significant experience in .Net, Scala and NLP but I can't even get companies to reply to my applications. I despise my job but I can't quit yet and I can't find another job. I'm working on side projects to build up my portfolio but I right now I just really hate waking up and going to work each day.

My advice: keep learning cool stuff, go to networking events (e.g. meetups, especially related to NLP, machine learning, and Scala) get some job leads there. Cold-emailing your resume rarely gets you anything, especially when you're in a career sand trap. But you have a skillset that, if you're strong in what you've cited, is desirable.

You will eventually get a feel for how much investment you need to put into your day job to keep it, and you can use the remainder of your time (which may be 20-35 hours of your work week) to keep current with the skills you want. If you're writing code on company time, be careful and make sure not to use it for any closed-source purpose, because you don't want your employer asserting ownership.

You can get out of the sandtrap but you'll have to break the rules to do it. Stealing an education from a boss feels dirty when you're young and naive, but it's a necessary survival skill and, in a world where bait-and-switch hiring is common as dirt, not at all unethical.

I do extremely little for my job, so little I honestly started to wonder if I was missing something. Then I slowly realized my coworkers are morons. A few weeks ago, I was asked to create a directory structure for a bunch of incoming data files, a few hundred directories all told. Another person was tasked to do the same on a different server.

I spent about ten minutes on it because I wrote a shell script. He did it by hand and spent all day. Which, of course, is what I told my boss it took me as well. I just happened to use the rest of my time to read Akka in Action.

I can't promise equity in a company, because this idea was looking more towards being a lifestyle business. Revenue/profits of operation would thus be the thing at stake.

But I agree with your basic point. If you can get good at spotting talent that has "flaws" that would disqualify them from the more typical elitist hiring mills, then you have a huge advantage.

What it comes down to is that the engineers are better suited to evaluate engineers. Obviously, you've got to have final say as leader of the company, but the engineers are your best asset here. This is why good VC's always have a luminarie in their back pocket to send out to evaluate tech before an investment. Even technically competent VC's would rather send an experienced database developer to look over a new DB company rather than rely on their now 15 year-old experience.

Depends what you're building. If you need cutting-edge programming expertise or knowledge of computing's deep magic, then yeah.

If you're building a CRUD database or a local-mobile-social app, you don't need the expertise that much.

This I disagree with. Causes of success and failure are hard to tease out, but people know when software sucks. Also, if people can't use it, then all the cleverness in the world (in optimizations, for example, or in feature set) doesn't matter. It still sucks.

But the broad middle is a muddle. People with talent that has been squandered by circumstance, people without talent who have succeeded due to accident and luck, etc.

That's very true.

This is part of why I think Valve's self-organizing open allocation is superior. Management rarely knows who the best programmers are. The group of programmers, if they're good, usually can figure out an appropriate leader on a per-project basis. This doesn't generate the permanent, entitled leadership that management wants to see, but it gets the job done.

You also need to create room to fail. If the software project can't be saved, let it die so people can allocate their talents and energy to something that has better odds. Give the architects respect for trying and ask them to write up the challenges they encountered, so as to keep the knowledge in house. If you fire people for failing projects, then you lose that knowledge and will probably fail in the same way again.

You talk about software that sucks. However, one of the quirks of software is that sometimes even when it sucks it can succeed and even when it doesn't suck it can fail. For example, technologically google wave didn't suck, but in terms of actually providing useful features for people that justified its use, it didn't have a leg to stand on. Then look at wordpress, which started out sucking but because it was open and because it had developed a strong community around it ended up getting better and better to where it was finally sort of decent. Or look at PHP. As a language it definitely sucked at the start, and there's a strong argument to be made that it still sucks. But it is perhaps the most popular language for web development in history.

Most software is even more difficult to determine success or failure with because even though a software project might not be a success immediately it could be a success down the road. Another case in point would be the Mozilla Project. At the outset it sucked, but eventually it became pretty awesome. How much of the awesome of today's firefox is rooted in the code from the early days of mozilla and how much is due to subsequent dev. work? How do you tell the difference between software that sucks because it is rotten through to the core and software that sucks because it has a layer of crap on top of awesome internals?

And then how do you track everyone's contribution to software? Sure you can keep track of commits, but that doesn't track inspiration and ideas. Sometimes the fundamental design or mechanism for a given piece of software will mostly be due to a different dev. than the one who implemented it in code, and often there is no paper trail whatsoever that that's the case.

Ultimately there's no objective way to measure either talent or success except in the extremes. Some people's subjective estimates can still be reasonably accurate though, but usually it takes a talented and experienced dev. to be able to judge another dev.

So, if people have an unfailing sense of good and bad software, which they might not, it would be eventually consistent at best.

You're also right that individual talent is, for the most part, impossible to measure.

Valve's ideology is "We hired you, we trust you." That doesn't mean that they give new hires all the keys, but the going assumption is that anyone who gets in is a competent adult who doesn't need to be restrained with typical, military-style subordination. If you're going to hire someone, then trust that person with his or her own time. If you can't, then don't hire.

Most companies grow too fast and end up hiring before they trust. This causes a loss of focus, because they need to generate busywork projects for the new hires, but it also creates a dynamic where there are Real Fooblers (for a company named Foobar) and everyone else, and the company has no problem generating a bunch of shit work for the "everyone else" category so the Real Fooblers can work on the fun stuff.

Talents of leadership and architecture can be assessed later on, but everyone worth hiring should start out with the basic right to direct her career and, when the time comes, prove herself.

When I say that it's not legitimated, I'm not saying that it's illegitimate philosophically (although I think it is) but that such managers do a poor job of convincing other people that their power is legitimate.

In the military, most people buy in to the rank system. A major component of the abuse inflicted in basic training is to tear someone's ego down and build the person back up again as someone who can take orders. The result is that a lot of them come out of the process genuinely believing that the commanding officer's power is legitimate. In most companies, nothing happens to convince the grunts that the managerial power is legitimate, since most managers are puppet leaders rather than the leaders that the group would pick.

Don't they convince their shareholders? These companies keep making it big even with tremendous overhead of useless management and they seem to do it by pitching investors and locking down markets.

Then you shouldn't. Period.

EDIT: It is unfortunate that the types of people (quick learner, genius at a subject) are not considered for engineering jobs usually because the interviewers don't know how to interview. They ask stupid technical questions instead of having a technical conversation and what code on a whiteboard.

Why not do calculus with an abacus?

Not to mention the problem of people who make hiring decisions having no idea of what the problem they need to solve is and hire unqualified people under false pretenses. The old bait and switch.