Companies have not found a process at scale where they can train their employees to systematically gauge candidates.
Small and startup companies where leadership is still involved in the interview process can gauge candidates based on their own technical intuition. So it's false to say NOBODY has a proxy for good software engineering skills. The founders can also look at the candidate's public code contributions.
The problem is that this method does not scale. If you trusted employees to hire based on feel and intuition, you open up the door for people to bring in incompetent friends.
The real problem here is incentives. Founders have a financial incentive to not hire bad candidates. Employees don't, and that's why a standardized process is needed.
> Small and startup companies where leadership is still involved in the interview process can gauge candidates based on their own technical intuition. So it's false to say NOBODY has a proxy for good software engineering skills.
Is there any proof that this method actually works better though? I've heard plenty of stories of people in smallish (30 FTE or so) startups working alongside some really disastrous hires.
Sure. I built a team of ~35 this way and it worked out pretty well. There were very few dud hires, and of those that happened on investigation it was always because the process I put in place wasn't followed (on one occasion it wasn't followed by me!). It got noticed; we had customers telling us they had to deal with us because they couldn't match the talent we'd been able to attract. We had employees leave and come back because they wanted to work in a good team again.
The process wasn't anything special and the questions weren't particularly hard. It was very heavily standardized though.
No system is perfect. You'll get bad hires no matter what, and miss good hires. The question is only how many.
Oh, rereading maybe I misunderstood the intent of the OP. What I meant is that small startups can also use highly standardised processes that avoid dud hires, you don't have to be big to do that.
As a manager in a scale up, my incentive is to hire competent people into my team so we can meet our goals. Sure, it's not the same as the company being mine, but it's a good enough incentive. I think a standardized process is better because it reduces unconscious biases.
That results in biased hiring based on "I like this person, they think/talk like me".
> look at the candidate's public code contributions
Also biased hiring on candidates who have the time & inclination to do extra-curricular coding at the required level.
> Founders have a financial incentive to not hire bad candidates
All companies have financial incentives to not hire bad candidates, the cost a bad hire on productivity and cost to manage them out is high. In fact, the incentives are so strong that the entire hiring system is optimized to avoid bad hires at the cost of missing out on good hires.
That's why the system is so frustrating - only strong candidates with clear signals get hired.
All hiring is biased. That's the entire point of behavioral interview questions. To make sure the applicant acts/responds similar to how the company expects/wants. And I've yet to encounter a company which hires without at least one behavioral interview component.
There's a difference between "I used my intuition to decide to hire them" (bias-driven) vs "I collected behaviorial examples that I compared to my company's values" (data-driven).
It's OK to say "we didn't hire this person b/c they gave an example of how they reacted to criticism that doesn't reflect our corporate values".
It's not OK to say "we didn't hire this person b/c I had a bad feeling about them".
That's the thing I was struggling with, I can solve many of problems for the avg company – but I don't know how to solve a leetcode problem. I bet there aren't as many companies solving the complex problems on a scale.
But if we look from the perspective of the company – maybe it's just an filter to get someone very dedicated and nerdy about programming. Less churn, less questions.
I have a controversial, philosophical explanation why that is:
Leet code is useless in the real world, everybody knows it. But one would find that people that are able to sit down and learn something useless and stupid only because their boss asked it makes for better employees.
The modern corporate world does not like free thinkers, if not within easily controllable parameters.
The solution for the ones that have better to do that to memorize dumb quizzes for weeks, only to get paid to write basic React components all day, is to go work for yourself, become a consultant, or move to another career.
I half agree. I know people who have a natural curiosity of algos & DS - most predictably, people who did algo/ds competitions/competitive programming. To them, leetcode is easy shit, and just naturally so. E.g. I had a group of friends who would ace Advent of Code every year while I muddled through (and always get stuck on day 7 or 10).
So I think leetcode interviews get both those groups - the ones you mentioned, who work hard and study (and thus demonstrate good qualities of being employees), _or_ those who are so good at this stuff that leetcode won't even make them sweat.
I don't think that's controversial at all - maybe only in the sense that people generally don't like hearing it, but I'm sure that anyone who has ever worked in a corporate environment knows this.
Well the vast majority of casual HN readers aren't actual bonafide geniuses nor are they super hard working. So of course they don't want to consider it and so these posts keep getting voted up.
I believe it is simply an efficient way to discriminate applicants based on their age: a fresh school graduate in his/her twenties is the one who is more likely to solve LeetCode like algorithms than a senior developer in his 40s who has a respectable knowledge and insight on how software should be done, deployed and run.
Usually the issues companies throw at their applicants aren't that difficult to solve. Every dev should know how to split data into structures you can work with.
Google asked me a question… I could solve it in O(n²) but not in O(n).
After, with calm, talking about it with a friend who is a professor at university and has a phd in mathematics, he reflected that since my problem only used positive numbers, I could have exploited a certain property and simplify my algorithm.
So in the end, if you're lucky and have faced that particular completely artificial problem before, you can do it.
Otherwise you need some calm to think, and you can't do that while there are two people harassing you because you named your variables with a name they don't like, and a clock is ticking.
In the end you're selecting for "has this person heard of this problem or a similar one before" and not "can this person think".
These kind of tests are rarely about seeing if an applicant can actually successfully solve the issue. It is about seeing how the applicant approaches a difficult problem.
Maybe google, amazon and whatnot see that differently but I would not personally wanna work there anyway.
Thank god I never had these interview questions at the companies I interviewed for, here in Germany. Most talks were simply a 30min chat with the actual developer there. Mostly about the job, what they do, and some of their current issues. Then they would ask me how I would tackle them.
Yup, can sign that. Never had a coding interview so far... It is all about sympathy and being knowledgable. Had the most success by being brutaly honest about not knowing certain technologies. But I also show a lot of interest once I do not know something.
It's roughly the same in France, aside from some startup bros who have been told leetcode and brainteasers are the way of unicorns. I think it's in part due to the higher trust we put on credentials like Eng degrees.
> it's just an filter to get someone very dedicated
Dedicated is a nice way of putting it.
The school system was funded by industrialists who needed well behaved and obedient workers. Academic and Leetcode problems which exist in a vacuum select for those who follows the rules. Tech companies don't actualy want people with a mind of their own.
I've asked non-leet code questions that aren't directly related to the work the candidate would be doing. It's usually a smallish self-contained problem, solvable in less than a dozen lines, in about 5 to 20 minutes. What I like about it, is that it's novel enough, that I get to see the candidate think through the problem. I get a glimpse into their problem solving capability.
It's not a perfect proxy for how well they'd solve real user facing problems, but so far it has given me a decent indicator of their motivation/passion and apetite for problem solving. There's a dozen other, mostly non-coding questions in the interview, but this one has been the best predicator for me.
(The problem is closer to a more elaborate fizzbuzz with some math parts, than it is to a memorizable b-tree or other leet-code algorithm question)
The thing is that for problems that are mildly complex, it is rarely that it is solved by a single person. So getting a person to solve complex problem under interview conditions is totally unrealistic.
If they are really using this as a filter, they are really doing themselves a disservice.
The reason I've been told is, they already have a bunch of dedicated nerdy candidates, and use leetcode to pick one of them. For the future tasks it's just as good as picking names from a hat, but that's what gets used. Of course if you're not Google and not have that bunch of candidates to start with, the leetcode will be a bullet heading for your leg, but reality tells not everybody sees it.
Same reason the NFL combine has athletes do the bench press and the 40 yard dash. They aren’t going to do a bench press in an NFL game. Neither are they going to a straight 40 yard dash.
However, those activities tend to reveal information about their abilities that are useful in an actual game.
Similarly, while you may not be doing those coding problems in day to day development, those coding problems can (ideally) reveal information about your knowledge and abilities in hat does apply to daily coding.
Could not read the whole article but get the gist. Time was that I would have agreed. But I have changed position.
Candidates: you need to appreciate how difficult it is to hire good people, the dishonesty of many candidates, and how limited the tools are for identifying talent and filtering out weasels. It is true: party trick Big O questions have low utility in most jobs. It is true: there is talent out there which sucks at Big-O questions. It is true: there are people who are good at little else than the party tricks. Nevertheless: those questions provide an objective judgement about a candidate. That objective judgement is valuable.
You spent several years doing a degree to make yourself employment-compatible. You will be able to achieve this standard in far less time, and there are cash rewards at the end of it. Accept that you will need to burn 1-2 months to develop this capability, and that you will feel awesome once you have.
Get /Cracking the Coding Interview/. I was not impressed by the object-oriented design chapter, but seek to master the other first-11 chapters. Take what you need from the remaining chapters. Develop a set of flashcards. Find problems online. Work intensively for at least two weeks but ideally a month. Then book some interviews at firms where you don't mind if you fail. Some of the FAANGs have awesome pre-interview study material. Try to get into process with a firm like that. Keep training on your flashcards and doing a few problems a day to keep in form. Expect to struggle with early interviews. Learn from those experiences. Revise what you made a mess of, keep working with your flashcards until you have mastered your weak areas. Now start applying at the firms where you want to succeed.
> It is true: party trick Big O questions have low utility in most jobs. It is true: there is talent out there which sucks at Big-O questions. It is true: there are people who are good at little else than the party tricks. Nevertheless: those questions provide an objective judgement about a candidate. That objective judgement is valuable.
That objective judgment is valuable at something different from finding out whether people can actually do the job. So, not very valuable at the purpose that people try to use it for.
My last job interview cycle (January 2023), they asked me a ton about the specific details of C++. It was going to be very difficult to do as a dishonest candidate. I mean, you might make it as a language lawyer who doesn't actually code, but you'd have to be a genuine language lawyer. One phone interview, then a 2-hour in-person interview, then another phone interview with higher level people. That was it. There was one "what does this code do" problem, which takes people 5-15 minutes.
So my point is, you don't have to grind leetcode to get a software engineering job. You don't have to use leetcode to hire good software engineers, either. And if you do use leetcode as an interview filter, then you hire people who are good at grinding leetcode, which is not the same skill as writing software.
If you'd like an article giving an explanation for this that doesn't require upgrading to a paid Medium subscription, try this one I wrote a few years ago:
It is, natch, hosted on Medium, so you will see a banner, but unlike this paywalled article it's dismissable. Just click the X to make it go away and then you can read the whole thing for free.
The summary is that many interviewers ask "unrealistic" algorithmic questions because:
1. They fit in the short amount of time available.
2. They get people writing programs that cover all the basic features of the language.
3. They don't ask people to do excessive amounts of work (e.g. takehome assignments)
4. They wash out people who lack basic skills you'd expect programmers to have, like actually starting a new project and being able to compile/run it in their self-chosen editor.
5. They are general and don't tend to require knowledge of specific frameworks or even specific languages.
The questions are unrealistic because they're designed to be fast ways to extract information in an interview setting, not to actually be an accurate sample of the daily work (which in the time available may only cover a fraction of the skills required of a working programmer).
Claims 2 and 4 don't apply to the way these algorithmic questions are usually administered at large companies, which is pure whiteboard coding where you can't run any of the code you're expected to write.
Whiteboard coding used to be common but I haven't run a whiteboard interview for over a decade. I don't know how many companies still do that, but it's definitely a bad idea. You want to see how well a candidate knows their tools, and they will be more comfortable working in an editor, so it's a win/win to do that. Also if you do coding interviews from home it's easier for candidates and they're in a more comfortable environment.
So yeah I'm sure some firms do this, but there are plenty of others that don't.
I think whiteboard interviews for coding definitively went away with the COVID pandemic - everyone uses some kind of live editor, usually requiring you to run the code as well.
If I didn't know ahead of time that Fizzbuzz is to weed out people who pretty much don't know how to program at all I could actually fail it on an interview. It would go something like this.
They would ask me to write Fizzbuzz. I'd see how simple the problem is and immediately think of something like this (assuming they want it in Python):
def fb(n):
for i in range(1,n+1):
if i%15 == 0: print("fizzbuzz")
elif i%3 == 0: print("fizz")
elif i%5 == 0: print("buzz")
else: print(i)
fb(100)
But then I'd think that this is too simple. That's something you could write just after glancing through a Python book at a bookstore. Surely I should use more language features to try to stand out among the candidates, right?
So I'd probably end up trying something like this:
def fizzbuzz(n, *args):
cur = ['' for x in range(1,n+1)]
for m, postfix in args:
cur = [y+postfix if x%m==0 else y for x, y in zip(range(1,n+1), cur)]
cur = [str(x) if y == '' else y for x, y in zip(range(1,n+1), cur)]
return cur
print("\n".join(fizzbuzz(100, (3, 'fizz'), (5, 'buzz'))))
But Python isn't my main language and if this was a whiteboard problem I'd probably make some mistakes and they would be very unimpressed.
There are problems between fizzbuzz and the leet-code algorithms that are small enough to be solvable in 5 to 20 minutes. I like using one of them to complement other non-code questions.
Fizzbuzz was the leetcode of yesteryear. It's exactly the same concept, just a different unrealistic question.
The leetcode issue is to a large extent caused by, literally, leetcode.com the website. When I first started doing technical interviewing this website didn't exist, so if you developed a question that wasn't too easy or too hard, covered the bases, fitted well into 45 minutes, that you could calibrate the difficulty of on the fly etc - basically a good question - then you could make it last quite a long time.
At some point people started sharing questions and pre-canned solutions online, and many candidates will cheat by looking up answers and copying them from another screen if they can. Also some shady recruiters started leaking questions to candidates ahead of time. That forced interviewers to start constantly inventing new questions.
One of the points I used to make in interview training is that questions are like programs, you have to design them. Some are better than others, they can vary in efficiency etc. Often you can't really know how a new question will work out until you try it a few times. Making a good interview experience for the candidate is hard work and it's easy to screw up, lots of interviewers do and it leads to a lot of bitterness (e.g. questions that are far too hard or specific or don't fit in the available time).
Unfortunately if you're constantly having to invent new questions to keep ahead of the people sharing them and grinding the process, then the quality will inherently get way more variable. By the time you've calibrated the question it's been leaked already. And there are a finite number of programs that it's reasonable to ask someone to write in a short amount of time, so by now these giant databases of interview questions and candidates memorizing the answers at scale makes it harder to get accurate information, which was the only goal in the end. Because indeed the specific tasks themselves don't matter much.
Right, but points 1-5 aren't about these issues you're bringing up.
Sure, people can rote memorize "public static void main(String[] args)", but there's way you can test them their understanding on each of these magic keywords. This would achieve the goal of understanding programming language constructs without unrealistic algorithmic questions.
That's covered in the article, my bullet points were just an attempt at a summary.
Briefly, it doesn't work like that. Interviewing developers is very unintuitive. Many reasonable expectations are violated. There are lots of people who can talk very well and will appear to have a good understanding of programming, up until the point that you ask them to write a program (any program) at which point they can't even start. If you don't force candidates to write an actual working program you will pretty quickly hire people who just can't do it, and then have to fire them later. Yes this sounds totally wrong, everyone struggles with this fact at first. It's sort of like the Matrix. Nobody can be told what interviewing is like from the employer's perspective. You have to see it for yourself.
For largely the same reasons why graduating from top schools is given more credit than any old college, even if the level of teaching is similar. Getting in takes determination, intelligence, and discipline. Same with getting good at Leetcode.
This feels like a retcon. The real reason is most likely that Google started interviewing people this way and then that's the culture that developed in the industry. Once a culture develops, it's very hard to change.
It's also noteworthy that if you're building a search engine, these types of questions are actually pretty relevant. As a field Search demands both a breadth and depth in CS fundamentals in a way that rare in most other places.
Most Googlers joining Google today aren't gonna be tasked with the elbow grease of building a search engine from scratch, not even close.
You'll be a small cog in a large machine, engaging in theatrics justifying in length why your 10h/week of serializing/deserializing JSONS was so difficult and important to the company, while you rest and vest playing videogames at home, asking on Reddit if everyone working in FAANGs is as bored as you.
That's true, but these practices date from back when Google was a search company. My point being there existed a time when Google asking these questions made a lot of sense.
This is the correct answer. Microsoft used to ask things like "Why are manholes square" or "How many dentists are there in USA". For a while it got trendy to ask those sort of brainteasers in interviews (thankfully it didn't seem to stick).
It is very hard to change established processed - especially when the people doing the interviews have gone through the process.
They had to suffer, so why should the people after them get an easier ride?
>They had to suffer, so why should the people after them get an easier ride?
People in the middle ages and WW1 had it pretty bad, why should people after them get an easier ride?
Because humanity progresses when we make things easier for the next generations instead to clinging to the 'crabs in a bucket' mentality of perpetuating the cycle of collective suffering and suckage.
I hold a very controvertial take on the dentist question: this and similar questions select candidates who can come up with convincing lines that lack any basis in facts (i.e. bullshit). This seems to be a good foundation for a terrible engineering/company culture.
That still doesn't say anything about the candidate besides the fact that he/she farmed algorithm questions.
It's an extremely poor predictor of a candidate's quality.
It should only be used in those companies where the number of applicants is way larger than the number of open positions where you can accept the trade off of losing few great applicants if you can also lose the many more bad ones.
It took me a while in life to realise that some people are just made to succeed. These people are typically good to great at anything they put their minds to.
Just because you say those questions are an extremely poor predictor of a candidate’s quality doesn’t actually make it so. It might just be one of the best and most cost effective ways of finding good candidates.
It's a way to get rid of many false positives even if you lose some false negative.
Again, companies like Google can afford to lose amazing devs that can't be bothered to farm leetcode if the trade off is getting rid of those that would not be able to farm those effectively.
The answer is usually self selection and calibration at scale. Self selection is about people looking like minded colleagues. If everybody is crazy about solving riddles and participates in programming competitions that's what they would be looking for.
Now, about the scale. Anything run at scale needs standardisation. You need to hire 100 senior developers. How do you know they have more or less same level? If every single interview is hand crafted, you'll either need to get all devs envolved (and not everybody is good at coming up with interview questions) or get standard questions and answers to allow a smaller group of people to deal with candidates and coding puzzles fit great there because they're slef contained and isolated. Every realistic question has multiple facets and that's what you get at the system design interview step.
Another problem I've observed was that the more you give the same puzzles to candidates, the easier they look to you. What the means is that you as an interviewer either need to keep your self in check regarding your reactions or you're risking giving a bad interview score in situations where it's not really warranted or there is a chance that the next question you take will be the one you find more complex (to be equal to position level in your head) and that would push the level of questions up. That's another observation I had when some interview questions would become so complex that I knew some of the existing devs would fail the interview for sure.
Does that give good results on the other end? I don't know, but what we definitely know is that there is the whole industry around leetcode to train peeople to pass these challenges specifically and that means that they only thing the interviewer know is that the candidate has put some effort to prepare for the interview meaning they're motivated to join the company. And maybe it's not the worst data point! Some companies actually explicitly mention this fact on their hiring pages.
To add to it, big corps have different ways to make interviews objective in whatever sense they think and that by definition reduces the personal impact. "Why did you ask this question? We've never done it before, we'll need to have a group call to calibrate". After a handful conversations like this you'll just stick to the standard process.
Is there a way to come up with a more human approach? Personal recommendations with some skin in the game I guess. I'm sure in some niche areas like browser engines all good devs know each other well and no interview is often necessary.
More or less the same reason why psychologists use the Rorschach test [0]
Sometimes they are not worried about the specific answer, they're trying to find out your (software engineer) unconscious mind reveals during the process.
> trying to find out your (software engineer) unconscious mind reveals during the process.
For many decades, Floyd's Cycle Detection algorithm was unknown in academic circles as the most efficient method of cycle detection.
It is not uncommon to encounter this problem in an interview. There's nothing revealing about the unconscious mind and process, other than that the candidate remembered the solution to this problem.
If it was so readily obvious to derive and arrive at a solution, you would have expected the army of PhDs and academics to have done so.
Because they have a different problem compared to smaller ones: a huge influx of applicants, and this is an easy way to filter and standardize process across a loooot of interviewers. Is it perfect? Hell no. Does it result in false positives/negatives? Sure. Is it OK for them, considering the volume? Yup.
These tests and questions are often in place so people feel they can manage the risk/uncertainty of hiring. Some studies show that previous performance is the key indicator of future performance. Interviewing and testing in the hiring process are negligible at best.
I'd far rather be subjected to 15 bizarre questions than be subjected to 5 rounds of interviews with everyone. Some questions you will get right and a few you may have heard before. Add a little luck and you have a job.
Isn't all about proving that the candidate "took the time to study all our questions"? It's about showing that your company is better because people take 6 months studying to apply there!
Yep. Strong negative signal from me. I can’t remember the last time I actually clicked the link. Anyone that thinks that the Medium experience is remotely acceptable is living in a world very different to mine. Just as importantly, the quality does tend to be crappy off-cuts from someone with an obvious content creator fetish.
Wow, what an incredibly low quality article (TL;DR the questions are hard because the interviewer wants to lord it over the candidate) for something behind a paywall. And he ends with a call to upvote and follow, classy.
Yeah, you're a software dev, spend an hour learning Hugo ;)
(Take this with a grain of salt, I don't want to tell people what to do, I just stop reading your blogpost when it fades out and asks me to sign in...)
I agree and is not because they are making money out of you (that's fine if you agree) it's mostly because they aren't censorship proof or ideologically neutral.
Medium started expressly for the money. How to let authors monetize.
To criticize it for doing what it was created to do, seems odd.
Sure, there are a lot of software packages out there so any dev can create a new Medium. Who cares? The point was to pull a lot of people to one site to monetize it. They would browse around and find other authors, which is not possible if every dev out there creates their own site.
Just like Twitch isn't it?
Are you saying it is being censored? To what end? What ideology cares about programming. "Those Democrats and their Objects are using the wrong patterns again". "Republicans and their Emacs, holding on to the past".
I already mentioned that monetizing is fine if you agree so no point in being triggered by that.
Regarding to censorship your reduction to the absurd attempt is making a non-point here. The cultural war is overt, well understood, widely understood at this point and no human discipline is immune to the ideological mind viruses. These might not change the patterns you code but certainly boycott hiring you without feedback, unconfessably, for the revolutionary goals of advancing a power grab agenda based on either true self-deception or lies.
I actually want to know. Is there a group in Medium management that is removing stories? Which direction? As far as I can tell, it might actually lean right, but I can't tell.
This is first I've heard Medium criticized for censorship, and while flippant, I am wondering how/why?
For monetizing. Maybe I was reacting to a few different posts. I only replied to one, sorry it was yours, but it seemed like a common theme that devs should just roll their own Medium. And I thought that missed the point.
Couldn't read the article due to pay/registration wall.
If this is about screening potential software engineers with programming questions during an interview, hasn't this always been a thing even before Google popularised it?
I started my career after the dot-com era but it was in Systems Administration, so I didn't get to experience any programming style interviews until I changed careers and attempted an interview in 2008 with Google.
But hasn't this been the standard modus operandi for big tech companies like Microsoft, Sun, Oracle, IBM, etc. even in the 90s? I recall reading this [1] article from Casey Muratori about programming questions for a Software Engineering intern at Microsoft.
I'm not against this style of interviewing, but I do also think that some questions can be absurd or unnecessarily tricky. I've had my fair share of programming interview questions, and I found my solve rate to be around 3/7 for my last interview with Google back in 2015. Some of the questions posed were really tricky, and I just don't have the ability to solve in a timely manner without properly experimenting with the problem. From my impression and perspective, interviewers would sometimes choose questions with high coolness/leetness factor instead of choosing something more practical for a 30-50 minute session.
Can't view all the text without signing in to Medium. Even after trying to sign in with Google, I'm repeatedly taken back to the home page. Does anyone have an archive link?
