This question could be improved as follows:

* present the naive solution. Explain that this is a PR that was submitted in 20 minutes by our new intern from Waterloo. Ask them to help improve it.

* suggest to them the intern’s argument why the lookup is O(1) and explain that it’s actually O(n). Ask them to write a PR comment to break the news to the intern.

* explain the optimal solution with tries. Ask what happens if the vocabulary is big enough—and the prod environment is resource constrained—so that the trie is rarely in memory when the function is called. How would the trie solution compare to a binary search through a length normalized list of words?

This kind of question is barely useful to interview new hires—when you still may have basic questions about how much undergrad material “stuck”. Within a few years when they hit full-performance level you should care much less about this kind of question and much more about “higher order” thinking about engineering.

How do you communicate bad news? How do you identify, communicate, and mitigate risk? How do you plan large tasks at at appropriate level without over engineering the solution? What questions should you ask at different project stages to capture the right requirements, anticipate issues, and successfully execute over time?

But whatever, I’m sure you get lots of signal out of repeating “a hash table lookup is O(1)” “nuh uh” on loop for 10 years. :eyeroll:

Maybe the problem has something to do with the way they hire. The article mentions that this string-processing problem had to be approved by a committee:

”Twice I asked the hiring committee if I should stop using this and they had no problem with it.”

Not even the Soviet Union had a process where job interview questions were regularly approved by a committee. The level of bureaucracy in administering these brain puzzles at mass scale is absurd.

The article mentions that the candidate who did best on this question was “the type who did high-school competitions.” I wonder if anyone stopped at any point to question whether that’s the ideal type of hire.

I think this is the most telling bit. Time after time, I've seen excellent competitive programmers end up being mediocre engineers. Almost none of them are bad and it's also a very objective (at least on paper) process which is probably why Google is happy to stick with the process but most of them also aren't amazing innovators.

I'm sure that university student who aced the interview turned out to be very smart - they went to a great university and can solve challenging puzzles. But I don't think they'll end up being the engineer that builds Google's competition to ChatGPT. I wonder whether a course correction where Google starts looking for people who create really cool projects even at the risk of not being able to solve any coding challenges would actually help.

Oh, wow, if I could get that kind of result ("almost none of them bad", my addition: "possibly great") from an interview, I would stick with that process.

I have also conducted interviews in a company with heavy focus on code tests, which obviously favors competitive programmers, and my anecdata was that I got very good results compared to the alternative (i.e. not doing that).

And I believe this is why this discussion always keeps popping up. And it makes you discard a lot of "possibly great" candidates at the cost of "almost none of them bad".

It is the eternal precision vs recall tradeoff.

OpenAI has a lot of strong competition programmers. Search IOI or ICPC in their team update blogs:

https://openai.com/blog/team-update

https://openai.com/blog/team-update-august

https://openai.com/blog/team-update-january

I've seen the opposite. I know many extremely talented competitive programmers (IOI, CCO, USACO platinum, Codeforces grandmasters, ICPC, and the list goes on) and they are all excellent engineers. It's not about having a wealth of knowledge about DSA (although this occasionally helps, depending on the type of work) or competitive programming itself, but rather the skills it teaches. They all know how to learn new things very quickly and thoroughly, and it goes without saying that their general problem-solving skills are incredible. They've interned at companies like Snowflake, Google, Jump, Citadel, Jane Street, Waabi, Uber R&D, DataDog, SingleStore, etc. (all before or during their 3rd year, which is practically unheard of for "normal" students, especially during the tech recession) and have consistently received the highest performance grade (our co-op program requires employees to give them a rating on a scale from 1 to 7). There are also a few I don't know directly who are in fact working on things like Bard, or have founded their own companies (TabNine was one of them IIRC).

> I think this is the most telling bit. Time after time

If I can rephrase the question as, does anybody check if the rating somebody receives in an interview is positively correlated with their performance rating in subsequent years?

The answer is yes people do analysis of that question and yes it is.

But if they have any poor engineers, their process clearly doesn't work so conclusively. Because those engineers passed their process. And they have no control group, unless they've secretly been hiring a group of engineers without asking them these sort of questions (or ignored the results for a group). I just don't see how Google could determine the process works without studying what happens if they don't do it.

I'm not sure. The author answered this question with a "no". They admitted they don't analyze or check how actual performance on the job correlates with doing well on this kind of interview questions. Maybe others do, but the author doesn't. I wonder if anyone at Google checks this.

In general, even people internal don't bother to use moma to check this so your opinion is not alone.

Correct.

> Because the author explicitly said he doesn't do follow ups and so he doesn't know if his question is a good predictor of actual performance.

This would be difficult as a random engineer. People Ops has different access to performance/interview data so they can do this analysis; your interviewer can't (unless you share your future ratings with them and they wrote down all of the ratings).

You can do the analysis on yourself though but you'll have to jump through some hoops to get your own interview data. It's also a single data point so unless you can convince a bunch of other people to also do this it really isn't significant.

> Strange that the author doesn't mention this though.

I'll be honest, I think maybe 1k employees know that anybody has bothered to do this analysis. It would not surprise me that the author is completely unaware given most of memegen is as well.

On the point of committees, though. Hiring committees make a hire/no-hire decision. They don't approve questions, but they do make decisions based on the results of asking a question. So they rely on the signal that the question generates, and if a question is not giving good signal, they will have an opinion.

There's a different reason for wanting to standardize on interview questions. It's very easy to embed cultural bias. If the topic of a programming problem uses the rules of a game, for example, that will probably mean that people who are familiar with the game will understand the gist of the problem much faster than people who've never played the game.

That being said I am easily the best/fastest programmer in my organization despite there being people who have 20 years more experience.

I have found this to be a very good signal in the hiring process.

People who have technical side-projects (machining, 3d printing, electronics, game programming, open-source) tend to be really good at conceptualization, problem-solving, cost-benefit analysis, problem-solving, staying-power, which are critical elements to the success of a software project.

Trivia CS knowledge questions, such as the one presented here, have their place too: they test whether someone can think on their feet, which is not a bad skill to have. However, all things consider, who would you rather hire? - someone who can do an O-time analysis, but has not finished any real project? - someone who failed the O-time analysis, but has built something to the end?

I'd obviously prefer someone who can do both, but the person who has released an open-source project for their ultra-niche hobby (say, a sock knitting pattern designer, used by 5 very passionate individuals; or a climbing wall with LED lights showing you the path) gets my vote otherwise. They may not have come up with the trie solution, but what they've done is conceptualize, build and deliver something concrete to users.

Which means they have proven and refined the precise skills every software project needs.

Well, the author does mention a surprisingly high number of allegedly senior programmers fail at coming up with the simple brute-force solution. At this early stage in the challenge, this isn't "a trivia CS knowledge question", it's simply knowing how to code the simplest iteration over a string and knowing there is something called "a dictionary" which allows lookups! If you don't know this, what do you know?

Not defending Google's practices here, but this claim is nonsense. Many public sector/third sector jobs in Western society will only use interview questions which have been pre-approved by explicit policy.

Hiring committee's job is to take the interview feedback and provide a "Hire" or "No Hire" decision. 99% of the time this is done entirely based on the interviewers' feedback without any further interaction, but occasionally the hiring committee will reach out to ask clarifying questions or provide feedback on the report. Sometimes that feedback is along the lines of "this question is more complex than the candidate's target level" and sometimes it's "hey, that question has been widely posted as a 'Google interview question', so people are studying it specifically; you need to stop asking it." Good interviewers will proactively ask for feedback from hiring committees so that they can provide a better signal.

Also, it's worth pointing out that there is no "one" hiring committee. Many people are in the pool for that task (it counts as a "citizenship contribution" during perf) and you meet for a few hours once every few weeks and grind through a stack of feedback. That's why he asked multiple times; he effectively asked committees made up of different sets of people.

Also I recommend you try these "high school problems" you seem to be talking down on. I for one greatly respect people who can solve those types of problems, regardless of their age. These are not trivial problems, mind you. They require a lot of mathematical and logical intuition that 99% of people (and I'd wager the majority of SWEs) don't have.

It's unavoidable when you allow too many different interests to have a say in the process.

I think any discussion of complexity for this problem is misguided.

I mean, it's fine to talk about a hash lookup being o(n) for pseudocode in some hypothetical context. But if you're talking about actual code that you intend to run in Python or JS or whatever, it doesn't make sense - are the strings interned or not? When your language hashes the key of a dictionary lookup, does it cache the result? If you're using a JITTed language, won't any native o(n) hash lookups vanish in comparison to your non-native o(n) for-loop?

On top of all of which - why are we even talking about o(n) complexity for an n that cannot grow? If we're given a fixed dictionary then we can ignore any substring longer than its largest element, so the "n" for hash lookups is not the size of the original input - which is the n we realistically care about!

Basically, once TFA mentions complexity I feel like the only correct answer is to say "this isn't a situation where complexity analysis would be useful - we should benchmark the code".

All of these are useful signals for a hiring decision. Much more than quibbling about big-O of hash table lookups.

But you don't try checking if the entire string is in the dictionary regardless of n - you only ever check if it's in the dictionary if it's shorter than X+1 where X is the longest string in the dictionary. As the input string size grows, the dictionary lookups are capped.

If the author asked about complexity as an open-ended question with no fixed answer, that would be a very reasonable way to hear how the candidate thinks about algorithms. But TFA makes it clear that the author treats it as a gotcha question - he expects candidates to say o(n) and then pushes them to give the answer he wants. That's not a great approach, even if the interviewer's preferred answer is unambiguously correct - and here it's not.

Like these can’t be prepared for? I think there are now whole chapters in those interview prep books/courses on how to answer these.

Even if there weren’t this is a good way to hire people who sound very professional on slack and in meetings but don’t produce any actual code (unless you got very lucky)

You would be absolutely stunned how many people cannot give you back the correct answer even after you literally give it to them. If you don‘t know some programming and I give you code and explain what it does, that still doesn’t help you.

There’s some tiny signal of your ability by asking you to memorize and regurgitate something; and maybe a tiny bit more signal in this question by implicitly testing “how do you respond to unexpected technical details in a high stress situation”. But they aren’t worth the trouble. You could measure both much more directly. You’d get more detail about their coding ability by directly asking about code they’ve written and know well.

> The fastest I had a candidate solve the entirety of the problem with all bells and whistles was in 20 minutes, by a freshman from the University of Waterloo, the type who did high-school competitions.

This should’ve been the interviewer’s first clue.

I don't get why the dictionary solution is actually O(n).

Surely you can have the hash function be O(1) by only looking at a subsets of characters something like

hash(s) = (s[0 %s.length ] ^ (s[6 %s.length]<<c1) ^ (s[7 %s.length]<<c2) ) % (size of the hash table)

You can trade-off memory for speed to avoid collisions by having a mostly empty big table (like 9/10 empty table). And you can also use multiple hash functions that must both simultaneously collide to avoid having too big a table (kinda like fronting your hash-table with bloom filters).

And you can pick the selected sub-index at random such that they split well your dictionary. Maybe you can even build a "Perfect hash function" so that all your dictionary words have distinct keys.

You only have to check for the whole string when the hash match, probability of which you can make as small as you want.

> > About Part 2 :“Write a function that given a string, determines if it’s a concatenation of a number of dictionary words.”

Isn't it just recognize if a string belongs to a regular language, where the language is defined by words in a dictionary ? So you just build a parser for your language and you test for belonging.

I'm quite rusty in non-deterministic finite automatons and regular languages, but surely there exists algorithm that build an automaton from the dictionary to recognize the language in something better than n^2, no? (We are moving the states of the automaton once per character, and we likely won't have n active states (except maybe for pathological dictionaries), complexity should be something like O(k*n) where k is a constant depends on the dictionary).

What this means in practice is that there is no polynomial curve for very long words. You don't bother hashing them, so the curve peaks and then flattens.

Obviously a trie is an interesting and possibly faster solution, though the hash has the benefit of being more cache friendly. But the big O analysis presented for this problem is flawed.

On these kinds of problems, I like to suggest that we might be thinking about arbitrarily long "words". Get out of the headspace of language dictionaries, look at things more abstractly. Otherwise a lot of big-O analysis becomes degenerate.

Also, don't forget hostile inputs!

I seem to remember that in the very early days of Java the standard hashing method for strings only looked at a small prefix - turns out that very common use case was to be storing URLs as keys which, of course, mostly start with the same letters...

Eg you can’t just do equality on collision, you need to do equality on every lookup at least once.

> you need to do equality on every lookup at least once.

I still don't think so. You only need for when the split would correspond to a good partial match.

There are also other smarter hashing thing you can do to help convince you.

For example if hash( s ) = sum_over_i( g(s[i]) ) if you pick g(x) = x or g(x) = xx or g(x)=xxx (or any function you are free to choose) (it's kind a like matching statistical moments of words).

You can calculate this hashes in amortized O(1) because of sliding window trick.

You can also index on split length.

You can filter your set of candidate split positions so much that you only have very probably valid split positions in your candidate set, and therefore the complexity is just verifying the validity of the candidates splits, so still O(n(number of candidates after filter) ).

> > Part 2

A solution based on filtering a candidate set could also probably be made to work for multiple words. You use the matching statistics to get candidates starting positions and length for words of the dictionary that appear in the string and you verify if they touch each other. Pathological cases are when words overlap each other but this may not appear very often in practice.

You can also add some filter based on pairs of words (or triplets..., or n-grams) .

All in all, I think the problem is rather bad because smart candidates will often have a solution that the interviewer may not be able to comprehend, and as a candidate you have just put yourself in a possible ego battle with the interviewer were even explaining the proof is not easy. In fact if the alternative solution is more interesting than the rest of the problem, you might get your candidate be focusing on that instead of your simpler but boring trie based solution.

My initial hunch was an example where every prefix of the word is part of the dictionary, then on an example like `abcdefgh` you'd test `a`, `ab,` `abc,`... But on such a case you can be clever and defer the string match until you verify that the hash of the suffix also exists in the dictionary, which it won't.

So it would seem a pathological example needs both prefix and suffix matches, but this just means that it's very easy for us to find a valid segmentation, so we will find one very early on.

I guess this is probably what you meant by "split would correspond to a good partial match" though.

This seems like a really great question, thank you :)

Great prompt!

Instead of doing stuff like this, you can just give candidates realistic programming challenges and have them write code. You don't even need to be in the room with them; after all, you won't be when they're actually doing the work.

I've given this same advice for _years_ for designing technical interviews. Take a problem that you or your team has actually faced in the last n months. Distill it down to remove tribal knowledge and get it to something you or a capable colleague can hammer out in m minutes. Give the candidate 3m minutes to solve it. Typically this means it should take you 15 minutes to solve and give them 45 minutes.

Our biggest hit for a question that followed this for an SRE type role: we had a cron job that was legacy and buggy and caused a couple of issues over 6 months. We went back and created a very similar script with very similar bugs, added some similar data and re-seeding scripts, and told the candidates "customers say that feature $x is not working, and that is ran by this script. Here is how to run it, here is the user's id that can be found in the data seed." We added some other data validation problems. After they fixed it, we asked them what they would do to make this more production safe and dig in on what they have and have not done for improving existing systems. We expected discussions on structured logging, metrics, alerting, testing, code layout, etc.

Over time, the roles that you are looking to fill and the weaknesses on the existing team that you want to shore up will change and you will need different interview questions. Maybe more towards distributed systems or more towards testing or observability.

Importantly, it also allows you to examine fluency. The candidate who can write a code/pseudocode solution as fast as they can type, and explain it verbally at the same time, is much more impressive than the candidate who pauses for a long time before starting haltingly and unconfidently. Fluency in solving leetcode problems just tests for having practiced leetcode, but fluency in writing real code tests for actual productivity.

If you have a question like this, you can drill down into performance characteristics just as the question in TFA does. Not necessarily using big-Oh notation, but "what if you have to do this millions of times? what if you have to provide results immediately? what if some of the files are too big to fit into RAM?". But you can also drill down into other aspects of the problem like changing specs, noisy input, constrained resources, observability.

FWIW, when I came up with a question like this (with the constraint that it had to represent a problem I had solved that morning), ChatGPT was not able to solve it correctly.

No wonder it is a hit. It is an actual problem that makes me want to go for it - I want to take an interview like this, instead of the usual of dreading interviews! Keep it up.

I think they're using "m" there as a variable, to frame that you should give the candidate 3 times however long it takes for an insider to solve it.

They mean 3 x m where m is how much time you think it would take for you to solve; so the example is: if it takes you 15mins, then give the interviewee 3 x 15 = 45mins

And in my experience listening to auditions over the last several decades, what you have now is a bunch of musicians who can only do what's on the test. When you try to plug them into an orchestra, they are completely fucked.

My experience interviewing and hiring tech people is that this is one of the worst possible approaches. Anyone who needed to could figure this out. It's not relevant. What you need to figure out in an interview is if the person is curious enough to find the answer.

I don't ask for code or algos in my interviews. I ask about the person. What's your favorite programming language? Why is it your favorite? What do you not like about it?

What's the project you are most proud of? What was hard about it? What made you happy? What did you find challenging about that project?

These are 100% totally humane questions that tell you everything you could possibly need to know about a candidate in about a half hour. Maybe less.

If you can't figure out if this person is technically competent from this set of questions then you aren't competent to interview or hire.

Interviews don't have to be only about how good you are at interviews. They could be about trying to understand if the person would be good at the job and fit in well with the team.

Episodic recall and emotional recognition are both talents that vary across the population. If I want to do well at an interview like this, I will have to spend a bunch of time in advance refreshing my memory of various projects I have worked on so that I can simulate a neurotypical response to those questions. If I don't, I may just stare at you nervously and break out in a sweat when I realize I don't have anything filed under, "project I am most proud of". Or that pulling up the narratively engaging details of a project I haven't thought about in years is a process that will take me hours of specific effort. And it's not just me; I've hired great developers who were also terrible at "humane" interview questions but who did great when we just sat down and coded together.

And lots of good developers skew anxious! What in normal life counts as being overly careful or or overly sensitive is just good practice in coding.

That's why I think it's especially important to make interviews as low-anxiety as possible. They'll always be worse than the job, of course, but the closer you get them, the more you're likely to see how people will really perform on the job.

Not only do I not remember past successes very well, preparing for these questions is literally the first time I asked myself what I'm proud of. I didn't even consider doing this and reject it as irrelevant before. It just never came up.

I've always had trouble with the kind of behavioral interviews that are "Tell me about a time when . . ." since I can often recognize that I've been in relevant situations but can't for the life of me remember the details (even with time to prepare).

This honestly makes me more nervous about interviews than leetcode shenanigans. At least with those, I know how to improve.

(At work I rely on a combination of having the code in front of me, written notes, and whatever project management tools we have so it's not much of an issue, but most of that isn't available to me in an interview setting.)

I usually have a project or two that I can speak-to in depth, to give examples of decisions made, challenges handled.

In the end all you could actually do is guesstimating who would do best at the job. And sometimes the answer is: "none of the ones who showed up".

If you are a good engineer or programmer yourself you can tell a lot about a person by talking to them for an hour. You can learn what is important to them in the craft, how they deal with criticism, what they aspire to, what kind of problems they tend to work on, if they are more autodidactic or more influenced by other's opinions and so on. This is all knowledge directly inpacting the question whether they are the right person for the job.

And who the right person is depends on the job, so there might not be "right" answers to the questions. For a very niche database job you might actually want someone who is very accurate, very in the detail and very focused. For other jobs maybe some entirely different traits are better.

Of course the problem here is that the people you hire could just be very good actors or liars who cannot do the things they say, so a little technical testing might also be needed.

While yes, you can get someone who has a good track record of being a good judge of character to make hiring decisions. If that decision goes wrong and the blame game starts then you inevitably ask this person to explain their hiring decision. This person will then reply: well it was my opinion that X, Y and Z.

In the alternative case where you just get someone to follow a rigid process you can answer: Well he scored well on this exercise and the opinion of the team was in his favour and the work history ticked the right boxes. This effectively dilutes the responsibility. You can no longer blame one person for the bad hire.

Lastly there's also the diversity aspect. With companies increasingly being scrutinized for their hiring practices, "it was my opinion" just gets translated to "this person's internal biases guided the hiring process and as such it was not fair."

I agree this is all making hiring much harder than it needs to be. And certainly you can still find companies where the hiring decisions are made on highly experienced and well honed gut instinct (its the only kind of interview I have been a part of) but it also makes sense how in our increasingly overcomplicated world we are developing increasingly overcomplicated hiring processes.

My recommendation is if you want a sensible hiring process (with maybe a telephone screen followed up by one normal interview) you should stick to smaller and less corporate companies.

I know plenty of smaller, less corporate companies that still use careful hiring processes with thoughtful rubrics. They do that not because of risk avoidance; they are generally still ok firing people who don't work out. They do it because they want to run a fair process while maximizing ROI.

Both of mine did. Much of their advice was irrelevant (nobody in tech gives a shit about your suit or how firm your handshake is) but they both stressed the importance of preparing in advance to show how good your skills are; and giving thought to cliche lines of discussion like 'what's your biggest weakness?'

I would still require the candidate however to write some code. Something rather simple, not leetcode, ok, not fizzbuzz either, and something that proves their familiarity with at least one language and platform.

I should be a GM by now simply watching chess analysis videos on Youtube, learning the names of some openings etc. I can talk about it all day long. But since I don't play at all (I'm not interested) when I actually do play on occasions I revert to 500 ELO...

You can easily tell a seasoned veteran from a rookie just by looking at them code for a few minutes, working on a simple task. If you can't, you have no business in interviewing others :)

Someone who can explain why they love Rust and hate Python (or vice versa) is a convincing candidate but I think you need to see a couple of other things from them. In addition, sometimes you just don't hit on the right question to get evidence of their aesthetic judgement. (Like the candidate who doesn't care about Rust vs Python, but who hates Postgres and loves MariaDB..)

Source: Multiple orchestra musicians and management speaking about the audition process and the now-defunct myauditions.com website

Process:

Show early talent

Get the right instruction

Get into a preparatory program

Make it into a conservatory

Do well once there

Get a spot in an ensemble, probably lesser-known (not easy) and establish a reputation

See an audition notice with repertoire to prepare, apply

Spend a few weeks or months intensively practicing (on top of other commitments)

Maybe get an audition, realizing the Music Director may be promoting his buddy past these rounds

Travel at your own expense to the audition

Play demanding repertoire behind a screen and respond to any directions

Most likely get rejected

Play in any following rounds

If not rejected, maybe you get a qualifying week rehearsing with your potential colleagues and playing a couple of concerts when the Music Director is in town

If you beat out the couple of stellar artists who've also gotten that far, you may get an offer

Or, the MD's buddy may get the offer

Or, there may be a no-hire

If you get and accept the offer, you are "on approval" for a couple of years at which point there's the Tenure Decision

IMO, the disdain from LeetCode from developers vs. its continued use disconnect stems from the fact that we developers tend to assume that the goal of the hiring process is to find the very best possible candidate, whereas the goal in reality is to find somebody "good enough" as efficiently as possible.

At the end of the day, you can't test for negatives and just pick a random person that doesn't fail anything. You need to have a defined set of values to look for in a candidate, and to design an interview that tests for those values. That's why I hate DSAs -- for most (all?) companies, they're treated as dogma rather than a pragmatic solution to a difficult problem.

I ask multiple related questions, and if a candidate struggles with one of them, I probe for strengths in other areas.

No one answer to a question in that interview is a pass or fail. But the interview, holistically, is a pass or fail, or a conditional[1] pass.

[1] I was impresssed by some aspects of the candidate, but there are notable, significant weaknesses. Please determine if other interviewers have detected similar weaknesses, or if the candidate just got unlucky with my particular question.

This is also an unproven statement with dubious validity.

I come from the trading world where the leetcode equivalent is a brain teaser. After much deliberation, the only thing we could think of for why a brain teaser was useful was that it often proved someone wanted the job so badly that they memorized all the brain teasers. And that type of dedication and drive is what we really wanted.

To me, I don't understand why someone who develops with advanced IDEs, using complex tools and libraries, is in the least bit judged by putting them in front of what is essentially a text editor, and asking them to solve a problem that's of no relevance to their job, using no tools that they're comfortable with

Period. If someone tells me they can develop software in a language, and they can't write ~10 lines of code, that tells me they lied. It's like when getting lifeguard certified they require you to dive down to the bottom and grab a brick. Nothing at all in the day-to-day job of lifeguard requires retrieving submerged bricks, but the set of people who possess the swimming skills to be a lifeguard is fully included within the set of people who dive to the bottom to grab a brick.

Programmers skew anxious. In theory there's no difference between writing basic code on a whiteboard and writing it in an IDE. But in practice, an interview is a novel, high-stakes environment with a bunch of judgy strangers. So if you then have people do the interview using tools that are unfamiliar, a notable percentage of them will be thrown off.

Once I switched to encouraging people to bring their own tools (own laptop, own editor, preferred language), I saw a marked decrease in people bombing interviews. Some of the people you might have put in the "they lied" bucket were, at least for me, just ones who got really nervous and so were unable to grab the brick.

To me, that's a bad interview, because the job isn't going to be high-stakes whiteboard coding. It's going to be sitting quietly in a comfortable and safe place and gradually improving things. The more an interview can be like that, the better.

I'll take someone whose Github repo shows talent, dedication and passion, but who misspelled an include header in their brain teaser over the inverse any day of the week. But you do you. Neither you nor I have a lot of evidence to support our preferences either way.

Digging through a couple GitHub repos is a good way to find good developers versus poor ones. The goal of a leetcode question is to separate the developers from the non-developers. And you can do that pretty adequately with 2 questions: "Tell me about something you've worked on" and "Hack together a solution to this trivial question".

The question did not "weed out someone who couldn't code." It weeded out someone who couldn't use a totally unfamilliar dev environment while being stared at and asked to "walk me through your thoughts" instead of letting me think. It left me distracted, thinking both "this is easy, what's the catch" and "wtf is the point of this?"

"Tell me about something you've worked on" is a totally fine question. I'm not arguing with that. But anything leetcode is not, IME. This becomes increasingly true as people's careers develop. If you've spent your last two years writing hyper low latency deserialization engines and packet processing, then you'd know that's what the job is about. In the macro, trading engines and game engines are one medium leetcode problem from an algorithms point of view, but they require teams of people and years of optimization and customization to make truly effective. I don't want to test for the person who can make a really basic solution in 30 minutes. I want to test for the person who can optimize a tiny fraction of it in 2 days.

1) most people here, I believe, don’t have a problem testing a developer’s code writing acumen.

2) People weigh something. Bricks weigh something. It’s the same exercise. The analogy in software would be the difference between asking someone to write _an_ function that can do x vs write _the_ function your company uses to do x.

3) the lions share of counterarguments here propose using real world exercises vs toy problems

I've always assumed the brick was meant to simulate a drowning swimmer, whose rescue is pretty clearly part of the lifeguard's job. I guess a mannequin would be more realistic, but it still seems pretty close.

Your question has no scientific justification. It just is a problem you can do, and you like yourself and think you're awesome. So you think it finds people like you. But even that isn't scientifically supported.

This qualified me to sit next to a pool in a condo complex where no one ever swam for eight hours a day. Best job I ever had.

I suppose that's why mathematicians came up with the idea of "optimal stopping." There are literally tons of scenarios out there where the goal isn't to get "the very best possible candidate," but to get someone "good enough," or maybe even the best person you can get after putting a reasonable amount of time and effort into searching.

https://en.wikipedia.org/wiki/Optimal_stopping

Sadly everyone who makes it to an on-site is now a competitive programming enthusiast who is probably a really poor fit to actually working in software development.

Many people are vaguely aware of this problem, but nobody has a strong incentive to put in the work to come up with anything better.

Unfortunately the OP seems to be using it in the former sense (when applied to the tries solution that he things will separate the "good" candidates from the plodding lunkheads).

And most programming jobs involve no complex algorithms at all. At most you have to be a little careful to avoid N^2 occasionally.

Leetcode is great for filtering out bad candidates but if your solution involves dynamic programming it is too hard for an interview.

- While DP might be rare, memoize is quite common alternative. Sticking a @functools.cache in Python for example. They are functionally equivalent in many cases,

- I believe knowing their data-structures and algorithms is the difference between an okay programmer and a good one,

- When interviewing entry-level software-engineers, there's not much else you can ask them except for what they learned in class, which is these kinds of questions.

For me, the difference between and okay programmer and a good one is: how much Linux/Unix they know (processes, core system calls, networking, sockets, awk/sed/bash). Whether you are writing apis for ecommerce platforms or writing custom k8s providers, Linux knowledge is a must. Knowing how to implement a trie? I bet most of us have never been in that situation.

Likewise, there are jobs where being able to reason on complexity and data structures is important.

They're not just functionally equivalent. They are the same thing. There are two things you need for a dynamic programming solution:

1. You remember the answer every time your function is called. ("Memoization".)

2. You structure your function calls so that they will ask for a lot of answers that have already been computed.

What you don't want is: say you're computing the sums of contiguous subsections of an array. You've already figured out that indices [2:5] sum to 3 and indices [5:8] sum to 8. When you need the sum of indices [2:8], you need to make sure you get it by asking for sum(2,5)+sum(5,8), and not by asking for sum(2,6)+sum(6,8).

At a high level they are the same, but it makes sense to distinguish between the techniques based on whether you need random access to the already-computed data.

For example, if you are computing the nth fibonacci number f(n), a DP solution needs only the state f(n-1) and f(n), but a typical memoized solution keeps around all the state f(1) through f(n-1). For tougher problems, you can't always convert between them trivially while keeping performance the same.

That's all true, but is it comparing like with like?

The "generic DP" solution to the problem is to allocate an array of n elements and fill it in from f(1) to f(n). That's the same amount of storage the "typical" memoized solution uses; cutting it down to an array of 2 elements plus an (implicit) offset between the part of the array that's in use and the full array seems to be an optimization on top of the DP solution.

You could apply the same optimization to a memoized form of the solution. I certainly agree that you most likely wouldn't do that, but you could.

What do we learn by comparing a tightly optimized DP solution against a "typical" memoized solution? Why do we give the DP solution credit for not using what it "doesn't need" while penalizing the memoized solution for what the programmer probably added unnecessarily?

Your broad strokes don’t account for the domain knowledge of a job. If all you’re doing is interviewing college grads, then I guess you have no choice, but this interview question sucks in just about any other context.

I worked a SWE job years ago which was heavily reliant on dynamic programming and string search/alignment/etc. It really depends.

Give me any problem related to algorithms+ds to be solved in 45mins without internet access and someone looking over my shoulder and I will give you a poor solution. But give me a day and internet connection and I will solve the problem in the most efficient/readable/maintainable way possible.

The interview question is not just about knowing data structures and is not about Behdad showing off either. It's the starting point for a Socratic discussion.

OK, but every program is literally one line of code. We only put line breaks in to make them easier to read.

The author makes this terrible comment about his "one-liner":

> Most candidates though write a for loop, which is equally acceptable.

He has also written a for loop. Specifically, he's written this for loop:

    for i in range(1, len(s)):
        if inDict(s[:i]) and inDict(s[i:]):
            return True
    return False

OTOH re the concern for O(n^2) time when n is bounded by word lengths, and when the inner loop was in already-optimized string comparisons in your language's hashtable library -- I think that part is very artificial.

Yes! The candidate already optimized for the important thing (which the author forgot about and added in a footnote), which is that the list of words could be very long, and that the words in an English dictionary are relatively short.

Optimising for the important factors based on an assessment of the problem domain is actually a more important skill than regurgitating the best algorithm from the book. If the author wanted to suggest a solution suitable for very long needles, he should have used a different domain like, say, gene sequencing.

It was great because he was testing the very thing I was going to be hired for. Additionally, he asked how I would design X or Y and I explained the pros and cons of certain designs and I could show him right there on the screen how I'd do it. This wasn't a whiteboard where I'd have to slowly write out code in a medium I'm not used to. I literally was typing on a keyboard, using intellisense, etc.

As someone who does iOS development day to day, it was a very easy way for me to show that I knew what I was doing and the guy was quite pleased when I plowed through all the requests he had.

Not precisely a SWE, but every single piece of code I write at work is more about knowing how the immense codebases are laid out and where to insert my change which is typically simple.

Given this, any "write self contained thing" is unrealistic. The only thing a coding question tells me is whether someone is happy writing simple code to solve a vaguely fun problem.

That still feels useful?

What is important is that the candidate’s skills generalize well to both the challenge in the interview as well as the day job. A question like this might be a good indicator or not depending on the job.

If people completely fail at this question it can reveal a lot of relevant information.

    def is_concatenation_of_two_dict_words(word, dict_words):
        """
        Returns True if the input word is a concatenation of two words in the input list of dictionary words,
        and False otherwise.
        
        Args:
        - word (str): The input string to check.
        - dict_words (list[str]): A list of dictionary words to check against.
        
        Returns:
        - (bool): True if the input word is a concatenation of two words in the input list of dictionary words,
        and False otherwise.
        """
        for i in range(1, len(word)):
            prefix = word[:i]
            suffix = word[i:]
            if prefix in dict_words and suffix in dict_words:
                return True
        return False

    def is_concatenation_of_dictionary_words(s, words):
        if not s or not words:
            return False
        n = len(s)
        dp = [False] * (n + 1)
        dp[0] = True
        for i in range(1, n + 1):
            for j in range(i):
                if dp[j] and s[j:i] in words:
                    dp[i] = True
                    break
        return dp[n]

That said, I believe an engineer with the answers it gave, would easily figure out from its output what the right complexity is. (Figuring out the trie, may not be so easy.)

That said, meanwhile, ChatGPT is not yet at a point where it can write out a full working repository solving a nontrivial problem. It can help, but it is not autonomous; and realistically, if someone can get that help to reach a good solution for a test, they can do so for a salary.

With O(n) memory, precomputing the rolling hash in O(n) time, you can get a hash of a prefix/suffix (or any substring) in constant time.

https://cp-algorithms.com/string/string-hashing.html#calcula...

I agree. We do work sample tests, and in addition to the code and docs the candidates hand in, what really matters is the walkthroughs we do with them. Why did they do it that way? What alternatives did they consider? What are the pros and cons? What past projects did they draw on? How did they research this?

Candidates usually enjoy this - most programmers enjoy talking about a just-finished project, especially when they feel good about the result - and you get to learn a lot about them.

If someone turned in an LLM-assisted work and lied about it, I doubt they'd fare well. And if they did use LLM assist - could be an interesting conversation all the same. What did you reject? What did you correct? Why?

If so, how do you solve the problem of the assignment just taking way too much time compared to a more typical process, where you do a phone screen and 3-5 hours of technical interviews with a behavioral or 2 thrown in there? I know that if you send me a take home that's any like most of the ones I've gotten in the past, and your competitor says I can do a recruiter chat and technical phone screen to qualify for a 4-5 hour onsite, I'm going with them just on the sheer amount of time and effort needed from me for each process.

How do you get around the fact that if you give such a test to enough people, some of them will eventually put it up on Github for the world to see and crib from? Even for those willing to accept the time commitment, how do you account for the bias involved in giving the same assignment to working candidates and people just out of school? There's a good bit of implicit age discrimination going on there, since folks who have been working a while are far more likely to have families than those just out of school. And then there's the old "fraudulent candidate hires someone else who's actually good to do the assignment." Granted, that person won't succeed at the job, but it'll take at least some weeks before most companies will give up on someone they hired, even if they're completely hopeless at the actual job.

I could go on and on, but I think you get the idea.

B) you replace time in interviews with the test you don’t just add it on as another thing.

C) you change the test regularly and implement basic plagiarism filters on the tests you do have.

As a person who has been in the industry a fair bit I think the flexibility of take home work samples gives me a leg up over day long in person interviews (and the last time I interviewed it was multi day not 4 or 5 hours).

I don’t have to take time off of work or not pick up my kid from school or anything. Just do it in my free time.

Seriously though, you've basically outlined the only reasonable way to go about it. The problem is really that that's a lot of fucking work you're opting into with 3 little sentences, and after all that, you actually end up with a non-repeatable process, which is not really ideal. I'm still highly skeptical that it's not biased toward people without families, either.

But my real issue here is that at least 75-80% of companies that do this crap obviously don't implement any of that stuff, and most of the rest don't do all 3. You really need all 3 in order to maintain any integrity to the process, and that's, unfortunately, one of the strengths of whiteboard hazing from the employer side. I know I'm never doing another 2+ hour take-home again, and the fact that employers generally don't do any of that stuff are more or less my reasons.

To me it’s a strong signal if a company has a work sample test because it shows they have at least one part of their process that has a chance to be a good filter. It’s an even stronger signal if they’ve replaced the majority of their pipeline with it.

I won’t say I won’t accept a job that didn’t do a work sample component but it would definitely jade my thinking against the firm. And I’ve certainly told companies I wouldn’t continue with their hiring process after hearing what it entails.

Better than the candidates I interview would do, and probably better than I would do, I would probably need a hint to use a trie and I haven't analyzed algorithmic complexity in a quarter of a century.

That might be something you are looking for in a member of your team, but it's not a great indicator that they'll bring much additional shareholder value.

That is, if your reaction to the problem is "Eww what a 'clever' problem, are you showing off that you know such stuff?" then you are not someone Google is looking for. (Of course, whether Google is right here is a separate question ...)

The danger here is that when you explain something to someone, and they get it, you get a warm fuzzy rush of happy brain chemicals. You will feel well-disposed towards candidates who appreciate and enjoy taking the breadcrumbs you feed them. Candidates who didn't think your clever insights were as clever as all that, or who fail to respond to your tutelage and try to solve it their own way... you'll get a negative feeling towards them. They're not playing your game.

That's great if you're interviewing a student to figure out if they will enjoy learning from you, and you will enjoy teaching them!

But while teacher/learner relationships might be part of your relationship with colleagues at work, it seems an odd part of the relationship to compatibility filter in an interview.

Also, once you used the same problem half a dozen times in interviews, you no longer get any "warm and fuzzy" feelings for having to explain something which is now blindingly obvious to you: if anything, you'll give higher scores to candidates to whom you don't have to explain such things.

Of course, if you are a candidate and (hypothetically) your response is "Actually it can be less than O(N), because a hash function does not have to read the entire string, and I know how it works because I've been dealing with string hash functions for years in my professional career," then by all means, fire away, I'm pretty sure any reasonable interviewer would consider it as a big plus sign.

The length of that word does not grow arbitrarily long, causing the asymptotic behavior of your algorithm to dominate. Because it’s a word.

In the case of ‘is this word of length n the result of concatenation two English words’, the time complexity as n approaches infinity is constant, because once you get past floccinaucinihipilificationantidisestablishmentarianism, any longer word than that you can just return ‘false’ immediately.

It feels really disingenuous when interview questions start off posing a seemingly innocent problem where the intuitive and practical answer is usually the one that'd you go for if you wanted to implement this in the real-world, then as soon as you answer the interviewer turns around and throws a hidden assumption at you. You could argue that the point of the interview is to have the candidate suss out these things, but it's usually a one-way discussion where you're not allowed to even question if the assumptions make sense in the real-world (which is what you ultimately care about).

And if the words are truly big enough to be an issue then you better start being rigorous and benchmarking different approaches rather than just handwaving big-o.

Ultimately when you get down to it, the whole point seems to see if the interviewee can be led down the pre-determined golden path, even if the path is only fools gold.

After all, if we assume that all words are at most, say, 50 characters long, then there's no asymptotic complexity to speak of. Any string longer than 100 characters is trivially not a concatenation of two words.

Google used to look for people who were excited about interesting problems. This is a question about basic complexity analysis and I would have chastised the interviewer if his feedback was ever on one of my interviews.

One interview I had years ago that I really liked was where they gave a sample of code with a number of issues and asked me to describe how I'd improve it. I really appreciated that a) it was much more like actual work than many questions, b) you could start with obvious things and work deeper, and c) there were many valid ways to improve the code, so you weren't just trying to guess the answer the interviewer had in mind.

I strongly suspect many interviewers (myself included) would have a few "top issues" and significantly favour candidates who pointed them out. But as the candidate you don't really know which issues will seem important to the interviewer.

It's not an irrelevant exercise though - code reviews are a real part of many roles and making sensible choices about what to comment on is a necessary skill. But it is hard to avoid turning the interview into an exercise in "guess what's on the top of my list"

These folks were obviously interested in differing approaches; we had a good conversation about it. And that's how I use broad questions.

But I agree a huge problem of interviews is the "guess what I'm thinking" interviewer. Most interviewing processes are not primarily about making good hires. They're about making interviewers feel smart and important.

For part 1 using a hashtable to look up the dictionary words, the complexity is not O(n^2) where n is the length of the input string. It should be O(w^2) where w is the length of the longest word in the dictionary.

The hash key material of the dictionary words has at most w characters. When computing the hash key for the input string, it can be truncated at w, as any longer input won't lead to a hash key making a match in the hashtable.

For part 1 using a trie, the complexity is O(w) rather than O(n). This is actually more obvious since walking down a trie tree will stop at the last character of a dictionary word.

Again for part2, the complexity is O(w 2^n) using bruteforce with hashtable, O(w^2 n) using DP with hashtable, and O(w n) using DP with trie.

Edit: Also for short strings like dictionary words, O(w) is kind of meanless since with SIMD or SWAR the short word can be folded into one or two DWORD. You can effectively treat it as O(1).

I don't think the function "inDict" is even defined as either local or remote query (unless I missed it), which really influences the answers. Querying a short string against in-memory structure will be based on "n", but querying a long string against any remote database will be based on "w".

As I age, one thing that I realise about interviewing: It is scary as hell. It is basically performance art. And, most nerds are terrible about public performance & speaking. Some days you are really on it; other days: way off. Every single job that I ever got in my life feels like a lucky day. Also, the tech universe is infinitely wide at this point, so somebody somewhere can ask a question that you will fail.

Since this industry is so dominated by men, it also feels like a lot of machismo. So many interviews are just a jerk dude on the other side of the table (probably working for an amazing company) that wants to make himself feel better by making me look foolish. There are too many stories to tell. I have also kept my mouth shut when interviewers were insistent about a technical point that was 100% verifiably incorrect with a simple Google search. It helps to stroke their ego during an interview. When they give you a hint, or explain a better solution, try: "Oh, that's really impressive! How did you discover that algorithm?" You are more likely to make it to the next round. The goal of interviewing is to get the offer. You can always turn it down. If you had one jerk in the interview stream, but they work on a different team, it should be OK to accept the offer.

Another thing that I say to all interviewees now: I know that interviewing is scary. We are interested in what you do know, not what you do not know. Together, we will dive as deep as possible on a few topics to find the outer limit of what you know. If we come to a topic that you don't know about (or not deeply), tell us; we will move to a new topic.

I think I read on here about the concept of successful people really just doing more things to increase their luck surface area.

In other words, you set yourself up to interview as best as you can, then make a bunch of connections and schedule a bunch of interviews, and (hopefully) at least one of those will be on a good day.

It's such a weird and, frankly, shitty way to make hiring decisions though. But at the same time, we don't have much better.

    I think I read on here about the concept of successful people really just doing more things to increase their luck surface area.

The trick for me was diligently applying to as many places as possible (that were at least somewhat of a fit), and eventually coming across one where the main technical part of it was "take home". Coding and discussing thought processes on the spot for an audience isn't something I'm good at it turns out, but solving a practical problem on my own time was fine.

It's ultimately a numbers game, the more places you apply to, the more likely you'll find an interview process that fits. That is the most important thing to keep in mind--shotgun blast those applications. It's hard not to take rejection personally, but don't give up and settle for a toxic job, that's even worse.

After a while, either they will make you offers to work for them on the spot, or they will at least "fast-track" you through the interview process.

It doesn't work in all software areas (it is pretty difficult to do it if you are an embedded systems developer) but it works especially well in others (e.g. cloud)

and you suggest providing free labour for an entire year to some companies. No wonder GP hates this industry.

>For the next year try contributing to open source projects that you like and that have companies behind them.

You studied field X to work in field X, and in a job interview the "smarmy git" considering to employ and pay you has the temerity to ask you things about field X to see whether you've understood the basics of field X, can apply them, explain your reasoning, and think on your feet?

And that engenders (and I quote) hate, sadness, anger and despair?

I have interviewed a lot of people for coding jobs, and I usually try to test ability at doing things like this, and when I find someone junior who already knows how to approach problems like this, I am always very in favor of hiring them. Based on the discussion here I'm definitely not the only one.

That is who you're optimizing for.

(I still think the question is unfair, but my objection is more about the stressful interview situation where the real problem you're solving is "desperately trying to pass this or N other interviews you're currently doing").

What's the function going to be used for? Are we talking about strings of length 32 or strings of length 32k? Is this going to be run a billion times a day or 12?

We're not competing at some contest, we're trying to solve real problems here. I'm not going to waste a couple hours hammering out some ultra-optimized code so the logAnalyzer4000 can run its hourly batch jobs 0.004% faster. If we need it to handle strings of length N with a dictionary of size M comprised of strings from length X to Y, tell me that, and we can plan (and benchmark) based on that.

It's an extremely simplified but recognizable version of a real problem: natural language input tokenization, which of course has real consequences.

I could ask a detailed question on cryptography, robotics or reverse-engineering, and it would be just as (ir)relevant to the SRE. Computer science is a much broader field than your niche area. Arbitrary goal-posts are arbitrary.

What folks need to remember is the employers who are primarily known for asking this generally have no problem hiring people who are good programmers and ALSO good at e.g. Dynamic Programming. And no, as much as some try to frame it, the two skills are not directly at odds with each other. Therefore, between a good programmer who can “Leetcode” and one who cannot, why wouldn’t they pick ones that are good at both?

And sure, if your company don’t have that luxury, don’t cargo cult. Hire a good enough domain expert for your specific needs.

Among those who pass, the two components might seem negatively correlated. This is because you can score a 10 on one section and pass with only a 5 on the other. However, among the general population both sections might even be positively correlated.

Do you seriously think that claim is true? That coin toss predicts software engineering ability as well as leetcode? I don’t think any reasonable person can take this seriously.

Back in 2015 Peter Norvig discussed an analysis at Google that suggested doing well at coding contests was a negative indicator of job performance: https://youtu.be/DdmyUZCl75s

Presumably they do well at these sorts of questions, but that doesn't imply the same actual effectiveness.

(Personally I would add that human measures of “job performance” can also be quite subjective.)

Yes, implicitly Google can only measure the job performance of people they actually hire.

And who is supposed to implement the robust canned implementation?

And that's fine! A specialized subset of engineers. Maybe that subset is exactly what you need.

I wonder how the stress of the interview process selects out people who understand O-notation and to whom a Trie would make sense if explained to them, vs people who have these concepts fresh in their minds and/or can handle themselves in stressful interview situations.

I only know 1 guy who did coding competitions. He was in fact a good developer.

Why would you think a person that enjoys such a thing wouldn't be?

Its just another signal of aptitude, its not some magical catch all heuristic. However, people here suddenly expect like they are allowed to forget all the (very fundamental) basics and get into a job that involves coding without any coding at all. Now that leads to a bad hire.

Isn't that in itself is a flag that this is in fact not a great interview question, unless it's being used for hiring candidates fresh out of college, say during campus interviews.

It's a whole other debate that most software engineers, even at BigCo end up writing CSS code to move pixels only and lose their DSA knowledge over time...

It's a bad question because _how well_ a candidate does past that minimum threshold has _no correlation_ with their ability to do the job.

You can treat it as a binary question for that minimum floor. To use it for rating, you're doing your org and the candidate a disservice and you're wasting interviewing time that could be spent more productively and humanely.

They probably did know how to code, but they’ve spent the last 5 years working on something quite a bit different from string wrangling in Python. Add in the stress of a (first-time?) interview and a reasonable candidate can look like a total fool…

Is this common at Google?

If by DSA knowledge you mean leetcode puzzle knowledge, yes, quite certainly. But from my experience fresh grads are less effective engineers before they mature by working with industry scale systems for a few years. And at that point they would have been less able to do your problem but a far better engineer.

Which means your interview question is aimed backwards, maybe if you only used it against fresh college grads then the gain function would be positive, but you applied it against everyone.

Actually the best questions I ask, and the ones that give me the best candidate "signal" are what I call the "clean code" style questions (as in code that is simple, easy to extend, not necessarily Uncle Bob Clean Code).

I'd rather not post examples here because I still use these questions, but the theme is as follows:

* These questions ask to implement functionality that you could see in a (super simple) real world application.

* They start super simple, and include several follow ups that increase in complexity.

* No data structures and algos beyond the basics. Loops, lists, hash maps and little else. What most programmers would use on the job on a daily basis.

I found in practice these work really well, specially with experienced programmers who can flex their low level design skills. People who ace leetcode style questions tend to do worse here, but no wonder, since they've been trained in a complete different style of interview.

Key point: I've never met anyone in the industry whose job it is to implement algorithms without reference materials all day long. It's more like... you might find yourself needing 1-2 clever bits a year, and the rest of the time you're tweaking model classes and hunting down UI bugs.

Algorithms are both easy to Google and these days, CoPilot will do the hard 4/5ths for you if you actually need them. It's also common that the really intense stuff is written by the PhD braintrust cofounder, not the people being hired after the fact.

What I want to know when I'm interviewing is simple: how do you approach solving problems; do you have an interesting personality that I'd like to interact with every day; are you the sort of person who has a strong instinct about what the next thing to do is, and will default to working on that thing without prompting?

Do you think learning new things is exciting? Are you able to admit you don't know something and do you feel comfortable asking for help when you're stuck?

Do you seem reliable? Are you a starter, a finisher or both?

Do you have a sense of humour? What do you do on the weekend?

Ultimately, I care what you know... but I am far more interested in your capacity to quickly master new things.

That's not what the author's question was about. Rather, it presented a problem, and wanted to see some approaches to that problem and a discussion of their properties (runtime, limitations, etc.).

> how do you approach solving problems > Do you think learning new things is exciting? Are you able to admit you don't know something and do you feel comfortable asking for help when you're stuck?

All very reasonable criteria. So how do you evaluate a candidate on those metrics, if not by giving example problems? (ideally different ones: a few theoretical ones like the one under discussion here, a few practical ones with access to google/Stackoverflow/etc.)

> do you have an interesting personality that I'd like to interact with every day

That's your prerogative, of course, but I wouldn't make that a hiring criterion. I have colleagues who are incredibly boring, yet deliver solid consistent results. Who cares about their personality?

I suppose I perceive the narrative differently; it really sounds like the author has a complex roleplaying scenario in mind, where there's a base assumption that he is the smartest person in the room and pre-plans 3-4 moments where he can assert this. If I was the candidate, I would be praying that this guy isn't my direct boss.

> So how do you evaluate a candidate on those metrics, if not by giving example problems?

Simple: I talk to them and get them talking about how they have solved problems in the past. I will dig in on interesting points and take careful note of telling details.

Perhaps this will help: I have enough experience as a developer that I can tell very quickly where the person I'm talking to falls on the skill spectrum. Part of my negative reaction to this post is that I sense the author does as well; however, where he chooses to make them feel small, I just keep asking questions because I want to give them more opportunities to sell me on their strengths. Their ability to whiteboard leetcode in an interview has laughably close to zero relationship to how they will perform on a Wednesday in jeans without feeling like they are being interrogated, so I largely skip that.

If you cannot quickly tell someone's experience (or BS level) from talking with them, then this interview style will not work for you.

> Who cares about their personality?

We have to agree to disagree here. Personality is huge, probably just behind potential. In real terms, if I am going to spend more time with my coworkers than I am with my life partner, they better be trending towards great company.

One of the questions I ask myself during an interview is whether I would want to be in a band with this person. If I wouldn't enjoy driving across the country in a van with them, what makes me think I would be able to do my best work if I'm trapped doing it with someone I don't like?

There are enough brilliant nice people in the world that I can comfortably avoid hiring brilliant jerks and come out way ahead on every metric.

> Do you have a sense of humour? What do you do on the weekend?

It's none of your business what I do on the weekend. It doesn't need to be similar to what you do.

Otherwise you'll tend to only hire people like you and reduce the pool of talent because you can only work with a very specific type of people.

My intention for the question is not to judge the answer, or compare it to what I like to do; the important issue is that they have an answer!

I cannot tell you how many folks I've interviewed (and not hired) who, when asked about their hobbies, immediately start listing programming languages and frameworks. This is not the answer I'm looking for.

In fairness to me, there's an almost infinite number of great answers to the question. You can say "music"! I will really respect someone who tells me that they like television. That person is honest. Cooking! Model rockets. Literally just say anything other than a bunch of acronyms.

I'm specifically not asking about which church or sex club they go to, and that's made contextually apparent from the conversation leading to that point.

I have a strong belief that a well-rounded candidate has interests beyond coding. If they live to code, they probably aren't going to be a good fit, so I save everyone the trouble.

When you never did any hiring, you assume you're mainly looking for good candidates. When you actually have the experience, you know that at least 30% of applicants can't even code the most basic thing.

So I was wondering how you, hiring more than 100 developers, weeded out those candidates who "do the talk but can't walk".

Another thing: how many of those 100 developers you hired were bad hires?

Perhaps this will help: I have enough experience as a developer that I can tell very quickly where the person I'm talking to falls on the skill spectrum. Part of my negative reaction to this post is that I sense the author does as well; however, where he chooses to make them feel small, I just keep asking questions because I want to give them more opportunities to sell me on their strengths. Their ability to whiteboard leetcode in an interview has laughably close to zero relationship to how they will perform on a Wednesday in jeans without feeling like they are being interrogated, so I largely skip that.

If you cannot quickly tell someone's experience (or BS level) from talking with them, then this interview style will not work for you.

I have had a dud rate of about 8-12% over the past 15 years. However, only 1/3rd of those were duds because they fooled me. It was almost always personality issues that I didn't anticipate; not getting on well with other people on the team, etc.

One of those that fooled me came in hot because he had started a popular testing framework, so I didn't dig. I fooled myself by accepting him based on the reputation of the project, not the person. Lesson learned.

However, I would like to turn the question around and brag a moment about how amazing some of the hires were. I hired the creator of Sass, a future Basecamp developer, several future founders. No whiteboards in sight.

The things they can tell me is basically their curriculum. What I found was working the best was showing them code and they they explain to me what's happening, and I ask some practical questions on that code ("what would happen if I change this?", etc)

I moved from your kind of inteview to a more technical "explain this piece of code to me, what would it do?".

But it's possible the kind of developers I see have different experience level than the ones you see.

I also still interview seniors (maybe 1 out of 10). Maybe I have to think about changing their interview a bit.

The thing is with frontend, you can get a senior that basically did mostly html and never serious programming. And so if you build a complex web app, you really need to see how far they can go with their development skills.

To be honest, it's super frustrating that we can't have that conversation without an army of bootcamp grads accusing us of elitism. Only a decade ago, it wasn't controversial to expect that people applying for highly demanding technical positions were ready to start on Monday.

I digress.

As to the senior HTML artist... I feel like that's on you to suss out in the conversation. Another one of my elitist views is that someone who can only do HTML - literally, text markup - is not a senior talent.

If you want to make sure that a candidate understands how an MVC framework works, ask them to talk about their thoughts on service objects, or when they use Redis vs Postgres, or whether they have used Sidekiq jobs to do cool things. You can interrogate them to a wild degree and check off all of your boxes while making it feel like you're just shooting the shit.

It's almost impossible to bullshit your way through explaining how you implemented a key-based authentication mechanism over websockets. Writing out the code on a board accomplishes nothing. I know how it's done, and I'll know in 15 seconds if a person shuts down, terrified, when talking about it.

This kind of 'let's watch people struggle with my favourite LC puzzle, which I think I know inside-out' just seems very inconsistent, not very predictive, and just quite flaky to me.

On the job, my very first response to any kind of problem that looks algorithmic is to google the hell out of it, even if I think I know the answer already. In future, it will be to get gpt to write it.

I think the actual challenges in real-world software are not algorithmic, but architectural and beyond, and then only after thoroughly understanding the problem in the first place.

> I continued to use [the interview question] because I got good signal from the candidates.

Working at Google, this is something I hear a lot: that some question, or other interview thing, is good for getting "signal" from candidates. But how do you actually know?

Like, I feel to definitely know a question is good you would actually need to track candidates after they start working and see if their performance on the question correlates with their job performance. (Unfortunately, it seems impossible to track candidates who don't get an offer.)

As an example, the author of TFA clearly is selecting for people who have the patience to sit around and be told how smart the author is. That is the signal in this “great interview question”: candidate will suck it up for hours while I tell them stuff.