> I don’t believe we ever lost a candidate due to them not being willing to take our test. The strongest candidates take their job search seriously and are willing to devote a fair amount of time to landing the right job.

This is the part that didn't ring true, at least for me. I believe that there are some candidates would not complain. But if I am good engineer, odds are that I already have a job that works well for me. Maybe not the perfect job, but working well enough that you'll have to give more incentive than just a job being available before I'd spend a day on a take-home, unpaid project. So according to the quote above, I'm not the strongest candidate because I'm not willing to devote a fair amount of time to landing the right job.

Maybe from the hiring manager's point of view, that is even true. But I've already landed the right job. I work there, every day. If you want to poach folk like me from a job that is already right, 6 hour take-home tests aren't the answer. Give us something quicker. Maybe not easier... I believe a challenging interview process is fair. But quicker.

I also don’t think this rings true. Picture this: you have a high-stress job that also takes over your weekends on occasion. You have kids or elderly parents that need support. You have a long commute.

How can you fit in _another_ project (albeit small) in a week’s time (heck, make it two) _and_ manage any two of the above at the same time, when setting aside an hour (or more, if it also entails on-site) is already a very significant investment on your part?

I saw a role open at a competing company for a bigger job that I would have been perfect for. They wanted me to take a personality test, which was supposed to take an hour, but could be paused/picked up again, and then a coding test which was going to be a block of 3 contiguous hours.

I volunteer teach, I am president of my condo board, help run a local meetup, have a demanding job, oh and a family that needs love too- my free time is precious.

I started with their personality test (Calipers). It actually took 3 hours over 45 minute sessions that took the better part of a week. Fitting that in was extremely stressful and took away pretty much all of my personal time. When it came down to blocking out 3 hours for their coding test, I just could not be bothered- this would require me to more or less have to lock myself in a room for a good part of a saturday or sunday, toiling away still to only have a chance for a new job that might work out better than my current (I quite like my current job, this was just a potentially bigger role), but maybe not.

And I wasn't even in the market- Imagine now I am doing this for 5-10 (if not more) different companies. Its completely unreasonable IMHO.

That is deceptively easy to write when you're not having to do so in order to "keep up" because your work culture is so toxic that there is clear-cut hypocrisy between "work-life balance" and "we want you to be at the customer 300Km away tomorrow morning".

I am a single dad with custody of two kids and I would be able to take the six hours. I've consciously created this life. There are other parts of my life I'm sure that I do not do as well as you (for example my workout regime has gone to shit) but it's entirely doable.

That being said I also haven't needed a job for a very long time so...

I think taking the 6 hours _once_ can be done by pretty much everybody, that should not be a serious problem.

What kind of gets me thinking is that, in my experience, people will take a look at multiple options before committing to a career change. This is where this concept seems to fray a bit.

Still I think it is a valid concept to minimize your exposure to false hires so it’s ok.

At my shop we opted for a similar approach but exchanged the project with a pair programing task that candidates get up front (and yes, we created reference code bases in many, many languages).

One other departure is that we don’t subscribe to the hard comparisons mantra. Candidates get to code with someone from their future team(or two sometimes) and it’s just a thumbs up or thumbs down signal. Thumbs down would usually mean that the process also gets cut short.

Shouldn't that make you extra motivated to find a new job, though? ;)

another question is: do I care enough about your company in order to invest this time? there are many job opportunities, why should I choose to invest my free time into this?

you guys are forgetting that this is a partnership, you also have to sell me your company.

if you're ok with that, I think you're missing out some good candidates.

I didn't mean to make a personal offense, just that this shouldn't be the normal.

Spare time - and time in general - is managed by priorities. While I did (on both sides of the interview process) a take-home technical test, it was always AT MOST a 2 hours test. A six-hours test is cutting away a lot of potential candidates which have higher priorities in their spare time (kids/parents above all).

Wouldn't you actually be motivated to take the 6 hour assignment to get away from high-stress workplace that is taking also weekends and with long commute, to land a position that has proper work-life balance, higher salary and maybe closer too?

Sorry, you are describing a candidate who's working life is hell and sounds to me they would be willing to do whatever it takes to get out and get on.

i mean, if they want me to do a 6 hour take home test for a chance at interviewing, that would be a strong indicator that they DON'T very much value work life balance.

Now, consider if they instead said, we don't need you to come onsite for 4 hours at all, just take this 6 hour at home test at your own leisure at your own time, and we will give you an offer based on it. That would speak volumes about the type of company and culture that they are.

I would very much doubt that a company that demands 6h of overtime even before we sigh a contract actually offers good work-life balance. If I was cynical, I would wonder whether this test's purpose is to filter for that reason.

I agree. I probably wouldn’t complain about a 6 hour take home. I’d probably just ghost them. There’s no way I’m spending 6 hours just to get to an on-site when I can do it after at most, a recruiter call and a ~1 hour phone screen.

I remember getting a 6 question take-home assignment from GE, after telling me they will pay me lower than I'm currently making for a higher position, in a higher COL area. Promptly never responded to that email thread

Why didn't you just tell them those reasons in literally one sentence? Why do people think ghosting or not responding back is some sort of power move that's worthy of bragging? It lacks even basic courtesy.

Almost every time I didn't get a job offer during the interview process I got ghosted. They probably do this for liability reasons. But why would I afford them the same courtesy?

> But why would I afford them the same courtesy?

Because you're NOT affording THEM the courtesy. You're doing it for US as a community of software engineers.

I read a lot of complaints on here about companies do x, y, or z in their hiring processes and we don't like it.

Fine: tell them.

You know why? Because companies often listen to this feedback. Sure, some don't, but plenty do, and we certainly do.

If you are hearing the same thing over and over again from great candidates about why they don't want to go through your hiring process[1], and you're not able to hire engineers in the calibre and numbers you need, you'd be pretty obtuse not to do something about it. Even if you are able to hire the people you need, you might choose to make changes so that even unsuccessful candidates have a good experience and are more likely to speak well of you, as well as to increase your pool of better candidates. Sometimes those changes will take an awfully long time to occur, but companies do listen, and they do make changes.

(There are of course some companies that have so many people who want to work for them that they can afford to have what many of us would perceive to be an unduly arduous selection process and they're not going to change however many people complain about it, because they have no need. They have a different problem that they're optimising for.)

You might complain that this is not your responsibility, and that's fine: it's certainly your choice. But, if you want to see a net improvement in hiring practices across industries that hire engineers, you may want to take some very small portion of the responsibility that we can all choose to take for making that happen.

Let me be clear: it is much more effective for you to give companies with poor practices feedback directly (and politely, even if through gritted teeth!) than to complain on a substantially anonymous internet forum.

With all of the above said I'm now going to move into perhaps more controversial territory.

Not everything is about how good an engineer you are. In fact, your technical skills, whilst absolutely essential for the job, are not the most important factor.

We care about character. We care about how well you're going to work with others. We care about attitude. We care about maturity. We care about entitlement (or rather we have a strong preference for the lack thereof). We care about listening skills. We care about humility. We want people with initiative and skill, but we don't want divas.

Sadly this sometimes feels like we're spending more effort on weeding out negative traits than selecting for positive ones. Still, as I've said these character attributes are quite a bit more important than candidates' technical skill.

Fundamentally, we want to work with people who are easy to work with because in a team context you want everyone to rub along well together. We want people to treat eachother courteously. We want to avoid unnecessary drama.

[1] I've chosen that phrase carefully: I'm deliberately not discussing people who don't want to work for you, because there can be many reasons for that that have nothing to do with hiring process, culture, salary, or whether you're actually a good place to work (location, product, sector, etc.).

[2] Note: character, not personality. I don't mind what your personality type is (extroverted, introverted, whatever). We can work with anyone, as long as they're not an asshole.

Take one look on glassdoor for almost any company and you'll find a slew of feedback on their interview process. How much of that is taken into account when updating their process? I've found almost none. I can almost always use glassdoor negative interview reviews as a accurate indicator of what the interview is like - 9 times out of 10 no hiring team incorporates the feedback. And why should they? I'm sure right after my interview they'll find someone who 'gels' with whatever process they came up with and hire them. Sometimes, why even go through that process? If I hear something upfront I don't like, rather than listen to a hiring manager tell me they'll log me in to their system and try to barter and convince me to overlook their company's flaws, I'll ghost them with no remorse.

And also I've worked at a company that was linked to a callcenter, I can say with 100% certainty, how someone behaves during an interview is never an accurate portrait of how they behave on the job

Some hiring processes are accidentally bad and benefit from feedback. Other processes (like 6hr take home tests) show an irreconcilable difference in value systems that mean it was never going to be a good fit. No amount of feedback is going to change that.

That's a good point.

It literally doesn't make a difference in those cases. Companies of that size have position leveling and paybands for each level. They set those bands based on market reports from companies that collect salary data in part for this exact purpose. They'll target percentiles of the market rate for hiring based on their budgets and the minimum level of skill they need to function (they're not looking to "hire only the best"; that's a polite fiction that everyone sees through).

It isn't intentional usually, you just procrastinate the rejection until it has gone so long that it would feel weird to respond.

I have never ghosted anyone or anything, but I can see the appeal if you are not a very confrontational person.

A lot of people get angry if you tell them the truth. If you try to soften the blow then they misunderstand and clearing up the misunderstanding then leads to them getting angry.

I've had the same experience. After two on-site interviews, an assessment, and a series of tests. I got offered 2k/month for a full-time job (despite my current job making about 25% more at the time). I declined on the spot, then got a mail with the same offer (including the standard "we are confident we're giving you a good offer". I mailed them that I declined, got no reply, and got called the next day (during work) if I could please respond to the mail.

Weirdest interview experience ever.

There is no harm in responding and letting them know. I did this for a job and after they investigated they overhauled their process quite publicly.

Agreed, I just turned down an online coding interview as it required 3 hours dedicated time, especially for those of us with families 3 hours at home is precious and not usual uninterruptible. Companies who set tasks like this probably aren't the sort of company you want to work for.

Prior to having a family, I'd spend 12 hours on a test but claim it was only 6, now I'd be pushed to spend 3.

It’s fine to have different priorities, but don’t expect to get rewarded for them. In financual firms where I have worked, everyone puts in at the very least 10 hours a day, and at least 8 hours total on the weekend, regardless of whether you have a family or not.

People who give their firm everything they have generally get rewarded for it. Those that don’t in some industries (like finance) are quickly fired. In others, it just means you get passed up for promotion and make less.

Just realize that everything has an opportunity cost and that nothing comes for free (both spending time with your family and spending time at work). Also that people have different priorities so don’t make blanket statements about what people want.

> People who give their firm everything they have generally get rewarded for it.

The first job I got out of college, there was a sea of Solution Architects. They all worked around 60 hours a week, if not more. If a boss said jump, they would always ask how high. If a boss told them they needed to take a red-eye tonight, they always did it. Yet every time I went into the main office, the group was entirely different.

That's an example of a group of people who drank the Kool aid, who were "generally" supposed to be rewarded.

> Just realize that everything has an opportunity cost and that nothing comes for free (both spending time with your family and spending time at work).

It's a sign of a deeply sick culture when everything has a transactional value.

Opportunity cost is a property of time, not culture. Unless we figure out how to live forever or stop time, this will always be the fundamental constraint on life.

> Opportunity cost is a property of time, not culture.

I'm aware, I have a degree in economics. I'm talking about cultural values that force constant optimization onto individuals as being deeply sick.

Regardless of if every action has some minmax strategy, it's not healthy to have individuals constantly running Bayesian analysis into if they should spend time with their family, or take a second job.

Do you have an alterate paradigm?

Letting people breathe

As someone trained in economics you should already be aware that everyone chilling isn’t a stable equilibrium. For example let’s say that everyone only works one day a week. Now I can double my output by working two days. So we can look at the relative productivity gains of working one extra day as a function of how many days on average everyone else works:

1 day: 100% 2 days: 50% 3 days: 33% 4 days: 25% 5 days: 20% 6 days: 16% 7 days: 0%

As you can see, the less everyone else works, the more value your extra work is. I’m not saying that the equilibrium is gonna be 7 days. But it’s not gonna be less than 4, at the very least. This is also the reason why the 20 hour work week will never catch on.

> As someone trained in economics you should already be aware that everyone chilling isn’t a stable equilibrium.

Wow TIL.

> As you can see, the less everyone else works, the more value your extra work is.

It's surprising someone like yourself subscribes to the Labor Theory of Value.

LTV is a fundamentally classical liberal idea, so it really shouldn't be that surprising. Even so, the supply and demand model offers a much more coherent vision of prices in the marketplace.

As such, we can view working more than anyone else in supply and demand terms. People who produce the most are always in high demand, and the less of them there are, the more value a high productivity individual provides.

Ah yes the soothing feeling that at any one time no matter what I am doing I am loosing out on doing something else. Such are the inner workings of the ever rational and goal focused financier who embraces this reality in its fullest form and yet still decides it best to prioritise their pride. "Do not fear the doubters" says the fund manager "they are simply jealous of your achievements and commitment to the fund". Of course this is not said out loud but rather communicated subtly through the language and the ideology that underpins the language of finance.

I would do the 6 hour test, but not if they told me the following:

Finally, we would all vote again, and this time, a “hire” decision needed to be unanimous to move forward.

Maybe they better start with the social part first then, not wasting your time for in case in the end someone happens to not like your face.

Exactly, the ease with which companies put you through multiple rounds of timewasting just to reject you on something small (or even not telling you the salary range) is staggering.

I've gone through rounds of interviews and take-home assignments and calls just to be given a very low offer. Start with that, save time for us both!

Very recently went through 5 rounds with a company before they told me my salary ask was too high.

They’re probably not going to convince you anyway so filtering you out at the top of the funnel is helpful to them.

I once had a founder offer to intro me to other companies if I wasn’t interested in his, which I thought was a pretty brilliant move that ended up saving us both time.

As the first stage of the interview, we give a take-home test with no constraints. We just ask them to put together some code that works with one of our public APIs.

I think this is decently respectful of the candidate's time, while giving us the opportunity to review some actual code written by them. And yet, most of them don't even bother... Well, I get that you are busy, but I don't feel comfortable investing time in an on-site interview without seeing any code first.

Disclaimer: If they provide a link to their github with some solid projects or OSS contributions, we skip the take-home altogether.

So I have a slightly different take. I actually prefer take home tests (that are time bounded to say 3-4 hours at most) as that gives me the ability to think "naturally".

But, I would only do this for companies I am desperate to get into (Ironically FAANGs for 'now' as they are the ones who are least likely to pull a WeWork/Uber/Theranos on me) and if I was pitched this by someone puritanically Id show them a finger. I do take my job search seriously and any hint of a company devaluing my personal time is a huge affront to me (and my family).

How would you give an interview that is challenging, fair, and quick?

I've had the most success doing a remote, one-hour pair programming exercise via screen sharing with the candidate. They're allowed to look at docs, do any google searches, and I don't penalize them if they don't finish. I'm more interested in the process and the way they navigate their environment to solve problems. I continually coach them to speak aloud whatever they're thinking, especially if stuck on something. My last hire was someone who didn't finish the exercise, but I really liked their approach to solving the problem. It's the closest thing to "what would it be like to work with this person" that I've found. This has been super effective for me after many years of making bad hires through "technical interviews" and take home assignments.

I encourage you to drop the whole "see how they think" process. You can't read minds, and it's not possible to see how they think.

I don't like complaining without a suggestion, so... consider letting a candidate code on their own, and then ask them to walk you through what they did. You won't be disturbing the process by staring at them, and you'll get to see a bit about how they work independently & articulate what they've done. Unless you're a pairing shop, that's really what you're looking for anyway, yes?

There is no “one true way” for hiring people. It depends on the position you’re hiring for, the team, the office environment (if there is one), the project, the budget, and so many other things.

The right hiring process for one position may be completely wrong for another. And that’s OK.

I have a strong dislike for the 'see how they think' or 'code like you would in production' style instructions for take-homes - they're not specific enough.

I don't think that necessarily applies to real-time pairing exercises though. The interviewer has significant latitude to interactively question and prompt to get at the skills and values a candidate possesses.

One time a guy asked me to bring in my laptop with some code I had written.

I showed him, we talked about it, he asked me to add a simple feature, I think I taught him something about the tech stack that he didn't know. It worked well for both of us.

Do a bunch of algorithm whiteboard interviews with a super huge false negative rate.

The trick is to get them to review your code, rather than the other way around.

Had one company paid me for my time developing their test interview app. That was the first and only experience I had. Took a day to do, but I made sure it was really polished.

We pay our applicants $50/hour for the later stages of skills tests because they can take 15+ hours to complete. More details on previous comment if interested: https://news.ycombinator.com/item?id=20710884

If you pay only for the later stages of skill tests that's 4-8 hours. You probably don't pay over $250.

But you do use the work to build your backend/office applications. In the end you are paying someone $250 for a 15 hour project + two days of in person development time.

Great idea. Many with follow.

"The costs of travel and lodging are compensated at this stage, but we are not paying them hourly. The work they do for us is not customer based work and does not generate revenue for us."

> What’s so compelling about your company

Paying for the interview is compelling. Almost nobody does that, so it's a differentiator. My first assumption on hearing that is "it sounds like they respect their engineers as professionals more than most companies do".

3-4 days of testing with two being on site.

They don't hire people who have jobs because no one would/could do this.

We often hire people with jobs. At most, it's three work days needed. If that's a problem, we can do the proctored skills test on a Saturday and we can cut the on-site down to 1 day. That's then one business day needed.

Most devs with existing jobs have good jobs with decent vacation/PTO policies. So they have days to invest if they want to.

Don't be so quick to assume the worst. :)

I think, it ends up being different strokes for different folks. We have a very detailed, and for some, a very compelling job description and offer. The time commitment is relatively small up front and as they engage with us, they get a sense of who we are, how we operate, and they like it. So, by the time we ask if they want to put the time into the skills test, they've had a chance to evaluate us and ultimately feel like it's worth it.

Simplisticly, it might boil down to something like a preference for quality* interviewing over quantity.

* not saying our process is definitively better than others, just that some candidates want fewer interviews that they feel are a better potential fit for them.

I get paid for it, and also get to work more closely with potential future teammates to better test the relationship.

Having had interviews at companies where it was NOT as advertised during the interview phase, I'd value that.

If I was already interviewing for another job outside of my existing one, I'd gladly take a week off to do it.

But if they were trying to get me to leave a job I already liked, I wouldn't commit that much time for it.

I had a YC startup invite me to work for them for a week, for which they paid me a decent sum of money. I was able to push a decent amount of code, considering the onboarding process was “here’s a laptop with the repo and some basic tools on it.” They did not hire me.

You should highlight them by name, more companies should do this.

Those answers should be “yes,” and “no,” respectively, because this scenario is properly a very short term contract. The employer should issue a 1099.

Some payouts are under $600, so we just cut them a check. Over $600, we issue a 1099.

Yes, it was basically a contract work.

This doesn't really satisfy the "quick".

You have stumbled upon the iron triangle of interviewing software developers.

This is especially true if the startup is unknown or not perceived as glamorous/prestigious. It may have the effect of skewing the experience curve towards younger/less experienced candidates.

indeed. it is selection bias.

Just an FYI. Every company talks about the great culture, work life balance, competitive pay and excellent benefits they offer. Jobseekers have to assume you are lying about those things, because most of your colleagues are. (this is why I suspect firms like new collage grads, they haven't learned that you have to give a price for a 50 - 70 hour week even though "overtime is rare") You can't really know how a company really is until you have spent some time and worked through a crisis or two. I used to think I worked at a great place, then the day my daughter was born I received dozens of slack messages and phone calls about testing environments and people were legitimately upset that I spent time with my daughter on her first day on earth rather than help them with their poor planning that could and did wait for me to get back.

Wanting me to jump through a bunch of performative hoops makes me assume you are in the "call you on the day of your daughters birth" camp.

So - the purpose of the take home task is to weed out the people who are not interested enough in the company?

I mean in the article you say "The strongest candidates take their job search seriously and are willing to devote a fair amount of time to landing the right job." the times when this has been true of me has been times when I do not have a job. If I am being headhunted for a job and I am already in a good job, I have a hard time envisioning a scenario where the job being offered would be so enticing that I would be happy to make a significant investment of my time although I might do it grudgingly if the offer is good enough.

When I took the job at a small agency I'm in now I had stopped a consultant task that had run out of funding. I was in a hiring process with four other companies, the small company in the interview asked me how much I wanted I told them my price they said "that's a lot but we expected it would be something like that so it's ok" and then they said something that really cemented my interest in working there than any of the other places that I was interviewing with - they said "we generally have a coding task to do but in your case I think we can skip that".

thank you for saving 4-6 hours out of my week! My family thanks you too!

In fact now that I think of it this testing is another reason why I give people a significant raise as a requirement for moving from my current job. I am earning pretty high up for my market, the market is like many tech places a sellers market. When recruiters ask me to interview with the companies they're selling I name a 18% increase of my current wage and equivalent perks as a requirement to consider moving. Maybe I would only want 10% if I knew that trying for the job wouldn't require me doing a bunch of unpaid work.

> it only makes sense to invest time in someone who really is excited about switching.

Is this necessarily the case? The Clippers had to bend over backwards to get Kawaii Leonard on their team. They almost certainly would never have gotten an NBA Finals MVP if their philosophy was only go after people who were already excited.

The point is there some tradeoff in the design space. It's nice to have people who are excited to work for you, but chances are the superstars that will take your team to the next level, already have a dozen suitors. Kawaii Leonard isn't doing a six hour take home tests.

I'd suggest that maybe Firebase's experience here is not representative. It was one of the hottest startups in the Valley at the time. So maybe it could put up a lot of onerous requirements and still have talented engineers beating down the doors. If you're the Lakers, Lebron James comes to you. But for the 99% of hiring managers that are at the equivalent of the Clippers, maybe the same approach doesn't work.

Professional sports performance is the probably the most visible and normalizable of any profession though. The problem with software is that talent is infinitely harder to evaluate, especially in the context of different teams, systems and domains.

Whenever I am ready to move jobs, I have 2-3 recruiter chats/phone screens per day on top of my normal job as well as some time researching potential employers. I've never joined a company that's given a project because it makes the process slower, it takes more energy and time (during a period where those are typically in short supply), and it's really hard to find multiple uninterrupted periods of time for a project compared to a timeboxed phone call or onsite.

I'm not against projects in concept and I've never complained about them to my point of contact, but they make the process slower (your process takes about twice as much of my time as everyone else's) and less exciting. Every other step of the recruiting process is really fun - talking to people about their company and the work they do, but a project is just homework that I have to do alone after work or on the weekend. I've really liked some companies that gave me a project, but I struggled to find time and then I got to the offer stage much faster with other companies that I also liked, at which point its hard to find the motivation to complete the project knowing that it isn't even the final step.

While you say you never lost a candidate due to the project, I would challenge you to consider whether you lost candidates due to the perception (fair or not) that your process was less enjoyable and more time-consuming than your competitors' and so candidates received and accepted other offers first.

How can any candidate without a solid networking connection inside your company possibly have enough information to be genuinely excited about switching to it?

"We want the best".

"We don't want the best if they are happy in their current job".

This comes across as really arrogant. And is a big big red flag for me.

It's not arrogance, just heaps of confirmation bias and mistaking a large amount of luck for skill.

I don’t think it’s arrogant, just realistic - if you aren’t interested in leaving your current job you probably aren’t going to no matter what the company trying to poach you does. So for them bend over backwards to convince you to interview is a waste of time.

That’s a huge false assumption, of course you might consider leaving if the interview process was less demanding and impractical? It doesn’t seem realistic at all. The company is just continuing with their bad interview process with their head in the sand, telling themselves “it’s okay, the people who don’t like our interview process probably aren’t any good or wouldn’t move anyway!”. I mean how would they know.

Agreed - that is why I said the statement may still be true.

The unasked question is where you believe the engineers are who can make your project a success? Are they out looking for work? Or are they happily ensconced in other work?

There was a prevailing belief, at least a number of years back, that the best engineers were not found on the open market, so you had to actively seek them out. Maybe that isn't the way hiring managers approach the field anymore, though.

Ahh, I understand what you're saying.

My blog post goes through the process for inbound candidates, which accounted for the majority of our hiring. We did actively try to poach folks as well. I spent literally 4 years trying to recruit one of my friends out of Microsoft.

For these folks, you need to sell, sell, sell your vision, product and team. They're not going to switch unless they're really pumped. All of this work to get them excited happens before the interview process even begins.

I think for these people the difficulty of the interview process is a small issue relative to the risk of switching away from a job they already like.

I was going to comment something very similar before I saw this.

Speaking from the candidate's perspective, as someone with performance anxiety, I abhor live coding sessions. Best case, I'm thinking at an impaired level due to the anxiety. Worst case, I can't think at all and freeze up completely.

Therefore, I absolutely love it when companies offer a take home project. And you're right: I'm probably not going to do it unless I'm really interested in the company.

I'm actually going to be doing one this weekend, and I don't mind spending 4-6 hours on it, because at this point in the interview process, I'm really excited about potentially working for this company.

I hate both. Take home exercises require far too much time. White-boarding is just so hit and miss as to whether a solution comes into my head at that time. And its kind of annoying / distracting having someone watch you code / think.

I get the sentiment - but at some point, a hiring company absolutely has to test your ability to make a judgement about whether to hire you. Speaking from experience on both sides of the table, I'd rather that be through writing actual running code than through some awkward or irrelevant whiteboarding exercise.

As I pointed out in another post, the best interview experience I had was when the guy asked me to bring in my laptop with some code.

I brought in my half finished side project. We talked about it, he asked me some questions, I showed him a couple of things about the tech stack that he didn't know. He asked me to add a basic feature. It worked well for both of us. If I had been faking it I think it would have shown.

I think that's a great approach. I agree absolutely about not being able to fake it either; in practice I've never caught anyone putting forth someone else's code as their own, and I don't think it's a huge concern for the reason you mention.

Do also spare a thought for the people who don't have side projects, or any code that they can legitimately share. There are plenty of talented engineers in that position, and it'd be a shame to have a hiring process that leaves them out in the cold.

I took that interview, and got the job, emigrated to the US from the UK and stayed for 4.5 years. I had total belief Firebase was the future and I wanted to be part of it. The challenge was mentally stimulating, harder than most academic planning problems. It was really hard to exploit the structure. I spent the full 9 hours on it after restarting my efforts at the 5 hour mark. 9 hours of sustained effort. Its not the kind of effort I would blow on most interviews, but totally worth it for that opportunity. And indeed, joining Firebase changed my life fundamentally. In retrospect.... it was such a great move, time spent exceedingly well :)

Edit: I wonder how much was luck. I definitely aimed for Firebase. I used it once after seeing on HN and fell in love with it immediately. It felt like the future, so it wasn't dumb luck, but then, it worked out so very well, I wonder if I saw a similar opportunity I would be able to leap at it again. Would it work out again? I guess time will tell.

Thanks for sharing. It's great to see how leaping at a seemingly small, fleeting opportunity based on seeing something once on HN had such a great impact on your life.

At a previous company, we too would administer a technical test. Our pass rate was close to what was described in the article (40% for ours vs 25%). However, our test was incredibly simple. At most, it should have take a competent developer two hours to complete [including writing comments and a README].

The assignment was to read a file containing a list of numbers (some formatted incorrectly, so there was some very simple parsing logic involved), call an API using each correctly formatted number as a parameter, and store what the API returned to a file. I am to this day stunned that 60% of people who passed a phone screen could not solve this task. Note that we gave them the input file, so it wasn't a matter of an edge case tripping them up or them getting one input file but the test input file having some other edge case.

My point here is that it may be possible to get the same screening value with much less investment from the candidate.

I hate the assumption that "this should take 2 hours", as I have been given tests like that. It involved setting up a oAuth token for Instagram or some similar service. I wasted two hours trying to get that done only to be told that I would have to wait a week for it to be approved.

I am sure half of these things are never thought through. In Python setting up a new project and downloading dependencies may involve needing to install a load of other crap and often takes more than two hours. Some libraries are incompatible with others.

If you are making assumptions that the test will take two hours, make sure that it involves minimal dependencies on third party stuff.

Hate it all you want. In this case, it's true. There are no hidden factors in my description. There was no token and it was a public API.

I'm sorry you've been burned, but that doesn't mean there aren't tests that actually take < 2 hours. I can't speak to every language, but what modern toolset can't open an input file, make an http(s) call, and write to a file?

I also don't understand why we shouldn't figure out how long something takes before administering it. Several people took the test and the time ranged from 15 minutes to an hour and a half, depending on language and experience level. I will say that if someone couldn't do it in 2 hours, they wouldn't have been a good fit for the team. If several team members took it, of course we're going to make an assumption about how long it takes.

Furthermore, since we didn't prescribe a specific language, there's no reason why someone wouldn't have all of the tools pre-installed. Even so, if you had to install your favorite development environment, you'd have been fine. That also wouldn't have been part of the two hour time frame (which wasn't a limit, BTW, just how long it ended up taking competent developers).

At my current workplace we also administer a technical test. It is designed to take less than 15 minutes and this is communicated when it is sent out.

It consists of a small chunk of code in Java/C#/Go or otherwise that has some obvious and other not so obvious mistakes. The candidate is asked to point out any issues they see in the code.

It takes them 15 minutes to do and about 15 minutes to review the response, which I feel values time on both sides.

That could be an even more efficient version of our test. As long as it screens for what you're looking for, I would definitely agree that shorter is better.

It feels like it tests something different than writing code, though both may be a proxy for "quality candidate".

We also did similar take homes that at most should take an hour or two if someone really went overboard. I was amazed at what was returned. It wouldn’t compile or the candidate didn’t follow simple directions. I called it our version of a take home fizz buzz.

We’re the other numbers in your funnel similar as well (phone screen pass, on-site pass, offer accepted)? I ask this because the numbers I saw in the article look remarkably similar to approximate numbers I’ve seen or been told about other companies.

I don't have the stats in front of me, but my recollection is something like (using the same base number for Phone Screen but our rates):

Applications: 5000 candidates; ~20% pass rate (vs unknown)

Phone Screen: 1000 candidates; ~50% pass rate (vs 40%)

Technical Test: 500 candidates; ~40% pass rate (vs 25%)

On-site interview & reference checks: 200 candidates; ~50% pass rate (vs 40%)

Offer: 100 candidates; ~80% hire rate (vs 60%)

Hired: 80

So by some arguments you could say Firebase was 2.5x as selective (40 offers vs 100). With a funnel like this, even small changes to the percentages end up having a larger overall effect.

Unfortunately, we don't have the Applications number from the blog post, though he says "we considered a great deal more applicants than that [1000] on paper." I suppose a "great deal" could be anywhere from double to 10x...

What it looks like to me is Firebase put more emphasis on the technical test. If you keep your exact numbers, except change the test pass rate to 25%, then you come out with 62-63 offers, which, by the argument you reference, would mean Firebase was 56% more selective.

That makes sense to me, because a smaller company needs to filter out as many people who couldn't possibly get hired at earlier stages, since the later stages are even more time intensive than code reviewing the technical test.

*were

The ROI is probably low on the LeetCode hard questions. Most people that can solve LeetCode easy, plus have some sense of Design Patterns (so you know they can think big picture too)-- and you've got yourself a good candidate.

> Then, after they submitted their answer, we would provide them with a thorough and detailed code review (usually by the legendary @mikelehen).

> We treated this code review like a real production code review, and we did it whether or not we planned to move forward with the candidate.

First, thank you for actually investing time in giving feedback to candidates regardless of their performance. Second, did you tell candidates that they would get detailed feedback or a code review as part of the instructions or phone screen? If yes, I figure that this would greatly affect how willing people were to doing the assignment.

I've always hated the thought of take home assignments because I have never been given detailed feedback and only rarely been given any feedback. Requests for feedback are always ignored regardless of performance. I sense that part of why people have greatly soured on take home assignments is because of how lopsided the time investment has become.

If people knew they were going to get a free code review by a notorious developer, they might submit without any intention of taking the job. Having someone who knows there stuff look over your code and give you detailed pointers on what's good and bad, is like finding water in a desert for most devs.

Any good resources you can recommend to improve my own code review skills? I'm keen on drilling into the most valuable skills needed in the industry right now. It sounds like good code review is super important but lacking.

Google released a Code Review guide which I find extremely useful [1].

[1] https://google.github.io/eng-practices/review/reviewer/

It's really difficult to actually do a thorough check of logic , exploration of alternate approaches, performance concerns, etc. in all your code reviews, it would be an insane amount of time spent and probably no real return on investment for the company (obviously devs would love to spend a couple hours weighing the merits of different approaches to a problem).

I have a list of checks to run through, some general (Sass variables used, reusable code/components used if possible, variables have meaningful names), others more specific to our codebase conventions. I add to the list as new conventions get prioritized and check it. Code review is more about maintaining codebase consistency than suggesting alternate solutions to problems for me.

A rare post in that it's entirely a coin toss whether it's meant to be irony or not.

Awesome. Thank you for the distillation. I found an insane amount of info as I started searching, so your breakdown of what you consider during reviews helps.

I have two answers to that:

Measure and adjust: Find a source of high-quality code reviews; the easiest source may be the high-level engineers at your employer, but if that's not possible/helpful, I'm sure there's free/open sources. Make your own code review before reading others, then compare and contrast. What did they flag that you didn't notice? Can you generalize to categories you could prioritize more? What did you flag that they didn't? When you both flagged the same thing, whose phrasing was more helpful?

More personally, here's my approach, which may or may not be helpful for others:

* First, skim the entire PR. On a raw, visual level, get a sense of what it looks like. A couple big edits, or a bunch of small updates (like changing a variable/method name, changing an imported library, re-ordering some interaction, etc)? A bunch of indentation changes without much else? An extension of an interface? This is to familiarize myself with the general question of Just What's Going On Here, basic context to use as the foundation for the real review. This should take just a minute or two.

* Second, perform a close reading of the diff. Don't worry about high-level conflicts like ordering of operations or other semantic considerations; stick to syntactical-level concerns and context-free mistakes. Are the variables and other symbols well-named? Is exception handling done properly, is this library not initialized correctly? At the "fit and finish"/polishing phase, this might be the majority of the effort. At the early-draft stage, it may be counter-productive to actually note all the possible issues. Either way, to use an analogy to compiling source code, the goal should be to build an AST of the diff, so you can begin to build a model to manipulate through the rest of the review.

* Lastly, address the higher-level qualities. Depending on whether you prefer a bottom-up or top-down approach to design, you might order this section differently. I tend to try to identify where on the abstraction scale the highest-leverage feedback resides, and then focus on that level. Depending on context, that could mean discussing which component needs to change, which repository/module this change should live in, which classes should be changed/created, what's the behavior of specific methods, etc.

Whatever your feedback, consider what you hope to accomplish with the comment. Are you trying to educate someone about an idiom they don't know about? Ensuring correctness of a complicated interaction? Maintaining a common code style or test coverage threshold? Make maintenance/debugging simpler? Is this something that you view as blocking, or is it more of an FYI? If the latter, it's pretty important to communicate that clearly.

Awesome. Thank you for the thorough response! This helps.

If there is going to be leetcode round after code assigment. I beg ppl to reverse the order, always have leetcode round first and respect candidates time.

Way too many companies have leetcode down the line after the candidates have invested considerable time in their process.

Ugh that reminds me of my least favorite interview of all time. Some years ago I did a take home project that I spent several hours on, felt pretty good about the code and how I’d addressed the challenge.

A few days later I was called back for a phone interview with the CTO who wanted to do a screenshare leetcode type thing. He had no idea I’d even done the take home task, and wanted me to solve a stupid graph traversal problem requiring the same algorithm I’d used in the take home project.

I've had actual comments propagated to me after performing take home assignments and I think not receiving any feedback works better for my own sanity.

The comments I personally got back were infuriating enough that convinced me of that.

I'm curious... what was it about the comments that you found infuriating? And did this provide some signal about the company itself?

I've seen a lot of companies that, like firebase, try to assess if a candidate "is excited about our company" early in the process.

It always seems weird to me. I barely know you- why would I be excited? Ask me at the end of the onsite after I've talked to a number of people across a variety of roles, seen your office, heard about the work, explored the culture, etc etc.

During the initial screen, I really only know about your business, which is not the biggest factor for me.

Firebase is probably a bit of an edge case on this one, because it was a paradigm-shifting developer product before dev products were cool.

If you were a top-tier developer, there was a >0% chance that you would be super excited about the product because you realized the pain point that it would solve for you if you were building a real-time product at another company. And that excitement or lack thereof was a useful litmus test for Firebase specifically.

I agree that "excited about our company" is not something you get up front if you are interviewing at a company that is building an on-demand AI marketplace for ML-optimized scooter rental office spaces.

Everyone wants a candidate "excited about our company", but last time I interviewed after being laid off, so I was more "excited about a job". I didn't really get excited for my current job until about 3 months in, when I knew more about the internal workings of the team and the products, rather than what shows up on the public webpage.

He did post a disclaimer that their approach wasn't designed to work for every company.

For such a small team, and an early-stage company, I believe it's critical to hire people who are absolutely excited about the mission. They're not simply looking for "an employee," but a partner who believes in what they're doing, coupled with having the ability to help them do it.

The cynical side of me thinks that candidates are screened for this because screening for things like technical ability and communication skills is actually really hard, so we screen for this easy thing instead.

I wonder if "excitedness about the company early in the interview process" is in any way indicative of on the job performance later on.

I find it's a lot easier to work for someone if I personally want their product; compared to a product that I have very little interest in.

They want you excited so you work harder and complain less.

It does seem odd. Unless I'm begging for a job, you should be telling me why I'd want to work there.

> The strongest candidates take their job search seriously and are willing to devote a fair amount of time to landing the right job.

... only for the 1-2 top companies they are interviewing for.

I have discontinued multiple interviews because they demanded 4-5 evenings of work for free on short notice, if I am going to spend 10-20 hours per company I have to take two weeks off from my current work to do it. On the other hand I have done the required work if I really want to join a company, but they are only top 5%.

Another problem is, usually what is expected and how the project will be judged is not mentioned beforehand;

- so do I write unit tests?

- do you want docs for running it?

- should it include deployment instructions/code?

- does have to support 5000 req/s?

- how important is clean code for this?

each add more time and these kind of question are never ending.

So unless you are Stripe, Airbnb, Netflix, etc. (Very strong Engineering and better than average culture), I will not bother, will say thank you and be on my way.

Finally, let's say you have done the project and they reject you, they never put a fraction of the time you spent to give you feedback. because that means a lot of time they will spend on someone that will not be hired. Also might open the way for litigation for discrimination in some countries etc.

On the other hand; Once I was managing hiring process of a new grad where we asked her to design a FE+BE system, after receiving it I spent a couple of hours reviewing it, we decided to hire her, she was very enthusiastic and smart, I also sent her our notes and how she can improve the project. Her response was along the lines of "even if you don't hire me I am very grateful for doing the project, did learn a lot" She joined and quickly became a very productive colleague.

Edit: I noticed that the author mentions they keep in touch during the project and provide a code review, had not seen that yet while writing the comment. In that case it makes sense, but the candidate should know they will receive feedback even if they fail which I believe is hard for the team to accomplish but kudos if they did manage to do it.

[author here]

I actually agree with your thesis! You're right that candidates are only willing to put in this level of effort if they are really interested and consider your company a "top 5%" big opportunity.

So it's critical that you convince them that you are in that top 5%! You don't want an employee to join you because their top choices turned them down -- you want to be their first choice.

We regularly won candidates away from Google, Facebook, Uber, Twitter, etc. If you want to build a great team, you're going to need to be able to do this too.

Re: code reviews -- yes, we gave everyone a thorough code review.

>You don't want an employee to join you because their top choices turned them down -- you want to be their first choice.

Why?

Unless you're offering FAANG levels of compensation, isn't it pretty arrogant to think you're anyone's top choice?

Candidates can easily lie to you about this. In fact in my last job hunt I was instructed to lie about this by recruiters who are actually being paid by the company I'm lying to.

I told every company I interviewed at they were my top choice.

If it's "arrogant" to think that a startup could be anyone's top choice, then startups would never be able to attract any good talent.

It's not arrogant. It's simply creating a process where you can filter out for folks who are truly excited to build what you're building and believe in the potential future impact. It might also turn out to be a big payday, or not.

Not everyone is simply chasing that 400k TC every year. If you lie about your excitement, a panel of folks who have went through the same process will easily sniff it out.

Some people are better at pretending to be excited than others. Recruiters for example are very good at it. Not all engineers are bad actors.

Well, then perhaps the test is working as intended! Candidates who aren't sufficiently excited about your company will drop out at this stage.

This doesn't make any sense.

Your intention is to get candidates to fake excitement or drop out?

I think the idea is that it's easy to fake excitement until you're asked to spend six hours on a take home test. At which point you'd actually have to be pretty excited to do it.

Or desperate.

As someone who has given take-home tests let me answer these for you:

- Unit tests? Yes. It will increase your standing.

- Docs? Absolutely. A must have. You will not be considered really otherwise.

- Deployment? See above.

- Support 5k requests? Not unless that is what the test is testing for, pay attention during phone interview about the role and it's requirements, the test will look for this and other generalizations.

- Clean code? Very, very important. Don't try to show how smart you are, show how audience aware you are. Most likely you will be working on a team, a team doesn't want a cowboy programmer.

Something you didn't mention but bodes really well for anything with a front-end interface: hosting. Executives hate running things themselves, so a hosted version will put you at the top of the list.

It's probably worth mentioning that any take home worth its salt will explicitly list what criteria you're being assessed on.

It's crazy how many coding tests are poorly designed and waste a candidate's time. I'm putting together a list of good practice for test design here: https://candidatecode.com/articles/coding_challenge_best_pra...

Feedback and contributions welcome

A lot of the time they are left out to see how experienced the individual is working in the environment. What do they think is the most important (outside of the outlined criteria)? Treat it as if they are testing your experience, not your literal skill in pumping out code. Anyone can pump out code.

The issue there is the problem starts to become unbounded for the candidate; they can't tell what you're looking for, so they don't know how to prioritize their time.

That's a problem even if you have reasonable expectations yourself, because the candidate probably has experienced some tests where the expectations haven't been reasonable. For a (real) example: marking candidates down for not having an analytics integration.

How does your candidate know that's not important to you, even though it was important to the last company?

> testing your experience, not your literal skill in pumping out code

Past a certain point, it gets much more efficient to discuss experience in an interview setting. Good coding tests should be designed to test the skills you care about most to produce a strong hiring signal for the next phase - it's difficult to showcase a full decade's worth of experience in 4-8 hours' worth of coding challenge.

> Anyone can pump out code

I've found that to be depressingly untrue amongst a significant percentage of engineering applicants.

I like how this is what interviewers are looking for and then you get hired and there are no tests, the code is a mess and the performance sucks.

And they ask a hard algorithmic/CS problem then your job is to build CRUD interfaces

You nailed it.

That's why they look for it ;)

Clean code is so subjective it hurts. Being audience aware basically means trying to guess the company and interviewers coding style from minimal information.

Great story. Instead of looking as recruiting as a cost center, make it a marketing investment. Spend hours with someone, give them a good experience, make them an ambassador for your brand and an external sourcer for their friends and their future self. Use them as practice for your staff to get better as communicators, mentors, code reviewers.

Yes, exactly.

This is also a double-edged sword, if the experience is horrible, you can be sure none of their classmates will apply to you or even their school if you blew it exceptionally.

There is an article by Joel Spolsky about Hiring great engineers that touches on this point also: https://www.joelonsoftware.com/2006/09/06/finding-great-deve...

> Even though only about one in three applicants who make it to the in-person interview stage passes all our interviews, it’s really important that the ones that do pass have a positive experience. Even the ones that don’t make it go back to campus thinking we’re a classy employer and tell all their friends how much fun they had staying in a luxury hotel in the Big Apple, which makes their friends apply for an internship the next summer, if only for the chance at the trip.

> I have discontinued multiple interviews because they demanded 4-5 evenings of work for free on short notice, if I am going to spend 10-20 hours per company I have to take two weeks off from my current work to do it.

Honest question: Would you feel differently if these companies had 1-2 days of onsite technical interviewing instead of take-home problems?

In practice, I rarely get pushback from candidates for our 4-hour take-home problem. We provide clear instructions about expectations, constraints, and boundaries. All candidates have thanked us for allowing them to work on their own time in their own coding environments.

I would, because interviews require someone from the other organization to be there. Time is a valuable resource, and as long as the employer is showing their commitment, I have no problem reciprocating. Skin in the game is the key.

Why does "skin in the game" matter? Why is that a useful signal about whether this company will be a good employer or not?

It's not a useful signal about if they are a good employer, but it is a good signal for how likely they are to hire you: asking everyone who applies to take several hours out of their life to do your test is free. Having someone from your team interview them for several hours very much is not so you'll be more careful selecting candidates you might actually hire instead of just taking a shotgun approach.

For me it's a useful signal, it means they value my time equally as theirs. I'm happy to do the take-home test after the onsite one when they only testing that I really know what I said I know but I'm at the finish line. A lot of companies require it right after the phone screening(usually with an HR person who knows nothing). I burned myself that way before. After finishing the take home test on the on site interview after 5 minutes it was clear they were looking for an ops person advertised as DevOps.

Right, having the employer invest something means they'll be more likely to check for a good fit before reaching for homework as a filter.

I interviewed with Andrew in Google for an internal team transfer as a Software Engineer. At the end of a very strange non-technical 30-minute interview where he sounded like an important and arrogant upper-management, I asked him what he was looking for and his reply was he wanted to hire someone that he liked and wanted to work with. That's seriously what it boils down to.

I cannot take this article seriously. And "a history of working on challenging projects with real deliverables" sounds ridiculous, when you know they hired people straight from bootcamps as engineers.

I’m sorry you feel I was arrogant.

I really do want to hire people “that I like and wanted to work with” though. I would hope most managers do.

FWIW we never technically interviewed internal transfers inside Google (we counted on Google’s normal interview process to maintain the bar). We were just looking for team fit in those discussions.

I’m disappointed you would belittle bootcamp graduates. Many of them are awesome, and we were lucky enough to hire one amazing one. And yes he did have a “history of working on challenging problems with real deliverables”, just not the CS kind in this case.

This article is worth reading just for the description about the take-home coding challenge.

I agree that 4 hours is a good target, and that the coding challenge must be fun and in an language of the candidate's choice.

> One common objection I’ve heard to take-home tests is that they are too time consuming, and candidates won’t be willing to do them.

As a candidate, I encountered this situation twice, but it was because the coding challenge was poorly designed.

In one case, the coding challenge basically required me to learn two major Java frameworks. As I had minimal Java experience and never touched the frameworks, I estimated that it would take me 2-3 days just to understand what someone else could probably do in 2-3 hours. I have no desire to be a framework jock, so I walked away.

Another time, I did a challenge for an open-source company. They had a list of some bugs, and I picked what I thought was easiest. The language wasn't one that I did any work in. No one responded to my questions, so again, I walked away.

I feel so fortunate that I have never been desperate enough to even entertain the idea of working for 8+ hours, for free, as part of an interview process.

Yikes.

I keep reading that there’s a huge shortage of competent software developers. Then I keep reading that employers are kicking applicants in the teeth over and over as many times as they can get away with. Something about this doesn’t quite fit.

The other thing that is often odd is they place an ad for a position and list very specific domain knowledge and they say 'before we will consider you, you need do this arbitrary four hour on line coding test'. Here in Sydney we have a couple of trading firms complaining they can't get staff. I often think now, being able to confidently do the job needed very well, is not a primary requirement. When I say well, I mean, well coded, well documented, good design, not just able to solve the algorithm at hand. Are we short of good programmers or do we have to many?

Not to mention they are still asking for references.

Big yikes, Firebase.

You should notice who says there is a shortage: employers. And what they mean, is that they can't find enough people to hire, meaning that few people satisfy their hiring criteria. This does not mean that few people apply, it means that their hiring process eliminates almost anyone.

I refuse to do so. The companies have no skin in the game and could easily be giving the same assignment to hundreds of people fighting for the job.

I would entertain the idea later in the interview process, especially if there were some token payment to show that the potential employer is just as invested as I.

They are paying their engineers to conduct interviews, write feedback, and sit on hiring committees. The hiring process is hardly free for them either.

On average it costs around 10-15k to hire an engineer.

how much does it cost to interview an engineer?

> $2,000: the engineering team interviewed three candidates in-house before deciding to hire Jeanne. Each interview took about 7 hours (phone screens, in-house, debrief) for about 20 hours total.

This is a rough estimate for the startup provided in this blog post, you can do this by taking the salaries, getting their assumed hourly rate, and then applying it to the amount spent interviewing.

See https://blog.usejournal.com/it-costs-50k-to-hire-a-software-...

This one cites 50k, on average though, most startups, the HR decks are generally around 10-15k with in-house recruiters. https://blog.usejournal.com/it-costs-50k-to-hire-a-software-...

It's not just the 8 hours though - It's the weeks on end of practice beforehand.

And before someone says, "you should be sharp on your skills regardless of an upcoming interview", I challenge you to try a mock interview for a similar company.

You should be sharp on your skills regardless of an upcoming interview.

Why should you be "sharp" on things you can Google for in seconds?

that means this filter is working as intended. from the article:

> Please note that this post is intended to help with vetting of software engineering candidates at early-stage startups with significant technical complexity.

candidates at early stage start ups need to sacrifice time, energy, and effort in a manner that's not based on hourly compensation for the company to hockey stick.

it's critical to filter out people who aren't willing to do this. the proof is in the pudding: all the people volunteering time/effort, freely, to get better scores on the gold mine game... after they're hired.

not to say those aren't redeeming qualities for talented software engineers. they're just using this curiosity/drive as a signal for "willing to work on cool stuff above and beyond normal salaried expectations, and not ask too many questions about how they will benefit".

there's an obvious tie in to age here as well (younger people are less committed to external causes e.g. relationships, have weaker boundaries generally, more able to be starry eyed, etc.)

I've once taken a 5-hour pre-interview test that actually took me from "eh why not" about the company to them being my first choice. My other offers were from companies that did not have at-home tests. I thought the particular at-home test was relevant to the job I would be doing, it (and the job) were in a language that I found during the test much more fun than the language I was working in at the time, and I found it in general a very appetizing job teaser.

Although yeah, I spend my evenings with my partner, and I was working extra hours at my real job, and it was a test you had only a few days to complete, so I had to pull an all-nighter for it.

Not only that, but having my references checked before an actual interview. No way. I don't think even the NSA does that.

We did the phone screen and the take-home test and asked them to come in for the on-site interview before asking for references. So this was pretty late in the process.

Yeah, I'll be sure to filter out any Firebase jobs after reading this article.

"As a startup, process speed and transparency are your secret weapons when competing for candidates with large companies."

Amen. I've known people who got through Google's interview process, but even then the response is "Great job, we'll let you know within a few months if there's an opening on a team you can join."

Not a great response when you're graduating soon and you've got concrete offers from other companies that will expire.

>The whole test was designed to take 6 hours.

Pre-Screening to get to an onsite interview taking 6 hours? I'm sorry, but this is the exact toxic interview culture that is poisoning the job hiring process in tech startups.

>Some of our employees enjoyed the test so much, in fact, that they continued working on their solutions after they were hired

Jee, I wonder if that had anything to do with the fact that they were, you know, new employees? As in, trying to put their best side forward and make a great impression? Did it even occur to you that you are inadvertently increasing their workload with your downright torturous screening process?

I'm currently on the job market, and have done a few of these types of take-home tests over the last two months. If they're well structured, as it appears here, they can be fun and hopefully build enthusiasm about the company. If they're not, it's a huge red flag. A company that doesn't value my time as a candidate is unlikely to do so as an employee, and isn't one I want to work with.

If you do this, make sure to treat candidates with respect and humanity. Give them some feedback at the end of the process, so they know their efforts aren't wasted.

2 issues i have with this "process":

1 - take-home task of "only" 6 hours. that's on top of the onsite (~6 hours) and a few hours for phone call and followups. that's too much for an unpaid job.

2 - they try to find people they know to sniff around about the candidate. that's usually bad as you can jeopardize the candidate's current job (usually people interview while still working at another place). overall this post made me want to work at firebase even less now.

Take home tests are fine by me, but before you ask me to do that you better have sold me on the job first. If I really want the job, no problem. I expect, however, that spending 6 hours of my weekend on unpaid labor translates into a shorter on-site if you like the quality of my work. The goal of the on-site should be to verify that I was actually the person who did the work for the take-home.

Things are a bit different when I'm some schmuck off the street begging for work. If you're trying to build a team and you want my skills, well, in this market I'm not putting a lot of energy into it.

Case in point, I've had a company trying to hire me for over a year now. They're in consumer electronics and it is a fickle market so I kept turning them down, but I had a bad week at work and figured I'd go for it. I'd be an absolute perfect fit given my experience and what they're looking for, so it seemed like things were going to work out great.

I accept an on-site, do well on a whiteboard coding test and a general review of my resume, and then one of their senior folks throws me an off-the-wall algorithm question(random sample a stream of data with uniform probability). Now, I'm an embedded guy, with an EE degree, and I specialize in "software engineering". They have many problems I'm perfectly suited to solving. They really, really need a guy like me(to design and fix drivers, abstraction layers, facilitate porting to new devices and enable third party clients). But here's this guy, asking me an algorithm question, and he didn't even explain it right. I googled the answer(reservoir sampling) when I was finished and understood the solution in a couple minutes, but it was too late. I was flagged for 2 more algorithm-heavy, 'cracking the coding' interviews the following Monday. I really didn't feel like spending my weekend reviewing bullshit I never use in my job, so I turned down the offer. It's too bad, I was a good fit for the team, and I was a great fit for the problems they were trying to solve.

A "take home test" of 6 hours is a waste of my time.

I'd rather spend those hours on a personal project which I can use for demonstrating to potential employers or perhaps use for a side-income.

Not a place I'd want to work, I think. Two red flags are:

1. They signal from the first point that all your time belongs to the company. Taking a test of 6 hours is not something I'd have time for.

2. Their company value "Give 110%, 33% of the Time" creeps me out. If someone is not satisfied when you give 100%, run away.

Candidates can cheat take-home exams by asking someone else to complete the assignment. After the assignment, you have to ask detailed questions about the code, solution approach and trade-offs.

If what you have to offer is above average (compensation, growth opportunities, reputation, culture) people will be willing to make an extra effort. If you are offering the same as everyone else, candidates will deprioritize your interviews.

"we had a small team of just 24 exceptional people that I firmly believe was among the strongest of its kind in the world"

I am absolutely sure that this is something A LOT of companies say and think about their current setup. Impossible to verify, just something that sounds great and makes the 24-people group feel special.

This overlaps a bit with RFP processes.

If you get a request for a proposal where the potential customer / hiring committee clearly just copy/pasted a set of items that don't even fit their company type or area, or that ask for unreasonable stuff from the start, then you will save yourself a TON of time by saying thanks but no thanks.

If someone has an interesting set of question or a clean and focused RFP, fantastic. GO for it, you will enjoy working with them.

Another item - if your customer will need to do some type of work, have a TINY bit of that in your engagement process. Ie, let's say client will need to write a lot of text, have them briefly respond to a question in the process. If they will need to fill out worksheets, have them do a tiny one.

Another one - if they won't tell you what they need or who they are - not worth it.

(rant)

Startup idea: take away home assignments for a fee. You send us your code assignment we make it for you. We also provide you with code highlights so you can understand the code and architecture faster and have something to talk about on the onsite.

Most of the work is not done by us but by "volunteers" .

Who are these "volunteers" ?

Desperate engineers looking for a role, who are thinking that our fake company is really wanting to hire them.

We also get in touch with hiring managers and convince them to send take away assignments in order to increase our market.

(/rant)

I don't like code assignments because they can easily be abused by both companies and candidates and regardless of my opinion on it I think Firebase is awesome. I was an early adopter before you got adquired by Google.

I wonder if the OP can comment on whether GoldMine ended up biasing their decision to hire in an unintended way. In hiring security researchers, we have been finding that a lengthy technical exercise is terrific to properly gauge applicant skills, but that the scores of the test could end up weighing heavily on the decision to hire or not, potentially at the expense of unrealized potential. In other words, the candidate may have technical strengths not captured. I am assuming the approach only works to the extent that the test is a model of the actual problems the company is trying to solve. And yet they are looking for generalists. Somehow the test seems insufficient.

I checked out the Gold Mine problem on geeksforgeeks (assuming it is the same problem at firebase). Can you explain why it is "potentially impossible" to get the optimal solution?

That's a different problem than the one we used. We added some additional complexity that required a search of the entire solution space to get a provably optimal answer, and then we made the maps & allowed # of moves large enough that brute force was impossible. So people had to build sub-optimal search solutions that pruned in intelligent ways.

Is the problem (statement, input and hopefully leaderboard) available online somewhere?

I figure it hasn't been used for recruiting in a long time, so there shouldn't be any downside of that nature to it being so. But there is the invested time in making it available, of course.

I hardly think I would be the only one thriled to try it just for fun. It would probably provide a great deal of goodwill for you guys, as well as further the industry's understanding of what constitutes a good take-home problem.

Did anyone use a SAT solver?

I took a quick look at it just now myself - it looks like it'd be pretty straightforward to solve by just brute-forcing every possible solution. Or is there some tortoise-hare type of "trick" I'm supposed to come up with?

The firebase problem states that there is no optimal solution. so it can be as easy as increasing the size of the gold mine in a way that a brute force solution it's not posible and you must find sub optimal maximums applying heuristics. The travelling salesman problem has a lot of literature to understand this kinds of problems that are usually some variations of this problem originally stated near 200 years ago.

Really appreciate all the skills and positive attitude they were looking for and I think every engineer should strive to live up to their standards up until this part:

    …is excited about our company.

Excitement should be deserved, which means it is a feeling which is acquired after working together for some time. It should not be required as the default position in a job interview. To build a great product, you need seasoned professionals, not gullible hipsters.

> One of my favorite interview questions (created by Vikrum) was: “What happens after you type ‘https://www.google.xn--com-to0a into a browser and hit enter?”

Wow, I had no idea who came up with that question first. (see, e.g., https://github.com/alex/what-happens-when ).

When was the first time it was asked? Where?

Could you elaborate on how you know "the amount of time a candidate took [on Goldmine] turned out to be a pretty strong signal."?

This is probably survivorship bias at work. A lot of startups have strong sense of self worth because they believe they are hiring/have the best candidates with their process. Let's be realistic, you can probably build a really good team out of just about any sampling of 1000 applicants with just a regular couple-hour interview process.

In a tight labor tech market , unnecessary friction in hiring will be detrimental until you build a brand (your company is hockey sticking or the day to day work has wide appeal to devs) or find the “true believers”. In a downturn , go crazy , set up your gauntlet , and you will find the best that your network or advertising can produce.

> If the candidate is traveling for the interview, your company should pay for all travel expenses and make sure they have a reasonable schedule and are able to get a full night’s sleep the night before.

We're lucky to have this in the software industry as standard practice. Has anyone had horror stories otherwise?

>The strongest candidates take their job search seriously and are willing to devote a fair amount of time to landing the right job.

Yeah, maybe if they're interviewing to work for Facebook, Google, Amazon, IBM, Microsoft or other big-ticket dream companies a lot of developers want to work for. But, Andrew was talking about hiring for an early-stage startup, before Google even acquired them.

As a startup, you're asking others to take a risk by working for you. Some engineers have families, debts and commitments to worry about. Working for an early-stage startup that might run out of money or collapse is a very real and likely risk.

It's hard to argue that Firebase was not a roaring success for Andrew and others. But, expecting a candidate to do a six-hour test for a startup that statistically might not have succeeded, and furthermore, listing out the benefits of a take-home test being less stressful and then proceeding to describe what sounds like micromanagement under the guise of helping (by calling candidates at the start and during of the technical test).

And we keep hearing there is a talented engineer shortage. Maybe the problem isn't a shortage, it's companies expecting engineers to be put through arduous and time-consuming interview processes like they're trying to get a job at NASA building human-payload rockets. For every company wanting to take hours and days of your time, there is another company that won't put you through a BS hiring process and make it faster.

I work for a smaller company, I had two interviews for a front-end engineering position. I came in, met with the engineering lead and another senior. They didn't make me do a technical test, they opened up my GitHub profile on a large monitor, handed me a keyboard and mouse and asked me to pick one of my repositories and run them through my code and decisions that I had made. I chose a Javascript library I had built, they asked me things about the architecture, if I had tests, my thoughts on Webpack and other bundlers. It was all related to the library and all answers I could easily provide because I built it. No BS, just a pressure-free interview.

The second and final interview was a coffee with the general manager and the engineering lead. They gave me more detail about why they're looking for someone, explained it was a new position and where they saw me fitting. I ended up having lunch with them and a couple of other team members who joined. It was all casual and pressure free.

This company offered me $50k more than I was currently earning. It was a MASSIVE pay bump for me, and I didn't have to sacrifice hours of my family time just to be told I didn't pass or get the job. It's also a completely remote position, no commute whatsoever. No wonder this company has been around thirty years and many of the original hires are still here.

Smart, compliant, friendly and enthuastic. No problem domain, life or business experience required.

Sounds neat until one realizes you can be replaced in an instant by one of the ever growing number of new "graduates" every year.

[author here]

Hi all - I hope you find this useful!

If you have questions or want to know more I’m hosting a live stream Q&A today on Twitter at 1 PM Pacific. Tune in and bring your interviewing and hiring questions! I'm @startupandrew on Twitter.

Your claim that a take-home test is "lower stress" is undercut by the fact that you time your interviewees on their homework. Why do you think it is important to gauge how fast a person might be able to solve your toy problem, and why do you expect your candidates to dedicate 6-12 hours (unpaid) of continuous time to your company before you've even met them in person in most cases? Did you consider that this might inherently discriminate against those with significant time constraints in addition to their current job, biasing your results to new grads and people without hobbies and/or families?

How does an employeee having hobbies and a family help with the survival of the company? It seems that hiring people with a life outside of work could be in the active disinterest of squeezing the most work for the least pay out of the human resources that corporations feed on.

So true. If you can’t code for 6 hours straight without ignoring the fact that a job change can have an incredible impact on all your life goals and personal relationships, why are you even a coder bro?

This process is so weak, when I heard about it in 2013 I was like...why would I ever want to work at firebase?

The alternative to a take-home test is on-site interviews with white board problems. These also take (unpaid) time and are timed (you have until the end of the interview). Doing this part of the interview at-home rather than in-person gives people more schedule flexibility, not less.

The total time spent is higher than most interview processes I think, which can be a struggle for some folks. But I do think it's the right tradeoff.

Regarding why time taken is an important data point -- we found a strong correlation between people who were able to get good solutions working quickly and people who were able to effectively defend their solutions during the "GoldMine review" in their on-site interview.

> The alternative to a take-home test is on-site interviews with white board problems.

That's not the only alternative though. Why not use a more humane interview process involving conversation about previous work, ideas about the future, strengths and weaknesses?

> The total time spent is higher than most interview processes I think, which can be a struggle for some folks. But I do think it's the right tradeoff.

I don't agree but I'm glad you thought about it. I personally think making your interview process more inclusive allows you to get better candidates, as opposed to structuring the process in such a way to exclude those that have different priorities and life choices.

> Regarding why time taken is an important data point -- we found a strong correlation between people who were able to get good solutions working quickly and people who were able to effectively defend their solutions during the "GoldMine review" in their on-site interview.

Correlation doesn't equal causation though. I also want to point out that if you are expecting someone to "defend" themselves, that means you are on the offensive. Interviews don't need to be adversarial.

Edit: I do want to put in one good thing I got from your article to offset my rather negative responses. You mention that even if you reject a candidate you gave them the full code review and feedback about why you rejected them. The culture of secrecy surrounding rejections is something that needs to die, and I'm glad that although I think your process as a whole is unfair, you took the step to change that toxic part of our industry's interview culture.

The problem with interviews that rely on discussions of experience alone is that they aren't effective.

It's easy for candidates to exaggerate or misrepresent their accomplishments. This is especially true when they worked on large team projects. For example, if they say "I was part of the team that built Google Search", that could be a really impressive thing if they played a key leadership or technical role, or it could be a really unimpressive thing if they hung out while others did the heavy lifting, and it's very hard to tell the difference.

Yeah this is the common excuse for torturous technical interviews but it just doesn't fly with me. What is the end game of the interviewer who is lying? They'll get found out when asked to deliver and lose their job eventually. This is not without cost to the employer, but it gets harder to explain so much turnover in your resume as you accumulate early exits.

In addition, there are levels of technical chats that become very difficult to fake, further reducing the pool of lying interviewees people claim to be defending against.

To be clear, at Firebase, we held these kinds of discussions in addition to the technical assessment. A portion of my section of the interview process was usually to do a deep dive on a recent interesting project that the candidate had worked on, either sourced from their resume or just by asking. You're right that you can often tell their level of involvement by seeing how quickly the well of details dries up.

There are tradeoffs, but I found it to be an overall net positive in addition to the technical assessment. Doing this effectively requires that the interviewer have a pretty broad exposure to lots of different technology, otherwise you can't distinguish between someone BSing their role vs something you just don't know a lot about / can't ask intelligent questions about. It's also incredibly subjective, which would make me hesitant to rely on it without more objective criteria.

> What is the end game of the interviewer who is lying? They'll get found out when asked to deliver and lose their job eventually. This is not without cost to the employer, but it gets harder to explain so much turnover in your resume as you accumulate early exits.

That's a pretty idealistic view of people. Most people exaggerate on their resume. The most reliable tool (at the moment) to cut through and establish some common metrics is the technical test. How do you compare technical "chats" between individuals? Duration of call? Gut feel? How do you ensure that the person on the phone has the breadth and depth of experience to thoroughly discuss any and all projects on the phone with the interviewee? Is it the same person on all of these calls? It can't be, so how would you compare between two interviewers' subjective feedback?

> This is not without cost to the employer...

This is a HUGE cost, especially at a startup. A mis-hire at a startup can sink the entire ship.

I do think that your approach has merit, and I personally dislike technical interviewing, but for a startup the OP's approach is, IMO, the best tradeoff.

> They'll get found out when asked to deliver and lose their job eventually.

It's the "eventually" part that's problematic, since it can be long, uncomfortable, and expensive to the company.

I read that comment differently. The idea is to look at the candidate's work history, and if there's a large number of short stints you can use that as a red flag that previous employers found them lacking.

Do you hire people to write code like in the example, plotting mine routes? These things don't test for relevant technical ability, they test for compliance, they test for algorithmic knowledge, and they test for being a young person without technically being age discrimination.

From the article: "In retrospect, I think our test was a bit longer and more difficult than ideal. If I were to do this over again, I would design the test to take about 4 hours."

Seems reasonable to me.

But the test time is actually unbounded, although of course they're hovering over you with a stopwatch.

Which means the candidate will be under incredible stress still, no matter what the supposed target time is.

Was there ever a time the testing and quizzing was wrong? How did you know?

For example, “Later that candidate did X, which would have been great for Firebase.”

How often did you take subordinates that weren’t fresh college graduates or former/current FAANG engineers?

Yes, we definitely had false negatives. It's always hard to be sure if you made a mistake though -- someone might be super successful at another company but not work out well at yours. I'm sure we made mistakes, though I don't have stats on this.

We definitely hired non-new-grad non-FAANG engineers. In fact, new grads and FAANG alums were a minority for us.

> someone might be super successful at another company but not work out well at yours

I don't actually buy this. Environment makes a difference, sure, but if someone goes off and succeeds technically[1] at a another company in a big way they would have succeeded at yours as well. If that person can't succeed with you, then there is something fundamentally wrong with your organization. My experience, though, is that such organizations have much lower hiring bars.

[1] I say "technically" because it's quite possible that the person is not very good technically, but landed at a place where that didn't matter and some combination of non-technical skills led to their success.

While it's certainly convenient to give everyone the same question, you'll find that questions asked by even smaller companies get posted online these days. You then give everyone willing to cheat an edge.

what kind of dev responds to the “what happens when you type google.com into the browser” question with “Well, most laptops have mechanical butterfly mechanism in their keyboards…”??

Right? Most browsers these days are run on mobile devices with capacitative touchscreens for input.

The kind you want to hire.

its a well known question, probably someone who has previously thought about it