Technical interviews are disguised IQ tests. IQ tests are actually not allowed in the US based on a Supreme Court decision[1] that said they were discriminatory.
We know that cognitive ability has one of the strongest correlations with job performance.
So "technical interviews" that are heavy on algorithm puzzles are just a loophole to get around the IQ test issue. They aren't actually testing domain knowledge.
The Griggs decision explicitly places the same restrictions on college degrees that it does on IQ tests. Obviously nobody cares about this, and it turns out the Griggs restrictions aren’t that strict and most people manage to ask about college degree just fine. This changes the argument to be about whether there’s complicated case law around Griggs which makes employers figure that college degrees will probably be accepted in practice, but IQ tests probably won’t be.
There's not actually a prohibition on IQ tests in hiring, and in fact, during my recent job search, one company I interviewed with had me take an IQ test (it wasn't explicitly labeled as one, but it had exactly the sorts of questions you would find on one). The same company also had me go through a more typical algorithm-heavy technical interview later in the process.
There's historical basis and case law to suggest that IQ exams have been used as illegal barriers on race, and it's not too far of an argument that your test could also be discriminating against a protected class. And it's potentially horrible PR.
> What is required by Congress is the removal of artificial, arbitrary, and unnecessary barriers to employment when the barriers operate invidiously to discriminate on the basis of racial or other impermissible classification...
> On the record before us, neither the high school completion requirement nor the general intelligence test is shown to bear a demonstrable relationship to successful performance of the jobs for which it was used. Both were adopted, as the Court of Appeals noted, without meaningful study of their relationship to job performance ability. Rather, a vice-president of the Company testified, the requirements were instituted on the Company's judgment that they generally would improve the overall quality of the workforce.
> The evidence, however, shows that employees who have not completed high school or taken the tests have continued to perform satisfactorily, and make progress in departments for which the high school and test criteria are now used.[8] The promotion record of present employees who would not be able to meet the new criteria thus suggests the possibility that the requirements may not be needed even for the limited purpose of preserving the avowed policy of advancement within the Company.
> In the context of this case, it is unnecessary to reach the question whether testing requirements that take into account capability for the next succeeding position or related future promotion might be utilized upon a showing that such long-range requirements fulfill a genuine business need. In the present case, the Company has made no such showing.
> We know that cognitive ability has one of the strongest correlations with job performance
There is a good argument that the technical interview success is much more strongly correlated to the number of hours spent practicing Leetcode-type questions than to actual cognitive ability.
The technical interview is not the only stage in the interview process. It might be better thought of as a way to filter out applicants who have poor innate ability and/or poor learning skills.
I would argue that being able to study leetcode and then apply that new knowledge you've learned is an important skill in and of itself in this industry. Isn't that what all of highschool and most of college is? Signalling that you a quick learner and a hard worker is important.
>It might be better thought of as a way to filter out applicants who have poor innate ability and/or poor learning skills.
It’s a way to filter out people who haven’t had time to study leetcode style quiz questions. It has no relationship to learning skills or innate ability. There is no way to distinguish between someone who hasn’t spent time grinding algorithms and someone who had with poor learning skills.
>I would argue that being able to study leetcode and then apply that new knowledge you've learned is an important skill in and of itself in this industry. Isn't that what all of highschool and most of college is?
Nope, it demonstrates that you had time to waste, nothing more. Candidates that built actual software to accomplish something are much more impressive because it shows they can write software attached to reality.
And just like the standardized test rigamarole, technical interviews can be gamed by investing in prep courses and materials for those who are willing to dump the money and time on them.
It's like saying "This guy spent 4 years in college trying to get a 4.0 GPA. Clearly, he only did it to game the system and get a good job/career. So I won't hire him"
A person willing to spend time and money preparing to succeed in a scenario that exists in real life (pass interviews), is one of the types of persons I want to hire.
The situation clearly becomes complicated when an ostensibly purely meritocratic measure becomes something that those with means can improve. If there was free and quality standardized test prep / technical interview prep for all, and then it was just a matter of dedicating enough spare time for the latter, then you might have a better case that it's good. Though you're also ignoring the fact that people have a limited amount of time/energy to devote to that, such as in the case as engineers with children.
I have recently passed several technical interviews at multiple large companies. The first one I botched completely. Then I looked at some algorithmic courses from Princeton on Coursera and passed the rest of the interviews in a breeze.
My point is, there _are_ free and good quality courses out there and if you have enough experience with actual work then it is just a matter of brushing up on the foundation. In the end I’m glad that I did.
Well, yes. However if you are already a good programmer then it will not take that much time, and if you are looking for a job then there is nothing wrong in expecting that you would prepare. And I would argue that it's not sunk time either. In my day to day job I do not use that many algorithms because most of the actually needed ones are hidden by the internals of the language, I got rusty and the refresher was useful.
>New data shows studying for the SAT for 20 hours on free Official SAT Practice on Khan Academy is associated with an average score gain of 115 points, nearly double the average score gain compared to students who don’t use Khan Academy. Out of nearly 250,000 test-takers studied, more than 16,000 gained 200 points or more between the PSAT/NMSQT and SAT …
But the company doesn’t really suffer if they don’t hire that disadvantaged developer who doesn’t have the resources to study for the test. Sure they have to spend some more $$$ on recruiting but there aren’t a ton of companies in America existentially threatened by their inability to hire these hypothetical engineers.
You're stating that the company doesn't lose out with false negatives. That's certainly true with big FAANG companies. But I suppose in my original comment, and when I brought up Goodhart's law in a descendent comment, also suggests that by creating a system that can be gamed by prep, you end up creating a situation that encourages false positives. In the end, is a company of Leetcoder elites and new CS grads going to be a more productive company than one comprised of experienced and capable engineers who don't happen to do as well on those tests? Which I suppose is the question for a different study.
In my experience hiring, the diligence to sit down and practice code tests correlates very highly with being a good employee. Just like strong academic achievement at good schools.
That's not to say there aren't gems in the rough who don't interview well, or skipped college, or who didn't go to good schools. But there's a huge time cost to looking for them.
Given that the former is dealing with decreased earnings growth and the latter is dealing with decreased user growth, perhaps all options should be explored.
> A person willing to spend time and money preparing to succeed in a scenario that exists in real life (pass interviews), is one of the types of persons I want to hire.
This brings us to an interesting dilemma. Do you want to hire people who know the problem domain you're solving --- or people who really badly want to be hired?
> There is a good argument that the technical interview success is much more strongly correlated to the number of hours spent practicing Leetcode-type questions than to actual cognitive ability.
Where is that "good argument" substantiated?
People say exactly the same thing about IQ tests, probably because it's something they'd like to be true. But it's not backed up by the evidence.
Ten years ago, I couldn’t program, so I couldn’t do leetcode problems. Now I can. If I go on the market, I’ll probably study a bit, and get even better at leetcode problems. During that whole time, my cognitive ability will not have significantly changed.
There probably is a decent correlation between cognitive ability and performance on these tasks, but it’s obviously mediated by practice.
https://www.gwern.net/DNB-FAQ contains a rather extensive review of this study and its follow-ups, critiques, etc, and comes to the conclusion that Dual n-Back training probably doesn't work to improve fluid intelligence.
That's quite a leap there. There is no contradiction between thinking IQ tests work and thinking that DNB increases IQ; I thought it did, and it was a bitter experience learning first-hand how exactly the Replication Crisis works.
If you think I am an unreliable source, why don't you look at the other meta-analyses like Melby-Lervåg, or Sala? Or look at the criticisms made by researchers like Redick whose experiments turned out to be nulls? (Redick's most recent comment on DNB is titled "The Hype Cycle of Working Memory Training" https://journals.sagepub.com/eprint/U8YNVXKM5SKF6ZY9TQBR/ful... , so you can probably guess what he thinks of DNB now.) Or look at what the field thinks in recent studies - assuming you can find any recent DNB/IQ studies, as everyone has realized that it doesn't do anything.
The process can definitely reject good candidates who are out of practice, but it’d be pretty hard to do the practice and give good performances without some underlying skill.
But look at it that way: a candidate that didn't practice before the interview, is either not so interested, a bit dull about basic human interaction or cannot even understand what an interview is.
It's honestly like coming to an interview without a suit and a tie. Sure if you're a pretentious startup it'll horrify you, but anywhere else it shows that the candidate just doesn't understand what's happening, that he could have just made the cheapest first impression effort and didn't even think about it, he came in sandals and tshirt instead.
Do you want him talking to clients ? Can you afford him not talking to clients ?
Is there anything wrong with preferring people who have spent a lot of time practicing leetcode?
Most jobs prefer people with certain kinds of experience vs raw cognitive ability. Raw cognitive ability doesn’t mean much outside of entry level jobs.
I don’t believe the interview process would be any more efficient or effective for software engineering roles if they started preferring pure “cognitive ability” to leetcode effort.
Assuming all other things are equal, obviously anyone will choose the candidate who can do these questions vs one who cannot.
The problem, obviously, is that all other things are seldom equal IRL. So, the issue now becomes : Would you rather hire someone who has a solid github profile but can't whiteboard algos, or someone who doesn't have any past portfolio but can ace whiteboard algos.
If you give me 20 years, I could probably get a PhD. Given ENOUGH time, anyone could theoretically achieve anything. But the differentiator between candidates happens when the clock is ticking.
I, and most other employers, are not looking for perfect answers. We want to know how you approach a problem under these constraints : short time, high pressure, unseen problem.
We want to know whether you are the type who chokes. It tells us that your desire to look good in front of someone (or conversely, your fear of looking bad in front of someone) fills up your mind more than your desire to solve a problem.
Would you choke if you mom asked you the same question - No, because you don't fear being judged, right ?
We want a similar trait in our developers. We want you to GIVE VERY LITTLE SHIT about what we think of you, at least for the next 30 minutes. We want you to have enough control of your mind to TEMPORARILY ignore our existence and focus on the problem at hand.
Granted, such scenarios occur rarely IRL, but it tells us a lot about how much control the person has over their psyche, how honest they are about their limits, what they do when they reach their limit etc. It's as much a behavioral test as an IQ one.
Having said all that, not all companies need non-choking geniuses. Choking geniuses work fine for many.
> I, and most other employers, are not looking for perfect answers
I wouldn't agree with that. Most of the interviewers I had at the end of the day didn't want the take the risk of "hire" if the answer wasn't like they expect.
> We want to know whether you are the type who chokes. It tells us that your desire to look good in front of someone (or conversely, your fear of looking bad in front of someone) fills up your mind more than your desire to solve a problem.
I think that can be better written in the job requirement. I would think twice before going to an interview which the solely purpose is to make an artificial stressful situation, out of an already quite stressful situation, just to see if I break or not.
Yeah, I can give zero shit if the stakes are low or if I am not interested in the job or the company for example. What happens is not my ego is hurt, but before I go for an interview I am already emotionally invested that I want to work for that prospective company.
All that said, people come from different backgrounds, imagine interviewing someone who is constantly judged in life and/or have struggled with bad managers prior to expect they would be free like a bird in front of failure.
The problem is that you're not paying enough for this. You're just not, even if you're FAANG. A smart dev can likely find a good answer for you---but it will almost always be true that it is a better answer, for them personally, to just go work somewhere else.
We are not paying you to SOLVE the problem with those constraints.
We are simply evaluating how you APPROACH a problem with those constraints.
If given such a problem, you throw your hands up in the air, or give me a lecture about how such a problem does not occur in real life, or how it doesn't represent your best-self, that itself tells a lot about you.
My question, as I repeat in other comments is this : WHY do you feel stressed ? Delve into your psyche for just 1 minute and it will boil down to - fear of being judged as incompetent. All we are looking for is someone who can set aside this fear TEMPORARILY.
You don't feel pressure if your mom throws you this problem, right ? Can you do the same even if a million people are watching, but just for 2 minutes ?
Even if we throw you a NP-Hard problem, can you objectively spend 2 minutes thinking about it, talking another 2 minutes about how you'd approach it, and then saying, 'This looks like it's beyond my skillset' - That clarity of thinking, mastery of fear, and blatant honesty is worth a hundred github apache open source projects.
People who can't do this lack the maturity to understand that in real-life, nobody really gives a damn about our achievements and failures except us (and very close relatives of course).
I think you are kidding yourself if you think the majority of these companies are satisfied with an answer consisting of "This looks like it's beyond my skillset". All things considered, they are going to progress the candidate who can solve the problem because it's an early stage test.
I wouldn't feel pressure if my mom threw me the problem, but I still probably wouldn't be able to come up with a solution (or even approach anything useful in a reasonable amount of time) that would satisfy an interviewer because frankly, I'm terrible at it and haven't spent the time to get good, just like many, many other successful developers.
Stress is only one aspect of these that people dislike.
>All we are looking for is someone who can set aside this fear TEMPORARILY.
Why should I do this? Fear is a survival signal, why should I ignore it? It's not fear of the problem, it's fear of the company. And is this not a justified fear? You say not to be afraid of being judged, but isn't that the entire purpose of the interview?
Now, there is a scenario I can imagine where I have some pride invested and am willing to be a fearless engineering badass, the developer equivalent of a Marine. But the Marines are loyal back in return, never leaving a man behind, etc. I'd also happily attempt such a problem for my mom, because I know our relationship isn't dependent on this particular problem. Is your company going to be as loyal as the Marines, or love me unconditionally like my mom?
Why do you say that FAANG isn’t paying enough for this? They seem to be, since they are hiring more than enough engineers in this manner to build even trillion-dollar companies. I doubt they give two shits about the developers who decide to work elsewhere.
Sure, they hire a good # of devs---who then mysteriously want to go into management, leave to start a startup, or simply complain about their bosses. No one with actual agency and power in the matter chooses to stay in that position, once they understand what it is, what their options are, and aren't handcuffed by a mortgage. Or perhaps the company isn't as dysfunctional as the interview, and they stay---in which case it is just the interview that's miscalibrated, serving as an arbitrary hazing ritual.
Despite any problems they have, they are able to retain enough devs to get their work done. Are they losing out on even $1 of earnings because of their inability to hire devs? Even with the arbitrary hazing ritual in place?
> Is there anything wrong with preferring people who have spent a lot of time practicing leetcode?
If you care about diversity and inclusion, yes. This interview format prefers people with formal education, extra time, good mental health, no kids, etc.
This can easily eliminate some women + caregivers + those with kids, the self taught, people that struggle with mental health or learning disabilities (including ADHD), etc.
Anecdotal story:
Studying LC because I'm self taught led me into burn out and I ended up with an ADHD diagnosis (common for women to get diagnosed at an older age anyway). My ADHD brain wants to do things it likes and studying LC isn't one of them so it led to being burned out faster. A former manager even told me to slow down and I didn't listen!
I struggled for a while with interviews and getting my current job. Each additional interview became extremely traumatizing for me (which is probably compounded by Rejection Sensitive Dysphoria that comes with ADHD). It's been almost a year since I got my job and I still worry about interviewing again in the future.
For me, being a woman in tech with ADHD makes it really hard to walk into an interview feeling confident. It's intimidating to be obviously different then the vast majority of my interviewers and I'm so worried if I'll keep my train of thought and remember things or talk in a way that makes sense. And it spirals into me worrying if I make women in tech look bad because so many people seem to believe women are inherently worse.
I've noticed that exactly. IQ presumably is something inherent.. But personal experience shows that technical interviews have nothing to do with IQ. They can be easily solved by creating a dynamic programming problem pattern matcher in your brain, which is what I created after doing 300 leetcode problems. I did not become smarter in general or at coding at all, I just got good at mapping problems to binary trees or dynamic programming or string search.
I don't get how IQ could be considered inherent. It's a test, so results are defined by talent and training. A person who spent a month solving the types of problems present in an IQ test will certainly get a higher score. By, the way, you did become better at coding by solving Leetcode problems, cause now you can easily spot them in the wild and use a proper algorithm instead of some naive solution.
aren't IQ tests the same?
You just learn how to solve some kind of problems (series of images, series of numbers and so on) etc.
This obviously does not make you "smarter", it just makes you good at solving some kind of puzzles.
But "smart" is ill defined and vague anyway..
That's the reason all real IQ tests (not the ones you find on the internet, the real ones) are secret/closely guarded: Knowing the tests beforehand skews the results.
So: yes, it is the same in that way and has been theorized as one of the reasons for the Flynn effect. Afaik there's also research into new types of IQ tests to mitigate this all the time, but a good IQ test has higher requirements than the typical interviewing test, so it takes longer.
It's really not. Knowing how to solve those leetcode questions has not improved my professional life one bit. I am not a better programmer.. In fact it probably made me worse. At my work I pretty much never deal with the kind of coding that leetcode requires. I'm more interested in writing larger scale software rather than throw away toy problems which completely ignore code quality or maintainability.
In fact, I'm not even good at solving leetcode problems that well so I wasn't able to get a job at the big N companies...although it did help with passing the interview at one of the smaller companies.
It depends- maybe if people devoted enough time to cramming algorithmic problems (more than they did in their CS classes, even) could they have memorized enough solutions and seen the patterns enough to be able to master this type of technical interview? One wonders, can that rote memorization + pattern machine process be trained into anyone- meaning that this is not a measure of intelligence but of brute force, like memorizing Rubik's Cube solutions?
And then the practical concern arises- does that actually make anyone a better programmer, and benefit employers?
Maybe they are at Google, but hiring managers are not generally savvy enough to have the ulterior motive you're attributing to them; most of them are just doing what they were taught to do at previous jobs.
I think your comment is a pretty gross mischaracterization of the relevant Supreme Court ruling. It's definitely not the case that "IQ tests are actually not allowed in the US". From the wiki page you link:
The Supreme Court ruled that under Title VII of the Civil Rights Act of 1964, if such tests disparately impact ethnic minority groups, businesses must demonstrate that such tests are "reasonably related" to the job for which the test is required.
I think it's pretty trivial for any company to argue that technical interviews are relevant to a developer job.
Note that the material you quote from Wikipedia is in error; the simple-to-meet “reasonably related” standard only applies for disparate impact cases where the discriminatory effect is based on age over 40, which is not under Title VII; the Title VII test is a much higher bar and once the disparate effect is shown requires showing “business necessity”, and even then disparate impact discrimination can be found if the employer declined to adopt available alternatives with less discriminatory effect.
> I think it's pretty trivial for any company to argue that technical interviews are relevant to a developer job.
Is it? Can you show, right now, that your companies technical interviews and job performance correlate positively? Cause from the things I've seen that's not checked commonly in companies and even if it is you don't hear "the technical interview is bullshit, let's do something else" as a result.
That's how I view them as well. It's why a lot of people can get through them by doing lots of Leetcode in the same way that one might do lots of logic problems in order to score higher on an IQ test.
Leetcode practice gets you through a particular kind of coding test. I would say that only covers ~half what a good coding test interview should cover. It makes you good at the "how", and solving algorithmic or textbook architecture questions.
The questions I prefer to ask are about writing a basic but readable and maintainable piece of code, maybe 50-100 lines long, that requires the candidate to talk about why they are making certain trade-offs, questions that look at when they would involve other team members in decisions (when do you ask a Product Manager or Designer for help?), questions that try to understand whether they can solve problems rather than the one problem in one language given to them in the interview. I typically do those interviews in Python but some of the best candidates have never written any Python – but their communication, intuition, and problem solving abilities were great.
For junior positions, I tend to ask very simple questions. Reverse a string or given an array and an int count how many times that number occurs (or variants thereof). Something that requires a loop and some ifs. It opens the posibility of endless discussions about extensions, refactoring, runtime, edgecases - it gives a solid bases on which to judge their ability.
For senior positions, it's slightly more advanced. But if they make it through to second round, we have them bug hunt and refactor some piece of code.
Programming is a craft and I need to evaluate that craft.
At least there’s considerable variance in the problems on an IQ test so your intelligence is diversified across many problem types. The type of shit I’ve come up against in past interviews has often been essentially an all-or-nothing single problem. Which is why I now refuse technical interviews.
Advice on getting a job as a software engineer without a technical interview? Literally 100% of the dozens of interviews I've had in my life were coding interviews.
Have projects available online that showcase your ability/ingenuity/creativity. Just reject the technical interview and explain your reasoning, offer an alternative.
Works for me because I don’t want to work anywhere where interviews aren’t being conducted by people competent enough to handle fluid requirements.
I don't see how asking a potential software engineer to write a 30 line function in 30 minutes is bad. Being able to write code is part of the job description.
I don’t need to “prove” that I can write code by memorizing an algorithm. I have made more than enough code available on my GitHub which I’d be happy to explain in depth with anyone who is interested. Not interested? Great, you’ve made it easier for me to make my decision.
Depending on what those 30 lines are supposed to do and whether they have to be asymptotically optimal at doing it, writing down 30 lines of code while talking about every decision you’re making, to show the “thought process” might very well be more than a 30 minute task.
I didn't mean to imply that this is the only way, just that it's a way that has worked for me.
I've worked with embedded systems and with desktop software that was used by my employers (electronics manufacturers) rather than sold by them, and I have never been asked to code in an interview.
IQ Tests ARE allowed in the US, as clearly stated in the page you linked ... "Broad aptitude tests used in hiring practices that disparately impact ethnic minorities must be reasonably related to the job."
Technical interviews are NOT a loophole about IQ tests ... they are tests that are related to the job.
This is questionable. There is data suggesting job performance is not correlated to interview performance. Also, none of the questions in my last interviews have anything related my actual job, not even the language nor the problems I solved (since I could pick any language to write the code). The link between them are very arbitrary imo.
We know that cognitive ability has one of the strongest correlations with job performance.
So "technical interviews" that are heavy on algorithm puzzles are just a loophole to get around the IQ test issue. They aren't actually testing domain knowledge.
[1] https://en.m.wikipedia.org/wiki/Griggs_v._Duke_Power_Co.