Why Developers Hate Coding Skills Tests (And What Hiring Managers Can Do To Change That)

By Geoff Roberts

When I signed on to join the team at Qualified last month, I ran the company through the diligence process that I use to assess opportunities at technology start-ups. Great product? Check. Strong growth trajectory? Check. Awesome customers? Check. A team I'm excited to be a part of? Check.

But beyond this list of important criteria my decision came down to a feeling that I couldn’t shake—one that my intuition told me was worth listening to above all else.

In 10 years of working with high growth technology start-ups, I’ve never come across a company that’s too good at hiring software developers. Everybody struggles with finding great software engineers.

I’ve sat in countless board meetings, leadership off-sites, and all-hands gathering where the story is largely the same—”We had 10+ open job reqs for development positions at the beginning of the year. We’ve filled three. One hire didn’t work out.” Queue the thumb-twiddling and downward glances.

Expensive employee referral programs ensue. Contracted recruiters come in to help. But all the subsequent efforts aside the common denominator is clear—there’s a shortage of software engineering talent and companies need to find better ways to identify, attract, and retain developers.

When I looked at the opportunity at Qualified the marketer in me saw a very real pain point and a market opportunity worth attacking; followed shortly thereafter by a lot of controversy surrounding the tools that are available to help companies assess software engineers.

The controversy surrounding coding skills tests

Any Google search or discussion with software engineers about the developer hiring process quickly surfaces a myriad of opinions on how coding skills tests are used in the recruiting process. You’ll find lengthy debates on this topic in HackerNews threads as well as not-so-subtle articles with titles like Why Coding Tests Are A Bad Interview Technique.

At first blush these sentiments should have deterred me from joining a company that makes a developer assessment platform, right? But I saw that the market for these products is exploding and I decided to dig in—a process that began by asking my former boss Dimitris Georgakopulous, Co-founder of Buildium and Outseta, for his take on coding skills tests.

“There’s absolutely a market and need for tools like this,” Dimitris said. “You simply can’t assess and hire developers without asking them to write some actual code.”

Joel Spolsky, CEO of Stack Overflow and Co-founder of Trello, provided further backing of this perspective with a simple analogy.

“Would you hire a magician without asking them to show you some magic tricks? Of course not. Would you hire a caterer for your wedding without tasting their food? I doubt it. Do whatever you want during interviews, but make the candidate write some code," writes Spolsky.

Encouraged by this feedback, I turned the question on the development team at Qualified. “Why do so many developers dislike coding skills tests?” Their answers surprised (and built credibility with) me.

“I've heard the complaint that interview tests are awful, pointless, and demeaning a lot— especially on social media,” said Phil DeJarnett, a Senior Front-End Developer at Qualified. “It's frustrating at best because it's clear that developers are looking at it from an (understandably) selfish perspective. ‘I'm a good developer, why do I need to prove this to somebody I don't even know?'"

Jake Hoffner, Qualified’s Co-founder and CTO, took it one step further.

“As a senior developer I’d be hesitant to take most coding assessments if I was looking for a new job,” Hoffner says. “I feel like I’ve built products and have a professional network that should preclude me from needing to do that.”

Prior to Co-founding Qualified Jake built Codewars—he’s definitely a person that I’d describe as a “developer’s developer.” But coming from a guy that’s now spending his professional life building a developer assessment platform, I was taken aback by hearing this level of empathy for the anti-coding tests sentiment.

“What’s this guy doing building a developer assessment platform then?” I wondered to myself.

I’ve spent the last few weeks talking to HR and engineering leaders to dig into this question in the greatest amount of depth possible. I’ve since resurfaced with strong conviction and evidence that there are companies using coding skills assessments in not only a developer friendly manner, but also in a way that directly leads to them finding and hiring the type of developers that immediately begin delivering on-the-job.

In fact, if wielded properly coding skills assessments can even help companies build credibility and excitement with developers throughout the hiring process. This is yet another aspect of your employer brand that can be optimized—and perhaps the one that speaks most directly to what it’s like to work on your company’s engineering team.

Shane Shown, a Talent Acquisition expert who has built engineering teams at companies like Facebook and Zillow, says it best.

“When a candidate does run into an interview that represents real-world problems that would need to be solved in the day-to-day of the actual position it’s MIND-BLOWING. I have had candidates leave an interview with a coding assignment that they were actually excited to complete, because they felt like they would understand the company’s problem and add real value.”

This post will teach you how you can move beyond the negative sentiments and turn your company’s use of coding assessments into a strategic advantage. But let’s start by reviewing some of the reasons developers dislike coding assessments in the first place.

The reasons developers often dislike coding tests (and how to change their tune)

While developers have expressed disdain for coding assessments for a wide variety of reasons, almost all of them can be overcome. These sentiments almost always come from coding skills tests being used as a blunt instrument to pre-screen developer candidates out of the recruiting funnel. Here’s how your company can intelligently apply coding assessments to flip the script and turn hiring developers into a competitive advantage.

Most coding assessments test only algorithmic skills

The most common—and credible—reason that coding skills tests get a bad rap is that the vast majority of assessments test algorithmic skills rather than actual programming ability. The tests represent a computer science problem that has little to do with assessing an engineer’s ability to deliver the work they’d actually find themselves doing on a day-to-day basis.

Glen McCallum, a Senior Software Engineer at INscribe Digital, draws a wonderful analogy between coding and music in his article Why Code Challenges Are Bad Practice For Hiring Senior Developers published just last week. Testing algorithmic skills is a technical exercise akin to learning to play scales in music. While playing scales is important when you’re learning music, it has almost nothing to do with working as a jazz musician.

“This week I was working on a Hackerrank problem for fun,” writes McCallum. “I had to pull out my old text book for data structures and algorithms in order to code up a merge sort from scratch. I felt inadequate, sent a tweet, and then a light bulb turned on in my head.”

“We all studied these algorithms in undergraduate computer science classes. But as you gain a lot of professional experience you learn how and when to apply them, then you refresh your memory as needed. I knew exactly what merge sort was for and I knew exactly where to find the reference algorithm. I just didn’t have the details in my head at that moment.”

McCallum continues building his point by returning to his music analogy.

“Does it make you a better professional musician if you can play all scales perfectly at any time? Absolutely not! In fact I’d argue that you’re wasting precious practice time. How silly would it be if you asked a professional musician with a history of outstanding performances to sit down at an audition and play a scale? It’s insulting. Then dismiss them before they even perform because they didn’t do the scale perfectly. The thought makes me sick to my stomach. But in 2019 it happens every day to senior developers applying for jobs.”

Luckily, there are developers out there who have been heavily involved in the hiring process that also understand the objectives of their friends in HR and the importance of pre-screening candidates.

“There's enough programmers in the world who aren't really skilled enough for the job (whatever that job may be) that it becomes necessary to offer some kind of litmus test that a potential job-seeker must pass," writes Ted Neward, a Guest Lecturer on development at the University of Washington’s iSchool. “I get that and it's not like all the programming tests in the world are created equal. But the ones where the challenge is to implement some algorithmic doodad or other? Shudder."

How to address the “coding assessments test only algorithmic skills” challenge Most coding assessments are built as generic STDIN/STDOUT tests. Instead of leveraging these tools, look for coding assessments based on unit tests—developers write unit tests on a daily basis. Also look for assessments that use language specific frameworks—preferably the same languages that candidates would be writing code in if hired at your company. Testing for the nuances of each specific language helps to better surface candidates who will deliver clean code on-the-job.

Coding skills tests are time consuming

Another reason that coding skills assessments are often bad mouthed is simply because they can be time consuming to complete.

“My biggest pet peeve with these types of tests is this: there are a lot of companies out there, and I’m sending out resumes to each one that I can find,” says Brandon Savage, Owner at Tailwinds. “I simply do not have the time to write fifteen code samples a day, just because you want to evaluate me against your coding test. Period.”

How to address the “coding tests are too time consuming” challenge While Savage has a point, this objection is far easier to overcome. From the perspective of the hiring company, all it takes is a little self-awareness and empathy for the applicant—make it a point to be mindful and respectful of each applicants’ time.

I asked Jake Hoffner, Qualified’s CTO, what he’d recommend with this challenge in mind. His answer was simple—for a first cut at a technical screening exercise, share an assessment that should take no longer than an hour to complete and ideally only 20-30 minutes. For a final stage technical assessment, limit the time commitment of the candidate to two hours.

“Even if they don’t complete the assessment in two hours ask them to stop at that point,” says Hoffner. “You’ll learn a lot from what they were able to accomplish in two hours, and you’ll learn even more from talking to them about where they’re at two hours in and what they’d do next anyways.”

From the perspective of the applicant, I’d also push back on Savage’s sentiment—if you find yourself needing to write 15 code samples, you need to reassess your job search strategy. This is indicative of a spray-and-pray strategy where you’re firing off a large volume of resumes (and are subsequently asked to take a high number of coding tests).

Regardless of the position you’re applying for, focus instead on applying for a smaller number of positions that you’re most interested in. This way you can devote significantly more time to each, which almost always yields a higher number of job offers.

Coding tests don’t reflect the real world programming experience

The focus on algorithmic skills aside, software engineers often cite that coding tests don’t reflect the actual experience of writing code as you would on-the-job. For example, many coding tests require developers to build something from scratch.

While starting from a blank slate is easy, it’s rare that developers would do this very often once hired—more often than not the real world, on-the-job experience would instead dictate that you familiarize yourself with an existing code base and learn to contribute to it effectively.

How to address the “coding tests don’t reflect actual programming” challenge There are countless generic coding tests out there because they are easy to create. Instead, look for coding assessment tools that give you access to a database of pre-built assessments in a wide variety of languages, scopes, and challenge types that are as similar to your own code base as possible.

If you’re hiring for more senior roles, look for a coding assessment platform that allows you to create and administer your own custom, multi-file assessments that directly reflect the process of contributing to your actual code base.

Coding assessments take developers out of their comfort zone

Developers often question the validity of coding skills tests because they’re asked to perform in an environment that’s not familiar to them. Understandably, they feel like they’re not able to put their best foot forward.

This complaint is also very legitimate—imagine asking Tiger Woods to play the Masters with somebody else’s golf clubs or asking a violinist to audition by playing the cello. Most developers have an IDE (integrated development environment) that’s been highly customized for their own personal tastes and preferences in a way that’s comfortable, familiar, and allows them to write code as efficiently as possible. How could they possibly perform their best when they’re thrown into an unfamiliar environment that they wouldn’t actually use if hired anyways?

On top of that many coding assessments are either timed or ask developers to perform on-the-spot while others look on—again contrived circumstances that add to a developer’s anxiety.

How to address the “coding tests take developers out of their comfort zone” challenge This one is simple to address—let developers operate within the confines of their comfort zone. Doing so will help you get the best read possible on how they’ll actually contribute on-the-job.

Look for coding assessment tools that allow developers to customize their IDE, or better yet use their own IDE altogether when completing coding assessments. If you do time restrict assessments, allow developers to work independently then use code playbook tools and pair programming sessions to discuss and expand upon the work they’ve already completed. This will give you a deeper level of insight into their thought process and problem solving ability.

Senior developers shouldn’t need to prove themselves

Senior developers are sometimes unwilling to complete coding assessments simply because they feel like they shouldn’t need to prove themselves—they’ve been working for years and have countless products and code samples they can instead highlight to showcase their abilities. You wouldn’t try to sign Lebron James by bringing him in for a workout and asking him shoot free throws for the first hour.

While there’s validity to this point—particularly with the very best engineers—there’s also at least some degree of ego at play here. As someone who has hired a large number of marketers and designers, when I’m confronted with someone who is not willing to deliver a reasonably scoped test project what I’m essentially hearing is, “I have an ego and I don’t want this job that badly.”

I can tell you with certainty that there’s a lot more risk with a marketer delivering a 90-day growth plan during the interview process than there is with an engineer completing a coding assessment. You’re being asked to share your own, original thinking and there’s nothing to keep the company from taking your own ideas and running with them on their own. There’s inherent risk there and you are being asked to provide value up front.

But if you are unwilling to do this, what does it say to the company that’s looking to hire you? And if you really check your ego at the door and take a look at this from the perspective of the employer, it’s really not too much to ask. For development jobs the employer is likely committing to paying you well over $100,000, to contributing to your 401k, to bringing on a person that they hope will contribute to their product for a long time to come. The employer is almost always taking on more risk than the applicant—particularly in smaller companies.

How to address the “I shouldn’t need to prove myself” challenge Make sure that whatever you are asking of senior developers to do, it’s actually providing some insight on how they’d contribute on-the-job—don’t make them jump through hoops for the sake of jumping through hoops!

If they’re truly a high quality, senior developer you can spend less time assessing their actual programming ability and more time probing on what they’re like to work with and how well they incorporate feedback. Create custom, detailed coding assessments to provide a common structure against which to evaluate senior developers, but purposely limit the amount of time they actually spend on assessments.

Instead use pair programming sessions or code playback to discuss with senior developers some of the architecture decisions they made or how they might expand a feature further. Rather than using coding problems in isolation, use the assessment as a starting point for a much deeper discussion that gives you insight into how candidates think and what it it would be like to work with them.

Automated scoring of code can lead to false rejections

Finally, there’s often skepticism or worry about coding skills tests because automated scoring of code can lead to false negatives—applicants getting rejected that shouldn’t necessarily be. Fair enough—for any coding problem there’s almost never a single “right” answer.

Consider the great works of literature; if assessed against the strict rules of grammar, almost all of them would fail an automated test. James Joyce’s Ulysses, considered one of the great works of English literature, is filled with fragmented sentences, slang, and words that are made up altogether. Joyce himself would fail to get hired at even the local newspaper if they were using an automated language assessment.

How to address the “fear of false rejections” challenge Automated assessments are a prerequisite for hiring developers at any sort of scale and save a company’s internal employees from spending a massive amount of time reviewing code submissions—any developer worth their salt will understand and appreciate this.

What you can do is look for automated assessments that have been informed, validated, and approved by thousands of other developers already through communities like Codewars (which forms the basis for all of Qualified’s assessments). Also look to use a developer assessment platform that provides a full suite of feedback tools beyond automated scoring of assessments—look for pair programming functionality, code playback, and tools to manually provide and capture feedback on specific segments of code.

Conclusion

Developers have voiced many hang-ups with coding skills tests, and many of their criticisms are well founded. But by leveraging the latest technologies in tandem with developer friendly hiring practices, companies can turn their use developer assessment tools into a competitive advantage.

Your company’s use of coding skills assessments represents a very real opportunity to build your employer brand, give candidates a strong initial impression of your engineering team’s culture, and create a hiring process that developers talk about glowingly (and share with their friends).

“How a company interviews people is a direct correlation with the quality of talent they attract,” says Shane Shown.

What does your developer assessment process say about your company?