loading...
Cover image for Let's end Timed Coding Challenges, Together

Let's end Timed Coding Challenges, Together

silvestricodes profile image Jonathan Silvestri ・5 min read

Introduction

Have you ever started an interview process with a company, and had them send you a timed coding challenge as a first round process? Do the questions feel like challenges that the developers of the company would not be faced with on a daily basis? If this has happened to you, and you're feeling frustrated, you're well within your rights to feel that way!

Image of red scantron sheet

Remember These?

Timed Coding Challenges are largely incorrectly implemented. There are a number of factors here that play into this, including:

  • Involving questions without any sort of context as to how it relates to the day-to-day challenges that the company's dev team faces.
  • Stuffing a lot of questions into a time frame that makes it very hard to provide good answers in a stress-free environment.
  • Writing incredibly long user scenarios that are time drains.
  • Not utilizing the results of the timed challenge outside of moving candidates to a next step (more on this later).

Alright Jon, what are you getting at?

I'm writing this in hopes of changing some minds on the right way to use this tool. Personally, I don't think timed challenges (or any long take-home challenge of any kind) should ever be used, but that doesn't make for a very interesting article. Instead, I'll offer some ideas on better implementation of this process and how companies can turn it into a stronger screening tool.

Let's get to the do's and don't's!

Don't: Send arbitrary challenges that your dev team wouldn't encounter in their day-to-day.

Do: Select a challenge that your team has found a solution for, and offer it as a challenge (or something close to it) at this point.

Image of a student with a mechanical pencil taking a test.

You may begin

Examine a challenge your dev team solved that has been in a stable condition for a long time. Perhaps you have some logic around filtering the response of an API. Maybe you have some interesting conditionals and approaches you use to rendering data in different forms. Asking someone to do this as a challenge would be a great way to see how they approach something from their vantage point. It will also make it interesting and easier for whoever is reviewing the code to formulate opinions on the candidate quality!

Contrast this with asking more algorithmically focused question, like doing efficient counts of matching pairs in an array. Does this question fit with the day-to-day mindset and work of your engineers? If it does, then by all means, go for it. Otherwise, it might be harder to get the person reviewing this code engaged in that process and could cause them to reject quality candidates because of an insufficient screening process. Consider both sides of the coin!


Don't: Overload the candidate with many questions because of a perceived large time limit.

Do: Focus on asking quality, engaging questions that can be completed with time to spare!

Close up image of a wrist watch

Tick Tock Tick Tock

I and many of my peers have felt the pain of a 5 question, 90 minute timed exercise where 1 or 2 questions required a significant portion of time to read and think about before attempting an answer. Truth is, there's just no value to be found in having a large question pool given to your candidates. Copy and pasting questions from popular coding practice websites and calling it a day, for example, is very disingenuous and often how these situations arise.

Instead, if you're going the route of a timed interview, why not pare down the question pool to one or two questions, and give a candidate just as much time to complete them as if you had given them five questions? What do you have to lose by doing this? Better yet -- do your current engineers work in an environment where they are rigorously timed like this? If the answer is "no", why are you exposing candidates to an unrealistic environment through which you evaluate them by? This sets people up to fail, rather than giving them every opportunity to succeed.

Consider a simpler warm-up question (this one can be generic and straightforward), followed by a tailored question that we discussed previously. Those 90 minutes you were giving them for 5 questions? Give them the same amount of time here, if you're timing things. That way, you significantly reduce the pressure a candidate feels and you'll be getting work that they've had time to think about, review, and optimize!


Don't: Define every single detail of your question(s) down to the last character.

Do: Get to the point quickly!

Image of many rows of old books on shelves

Stories on stories on stories on...

I won't linger on this one for too long. Your coding challenges do not need a full blown narrative attached to them regardless of what the ask is. One or two sentences of context are fine, and then clearly defined expected outputs with sample inputs capture everything you need.


Don't: Forget all about this coding challenge once you move a candidate to a next step.

Do: Craft a challenge that can be iterated on so you can continue to use it throughout any further technical assessments in your interview process.

Image of a group of people sitting together working on their laptops.

Collaboration is awesome!

I can say with certainty that the majority of timed coding challenges I have completed have been discarded once I move to the next stage in that process. Outside of off-hand references to the challenge, it simply would not feature in any way, shape, or form.

I am often left asking myself why any company would make candidates invest that much time in a challenge, only for it to be ignored as that candidates progresses through the process. Wouldn't it be easier to craft a challenge that a candidate could carry with them the whole way through? Think of the benefits!

  • You can structure your entire technical assessment of a candidate around how they built something from scratch with just a few guidelines and some assets. Simplification for a low cost.
  • You get a really in-depth look into a candidate's problem-solving process through a focused, well-defined challenge.
  • Your candidates get the benefit of learning what it's like to collaborate with the existing engineers, and vice versa.
  • You'll be doing candidates a solid by helping them to practice and refine their problem solving processes in different settings!

Conclusion

Hopefully this article gave you some ideas if you are able to influence the interview process at your company. Think a bit about what resources you have, what realistic challenges you can present to your candidates, and everything you want to learn about someone (and everything you want someone to learn about you!) before sending that timed challenge. You only stand to gain a better reputation, positive feedback, and higher-quality candidates by refining your process!

Posted on by:

silvestricodes profile

Jonathan Silvestri

@silvestricodes

Enjoys JavaScript, React, and video games.

Discussion

markdown guide
 

May be worth noting that such coding challenges are also a self-reinforcing pattern. The more people the process flunks, the more your people who passed it are likely to consider it valuable.

Three reasons behind that:

  1. The higher the failure rate, the more it makes some feel like those who pass are "the best of the best."

  2. The stronger that feeling, the more current employees may find their success at it feels validating, reinforcing a sense of self-worth.

  3. Like fraternity members who had to go through hazing to get initiated, there can even be a sense of righteous injustice that new members wouldn't have to pass through the same level of Hell to join their ranks.

If you're seriously considering changing your process, that reinforcement and bias may result in serious push-back against the changes.

 
 

Definitely can see how those biases will keep people from being okay with making these changes.

 

I think timed coding challenges are not the best way to evaluate candidates as a generic practice.

However, there are some fields in which these challenges are best used. Fields like research (programming), low level programming etc doesn't require the knowledge of a high level programmer. Instead of knowing which tools or libraries or frameworks to use, they must know how the language, programs and algorithms work behind the scene.

Although, I think the time limit might not be the best way to evaluate how fluent are the candidates with algorithms, I can't think of a better alternative that can be used in evaluation process.

 

Update: okay after thinking about it, I concluded I was wrong. These coding challenges are very important regardless of fields. We all should know how algorithms work. We all should be able to come up with many different ways to solve a logical problem as soon as we encounter it so that we have the option to choose the best one. The time limit is also important. Because if you are not able to come up with the best way fast, you will end up choosing a different way which will become a problem later.

However, the difficulty of the challenge needs to regulated based on domain.

 

I don't agree that not being able to come up with the best way fast somehow means the different way will become a problem later. I'm not sure how you came to that conclusion, but I would need you to provide some extremely measurable examples of that before I'd concede the point.

Timing people and expecting them to produce their best work is dubious at best, incredibly presumptuous and unfair at worst.

Let's consider a simple example. Let's suppose someone new to the field of development, knows how to use various frameworks and libraries to create working applications. But due to not enough exposure in algorithms and data structures, doesn't know that lists are O(n) and maps are O(1). In a performance critical application, he has to create collections of objects and implement logic to search elements in the collections. Due to lack of enough knowledge, he thought that using list as the data structure and using list.find(element) as the search logic is the best choice. He won't look for other possible options as he already has a solution that he thinks is the best one. Now he has written the entire app with list and it's not performing as expected. When he later finds the reason, he has to modify a lot of code to convert all the lists into maps.

Competitive programming requires us to have this prior knowledge of all the options we could use in such situations. We know a solution doesn't mean it's the best one. But we often choose the solution that comes to our mind first. And we often don't look for other options unless we get a hint that the logic we used is not the optimal one.

It's true that in real world, the time limit we get are much longer. Yet, we don't want to spend a lot of time on simple things like this. So we subconsciously set a time limit for ourselves to solve simple problems and go with whatever solution we find first. And most often this time limit we subconsciously set for ourselves is even shorter than what we get in timed challenges.

Your example rarely if ever occurs, as someone either catches that sort of thing in a code review, or the lists they work with will never be large enough to matter.

Competitive programming is an incredibly dubious proposition and I caution against anyone having that sort of mindset.

Your example rarely if ever occurs, as someone either catches that sort of thing in a code review, or the lists they work with will never be large enough to matter.

It's subjective. In domains like web development where the bottleneck is IO, data is rarely large enough, solution to most of the problems is to use some ready-made library, it might be the case. But that's not programming all about right? The reviewer needs to know too that maps are O(1) and lists are O(n). If everyone adopts "it doesn't matter" mindset, there will be nobody to point such things out or to write those ready-made optimized libraries.

My point is that an interview setting is not the place to uncover that knowledge. That is far better suited for teams that are working collaboratively together already.

 

Knowing a candidate can do a job is important so coding should be apart of screening. However, if you are going to do an automated coding challenge as part of your hiring process, there should be a couple of ground rules:

  • don't limit the candidate with things like "you can't use the Internet" -- being able to find a solution without reinventing the wheel is important

  • don't place a time limit -- that doesn't mean don't time the challenge, just have a conversation about why it took whatever amont of time. Did you use TDD? Did you struggle with a particular section?

  • assuming you have unit tests in the challenge that validate the solution, don't limit access to the test inputs and don't take away the simplest of debugging tools like writing out to the ui

  • don't assume that the way the challenge is written will be interpreted by the candidate correctly. It may not be as clear as you think and there should be two-way communication throughout the challenge.

  • understand that coding is not the extent of a dev job and communication and willingness to ask questions are equally, if not more, important.

I'd rather hire a candidate who will asks questions and demonstrates critical thinking over the rockstar that can hack out the fastest solution without tests. We canteach better coding, it's much harder to teach teamwork and communication IMHO.

 

Good points! Yeah -- coding being part of the screening at some point isn't the worst. I should clarify that I think that take-home challenges are not so great, and that I think doing the coding in a pairing session over a hangouts or phone call with a shared code env is what I prefer.

 

I like this post and share your sentiments! Recently, I received via email a Technical Review question to do within 24 hours to be built in Java (server-client application). I believe the review was pretty generic as the role didn't even ask for Java as a skill.

 

Ugh that sounds maddening. I wouldn't blame you if you didn't take the time to do it.

 

YOU WANT TO END JAMS?????!!!!!! ah okay I'm a fan of procrastination anyway