Codility === Sadness

My very first article on this site (https://dev.to/bytebodger/your-coding-tests-are-probably-eliminating-some-of-your-best-candidates-de7) was about the sheer ridiculousness of many "coding tests". So this rant is nothing new from me. But I just had to take a few minutes to tell you how much I despise Codility. If you're a hiring manager and you think that Codility is giving you a better measurement about who is a "real" programmer... you need to re-evaluate your premises.

Some background information

I've been coding now for a quarter century. I'd never claim - not even for a second - that I've figured it all out. Nor am I an expert in "all things coding". But let's just say that I have extreme confidence in my ability to crank out vast quantities of high-quality code in brief periods of time.

So maybe you'd think that I have no problem with coding tests? Maybe you'd think that I churn through those tests with speed and agility? But if you think that... you'd be wrong.

When I'm confronted with a request to take a coding test - of nearly any kind - it's always a crapshoot. Sometimes... I ace the tests. Other times... I fail miserably.

Now, you might assume this is because, despite my decades of experience, I'm still not much of a "real" coder". But I see the issue much differently. Nearly every time I've "failed" a coding test, I've come away from the experience thinking that the test was, in fact, a piss-poor measurement of my true skills. If you've been in this career field for even a short amount of time, I suspect that you may feel the same.

The setup

First I'll acknowledge that, as I've highlighted in many previous articles, I rarely even bother to take coding tests. I've got a few dozen NPM packages that I've written and maintain. I have a boffo presence on GitHub. I have numerous live sites, with openly-accessible code, that I maintain on the web. I have a great CV site (that, in its own right, is a full-fledged React app). I've written more than 100 articles on this site solely on the subject of software engineering. At the risk of sounding even more arrogant than I already am, the simple fact is that I have a really solid internet presence that showcases my coding skills.

Furthermore, I usually have a really good job in-hand. So when some recruiter calls me and says that they have a great opportunity for me to consider, and "all" I need to do is complete this little coding test to be considered, I normally tell them, "Umm... no thanks."

But every once-in-a-while, I find myself "between gigs". And when I do, I find it harder to tell someone "no" when they want me to jump through a few coding hoops for them. So that's how I found myself, once again, confronted with the Codility coding test.

The test

The test had a time limit of 2 hours and 10 minutes to complete 3 challenges. That seemed like an odd allotment of time, but whatever. I'm certainly not complaining about the time because I've seen far too many of these tests where you're given 15 minutes (or less!) to crank out what can sometimes be a tricky coding solution. So 2 hours and 10 minutes seemed more than reasonable.

My first task required me to write an SQL query. The task itself was extremely easy. But I was severely annoyed to see how long it took for each iteration of my query to run in their janky online portal.

Maybe that sounds unfair to you. Afterall, you can hardly expect online tools to run as fast as native clients. And to be fair, the first time I remember taking a Codility test - many years ago - their portal didn't seem like it was any slower than any other online coding environment that I'd encountered.

But this is 2023. Now I can flip over to something like CodeSandbox and watch my solutions compiling-and-running so got-dang fast - right in the browser - that I occasionally find it to be distracting. Yet Codility seems to be using the exact same slowwwwwww platform they were using a decade ago. That's friggin ridiculous, because those types of differences affect how people are performing on their tests.

Of course, you don't have to do all of your work directly in the Codility window. But I haven't had a need to run SQL on my local system for quite a while. So it's not as though I could simply open up my local query editor and spin up a comparable query.

Regardless, I finished the SQL exercise in about 8 minutes. Granted, this was about 6 minutes more than it would've taken me if I wasn't working inside their cheesy toolset. But I still had plenty of time to do the 2 remaining tasks and I wasn't too stressed.

I went on to the second task and finished it in about 20 minutes. It was a coding (JavaScript) challenge where I had to write a function (blah, blah, blah) that accepted (blah, blah, blah) and returned (blah, blah, blah). The wording of the instructions wasn't ideal, but neither was it horrible, and I pretty much knew straight away what I had to code. I had the second task finished in about 20 minutes, leaving me nearly 1.75 hours to finish the assessment.

Then I went on to the last task...

Shamefully obtuse wording

Here are the instructions from the last task:

There are M children, ordered from 0 to M-1, involved in a game.  
The N-th child is assigned a letter: L[N].  At the 
start the 0th child gives a note, consisting of one 
letter L[0], to the B[0]-th child.  When the N-th 
child gets the note, they add their letter L[N] 
to the note and pass it to B[N].  The game is over when 
the 0th child gets the note.  Find the final note.

Write a function:

function getNote(L, B);

that accepts String L and an array of integers B, both of 
equal length, and returns a string with the final note
given to the 0th child.

Examples:

1. Given L = "cdeo" and B = [3, 2, 0, 1], the function 
returns "code".  The 0th child gives a note "c" to the 
3rd child.  Next, the 3rd child gives the note "co" to 
the 1st child.  The 1st child gives the note "cod" to 
the 2nd child.  After adding the letter 'e' to it, 
the 2nd child gives it to the 0th child.  The final 
note, given to the 0th child, is "code".

2. Given L = "cdeenetpi" and B = [5, 2, 0, 1, 6, 4, 8, 3, 7], 
the function returns "centipede".

3. Given L = "bytdag" and B = [4, 3, 0, 1, 2, 5], the
function returns "bat".  NOTE: not all letters 
from L must be used.

M is an integer from [1... 1,000];
String L consists only of lowercase letters (a-z);
B is all integers within range [0... M-1];
L and B are both of length M.

First of all, the opening paragraph is an embarrassing, unnecessarily-obtuse word jumble. And to be frank, Codility does this all the dang time. I'm pretty sure that their tech writers are all frustrated, failed physics majors. I believe that they think they're being incredibly precise by using all this A[0]-th person, K-th, N-1 type of terminology. But what they're really doing is just creating a confusing mess.

I've been coding professionally now for 25 years and I've never had anyone give me specs for any task that were worded anything like this crap. This isn't testing your ability to write good code. Nor is it testing your ability to understand complex requirements. It's testing your ability to decipher what they believe is their clever riddle.

For me at least, I actually struggled to figure out from these instructions how I was to determine the "K-th" person. I read and re-read the instructions, and the examples, but it just didn't make any sense to me. (The fact that we're even talking about a "K-th person" should tell you that we're dealing with absolutely crappy specs.) Yes, I could look at the examples to see what I should get from those arrays, but it still wasn't clear from the instructions exactly how I should come up with that magical "K-th person".

The (inevitable) objections

After cranking out more than 100 articles on this site, I've learned that there will always be a certain population of tech curmudgeons who will snarkily dismiss almost anything I write. I could write that "Bugs are bad" and someone in the comments will chime in dismissively with something like "If you were a good developer, you wouldn't mind bugs." So here are some of the objections I imagine will bubble up from my commentary above:

I didn't have any problem understanding the instructions.

Good for you. You get a cookie. My point isn't that the instructions are utterly incomprehensible to everyone. I'm sure that some people will read those instructions and immediately know what's being asked of them. My point is that they are worded in a way that's unnecessarily obtuse.

I've worked through numerous Codility challenges before. They're almost always written in this exact same style. Sometimes, I read the instructions and I immediately "get it". It just... clicks in my head. And I can swiftly set about coding the solution.

The problem is that my ability to grok the instructions feels almost... random. Sometimes they're immediately clear to me. And other times I find myself reading (and re-reading) the instructions and thinking, "Umm... What???" If your instructions aren't quickly understandable for a vast majority of coders, then your instructions suck.

You're just butt-hurt because you failed the test.

Umm... no. I got 100% on the evaluation. I figured it out. But figuring out exactly what I was supposed to code was unnecessarily complex.

Sometimes requirements are complex, and coders must be able to handle complex requirements.

In a quarter-century of writing code I've cranked out some amazingly complex solutions. And sometimes, the requirements were, indeed, complex. Confusing, even. But there's a huge difference between complexity versus clarity. Just because something's complex doesn't mean that it can't be described in a way that's clear. If the requirements you hand me are extremely difficult to understand, that's not a "me problem". That's a "you problem".

Of course, in real life, if I'm handed requirements that I don't fully understand, I will absolutely go back to the person/group who formulated those requirements and ask (nay, demand) clarity. I'm not going to start cranking out a coding solution until I'm certain that I understand what's being asked of me. But when you're doing an automated online coding evaluation, you have no such ability to get clarity on those requirements.

Complex solutions may require complex instructions.

Again, just because the solution is complex doesn't mean that the instructions must be confusing. If you can't clearly describe the problem you're trying to solve, then you should rework your instructions until they are clear.

Besides, the task above is not a "complex" task. It only feels complex because the instructions were written in a confusing manner. The solution consisted of... SEVEN lines of code. And no, I'm not talking about seven lines that involve regular expressions and nested reducers and arcane JavaScript constructs. I'm talking about seven lines of dead-simple code. For such a simple "ask", there's no reason why the instructions need to be anywhere near that complex.

Codility has to provide challenges that have minimal ambiguity for a mass audience - hence the overly-precise mathematical language that they use.

I'm sorry, but this is just plain bunk. I do understand that they want to eliminate ambiguity in their instructions. But you can do that without resorting to phrases like the A[0]-th person. Removing ambiguity by increasing confusion defeats the whole purpose of evaluating someone's coding skills.

There's also another aspect we should consider here if we're trying to eliminate ambiguity: How many coders are not native English speakers?? If a US-born native English speaker like myself has problems parsing through those instructions, how do you think it feels if your native language is Hindi or Portuguese or Mandarin or any language other than English?

If you have any decent experience in dev shops, you know that non-native English speakers are extremely common. And you also know that many of those non-native English speakers are some of the best coders. So why would you peg your assessment of their skills to their ability to parse English instructions written in obtuse mathematical lingo?

Honestly? Codility isn't even close to the worst provider of online coding assessments. And if you've read any of my previous rants about coding tests, you know that I find nearly all of them to be a joke. But that doesn't mean that Codility isn't committing some serious effery here.

When you're giving someone a cursory coding test - the kind of test that simply establishes that, yeah, this person is a "real" coder - you shouldn't be relying on any verbiage that muddies up that central question. Codility could do much better. But I really don't think they care to.