Testing Developers is Hard!

One of my jobs is to hire new developers. It's something I really enjoy, as I love talking to other devs. But, unfortunately, this isn't my full time job, so I've got a finite amount of time available to evaluate and really pay attention to what our candidates are capable of.

Unfortunately, a lot of the applications we get are people who simply aren't developers. I get it. Being a developer is awesome. I can understand why they want to become one. But we get a lot of resumes for folks who really couldn't write a single line of code.

So, I had a great idea for a simple way to filter those out so I could dedicate more time to people who've worked really hard to build up their skills and get to the level we're looking for... (or so I thought 🤨). I decided to build a sort of CAPTCHA for developers!

I built out the ad site, made it look good, made sure to make the advertisement itself as specific and accurate as possible without being a pain to read... then set out to build my test.

The idea was pretty simple, I wanted to include a very basic programming test. Something that wouldn't tell me if someone was a senior developer, but would at least discount anyone who really just couldn't create even a basic function.

I found the VS Code Monaco which made it ridiculously easy to embed an editor that doesn't suck for folks to use. I built some javascript around it and described FizzBuzz.

I sent that out for some feedback and got a few things immediately. First off, FizzBuzz isn't really easy to write from scratch (unless you happen to know it). The simplest way to solve it requires using the modulo (or modulus) (%) operator, which returns the remainder after division. e.g. (5 % 3) == 2... Simple... but not common. It's rare I run into a real need for it.

So, I asked for some suggestions. I really wanted something that any reasonably experienced developer could solve in a couple minutes.

One of the suggestions made sense. It was based on a real world problem that comes up pretty often. The idea was to have a master array and a filtered array based on that. Then to remove the nth from the last object in the filtered array from the master array.... (nth means an arbitrary number, so if passed 3, then the third from the last) as I was writing out the instructions, it was clear this was more than a little convoluted. 🤪

I simplified it and made it a lot easier... just remove the nth from the last object in an array. That felt a lot easier to describe. In the editor, I supplied a function that accepts the array and the number from the last as arguments so there was a starting point.

And it was. I wrote out the instructions, built the test, shipped it for feedback again.

Everyone failed! 😲

It wasn't "difficult" and every person was fully capable of passing it, but it wasn't clear what went wrong if things didn't go perfectly the first time around. There was no feedback, it simply said it didn't work. That made it really hard to troubleshoot and also left it ambiguous as to whether or not that was the end. Did they only get one shot?

So I went back to the drawing board and iterated a bit more. I added some feedback to explain what their function did vs what I expected. I also added some extra text to make it absolutely clear it was ok to try more than once.

After all, I don't expect people to get things right the first time or to not need to play around with it a bit. Arrays can be tricky. Sometimes getting the exact position for removing something from an array can be a little fiddly!

This time the feedback was much better. The test was simple enough everyone could figure it out and the descriptions were clear enough they felt comfortable doing so.

It just really drilled home how difficult it is to make tests inclusive and open, but also effective. I wanted something that wasn't a real hassle for developers, but wasn't so simple that a non-developer could figure it out or copy/paste an answer from Google. (Mind you, I've no problem with getting an answer from Google, as long as you know what it's doing!) That proved to be a really difficult balance. I'm absolutely certain I've still not got it 100%, but what I came up with, I believe is a pretty reasonable compromise.

Time will tell! But hopefully it's a little quirky, a little fun, not too annoying and helps give me a little more time to learn about our candidates!