The first time I was involved in a technical interview I was asked to present and grade a coding question: "What does this script do?" The candidate was then handed a print out of a (relatively) simple VBScript that changed the computer name of a Windows PC.

Only one candidate answered correctly. Every other candidate failed the question with an "I don't know."

On first glance, this seems a reasonable question. Part of the job they were interviewing for often required working with scripts to deploy computers and software. I told them that we weren't looking for a specific answer, just a general idea of "What does this do?" The candidate failing to even make a guess seems like an interview fail. They couldn't think on their feet. They couldn't think creatively.

But was it really a failure of the interviewee?

As a test, we handed out that same question to a few of non-technical folks in the office. All of them answered correctly within a minute of skimming it by looking at method and variable names and guessing.

It's been a long time and many interviews since those first ones and I now look at their failure to respond as a failure on my part as the interviewer. If a non-technical person could answer the question correctly, most of these candidates should have gotten it right, too. So why didn't they? Why did they only answer with a "I don't know?"

Fear

Job interviews are scary. Unless they don't need the job, there's real risk involved in failing an interview. Paying bills, making rent, supporting a family, growing their career in a direction that really excites them. The more they want or need the job, the more that fear is present with them in an interview.

An important part of an interviewer's job is to help them relax. You're not going to learn the full range of their skills if they're too nervous to answer your questions. The easiest way to help a candidate relax is to help remove their fear of failing. You can do this surprisingly simply: Ask them more questions that they're going to fail.

It seems backwards, right? But the goal here isn't to catch them in a trap, but to show them that answering a question incorrectly is safe to do. Dig into their wrong answers, see how they got there. Ask similar questions to see if they know related information. If they look nervous, tell them it's ok to not know the answer.

Some people think interviews should test for "handling stressful situations". Unless your business is in high stress situations (think: disaster relief operations), then this is bullshit. It's your job as an employer to provide a safe place to get work done.

Part of what you have to do to measure a candidates full skill is prove to them that you're not going to judge them arbitrarily. Prove to them that one wrong answer will not keep them from getting a chance at a job.

So what should I have done differently? I was on the right track, reassuring them that guesses were fine, but I didn't go far enough in helping them relax or preparing them to answer. And part of that was due to the specificity of the question itself.

Digging into a skill

When you have a question with a specific answer you create a very obvious point where the candidate can fail. They know getting the job relies strongly on the correct answer, and that fear of failure will be eating up most of their thoughts.

To help mitigate this, you ask lots of questions. Think about it like this: Would you prefer one chance, 0% or 100%, on passing a test, or 10 chances, where it's possible to get a 70% passing grade?

In the example of the computer rename script, the skill I was looking for was "The ability to work with scripts". Preferably creating and editing, but just being able to read them to see if it "did what you want" would have been fine.

A better question: "Have you ever used a script to automate a task before?"

"But wait!" you say. "That answer isn't measurable! How do I know if they got it right?!" Answer: You ask more questions.

For example: If the candidate said yes, ask them about the time they used it. Ask them about what it did, how many computers it ran on. Ask them where they found the script, and how they knew what it did.

Even if they said no, they still haven't necessarily failed. Ask them if they think they could "figure it out." (I've yet to interview a technical job candidate who wasn't confident they could figure something out that they didn't know given time and google). Now do the handout. The handout in our interview was an actual script we used daily. It was an accurate representation of the job. Tell them that. It gives them incentive to really try figuring it out since it's the job they want to do. For an even more accurate job representation, hand them a laptop with Google ready. Letting someone know that you won't arbitrarily limit their tools to get a job done does wonders for relaxing them.

Measuring a skill

After reading the above, you're probably thinking "But that question sounds too easy with all that help". You're right! And isn't that great? Think about how many candidates could show that they could really do the job. We just went from one, to (likely) all of them.

On top of that, if you went down the first question path, you know how deep their scripting skills go. You can measure it relative to your own skill. Are they better at scripting than you? Worse? Maybe they've written lots of scripts, but in different areas than yours. Write that down. Those are all important impressions to use when comparing candidates against each other.

Before you begin an interview, you should have a list of specific skills you're looking for and a list of open ended questions that can help candidates tell you about times they've used those skills.

Once you have that list, talk to them until you can answer these questions:

• Do they know more than you?
• Do they know less? How long would it take you to train them to a level of competence needed to do the job?
• Do they have other, similar skills to make up for not knowing this specific skill? How deep does that skill go?
• How excited are they about the skill. Is it an interest? Something they hate but do anyway?

A helpful "trick" for getting to the depth of a candidates skill is to tell them exactly what you're looking for! They'll be eager to show you that they can do it, or to show you skills similar to what you're looking for.

Keep in mind the real question you're trying to answer: Do they have the skill and desire necessary to do the job? If you can't confidently say 'yes' or 'no', keep digging, keep asking questions. I really can't stress this enough. This is important. You're in a position of authority - in a position to approve or deny them livelihood. If you reject someone for a position, you should be prepared to explain the reasons why for your choice.

Note: I've found it common that interviewers don't feel confident enough in their own abilities to judge someone else. Two things to keep in mind:

1. You're measuring their skill relative to yours. You don't need to know everything about a subject to do that.
2. Someone, likely your manager, thought you were qualified because you had the skills they're looking for. You can do this.

If you really aren't qualified, though, make sure someone else in the room is. (For example: back when I gave that first interview, you wouldn't have wanted me to grade someone's diplomatic skills.)

Comparing candidates

Once you have the relative skill of candidates, it's pretty straightforward to compare them. Once again, though, you should be able to confidently describe why one candidate is better than another. Use specific terms. In a perfect world, you would be able to coach a failed candidate explaining the things they should work on.

Also, you should have multiple people performing these skill measurements, and the people doing those measurements should have the skill level you're looking for. (There's no real point in having your worst communicator interviewing someone about their email skills.)

One other thing to keep in mind: Sometimes a candidate comes along with a great set of skills that doesn't neatly fit into any one role. If you can make it work in your organization, these people make great employees. The diversity of those skills can lead to innovative solutions to difficult problems. When you can: fit roles to people, not people to roles.

Iterating your process

A lot of my opinions on interviews formed after my boss handed me a stack of developer resumes and told me I had 30 minutes to screen each of them for the necessary tech skills. Three or four interviews a day, generally given back to back, for a couple weeks.

During that time, I iterated. I wrote down which questions worked well to draw out a candidates skills and which ones didn't. Most of the time I could confidently tell within those 30 minutes whether they had what we wanted or not. And, if I wasn't confident, I passed them on anyway. I didn't want my gut 'feeling' to be the deciding factor on someone's employment.

But I was mostly surprised at how effective this method was. At the end of every interview, I could accurately communicate their skills relative to mine to my manager so he knew which ones to interview next, and I could repeat the results of the process with the next candidate.

Iteration is always important. Something I've learned relatively recently is to keep metrics at each 'stage' of an interview process. Every stage acts like a 'filter'. They're an arbitrary line in the sand that candidates have to make it across. "The top 10 candidates". It's important to know where people fall out and if great candidates are falling through. If you see a problem, fix it and try again.

Some additional reading on the subject: We’re Bad at Interviewing Developers (and How to Fix It). (I swear I didn't intentionally pick the same title. Great minds think alike?)

Conclusion - Interviews vs. Tests

Finally, I'm not Google or Facebook. When you start hiring on that scale, you need more specific metrics to be able to assure relative consistency of skills. Those types of questions have become the 'standardized testing' of jobs.

If you're not hiring on that scale, then you likely just want more developers like the ones you have. Let them interview and make good decisions from that.

And if you do need metrics, test, but don't grade based on the results of those tests. Try to avoid letting your measuring instrument influence the results of what you're trying to measure.