Originally posted on the Triplebyte blog.
My co-worker Daniel is really into Emacs. It's his primary editor, his grocery list (via org-mode), and #4 of the 6 things he can't live without on his OkCupid profile. Other engineers in the office, however, prefer Vim (or Sublime Text, or even RubyMine). Naturally, this leads to war. But after months of salvos (VimGolf one-upmanship, gratuitous references to this Stack Overflow question) no clear victor has emerged.
So last week, I set out to see what the Triplebyte data has to say on the issue. Take this with a grain of salt (correlation does not imply causation), but there is a correlation between what editor an engineer uses, their language and OS, and their ability to pass programming interviews. And, according to our data, Vim users pass interviews at a rate ever-so-slightly higher than Emacs users! [1]
About the Data
At Triplebyte we evaluate hundreds of engineers every week. Each interview involves talking to one of our interviewers (like me!) over Google Hangouts. We record what language and editor each engineer uses. After our interview, we introduce the engineers to other companies (they skip the phone screens).
To gather the data I wanted, I pulled the editor / language data for every Triplebyte interview over the last year, as well as outcomes (pass/fail) for both the Triplebyte interviews and the interviews with companies the same engineers did through our platform. Here is the breakdown of editor use:
Editors and IDEs
It's not just an Emacs and Vi(m) world anymore (was it ever?):
The #1 most common editor we see used in interviews is Sublime Text, with Vim close behind.
Emacs represents a fairly small market share today at just about a quarter the userbase of Vim in our interviews. This nicely matches the 4:1 ratio of Google Search Trends for the two editors.
However, I'm really looking for signals of quality, not quantity. Which engineers are most likely to successfully pass our interview?
Vim takes the prize here, but PyCharm and Emacs are close behind. We've found that users of these editors tend to pass our interview at an above-average rate.
On the other end of the spectrum is Eclipse: it appears that someone using either Vim or Emacs is more than twice as likely to pass our technical interview as an Eclipse user.
I was curious to see what happened after our interview, so I looked at how likely each group was to pass their final onsite interviews and get a job offer at the companies we connected them with:
For the most part, this looks fairly even across editors, which tells us that even if Vim users are stronger as a population compared to Sublime users when it comes to Triplebyte's interview, we're still identifying the correct subset of Sublime users that will do well at the companies we work with.
I was surprised to find one huge outlier here: Eclipse! Oddly enough, it appears that even after we pre-screen Eclipse users for their programming skills, they still get offers from only half as many of the onsite interviews we send them to compared with the rest of our population. I don't have a good explanation for this correlation, and it's something I'll be looking into. (In any case, don't worry, we still happily accept and find jobs for many great Eclipse users too!)
Programming Languages
You can use whatever language you like in the first section of our technical interview. Today, a plurality of engineers are choosing to interview in Python:
Hey, what happened to PHP? As a former PHP developer myself, I found it interesting that it's now just a small drop in the “other” bucket. As one of the interviewers here, I've personally never seen anyone interview in PHP yet (although others have).
Python is also known for being beginner-friendly, so will an Eclipse-like trend play out here? Well, not quite:
In this case, we find that the average Ruby, Swift, and C# users tend to be stronger, with Python and Javascript in the middle of the pack.
As we dig into these numbers, it's worth noting that there's an unavoidable strong pairing between certain editors and certain languages: 90% of Eclipse or IntelliJ users also used Java, and 99% of PyCharm users used Python. One fascinating and statistically significant result I found is that a Java developer who used anything other than Eclipse was 47% more likely to pass our interview compared to those who did use Eclipse.
Here's what happens after we select engineers to work with and send them to onsites:
There are no wild outliers here, but let's look at the C++ segment. While C++ programmers have the most challenging time passing Triplebyte's technical interview on average, the ones we choose to work with tend to have a relatively easier time getting offers at each onsite.
I'm not entirely sure how to explain this, but one hypothesis might just be to look at the overall supply and demand on each side of our platform. We see strong demand for talented systems and low-level programmers from the companies we work with. On the other side, we're constrained by a relatively small fraction of people applying using C++, multiplied by a relatively smaller fraction of those who we find talented enough to work with. Another thought is that perhaps more niche specializations might give more signal to our matching process. In any case, this tells me we should be looking for more talented C++ developers!
Operating Systems
If you've frequented a coffee shop in San Francisco in the past few years, it should be no surprise that the majority of engineers are interviewing on macOS today:
In this case, the most popular choice is also the most successful:
We found that Windows users struggle a bit more on average. In this category, I was surprised not to see Linux in the lead. As a former Linux desktop user myself, I'd have guessed that Linux users might have a wide range of technical knowledge related to their daily computing.
The trend continues in onsite interviews:
Curiously, the Apple fans we work with also seem to pass about 20% more of their onsite interviews than Windows or Linux users. (This reminded me of an old OkCupid blog post comparing data from their Apple and Android users.)
It's hard to know why exactly this happens, but there may be a hidden variable here. For example, while 60% of the people we interviewed used macOS, if you walk into any tech company in San Francisco you're inevitably going to count far more than 60% MacBooks. While our interviews are background-blind, having a few years of professional programming experience usually helps!
Conclusion
As an interviewer and engineer here, I thought it'd be cool to see what kind of data we had on the tools engineers are using today.
In our 2017 dataset, we found that an engineer writing Ruby in Vim on macOS has a better-than-average chance of passing our technical interview, while one writing C++ in Eclipse on Windows is on the other end of the scale. We don't explicitly use these indicators to decide who we work with, but this is a side product of data collected in our interview process.
At a higher level, we believe that the hiring process should be focused on discovering strengths, not uncovering weaknesses, and we've written extensively on How to Interview Engineers. Our best suggestion for companies is to allow candidates to use the tools that let them best show their personal strengths. In our case, this means we allow engineers to choose any language and environment of their choice, and do convenient interviews from home using screen sharing.
If you're an engineer and want to give our process a try, the first step is completing our multiple-choice programming quiz here.
If you're a company looking for engineers and would like to learn more about working with Triplebyte, you can get started here.
[1] To be clear, I don't think this has any predictive value for hiring, and I don't think engineers should change their editor based on our analysis. Emacs (and Eclipse) are great editors! ↩
Top comments (11)
Why is Visual Studio Code, The most popular editor according to the 2018 Stack Overflow Developer Survey Not listed?
This whole thing does seem a bit disconnected, but that may be because of the type of jobs and the technologies their recruits were applying to work with.
I have no idea what Triplebyte is or does, but if they do less conventional stuff or search for different kinds of developers compared to whoever answers SO surveys, would explain why these statistics look so different to what StackOverflow seems to find in their annual assessments.
Our data comes from software engineers applying to work at a few hundred top tech companies. See triplebyte.com/blog/we-ve-raised-a... for more background -- we've helped software engineers get more than $50 million in job offers in just the past 6 months.
Instead of relying on self-reporting from a low-response-rate survey, we record what editor and programming language all engineers actually use during a two-hour technical interview with us. (You can use any programming language you like!) We don't actually use these signals to decide who we work with. Instead, we have a structured rubric and are looking for things like coding productivity, coding professionalism, and debugging skills. I've personally interviewed a lot of engineers applying to Triplebyte, and ultimately we're looking to predict whether this person has what it takes to perform well as a software engineer at one of the companies we work with. And more importantly, we use structured data from the interview to match you to companies, so we can say, "Jane is a really speedy and natural Python developer who also speaks very well about database design," from which we can match you to companies we know are looking for those strengths.
You can see triplebyte.com/interview_guide for more information about the interview in which we collect this data.
It's there, shown as Visual Studio!
Are Visual Studio and VS Code lumped together? Or is the label just incorrect?
Lumped together!
If the interviewer knows the candidate uses a given editor/OS/language when making the determination, I'd interpret these statistics as suggesting that your interviewers may be using these as proxies for the competence of the candidates. That might be a hard bias to correct for without hiring Java developers to interview Java developers, Windows users to interview Windows users, etc.
(In the following, I use "external" to mean "interviews conducted by an agency that performs screening interviews on behalf of employers", and "internal" to mean "interviews conducted by the employers themselves, 'internally'". I think other terms I use should be pretty obvious, or at least I hope so, in context.)
Assuming no evaluation bias (a big assumption; we all have biases), I have a few wildly speculative ideas to offer for things to consider when trying to impose some kind of interpretation on these numbers:
Sublime is probably popular because it's free-or-cheap, and because it was the early free-or-cheap entrant in its specific editor-style market (including TextMate, Atom, and VS Code). Even on MacOS (TextMate territory) it's unsurprising, because many devs on MacOS are migrants from Linux-land.
Vim is probably popular in the interviews because it likely dominates among Linux users, and some migrants to MacOS carried that with them.
Atom users are probably mostly ex-Sublime users, and many of them seem to have trouble with it due to negative transference of knowledge (call it an "uncanny valley" effect, if you like) given the similarities with Sublime coupled with some differences that subtly trip up people used to Sublime.
Vim users are clearly well-suited to picking up skills involving systematic linguistic traits, given the fact the vi editing paradigm is basically a text processing language with a relatively simple grammar.
C++ is probably a very difficult language to use to perform the same kinds of tasks suitable to quick/easy completion in an interview environment. For exactly that same reason, if I was interviewing for a C programming job (because I'd rather have a C job than a C++ job), and was given the choice of any language at all to use in a skills evaluation for the job, I'd use Ruby. Once people who prefer C++ in the external skills interview actually manage to pass it, though, you've probably selected for the very best of them, so those candidates are probably much more likely to have a good chance of getting the job.
Ruby is a very productive language, and easily solves many problems more difficult in other languages, and it's thus possible that a lot of candidates do well in an initial skills test screen that are then judged not suitable to the job. A very smart bootcamper could lack the specific qualities to get past the internal interviews, for instance, even if that person excels in the external interview due to a solid grasp of the basic language and simple skills-test problem solving, while using a language that lends itself well to fast completion of tasks.
Emacs seems to be rapidly losing popularity for software devs overall, and I suspect that leaves those most "niche" in personal manner, preferences, and so on, which might put off interviewers at the target employer. They are also probably more GNU-favoring overall, given the pedigree of, and community around, Emacs.
Linux users are likely to be either fairly technically competent in a wide range of skill areas, or fairly overconfident in such skills, so you likely get a balance of the two resulting in a close-to-average overall success level for the group as a whole in the technical screen. Once they get there, those that are most skilled may also be the most "niche" in preferences, skill distribution, and so on, thus making them appear less desirable at the internal interview. This bodes well for me in terms of competence, or ill in terms of overconfidence, for a technical screening, but doubly ill for me at the internal interview stage; I actually prefer OpenBSD for my daily driver, and I have many eclectic technical skills (I like to explore interesting, quirky skill areas at times), most of which are more likely to make me less enticing to employers if they have any effect on the evaluation at all, thanks to the common human fear of weirdness.
I could be full of it. This is all just wild speculation, after all.
So your process is biased towards Vim and Sublime?
Correlation != bias :)
Also, Sublime was quite popular with engineers applying to Triplebyte, but Sublime users passed our interviews at a below-average rate. See "Triplebyte Technical Interview Pass Rate by Editor."
I think developers tend to use macOS not only by the OS itself (and Unix-like) but the hardware quality