"Hype" & "Garbage"? An Honest Look at the State of JavaScript Survey's Track Record

"#StateOfJS has been plagued with sampling bias for years"

"The State of JS is a measurement of hype metrics [...] and I hope no one takes it seriously."

"Any sort of take/decision, derived from something like State of JS, is garbage".

These are some of the statements about the State of JavaScript survey (live now, go take it!) that I came across on Twitter over the last couple days.

Now I got thick skin so I can take the criticism (even if I sometimes wish it'd be worded a bit more kindly). And hey, on some level maybe these people do have a point…

Sampling Bias

If for some reason all 5,000 remaining Backbone.js developers on the planet all decided to take the survey the same year and pick Backbone as their number one favoritest framework ever, then that year's results would look very strange for sure. You'd see headlines about Backbone's resurgence and recruiters would start asking for five years of Backbone experience.

This is sampling bias in action: by changing who you survey, you can completely change the resulting data even if the questions remain the same.

And because the State of JavaScript is an open survey, there's no practical mean of avoiding this problem altogether even if we do try to mitigate it by (for example) tracking where respondents came from to try and identify any weird patterns.

But just because sampling bias is a concern doesn't necessarily mean the survey's data is necessarily bad, just that the risk exists. So maybe one way to see if the data can be trusted is to just… look at it?

So let's go back in time, and for each category we'll see which libraries you would've picked if you had trusted the survey's results all the way back in 2016… and whether you'd regret it today!

JavaScript Flavors

The survey ran for the first time ever in 2016. Let's start by looking at the "JavaScript Flavors" section. Excluding ES6, the two flavors with the highest satisfaction ratio (meaning the ratio of developers who have used a library and would happily use it again) are TypeScript (83%) and Elm (84%):

Over the next couple years, TypeScript would not only go up to 93% satisfaction, but as all know basically take over the modern JavaScript ecosystem:

Elm on the other hand has steadily declined down to 63%. Does that mean the initial 2016 result was driven by hype that didn't materialize, sampling bias, or just that the rest of the ecosystem has now caught up to Elm? It's hard to say, but let's score this as 1 win and 1 miss.

Front End Frameworks

Here's what the 2016 front end landscape looked like:

Let's take a look at the top two again: React is at 92%, and Vue at 89%. Now compare this with four years later in 2020:

Although they went down slightly to 88% and 85%, the fact remains that an overwhelming majority of those who chose React or Vue as their front end framework of choice back in 2016 are still happy with it.

(And yes I'm aware that the survey doesn't tell us if those are the same people – it's technically possible that the 2016 people all ended up hating React and Vue and that the subsequent years' high numbers all come from new respondents; but that just doesn't seem very likely).

Based on this, I would give the survey a solid 2/2.

Data Layer

Back in 2016 the Data Layer category was split into State Management:

And API Layer:

While GraphQL has maintained a pretty positive ratio and seems here to stay, Redux on the other hand has seen somewhat of a fall from grace:

(And yes I'm aware GraphQL and Redux don't do the same thing at all – sometimes you just have to group things together the best you can.)

You could blame the survey for leading you to embrace Redux, but then again so was everybody else as the time. In any case, let's give this section 1 win/1 loss.

Back End Frameworks

The 2016 edition of the survey had a "full-stack frameworks" section for some reason:

The only entrant with a high satisfaction ratio is the "MERN stack" which stands for MongoDB, Express, React, and Node.js. In other words a standard Node.js React app backed by MongoDB (remember when liking MongoDB was cool?).

If we look at later years, we can see that Express always remains high, while Koa goes down a bit.

If we pick Express and Koa as our two recommendations, we again have a 50% win record.

Build Tools

Finally, let's look at Build Tools:

Only one build tool really stood out in 2016, and that was Webpack. And we all know that Webpack then went away and was never heard of again… not!

Let's give this one full marks!

Testing

The testing stars of 2016 were Mocha and Enzyme:

Four years later, Mocha got diluted way down and Enzyme got digested out of the rankings altogether (although it was at 71% in 2019):

It turns out underdog Jest went from a miserable 59% to a sky-high 94% in one year! You got to be kidding!

But enough testing puns. This is clearly a miss, 0/2.

Final Score

The final tally of this completely arbitrary scoring system is is 6/11, or 54%.

In other words, if you had based all your technology choices on the State of JavaScript survey back in 2016, you'd probably be happy with about half your stack right now.

Is that so strange though? With how fast the ecosystem is moving, you could do a lot worse than picking React or Vue all the way back in 2016.

What's more, 2016 was only the first year of the survey. Now that we have historical data you can look at trends over time to see if a high mark is a one-time fluke or a solid trend.

In Conclusion

So yes the survey suffers from sampling bias, and yes it hasn't been able to anticipate trends perfectly.

So maybe don't base your technological decisions only on its results and nothing else. But as one of many factors including other metrics (GitHub stars, NPM downloads, etc.), blog posts, and of course your own experience, I do think it has value.

So if it's garbage, let it be the good kind of garbage. Like a cool retro lamp you find in the trash that still works great, or a rusty cast-iron pot that cleans up to become a beautiful heirloom piece. We could all use more of these!

If you've read all of this and want to take the survey yourself go do it now and make your voice heard!