How the State of JS/CSS Surveys Are Run

You can find an updated version of this post here

People sometimes ask me how they can help contribute to the State of JS and State of CSS surveys.

When this happens, I typically launch into a convoluted explanation of how all our moving parts fit together. Somewhere around the third mention of our GraphQL API, I can see people's eyes glaze over. Or, to use the contemporary equivalent, I notice those three little "someone's typing" dots haven't moved in a while. And more often than not, they never move again…

To avoid this scenario, I thought I'd write down once and for all an exhaustive list of everything that goes into running one of our surveys. Whether you're interested in contributing or maybe want to run your own, hopefully this list will do a better job than my off-the-cuff explanations usually do!

Here are the different areas of work, in a rough chronological order, from survey conception to releasing the results, along with a "help wanted" section that outline what needs we have for each step.

Survey Design

The first step with any survey is to come up with a list of questions. This means balancing several criteria such as:

• Continuity: asking the same questions as last year means we can analyze data over time.
• Fairness: we need to have a good rationale as to why we decide to include some libraries or projects and not others.
• Inclusivity: we need to make sure the language we use is inclusive and representative.
• Brevity: the survey itself shouldn't grow too long.
• Ability to Process: we need to make sure we can actually do something with the data we collect.
• Actionability: will the data collected from a question make a difference? Are people interested in it?

All of this needs to be done in an open, transparent, and inclusive manner, and we use GitHub issues to hold that discussion.

And while this step isn't really visible from the outside, I basically never stop contacting as many people as I can in between each survey to ask them if they'd be interested in providing their feedback to make the next one even better.

Help Wanted

• Review the questions for accessibility and inclusivity
• Review the questions for technical accuracy
• Suggest new questions

Translation

Once the survey content is finalized in English, it can now be translated into other languages by our awesome community of volunteer translators.

At this stage although we've finalized the contents of the survey itself, we haven't started writing the contents of the survey results. Still, in order to centralize translation we try to already anticipate what that will look like, and especially share as many strings as possible between the questions and results (as well as between different surveys).

Currently we don't really have a great process for managing translations. Ideally we would either give every language its own repo (with some GitHub actions set up to sync everything up properly) or else rely on a translation service such as Transifex.

Help Wanted

• Translate the survey in your language.
• Set up a separate API to serve translation data.
• Help set up a better translation workflow.

Survey App

With the questions translated, we can now launch the survey to collect data.

In the past we've relied on third-party survey apps to do this, but we now have our own data collection app, which lets us do cool things like translate the UI in many languages and process data on the fly.

This app is itself based on a full-stack JavaScript framework I created, Vulcan.js. It works well enough for now, but in the future the app might need to be refactored in order to handle more surveys in parallel.

At this point there is also a normalization stage, where we copy the data inputed by respondents into a more standardized format, taking care of things like parsing freeform fields at the same time.

This being a full-stack app, there's also quite a bit of front-end work involved in addition to the back-end data handling, and it's basically almost a full-time project in itself.

Help Wanted

• Improve survey app accessibility
• Help port app to Next.js

Outreach & Promotion

A survey is no use if nobody fills it out. At this stage, I'll reach out to as many people as I can to help distribute the survey.

This is a key stage because the channels we use here will influence the data we get in return. For example, this is where efforts to diversify our audience (whether in terms of gender, background, language, etc.) can come in.

Although time-consuming, writing blog posts or guest posts at this stage can be a good way of generating additional interest around a survey.

Help Wanted

• Help connect us to organizations such as Girls Who Code, Black Girls Code, etc.
• Share the survey in non-English-speaking channels.

Data Processing & GraphQL API

Once the survey is closed, we can now process the data. Our infrastructure at this point is still pretty ad-hoc. The survey app stores data in a Mongo database, and we have a separate Node app that uses Mongo aggregations to output that data in a usable format through our GraphQL API.

This is another fairly complex codebase, as we need to not only correctly process the data (which means both designing the right aggregations and calculations, and also implementing them correctly) but also make sure the API is designed in a way that makes sense and is easy to use for others.

Help Wanted

• Write new aggregations and calculations.
• Rethink the API to better handle multiple different surveys.

Survey Results

The next step is setting up the Gatsby app that power the survey results sites like https://2020.stateofjs.com/ or https://2020.stateofcss.com/.

This is no small feat either because these codebases have little in common with the typical Gatsby app. To begin with, each entire site is dynamically generated from a YAML file instead of relying on static Home.js pages.

We also handle internationalization, as well as generating permalinks and metadata for each individual chart. And all of this needs to be done in an accessible manner (which isn't quite the case currently, but is definitely high on the list of things to improve!).

Help Wanted

• Help create a dynamic query builder to add filtering to charts.
• Create translation widget to translate strings in-line.

Image Generation

As an aside, we even have a separate little app just for generating image previews so that every chart can have its own thumbnail when you share it.

Help Wanted

• Find way to integrate this step into build process of results app.

Data Visualization

While data visualizations are technically part of the survey result app, they're almost a whole step by themselves.

First, we need to decide what we want to visualize, and how. This usually takes the form of either notebook sketches or rough Figma mock-ups.

Then, we can think about the implementation. We use Nivo for most data visualizations, but we do have a few that either use straight HTML/CSS, or use D3 directly.

Help Wanted

• Create new data visualizations.
• Help fix readability/accessibility issues with current charts.

Content Writing

We'll also typically at least write an introduction and conclusion, but often have section headings, picks, notes, etc. all of which will also need to be translated.

This doesn't usually take that long, but it can still be hard to find time to fit it in given everything else that needs doing!

Help Wanted

• Write guest intros/opinions.

More Promotion

I don't do this every time, but this can also be a good time to keep interest alive with preview blog posts or even small side projects to tide people over until the full results come out.

Help Wanted

• Create content around the survey.

Sponsorships & Shirts

Although profit is not the main motivation behind the survey, I'm a big believer in trying to make any project as self-sustaining as possible, so I've been exploring different ways to monetize the surveys since the start.

The two that have worked best so far are sponsorships in the form of sponsored resource links, and selling t-shirts.

Sponsorships take time in the form of finding the right partners, emailing back and forth, and actually entering the content.

As for t-shirts, while designing them is one of my favorite part of the entire process, it's also very time consuming. And so is setting everything up properly with the printer, to say nothing of handling logistics (although we now use Cotton Bureau which manages everything for us).

Help Wanted

• Help connect us with new sponsoring partners.
• Wear our t-shirts!

Project Homepage

At this point I'll often realize that the project homepage is woefully out of date, and of course it'll take me an hour of cursing at package.json to get the codebase to run again.

In the future, I hope to integrate these homepages with our APIs directly so that their content doesn't need to be manually updated.

Help Wanted

• Design better homepages.
• Add GraphQL integration to fetch homepage content through our API.

Launch & Livestream

After this ridiculous amount of work, it's finally time to launch!

By now we have a sizable email list which ensures we won't launch to crickets, but I still try to encourage people to share the survey results as much as I can. I'm especially thrilled when I see that a podcast mentioned the results or that a YouTube channel reacted to them!

We'll try to have some kind of special event, usually a livestream to discuss the results with other community members.

Help Wanted

• Share the survey results.
• Talk/write/podcast about the survey results.

---

So there you go. Even though we aim to reuse as many pieces of our stack from survey to survey, in practice we always need to invest time into maintaining or improving our processes.

Still, the reception we get from the community every time a survey is released makes it worth it! And if you're interested in contributing, hopefully this list will show you that there's definitely no shortage of tasks you can potentially help with. Just come by our Discord and say hello, or email hello at stateofjs.com!