(You can find an updated version of this post here)
(Note: this is an updated version of the original post from 2021)
People sometimes ask me how they can help contribute to the State of JS and State of CSS surveys.
To make our convoluted infrastructure easier to understand, I thought I'd write down once and for all an exhaustive list of everything that goes into running one of our surveys. Whether you're interested in contributing or maybe want to run your own, hopefully this list will do a better job than my off-the-cuff explanations usually do!
Here are the different areas of work, in a rough chronological order, from survey conception to releasing the results, along with a "help wanted" section that outline what needs we have for each step.
Survey Design
The first step with any survey is to come up with a list of questions. This means balancing several criteria such as:
- Continuity: asking the same questions as last year means we can analyze data over time.
- Fairness: we need to have a good rationale as to why we decide to include some libraries or projects and not others.
- Inclusivity: we need to make sure the language we use is inclusive and representative.
- Brevity: the survey itself shouldn't grow too long.
- Ability to Process: we need to make sure we can actually do something with the data we collect.
- Actionability: will the data collected from a question make a difference? Are people interested in it?
All of this needs to be done in an open, transparent, and inclusive manner, and we use our Working Group newsletter to keep everybody informed, and then GitHub issues to hold discussions.
And while this step isn't really visible from the outside, I basically never stop contacting as many people as I can in between each survey to ask them if they'd be interested in providing their feedback to make the next one even better.
Help Wanted
- Review the questions for accessibility and inclusivity
- Review the questions for technical accuracy
- Suggest new questions
Survey Config
The result of this survey design process is the creation of YAML config files, which we store in our main config repo, alongside with any survey-specific elements such as the GraphQL queries used to generate the survey's data, the data itself, and any survey-specific images.
Translation
Once the survey content is finalized in English, it can now be translated into other languages by our awesome community of volunteer translators.
At this stage although we've finalized the contents of the survey itself, we haven't started writing the contents of the survey results. Still, in order to centralize translation we try to already anticipate what that will look like, and especially share as many strings as possible between the questions and results (as well as between different surveys).
Help Wanted
- Translate the survey in your language.
- Set up a separate API to serve translation data.
- Help set up a better translation workflow.
SurveyForm App
With the questions translated, we can now launch the survey to collect data.
In the past we've relied on third-party survey apps to do this, but we now have our own data collection app, which lets us do cool things like translate the UI in many languages and process data on the fly.
Note that this app, like most of our other code, lives in a central pnpm-powered monorepo.
This app is itself is a Next.js app that relies on Vulcan.js, a full-stack JavaScript framework that I originally created for Meteor, and that Eric Burel later ported to Next.js. But we will likely phase out the Vulcan.js part eventually to make it a more standard codebase.
Help Wanted
- Improve survey app accessibility
SurveyAdmin App
At this point there is also a normalization stage, where we copy any freeform data inputted by respondents into a more standardized format based on our own list of "entities" (frameworks, features, people, etc.).
This is done through a separate Next.js app that I basically only run locally on my own machine. Since that app has full access to our database, keeping it offline seemed safer from a data security point of view.
Help Wanted
- Improve normalization logic
Outreach & Promotion
A survey is no use if nobody fills it out. At this stage, I'll reach out to as many people as I can to help distribute the survey.
This is a key stage because the channels we use here will influence the data we get in return. For example, this is where efforts to diversify our audience (whether in terms of gender, background, language, etc.) can come in.
Although time-consuming, writing blog posts or guest posts at this stage can be a good way of generating additional interest around a survey.
Help Wanted
- Help connect us to organizations such as Girls Who Code, Black Girls Code, etc.
- Share the survey in non-English-speaking channels.
Data API
Once the survey is closed, we can now process the data. Our infrastructure at this point is still pretty ad-hoc. The survey app stores data in a Mongo database, and we have a separate Node app that uses Mongo aggregations to output that data in a usable format through our GraphQL API.
This is another fairly complex codebase, as we need to not only correctly process the data (which means both designing the right aggregations and calculations, and also implementing them correctly) but also make sure the API is designed in a way that makes sense and is easy to use for others.
Help Wanted
- Write new aggregations and calculations.
- Rethink the API to better handle multiple different surveys.
Internal API
Note that we also have a separate Node API that mainly handles providing internationalization data.
We split out this part of the logic so that even if the data API somehow got overloaded or ran into any other issue, the separate internal API would still be able to provide translated content to the surveys.
As a bonus the internal API codebase is also much simpler and easier to maintain compared to the "main" data API.
Survey Results
The next step is setting up the Gatsby app that power the survey results sites like https://2021.stateofjs.com/ or https://2021.stateofcss.com/.
This is no small feat either because these codebases have little in common with the typical Gatsby app. To begin with, each entire site is dynamically generated from a YAML file instead of relying on static Home.js pages.
We also handle internationalization, as well as generating permalinks and metadata for each individual chart. And all of this needs to be done in an accessible manner (which isn't quite the case currently, but is definitely high on the list of things to improve!).
Help Wanted
- Help create a dynamic query builder to add filtering to charts.
- Create translation widget to translate strings in-line.
Image Generation
As an aside, we even have a separate little app just for generating image previews so that every chart can have its own thumbnail when you share it.
Help Wanted
- Find way to integrate this step into build process of results app.
Data Visualization
While data visualizations are technically part of the survey result app, they're almost a whole step by themselves.
First, we need to decide what we want to visualize, and how. This usually takes the form of either notebook sketches or rough Figma mock-ups.
Then, we can think about the implementation. We use Nivo for most data visualizations, but we do have a few that either use straight HTML/CSS, or use D3 directly.
Help Wanted
- Create new data visualizations.
- Help fix readability/accessibility issues with current charts.
Content Writing
We'll also typically at least write an introduction and conclusion, but often have section headings, picks, notes, etc. all of which will also need to be translated.
This doesn't usually take that long, but it can still be hard to find time to fit it in given everything else that needs doing!
Help Wanted
- Write guest intros/opinions.
More Promotion
I don't do this every time, but this can also be a good time to keep interest alive with preview blog posts or even small side projects to tide people over until the full results come out.
Help Wanted
- Create content around the survey.
Sponsorships & Shirts
Although profit is not the main motivation behind the survey, I'm a big believer in trying to make any project as self-sustaining as possible, so I've been exploring different ways to monetize the surveys since the start.
The two that have worked best so far are sponsorships in the form of sponsored resource links, and selling t-shirts.
Sponsorships take time in the form of finding the right partners, emailing back and forth, and actually entering the content.
As for t-shirts, while designing them is one of my favorite part of the entire process, it's also very time consuming. And so is setting everything up properly with the printer, to say nothing of handling logistics (although we now use Cotton Bureau which manages everything for us).
Help Wanted
- Help connect us with new sponsoring partners.
- Wear our t-shirts!
Project Homepage
Each survey homepage (stateofjs.com, stateofcss.com, etc.) is actually generated from the same codebase using Astro, based on our global survey configs.
Help Wanted
- Design better homepages.
Launch & Livestream
After this ridiculous amount of work, it's finally time to launch!
By now we have a sizable email list which ensures we won't launch to crickets, but I still try to encourage people to share the survey results as much as I can. I'm especially thrilled when I see that a podcast mentioned the results or that a YouTube channel reacted to them!
We'll try to have some kind of special event, usually a livestream to discuss the results with other community members.
Help Wanted
- Share the survey results.
- Talk/write/podcast about the survey results.
So there you go. Even though we aim to reuse as many pieces of our stack from survey to survey, in practice we always need to invest time into maintaining or improving our processes.
Still, the reception we get from the community every time a survey is released makes it worth it! And if you're interested in contributing, hopefully this list will show you that there's definitely no shortage of tasks you can potentially help with. Just come by our Discord and say hello, or email hello at stateofjs.com!
Top comments (0)