There's been some Twitter back-and-forth about the way we've been running the State of CSS survey, and since this raised valid concerns I thought I'd address them here.
The criticism in a nutshell: our survey is biased against the women of the CSS community.
Exhibit A: Survey Demographics
The first argument in that direction is that our respondent demographics are pretty biased: in 2019, 85% of our respondents were male.
Now I'm not sure what data source we would use to determine the "real" demographic breakdown of the CSS ecosystem, as the Stack Overflow survey for example has a 91% male population – but then again it doesn't focus specifically on CSS.
This is why I tend to push back when people make the jump from seeing a skewed result set to automatically assuming there are problems with the survey's methodology. This doesn't mean the survey doesn't have issues, just that a skewed chart by itself without more context is not evidence.
So what are those issues? I believe they lie in the way we distribute the survey.
Our main channels are social networks (Hacker News, Reddit, Twitter, etc.), which carry their own preexisting bias, and our email list of respondents from past surveys – which propagates any existing bias from one year to the next.
The consequence of all this is that our distribution methods are indeed biased towards a male audience.
Exhibit B: Survey Resources
The other thing that was pointed out is that our Resources section includes no women, which is also true.
But if you take a closer look, you'll see that it's pretty clear we've tried to stay away from individual blogs or sites altogether, precisely because once you start including Persons A and B, people will rightfully ask why Persons C and D were not part of the list.
The two exceptions are David Walsh, because his blog often features guest writers and is not quite a typical personal blog, and Wes Bos (in the "Sites & Courses" section) just because of the sheer size of his audience.
But the bottom line is that the Resources section can definitely be improved to better reflect the CSS community, and I think we will probably start by reconsidering our decision to exclude personal blogs for next year (leave your suggestions here).
Exhibit C: We’re JavaScript Developers
The first survey we launched was the State of JavaScript survey back in 2016, which we then followed up with the first State of CSS survey in 2019. So during those first three years, we did accumulate a very JavaScript-focused audience.
So it's understandable why some people might think we are outsiders barging in without trying to represent the real CSS community.
Moreover, as JavaScript has come to take more and more space on the front-end over the past couple years, it’s understandable that the CSS community might be afraid of this “invasion”. Especially if this threatens the things that make the CSS community so great, such as the fact that it’s traditionally been a more welcoming place for women and minoritized demographics.
Then again, the reality is that CSS is changing, and that there is no impartial authority to decide who represents “the community” and who doesn’t. A CSS working group member working on the next specification and an open-source developer hacking on a new CSS-in-JS library are both advancing the CSS state of the art, in their own ways.
Yet this “pro-JavaScript” bias in our audience is real, and one way we hope to mitigate it in the future is to give you tools to filter survey results by respondent source — such as people who found it through Twitter, CSS-Tricks, or our pre-existing mailing list. In the meantime, we do make all of our data public if you want to dig through it yourself.
So… We're Biased?
At this point you might be surprised if you expected this post to be an impassioned defense against the critics. Turns out I agree with them!
What I don't agree with on the other hand is the assumption that these biases stem from laziness, negligence, or lack of caring. The truth is just that this is a hard problem to solve!
We ask many different people to give us their feedback each year (and we also directly asked the 10,000 subscribers to our mailing list, too) and we’re hugely grateful for their help and suggestions; but as you'd expect many more are also too busy with their own projects to spend time and effort helping us.
And while suggesting we "involve more women" can seem like a no-brainer, this also puts even more burden on the very people who are being let down in the first place… We never want "involving women" to become a way to offload our own responsibilities onto them!
So we do our best with the time and resources we have. And if that best is not good enough for everybody, all I can say is that it’s not for lack of trying.
Making Things Better
All that being said, if you do want to help, what can you do?
First, you can give us feedback in anticipation for next year’s CSS survey. And while pinging us on Twitter is a good start, leaving us a comment in the dedicated GitHub threads for the State of CSS and the State of JavaScript surveys is even better since it'll ensure we can refer to it one year from now when we're working on the next edition of a survey.
Barring that, you can also give us time. Building the infrastructure to run these surveys takes time, and so does building relationships with key members of the community. Please remember that this is only the second time we’re running the State of CSS survey, and it’s been a tough year for everyone, us included.
If this post can show you one thing, I hope it’s that I’m not interested in making excuses or sweeping our issues under the rug. I’m interested in improving the survey and making things better, and hopefully so are you!
Note: I'm taking a break from social media so I won't respond to tweets or DMs, but I'll try to keep an eye on the comments here.
Thanks to Stephanie Walter and Christian Oliff for reviewing drafts of this post.
Top comments (23)
For a post that positions itself as trying to take responsibility for a mistake, there are certainly a lot of excuses provided here. Also, the "make things better" section really seems to put the onus of making things better on readers of this post. What are you all planning to do to make things better here?
So here are the actual steps we're taking:
Do you have any other suggestions?
(And by the way we've been talking about women right here but I'm fully conscious the same conversation could take place around BIPOCs; or even something like accessibility issues.)
I think these next steps sound really strong and I appreciate you messaging me back here to share this.
Reading your points above and responses to others, I do believe you are actively trying to correct the issues with inclusivity in the survey.
I'll continue to think on this and will be in touch if I have advice.
I... don't understand the problem. I deeply appreciate the post, and I am sad that the percentage of 'female' participants isn't higher, I just haven't experienced any different working in the web development industry for 20 years. I used to be the only female I knew in my immediate area, then one of a handful - and that hasn't changed. How can anyone be expected to increase 'female' participation when there just aren't as many working in the industry? I do not think I use any different resources than my male counterparts, so you could not have reached me any other way. I feel like I am missing something: everyone thinks the percentages are off compared to real life? Where's the statistics to support that?
My humble advice would be the following:
Use more diverse platforms: for example, Twitter is used by more Americans than Colombians, and it's possible that gender issues/state of CSS are different for Colombians than for Americans. Also, the second biggest group is Japan and I think your survey isn't in Japanese... I THINK (o x o;;)
Connect with female development/coding groups and organizations: Epic Queen, Woman in Web Development, Vue Vixens and Black Girls CODE come to mind, try to get yourselves known out there in those types of circles c:
Promote your survey to female dev groups, use female development hashtags, connect with important women in the development industry (in different countries if possible, you need that diversity!) that can help you reach girls like me: not popular but have insight to provide.
You need to find your market segment, it won't come finding you!!
Hopefully this small advice helps :3
It's good that you're aware of this pitfall, but I don't see this as a reason to continue on as you have. This strikes me like a too-convenient excuse. Many women in CSS would probably see it as an opportunity, not a burden. There's no harm in actively involving more women while agreeing to listen to them if they say they need more from the rest of the team.
To be clear, we absolutely welcome feedback and involvement from women, sorry if I didn't make that clear.
It's just that from my own discussions with women on this very topic, it's become clear that a common response to raising any kind of representation issue is to put the onus back on the victimized community to fix it ("oh our conference is not diverse enough? well why don't you send us a list of women who want to participate and we'll consider it!").
I wanted to communicate the fact that I'm aware of this, and for that reason am not expecting others to fix the issues with our own product. Sorry if that didn't come across properly.
Honestly, the implication here sounds a bit like "Women just aren't on the internet, so we don't reach them", which I find odd.
Twitter, Reddit and Hacker News are, at their core, very accessible platforms. If you have something to say, you can create an account and participate, or just lurk and read what others post.
So the assumption seems to be that there is a sort of shadow CSS community with a higher percentage of women, which the survey just doesn't reach because they're mostly inactive online?
With such a web-centered technology as CSS, where would such a community be hiding, and is there any realistic way to actually reach them for future surveys?
Or did I just misunderstand what the post is trying to explain?
I'm not in a minoritized demographic so that's also how I've always naively perceived these platforms. Just create a Hacker News account and leave a comment, what could be simpler!
But if not only minoritized individuals, but also the data itself, are telling us that there are biases on these platforms I think we should listen even if that doesn't match our own subjective experience.
So maybe we'll succeed in reaching out to a "shadow CSS community of women" or maybe we won't, but that doesn't change the fact that these biases exist and that we should take steps to fight them.
The thing is, that's not just a perception. If we're considering the technical side, then that's just a fact, so we can safely assume that on the technical side there's no problem.
The data certainly does indicate that there's factors we don't account for and I'm very curious what those factors are.
[edited: my comment is a response not to the post itself but to another comment, by a popular community member, that ridiculed the criticism the survey got]
I think I understand your sentiment that whenever there's a survey, there's a group of folks voicing their criticism for the sake of criticizing. However, I'd like to just point out that:
I'm not writing this response because I take an issue with how Sacha addressed the criticisms — I was just taken aback by your emotional comment, in which you seemed to mix research factors that do matter and that perhaps don't (thus downplaying the stuff that is important) just because you “have very little patience”. I appreciate how Sacha is open and honest about the limitations of the survey. Having worked as a researcher, I know what a horrible and unreliable tool surveys are and I usually just accept it as a fact and move on. As long as the survey is clear about methodology (who/how was prompted to respond, what the question were, etc.), I'm fine with the results.
In this particular case, I also don't know how much of a factor gender is in CSS-related subjects or how the gender of a respondent influences the response. Maybe it does when it comes to salary, for instance — but then again, this survey is not about salaries in tech and anyway, the salary data is also here all represented equally across countries even though only 21% respondents are from the US, which makes it irrelevant ($1k in Poland would be okay vs $1k in New York would be very little for this role). So, beyond just curiosity factor, I'd first ask if we even need the data on gender/age and whether the criticism is about the survey or rather the general feeling of not being included/considered in tech — these are two different problems.
However, if it should be kept, perhaps also comparing/representing the answers of women/men/nb/trans/other folks could tell us how the CSS experience differs and why.
If I can try and put myself in the shoes of the critics for a second, I think the issue is compounded by the fact that the survey (being named "The State of CSS") tries to speak for the CSS community – and our goal is definitely to make something the CSS community can embrace and be proud of.
So beyond the purely methodological criticisms, I also understand the frustration if you feel that something that's aiming to represent you does not in fact correspond to your own reality.
Thank you for this comment — just to be clear, I was responding not to your post directly but to another comment that ridiculed the criticism.
I am not trying to downplay the frustration people feel with not being represented in tech (I myself am rolling my eyes and speaking up often because of this). Still, I wonder whether gender does play any part in the results — seeing the differences and learning about reasons for them could tell us a lot about the community and CSS alike.
I really appreciate the transparency in your methodology and your openness about the limitations and imperfections of this survey. I’d be happy to help out in the next iterations even if by distributing it in my networks.
I'm missing possible action items for next steps.
And I think reflecting on your bias would also mean to reflect not only on the things you mentioned but having it even broader:
My feedback for 2021 would be to start reaching out to people, do some research and ask many more people (Woman, PoC) while developing this survey.
Some feedback related to what you write:
What is then your plan to increase awareness of women and non-binary people to support this? And not only that, this survey has a high range of responded being White (76%). There is definitely a huge community out there of PoC you are currently not able to reach out.
One way could be to reach out to different folks in these areas, even better reach out to organizations (Black Female Coders, Blacks In Tech,...).
The truth is, doing this survey every year is time-consuming for you (and the rest) and because of that you might accepted existing exclusivity inside developer communities.
For 2021 one major focus should be to a) get more people (Women, PoC) into your processes b) reach out to many of these PoC organizations / thought leaders and ask them What you could do to increase reach. Even better get some of them into your project.
This isn't easy, but it is important and necessary.
I doubt that this is true. It's true that they have a big reach, but I believe your impression of that itself is biased. In general, in 2021 I wouldn't put any person there or make sure to have a better (diverse) range of sites, podcasts etc. (just to name some: Emma Bastion, Sara Soueidan, Lea Verou, Sara Vieira, Shruti Kapoor, Shruti Kapoor, Tracy Lee | ladyleet,...)
Thanks Sacha, good job identifying some of the preexisting bias. It can be difficult to acknowledge this, and especially your own preexisting bias (which we all have). Harder still when I know how much work you team puts into the survey. It can feel very personal and uncomfortable to acknowledge.
If keep working on it it, that won't be uncomfortable and we can start to eliminate some of that bias and reach a wider community - good luck with it. As I've said before I'm willing to help if I can.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.