In general, we all tend to think that users are going to be like us, but that's not always the case, and the results can be really negative: websites that are not accessible for people with disabilities, face detection software that works great for white men but not so great for women or people of color, text that uses slang/terms that may have a negative connotation in some cultures, variables in units that are not the expected ones, etc.
Yes, this! We like to believe we can cover every scenario, but in reality, it's harder to remember to cover scenarios we don't personally relate to or have experience with. In theory, these things /should/ be assessed before the solution starts, but in reality, it's not. For example this study I am linking below. The study was done on tech by really large tech companies like Microsoft and Amazon. You'd think as tech giants they would be careful about their process before deploying things to market, but they still failed to a certain extent.
But wouldn't these details be paved on the requirements and problem assessment, even before the solution engineering starts?
Wouldn't these things be assessed before the algorithms and coding even began? I mean...
Coding is just a step in the SDLC. Taking into account diversity and inclusion are things that need to be done at every step of the way, because every contributor in a project can (and should) bring concerns up.
If you're gonna develop a face-recognition algorithm for "people" (generalistic approach), you need to assess first who this people would be, and indeed you would have to test your solution against these people.
For example, not only on white males, but white females, black females, black males, etc, Etc... (All people that correspond your target population), and if the algorithm is wrong, you would have to fine tune-it or retrain your models.
I mean... You wouldn't develop an app. that doesn't have accessibility options if your target is a lot of people there will be blind or deaf people.
100% agree with this... but still it doesn't happen. And there are many examples available: face recognition software is 99% effective with white men, but 65% effective with black women. Ask if Target and Domino's have blind/deaf people as their customers and how did it go for them accessibility-wise. POs, BAs, ScrumMasters, Designers, Developers, Testers... they all didn't realize something as basic as what you are saying. Again, because developers tend to think that users are like them. Even for the test data.
But if the workforce doesn't include your target audience, or representing players on yur target audience, you will have issues. Is that what you mean??
No, that's not what I said or meant. I said that developers tend to think users are like them. It's not bad, it's a normal bias. If you have a uniform team, chances are that some issues are brought later than what they would have been brought in a more diverse group. Different points of view and experiences enrich a team in that sense.
What I meant was that I would like to have more resources pointing me on the conclusion that an inclusive workforce is directly related to inclusive solutions, and that the design and development can only be produced if we have a diverse workforce.
I mean... If the factor is inclusive workforce, or an inclusive solutions approach that would not necessarily depend on a more inclusive approach but a wider, more sensible analisis on the solutions and the target market.
Maybe it's the methods, not the people. That's why I would love to know more with some studies or so...
About some studies, you can see this Medium post about diversity statistics with links to different sources and studies. (Which doesn't mean that I endorse any of them in particular, I actually disagree with the approach of some I read in the past. This was just the result of a quick search on Google).
I wish people would stop using the AI examples for this, as it is a very poor argument.
When training an ML model, or doing any kind of statistics, you must ensure your test set is representative of the population you are going to make a statement about. Sex and skin color are blatantly obvious cases in vision systems. There are biases that are far harder to detect but have the same result on an individual. Adding team members with different sex or skin colors might fix this particular symptom, but the problem is that your data-gathering is inadequate.
For instance, a little ago there was a post about a soap dispenser using AI/computer vision to recognize whether a hand was beneath it, but only worked for light skin colors. The argument was made that a more diverse team would have spotted this problem, completely missing the point that a cheap sensors would have been a more robust solution and would work for different skin colors, missing fingers, tattoos...
There's a further problem. Often, there just aren't enough willing participants to get a representative data-set. This is a well-known problem in academia. Many people just have better things to do than subject themselves to some tests they do not understand for a few bucks. While we should be wary of unrepresentative data-sets, often the only alternative is doing nothing at all.
There are good arguments (beyond public relations or social injustice) for at least male/female diversity, and there are excellent arguments for tearing down some of the 'soft barriers' keeping mostly women out of STEM. This just isn't one of them.
I understand the AI example may not be the best, but it's a sign of something bigger. And it is not limited to poor test data. As I put in a different comment, development is not only coding, it involves all the steps in the SDLC, and data gathering too.
The data-gathering is definitely inadequate, but it's not an excuse either. Training data doesn't show up out of thin air, it is created and gathered by people (or algorithms created by people), which may influence its representation of the population and neutrality.
Even if the data is wrong, and the training is wrong. Nobody realizing that the accuracy was so far sided is a sign that they were oblivious to a sex/skin color issue. No one thought "hey, we have 99% accuracy for white men, but 65% for black women"? And if they did, nobody did anything? That's not a data gathering issue.
I agree that it is not always possible to get a good representation of the population. But in this day and age, with many free sources available for images and portraits, having bad data for a vision system is a poor excuse.
I can't really see the point you're trying to make here. Nevertheless, I think there are a few problems with what you're saying.
it's a sign of something bigger
Yes, it is a product of a divided society. The reasoning "biased AI → we need diversity in tech" does not hold though.
And it is not limited to poor test data [...] data gathering too.
If you know a good example about how diversity in the development team can profit the company, use that rather than AI. Let's not dilute good arguments with bad ones.
You also appear to assume the entire team is responsible for the whole process, which is often not true. Essentially this issue only matters for QA.
The data-gathering [...] sex/skin color issue.
I think you've missed my point here. There are an uncountable number of biases your dataset might have. A good data-gathering process ensures samples are representative of the final use case. Skin color issues are an indicator that the data-gathering process is poor and produces bad results. That is a problem in and of itself. Adding a black woman to the team might solve this particular issue, but the team is still going to produce dangerously biased models, with biases that are far less obvious to notice.
and the training is wrong
This is unlikely to be the case. ML will just match the data, whatever that is. Beyond having a model that is too simple, which will result in low accuracy, bias of the model after training is a reflection of the bias in the input data.
with many free sources available for images and portraits, having bad data for a vision system is a poor excuse.
This would cause exactly the bias problems I was talking about. Data gathering is hard. You can't just download some pictures and expect it to be an unbiased dataset.
I'd like to reiterate: I'm not making and argument against diversity. I've had rather good experiences pair-programming with women; men and women have different ways of tackling problems and there's definitely a "stronger together" effect. I would, however, like to see the argument of biased AI go away.
If you add bad arguments to good ones, the good arguments lose credibility.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Yes, this! We like to believe we can cover every scenario, but in reality, it's harder to remember to cover scenarios we don't personally relate to or have experience with. In theory, these things /should/ be assessed before the solution starts, but in reality, it's not. For example this study I am linking below. The study was done on tech by really large tech companies like Microsoft and Amazon. You'd think as tech giants they would be careful about their process before deploying things to market, but they still failed to a certain extent.
news.mit.edu/2018/study-finds-gend...
insurancejournal.com/news/national...
Coding is just a step in the SDLC. Taking into account diversity and inclusion are things that need to be done at every step of the way, because every contributor in a project can (and should) bring concerns up.
100% agree with this... but still it doesn't happen. And there are many examples available: face recognition software is 99% effective with white men, but 65% effective with black women. Ask if Target and Domino's have blind/deaf people as their customers and how did it go for them accessibility-wise. POs, BAs, ScrumMasters, Designers, Developers, Testers... they all didn't realize something as basic as what you are saying. Again, because developers tend to think that users are like them. Even for the test data.
No, that's not what I said or meant. I said that developers tend to think users are like them. It's not bad, it's a normal bias. If you have a uniform team, chances are that some issues are brought later than what they would have been brought in a more diverse group. Different points of view and experiences enrich a team in that sense.
About some studies, you can see this Medium post about diversity statistics with links to different sources and studies. (Which doesn't mean that I endorse any of them in particular, I actually disagree with the approach of some I read in the past. This was just the result of a quick search on Google).
Thank for your patience!! This clarifies a lot of the issues. After this starting point, I can pick up searching and knowing more.
I wish people would stop using the AI examples for this, as it is a very poor argument.
When training an ML model, or doing any kind of statistics, you must ensure your test set is representative of the population you are going to make a statement about. Sex and skin color are blatantly obvious cases in vision systems. There are biases that are far harder to detect but have the same result on an individual. Adding team members with different sex or skin colors might fix this particular symptom, but the problem is that your data-gathering is inadequate.
For instance, a little ago there was a post about a soap dispenser using AI/computer vision to recognize whether a hand was beneath it, but only worked for light skin colors. The argument was made that a more diverse team would have spotted this problem, completely missing the point that a cheap sensors would have been a more robust solution and would work for different skin colors, missing fingers, tattoos...
There's a further problem. Often, there just aren't enough willing participants to get a representative data-set. This is a well-known problem in academia. Many people just have better things to do than subject themselves to some tests they do not understand for a few bucks. While we should be wary of unrepresentative data-sets, often the only alternative is doing nothing at all.
There are good arguments (beyond public relations or social injustice) for at least male/female diversity, and there are excellent arguments for tearing down some of the 'soft barriers' keeping mostly women out of STEM. This just isn't one of them.
I understand the AI example may not be the best, but it's a sign of something bigger. And it is not limited to poor test data. As I put in a different comment, development is not only coding, it involves all the steps in the SDLC, and data gathering too.
The data-gathering is definitely inadequate, but it's not an excuse either. Training data doesn't show up out of thin air, it is created and gathered by people (or algorithms created by people), which may influence its representation of the population and neutrality.
Even if the data is wrong, and the training is wrong. Nobody realizing that the accuracy was so far sided is a sign that they were oblivious to a sex/skin color issue. No one thought "hey, we have 99% accuracy for white men, but 65% for black women"? And if they did, nobody did anything? That's not a data gathering issue.
I agree that it is not always possible to get a good representation of the population. But in this day and age, with many free sources available for images and portraits, having bad data for a vision system is a poor excuse.
I can't really see the point you're trying to make here. Nevertheless, I think there are a few problems with what you're saying.
Yes, it is a product of a divided society. The reasoning "biased AI → we need diversity in tech" does not hold though.
If you know a good example about how diversity in the development team can profit the company, use that rather than AI. Let's not dilute good arguments with bad ones.
You also appear to assume the entire team is responsible for the whole process, which is often not true. Essentially this issue only matters for QA.
I think you've missed my point here. There are an uncountable number of biases your dataset might have. A good data-gathering process ensures samples are representative of the final use case. Skin color issues are an indicator that the data-gathering process is poor and produces bad results. That is a problem in and of itself. Adding a black woman to the team might solve this particular issue, but the team is still going to produce dangerously biased models, with biases that are far less obvious to notice.
This is unlikely to be the case. ML will just match the data, whatever that is. Beyond having a model that is too simple, which will result in low accuracy, bias of the model after training is a reflection of the bias in the input data.
This would cause exactly the bias problems I was talking about. Data gathering is hard. You can't just download some pictures and expect it to be an unbiased dataset.
I'd like to reiterate: I'm not making and argument against diversity. I've had rather good experiences pair-programming with women; men and women have different ways of tackling problems and there's definitely a "stronger together" effect. I would, however, like to see the argument of biased AI go away.
If you add bad arguments to good ones, the good arguments lose credibility.