Ethical implications of big tech using machine learning for decision-making

On May 11, 2023 the European Parliament Internal Market Committee and Civil Liberties Committee adopted a draft of the first ever rules regulating Artificial Intelligence, the AI Act. The draft was approved by an overwhelming majority, with 84 votes in favor, 7 against and 12 abstentions. This initiative is a major turn in a wide discussion about artificial intelligence and machine learning (ML), the risks, ethical and transparency issues associated with this technology. Let’s have a look at the major concerns about AI and ML and see how the EU’s groundbreaking initiative tackles them.

Bias and discrimination in ML algorithms

We are used to the idea that computers “do not make mistakes”, but they are being programmed and trained by people who do. This is especially true for ML, where data selection and learning algorithm mechanics have drastic effects on the results, sometimes leading to unfair outcomes. For instance, facial recognition models may have higher error rates for people of colour. Loan approval algorithms may turn out to be prejudiced against certain socioeconomic or ethnic groups.

The AI Act tries to deal with these issues by explicitly banning biometric categorisation systems using sensitive characteristics like gender, race, ethnicity, citizenship status, religion or political views. Also, it prohibits predictive policing systems based on profiling, location or past criminal behaviour.

But apart from just banning the high-risk systems, other rules might also be useful. For instance, ML models should be trained on datasets that adequately represent the population they serve. Special industry standards are required with comprehensible metrics and correction algorithms built upon them to ensure fairness and diversity requirements.

Transparency in machine learning-based decision-making

ML transparency is becoming an issue since these techniques are used to automate decision-making. First, there is a lack of transparency about the data being used for model training, while biassed data can lead to wrong decisions. Second, there is a “black box” problem, as many algorithms are too complex and involve too many factors to be easily understood and checked for consistency. Apart from the already discussed fairness and diversity standards, there is also an approach known as explainable AI. It focuses on making ML models more interpretable and explainable. Such an approach enables tracking the entire decision-making process from logical rules employed in the algorithms right to the data source that was used to train the model.

The AI Act includes obligations for providers of foundational ML models who have to guarantee “robust protection of fundamental rights, health and safety and the environment, democracy and rule of law”. Model providers would need to “assess and mitigate risks, comply with design, information and environmental requirements”. Also they would have to register in a dedicated EU database. Generative model providers (like Open AI) need to comply with additional transparency requirements. Their models would have to be designed in a way that prevents generating illegal content. They would be obliged to publish summaries of copyrighted data used for training as well as disclose that the content they produce was generated by AI. The draft also promotes regulatory “sandboxes”, or controlled environments, established by public authorities to test AI before its deployment.

These steps seem feasible as explainable AI is already being employed by many tech companies. For instance, IBM has developed an AI Explainability 360 framework, a tool for interpreting decisions made by ML models. Microsoft has come up with an InterpretML open source toolkit with a similar goal. Google TensorFlow toolkit also includes a module that can interpret and visualise the inner workings of ML models. FICO, a credit scoring company, has come up with a feature known as FICO XAI that explains the factors influencing an individual’s credit score.

The impact of machine learning on privacy and data protection

Using ML models involves collecting, storing and processing vast amounts of user data. Increasing concentration of such data in the hands of the tech companies raises risks of possible misuse. There is already an industry-wide consensus that user data should be anonymised, which basically means removing or altering any information that could identify an individual. However, this approach is not bulletproof. Re-identification is possible by inference techniques and combining and comparing different datasets.

Another approach that is currently gaining momentum is differential privacy. This method employs deliberately adding noise to datasets so that their statistical properties are preserved keeping individual-related data unidentifiable. Differential privacy has already been adopted by big tech companies like Microsoft, Apple and Google as well as some government agencies like the United States Census Bureau.

The importance of these issues drew the attention of government bodies across the world quite some time ago. Actually most legal frameworks regulating the digital industry have been focusing on personal data protection. For example, Canada introduced a Personal Information Protection and Electronic Documents Act (PIPEDA) as early as 2000. PIPEDA sets rules for collection, usage and disclosure of personal information in private sector commercial activities. Similarly purposed General Data Protection Regulation (GDPR) went into effect in the European Union in 2018. A California Consumer Privacy Act (CCPA) in 2020 enabled California residents to know and control the personal data being collected about them.

However the AI Act goes much further, addressing specific AI and ML features. It completely bans “​​real-time” remote biometric identification systems in publicly accessible spaces; “post” remote biometric identification systems, with the only exception of law enforcement for the prosecution of serious crimes and only after judicial authorization. The draft puts a ban on emotion recognition systems in law enforcement, border management, workplace, and educational institutions; and on indiscriminate scrapping of biometric data from social media or CCTV footage to create facial recognition databases, violating human rights and right to privacy.

Conclusion

From the political point of view, the European Parliament reiterated its commitment to human rights and liberties, rejecting the most controversial data-driven practices. Most importantly, the draft strictly prohibits all forms of “social scoring”, systems classifying people based on their social behaviour, socio-economic status, and personal characteristics. Such developments are mostly known to be working in China. However, there are concerns that some existing systems might mutate into social scoring, like Homeland card in Venezuela or DICOM in Chile. Some systems, so far mostly limited to financial scoring, have raised public concerns in the UK, the US and Germany for collecting a wider range of data than needed for purely credit history needs.

As for other aspects of the initiative, adding new compliance, “explainable AI” features and reporting to the regulators will certainly impose a somewhat financial burden on the tech sector. However, many of these approaches have already been adopted by the industry, partially compensating for the regulation lag, when technology develops much faster than the regulating legal system. The draft still needs to be approved by the European Parliament, discussed at trilateral talks between members of the Parliament, the European Commission and the Member States. Only then it can be approved in final form to become law. It is expected to happen before the end of 2023. It still remains to be seen if the draft that currently seems either prohibitive or vague is enriched by more practical measures by then.

At this point, joint efforts of all stakeholders, including big tech, governments and consumers is essential to strike a balance between efficiency, security and ethics for this technology. However, should this discussion be outsourced to politicians, tech tycoons and other decision-makers? It may be good for everyone, not just the industry, if all people involved started considering the consequences of their work. This does not seem impossible. “Climate ethics”, which was unheard of just several decades ago, is now a must not only to businesses, but also to millions of consumers across the globe. It is not only regulation that lags behind technical progress. It is also our perception of technology and its possible outcomes. That was the case with fossil fuels, nuclear energy, use of plastics – the list can go on and on. So it is probably high time to start thinking about AI in a similar way and decide what is your take.