By Sahar Karimi
Introduction
Imagine applying for a loan and being rejected, not by a human, but by an algorithm. No explanation, just a cold decision made by a system you barely understand. In a world increasingly run by artificial intelligence (AI), these scenarios are becoming disturbingly common. How do we ensure our fundamental rights do not get lost in the data-driven rush?
Automated decision-making (ADM) systems, powered by AI, are transforming how life-changing decisions are made. While they promise efficiency, they also pose significant threats to the right to privacy and data protection (Articles 7 & 8 CFR), non-discrimination (Article 21 CFR), the right to dignity (Article 1 CFR). As a result, there is an urgent need to ensure that automated systems are understandable, accountable, and equitable, respecting fundamental rights such as dignity, equality, and autonomy, as outlined in the Charter of Fundamental Rights of the European Union (CFR).
The Transparency Problem: Why AI Decisions Are a Black Box?
When algorithms decide who gets a loan, fairness and accountability remain essential, just as they have always been in human decision-making. However, many ADM systems are not always transparent, raising additional concerns. These so-called “black boxes” process vast amounts of data to make decisions, but the logic behind those decisions is often hidden, even from the experts who design them.
For example, a credit scoring system might deny a loan because of an applicant’s income level or spending patterns, yet fail to provide any meaningful explanation. This lack of clarity can lead to unchallenged discrimination, inequality, and a sense of powerlessness. Without transparency, how can anyone ensure fairness?
The General Data Protection Regulation (GDPR): A Framework Falling Short?
The European’s Union General Data Protection Regulation aims to safeguard individual rights in the digital age by imposing transparency requirements on ADM systems. Articles 13-15 and 22 of the GDPR provide individuals with the ‘right to explanation,’ allowing them to access clear explanations of how a decision was made and challenge unfair outcomes or those automated decisions that significantly impact their lives. This framework is intended to ensure that ADM systems operate in ways that are aligned with human rights principles, particularly those outlined in the Charter of Fundamental Rights of the European Union, such as the right to privacy and data protection, the right to non-discrimination, and the right to dignity. However, there are still some gaps in the GDPR’s ability to enforce meaningful transparency. For example, Sandra Wachter, a legal scholar, argues that the GDPR does not clearly define how companies should explain their automated processes; vague language in the regulation has left many organisations uncertain about compliance, leading to insufficient transparency measures that fail to empower users. Additionally, while the GDPR establishes critical protections, its enforcement varies across jurisdictions, limiting its effectiveness in ensuring universal accountability.
The AI Act: Strengthening Oversight and Accountability!
The European Union’s AI Act aims to regulate AI systems based on risk levels, placing credit scoring and loan approval systems under the high-risk category (Chapter III). These systems are subject to stricter rules to ensure transparency, accountability, and human oversight. The AI Act requires that financial institutions integrate human intervention in automated decisions, ensuring that AI-generated outcomes can be reviewed and, if necessary, overridden. It also mandates that AI systems maintain clear documentation to enable audits and verify compliance with ethical standards.
However, implementing these provisions imposes significant challenges. Financial institutions struggle to balance AI’s efficiency with the necessity of human judgment, especially given the technical complexity of AI systems. As algorithms grow more advanced, they often become harder to interpret or control, leaving even engineers unable to fully understand how these systems function.
The challenge: Why Existing Solutions Are Not Sufficient?
Part of the problem lies in the complexity of modern AI systems. Machine learning models often function as intricate “black boxes.” While tools like counterfactual explanations, offering alternative “what if” scenarios, can help users understand why decisions were made, they fall short of full transparency. For instance, a counterfactual explanation might tell a rejected loan applicant, “If your income were €5,000 higher, your application would have been approved.”
While counterfactual explanations are promising, there are some limitations on applying them. These models rely on patterns in data to make predictions or decisions, but their internal workings are often too complex to be easily understood. For instance, they do not reveal the full logic of a system, and even these explanations may fall short of providing meaningful transparency.
Moreover, as AI models become more complex, their unpredictability increases. Tiny changes to inputs, like a single pixel in an image, can drastically alter outputs. This unpredictability raises concerns, particularly for high-stakes systems like credit scoring, where small errors can have significant consequences. Even with access to system source code and training data, engineers often struggle to interpret the internal mechanics of these systems in ways that are meaningful to human oversight.
These restrictions of these tools highlight the difficulty of implementing the transparency, fairness, and oversight provisions outlined in both the GDPR and the AI Act.
What Needs to Change: A Call to Action
To bridge the gap between the technical limitations of full transparency and the ethical and human rights implications of the systems, legal and technical disciplines need to collaborate to enhance the accountability and fairness of ADM systems. Legal experts must explain how principles like accountability, fairness, and transparency, grounded in human rights law, can be practically applied to ADM governance. At the same time, technical experts must develop tools to ensure that these principles are meaningfully implemented within the current technology. To do this, private companies, whose aim is often efficiency in ADM systems, need to align their operations and private interests with societal benefits, such as reducing bias and improving transparency. Moreover, it is essential to go beyond pure engineering and consider the societal and ethical impacts of the ADM systems. Therefore, companies need to incorporate and invest in ethical education for their engineers and developers and reject technological determinism. This belief suggests that technology evolves on its own and shapes society in ways beyond human control. Instead, companies should position governance as a democratic process that puts ethical values into practice. Ultimately, achieving meaningful transparency in ADM is not just a technical or legal challenge; it is a moral imperative to ensure that technology serves humanity equitably and ethically.
Sahar Karimi is a doctoral researcher at the faculty of law of the University of Turku, Finland.