When an AI rejects a candidate based on their origin or gender, who is responsible for this injustice? Is it the employer using the tool, or the developer who programmed it? In an increasingly opaque supply chain, responsibility evaporates at every step. To end this impunity, the responsible AI advisory team from Mila Studios has published an analysis auditing these systems and proposing concrete solutions.
How Algorithms Learn to Discriminate
AI recruitment software promises recruiters massive time savings and selection free from human prejudice. Yet, once integrated, these "certified compliant" tools often reproduce racist and sexist biases. How can cutting-edge technology fail so profoundly on the grounds of fairness?
To learn how to recruit, algorithms are trained on CV databases and hiring histories. However, these data inherit decades of discriminatory practices. AI simply mathematizes the past: all qualifications being equal, some software prioritizes names perceived as "white" in 85% of cases. Even corrective attempts, such as using AI-generated "neutral" synthetic CVs, prove disappointing: they only make discrimination harder to trace.
Diluted Responsibility Among Stakeholders
Current legal frameworks are failing to curb these systemic biases. The problem is primarily structural: the AI recruitment market relies on a fragmented supply chain. Data providers sell hiring histories to developers who train algorithms, which are then integrated into HR software by publishers, and finally purchased by employers.
With so many stakeholders involved, responsibility is diluted. While the law designates the employer as ultimately responsible, they often have neither the technical expertise nor access to the source code, protected by trade secrets, to verify the tool. Conversely, developers master the technology but remain unaware of its real-world application. It becomes a game of "passing the buck": when a bias occurs, each actor blames the previous link in the chain.
When Regulatory Frameworks Hit a Mathematical Wall
Traditional anti-discrimination laws are ill-suited to the technical reality of hiring software. For example, the "four-fifths rule," a historical standard used by courts, requires the selection rate for minorities to be at least 80% of the rate for the majority group. If an employer hires 10% of male applicants, at least 8% of female applicants must be hired. However, if the algorithm unfairly eliminates 90% of female candidates during the initial screening, the employer may appear fair with the remaining profiles, ignoring the massive exclusion occurring upstream.
Globally, the regulatory landscape is marked by deep divergences and ambiguous terms: the EU AI Act requires "error-free" data, New York mandates limited annual audits, and Colorado calls for assessments with vague criteria.
Furthermore, evaluating fairness can be done using different metrics. Should we prioritize demographic parity (same proportion of minority profiles) or predictive parity (same error rate for everyone)? Satisfying both metrics simultaneously is technically impossible. The problem worsens when employers stack software from different vendors. If a CV screener optimizes for one definition of fairness while a ranking algorithm targets another, their interaction produces an incoherent system that ultimately discriminates.
Toward Continuous and Human Oversight
In response to these findings, the responsible AI advisory team from Mila Studios is proposing solutions to break away from the issue of static annual audits:
- Global and continuous auditing: Replace isolated evaluations with quarterly monitoring. An algorithm deemed "fair" at a given time can quickly drift following an update or a change in the candidate pool. This involves auditing the system end-to-end to verify how different tools (screening, ranking, evaluation) truly interact.
- Strict contractual transparency: Vendors must move beyond the opacity of trade secrets. They should be contractually required to declare which mathematical definition of fairness they use and document how their technical choices reflect legal requirements. In case of drift, technical and financial responsibility must be pre-established.
- The right to human challenge: Mila emphasizes training recruiters, who must be able to "unplug" the algorithm at the slightest doubt. Above all, every candidate should be informed of the use of AI and benefit from a right to human review in case of rejection.
Mila’s approach proposes moving from isolated audits to a global system evaluation, where responsibility is contractually shared across all links in the chain. Without this coordination, each actor can satisfy their individual obligations while the final tool continues to discriminate with impunity. Establishing such transparency is the prerequisite for building truly equitable AI, capable of delivering on its promises in recruitment as in all other high-stakes domains of our society.
