AJung Moon

Measuring What Matters: Connecting AI Ethics Evaluations to System Attributes, Hazards, and Harms

Renee Shelby

2025-10-15

Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (published)

doi.org

arxiv.org

Measuring What Matters: Connecting AI Ethics Evaluations to System Attributes, Hazards, and Harms

Renee Shelby

Over the past decade, an ecosystem of measures has emerged to evaluate the social and ethical implications of AI systems, largely shaped by … (see more)high-level ethics principles. These measures are developed and used in fragmented ways, without adequate attention to how they are situated in AI systems. In this paper, we examine how existing measures used in the computing literature map to AI system components, attributes, hazards, and harms. Our analysis draws on a scoping review resulting in nearly 800 measures corresponding to 11 AI ethics principles. We find that most measures focus on four principles – fairness, transparency, privacy, and trust – and primarily assess model or output system components. Few measures account for interactions across system elements, and only a narrow set of hazards is typically considered for each harm type. Many measures are disconnected from where harm is experienced and lack guidance for setting meaningful thresholds. These patterns reveal how current evaluation practices remain fragmented, measuring in pieces rather than capturing how harms emerge across systems. Framing measures with respect to system attributes, hazards, and harms can strengthen regulatory oversight, support actionable practices in industry, and ground future research in systems-level understanding.

2025-10-15

Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (published)

doi.org

arxiv.org

Measuring What Matters: Connecting AI Ethics Evaluations to System Attributes, Hazards, and Harms

Renee Shelby

Over the past decade, an ecosystem of measures has emerged to evaluate the social and ethical implications of AI systems, largely shaped by … (see more)high-level ethics principles. These measures are developed and used in fragmented ways, without adequate attention to how they are situated in AI systems. In this paper, we examine how existing measures used in the computing literature map to AI system components, attributes, hazards, and harms. Our analysis draws on a scoping review resulting in nearly 800 measures corresponding to 11 AI ethics principles. We find that most measures focus on four principles – fairness, transparency, privacy, and trust – and primarily assess model or output system components. Few measures account for interactions across system elements, and only a narrow set of hazards is typically considered for each harm type. Many measures are disconnected from where harm is experienced and lack guidance for setting meaningful thresholds. These patterns reveal how current evaluation practices remain fragmented, measuring in pieces rather than capturing how harms emerge across systems. Framing measures with respect to system attributes, hazards, and harms can strengthen regulatory oversight, support actionable practices in industry, and ground future research in systems-level understanding.

2025-10-11

ArXiv (preprint)

doi.org

arxiv.org

Opening the Scope of Openness in AI

Tamara Paris

AJung Moon

Jin Guo

2025-06-23

Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency (published)

doi.org

arxiv.org

Roboethics for everyone – A hands-on teaching module for K-12 and beyond

In this work, we address the evolving landscape of roboethics, expanding beyond physical safety to encompass broader societal implications. … (see more)Recognizing the siloed nature of existing initiatives to teach and inform ethical implications of artificial intelligence (AI) and robotic systems, we present a roboethics teaching module designed for K-12 students and general audiences. The module focuses on the high-level analysis of the interplay between robot behaviour design choices and ethics, using everyday social dilemmas. We delivered the module in a workshop to high school students in Montreal, Canada. From this experience, we observed that the module successfully fostered critical thinking and ethical considerations in students, without requiring advanced technical knowledge. This teaching module holds promise to reach a wider range of populations. We urge the education community to explore similar approaches and engage in interdisciplinary training opportunities regarding the ethical implications of AI and robotics.

2024-12-20

Proceedings of the Canadian Engineering Education Association (CEEA) (published)

doi.org

Roboethics for everyone – A hands-on teaching module for K-12 and beyond

In this work, we address the evolving landscape of roboethics, expanding beyond physical safety to encompass broader societal implications. … (see more)Recognizing the siloed nature of existing initiatives to teach and inform ethical implications of artificial intelligence (AI) and robotic systems, we present a roboethics teaching module designed for K-12 students and general audiences. The module focuses on the high-level analysis of the interplay between robot behaviour design choices and ethics, using everyday social dilemmas. We delivered the module in a workshop to high school students in Montreal, Canada. From this experience, we observed that the module successfully fostered critical thinking and ethical considerations in students, without requiring advanced technical knowledge. This teaching module holds promise to reach a wider range of populations. We urge the education community to explore similar approaches and engage in interdisciplinary training opportunities regarding the ethical implications of AI and robotics.

2024-12-20

Proceedings of the Canadian Engineering Education Association (CEEA) (published)

doi.org

From Silos to Systems: Process-Oriented Hazard Analysis for AI Systems

Shalaleh Rismani

Roel Dobbe

AJung Moon

2024-10-29

ArXiv (preprint)

doi.org

arxiv.org

How different mental models of AI-based writing assistants impact writers’ interactions with them