Alan Chan

Alex Leung

Alvin Kwock

Anthony Tung

Arisa Siong

Arthur Tea

BEN BUCKNALL

Benjamin Weinstein-Raun

He Bing Sheng

Liu Bo

Bryan Kian Hsiang Low

Chris Ngo

Clement Neo

Cyrus Hodes

Dan Hendrycks

Daniel Ross

Liu Dapeng

Denise Wong

Djordje Žikelić

Elham Tabassi

Fabien Le Voyer

Fazl Barez

Gabriel Nicholas

Henry Papadatos

Jaan Tallinn

James Petrie

Xu Jia

Shao Jing

Jonathan Barry

Julia Chen

Sun Jun

Karson Elmgren

Kat Lyness

Katherine Lee

Kristy Loke

Lee Kwee Geak

Leslie Teo

Meng Ling Yu

Lisa Soder

Madhulika Srikumar

Malcolm Murray

Mark Brakel

Mark Nitzberg

Mary Phuong

Matthew Jagielski

Max Fenkell

Miro Plueckebaum

Kim Myuhng Joo

Hu Naying

Neil Davison

Nicolas Miailhe

Niki Iliadis

Nur Syahidah Sahrom

Ong Chen Hui

Pradeep Varakantham

Rebecca Finlay

Renata Dwan

Robert Opp

Rumman Chowdhury

Saad Siddiqui

Sabina Nong

Sam Ramadori

Sami Jawhar

Samuel Boger

Sara Hooker

Ying Shao Wei

Sebastian Hallensleben

Shinyuk Kang

Sophie Toura

Sreejith Balakrishnan

Stephanie Kasaon

Stephen Clare

Summer Yue

Sunny Yuqing Sun

Supheakmungkol Sarin

Tian Tian

Tim Schreier

Tori Westerhoff

Urvashi Aneja

Wayne Tee

Lu Wei

Xu Wei

Zhang Wenxuan

Hu Xia

Yang Xiaofang

Pan Xudong

Xiao Yajun

Yifan Jia

Tan Yong Khiam

Yuejin Du

Yuma Kurihara

Tan Zhi Xuan

Frontier AI capabilities and autonomy are advancing rapidly. A growing number of real-world incidents make a trusted AI ecosystem essential … (see more)to embracing AI with confidence. The 2026 Singapore Consensus is an outcome of the second International Scientific Exchange on AI Safety, bringing together over 100 contributors spanning 13 countries from frontier developers, government safety institutes, academia, and civil society. Building on the 2025 report, it presents a global understanding of technical AI safety research problems of top priority, now with a dedicated focus on societal resilience and on managing the risks of increasingly autonomous AI agents.

2026-07-08

arXiv (preprint)

Open Problems in Technical AI Governance

Anka Reuel

Benjamin Bucknall

Stephen Casper

Timothy Fist

Lisa Soder

Onni Aarne

Lewis Hammond

Lujain Ibrahim

Peter Wills

Markus Anderljung

Ben Garfinkel

Lennart Heim

Andrew Trask

Gabriel Mukobi

Rylan Schaeffer

Mauricio Baker

Sara Hooker

Irene Solaiman

Sasha Luccioni … (see 14 more)

Alexandra Luccioni

Nicolas Moës

Jeffrey Ladish

David Bau

Paul Bricman

Neel Guha

Jessica Newman

Yoshua Bengio

Tobin South

Alex Pentland

Sanmi Koyejo

Mykel Kochenderfer

Robert Trager

AI progress is creating a growing range of risks and opportunities, but it is often unclear how they should be navigated. In many cases, the… (see more) barriers and uncertainties faced are at least partly technical. Technical AI governance, referring to technical analysis and tools for supporting the effective governance of AI, seeks to address such challenges. It can help to (a) identify areas where intervention is needed, (b) identify and assess the efficacy of potential governance actions, and (c) enhance governance options by designing mechanisms for enforcement, incentivization, or compliance. In this paper, we explain what technical AI governance is, why it is important, and present a taxonomy and incomplete catalog of its open problems. This paper is intended as a resource for technical researchers or research funders looking to contribute to AI governance.

2025-04-13

Transactions on Machine Learning Research (accepted)

openreview.net

Foundational Challenges in Assuring Alignment and Safety of Large Language Models

Usman Anwar

Abulhair Saparov

Javier Rando

Daniel Paleka

Miles Turpin

Peter Hase

Ekdeep Singh Lubana

Erik Jenner

Stephen Casper

Oliver Sourbut

Benjamin L. Edelman

Zhaowei Zhang

Mario Günther

Anton Korinek

Jose Hernandez-Orallo

Lewis Hammond

Eric Bigelow

Alexander Pan

Lauro Langosco

Tomasz Korbak … (see 22 more)

Heidi Zhang

Ruiqi Zhong

Seán Ó hÉigeartaigh

Gabriel Recchia

Giulio Corsi

Markus Anderljung

Lilian Edwards

Aleksandar Petrov

Christian Schroeder de Witt

Sumeet Ramesh Motwani

Samuel Albanie

Yoshua Bengio

Danqi Chen

Philip H.S. Torr

Tegan Maharaj

Jakob Foerster

Florian Tramèr

He He

Atoosa Kasirzadeh

Yejin Choi

David Krueger

This work identifies 18 foundational challenges in assuring the alignment and safety of large language models (LLMs). These challenges are o… (see more)rganized into three different categories: scientific understanding of LLMs, development and deployment methods, and sociotechnical challenges. Based on the identified challenges, we pose

2024-09-01

TMLR (accepted)

openreview.net

IDs for AI Systems

Noam Kolt

Peter Wills

Usman Anwar

Christian Schroeder de Witt

Lewis Hammond

David M. Krueger

Lennart Heim

Markus Anderljung

2024-06-16

ArXiv (preprint)

IDs for AI Systems

Noam Kolt

Peter Wills

Usman Anwar

Christian Schroeder de Witt

Lewis Hammond

David M. Krueger

Lennart Heim

Markus Anderljung

2024-06-16

ArXiv (preprint)

IDs for AI Systems

Noam Kolt

Peter Wills

Usman Anwar

Christian Schroeder de Witt

Lewis Hammond

David M. Krueger

Lennart Heim

Markus Anderljung

AI systems are increasingly pervasive, yet information needed to decide whether and how to engage with them may not exist or be accessible. … (see more)A user may not be able to verify whether a system has certain safety certifications. An investigator may not know whom to investigate when a system causes an incident. It may not be clear whom to contact to shut down a malfunctioning system. Across a number of domains, IDs address analogous problems by identifying particular entities (e.g., a particular Boeing 747) and providing information about other entities of the same class (e.g., some or all Boeing 747s). We propose a framework in which IDs are ascribed to instances of AI systems (e.g., a particular chat session with Claude 3), and associated information is accessible to parties seeking to interact with that system. We characterize IDs for AI systems, provide concrete examples where IDs could be useful, argue that there could be significant demand for IDs from key actors, analyze how those actors could incentivize ID adoption, explore a potential implementation of our framework for deployers of AI systems, and highlight limitations and risks. IDs seem most warranted in settings where AI systems could have a large impact upon the world, such as in making financial transactions or contacting real humans. With further study, IDs could help to manage a world where AI systems pervade society.

2024-06-16

ArXiv (preprint)

IDs for AI Systems

Noam Kolt

Peter Wills

Usman Anwar

Christian Schroeder de Witt

Lewis Hammond

David M. Krueger

Lennart Heim

Markus Anderljung

2024-06-16

ArXiv (preprint)

IDs for AI Systems

Noam Kolt

Peter Wills

Usman Anwar

Christian Schroeder de Witt

Lewis Hammond

David M. Krueger

Lennart Heim

Markus Anderljung

AI systems are increasingly pervasive, yet information needed to decide whether and how to engage with them may not exist or be accessible. … (see more)A user may not be able to verify whether a system has certain safety certifications. An investigator may not know whom to investigate when a system causes an incident. It may not be clear whom to contact to shut down a malfunctioning system. Across a number of domains, IDs address analogous problems by identifying particular entities (e.g., a particular Boeing 747) and providing information about other entities of the same class (e.g., some or all Boeing 747s). We propose a framework in which IDs are ascribed to instances of AI systems (e.g., a particular chat session with Claude 3), and associated information is accessible to parties seeking to interact with that system. We characterize IDs for AI systems, provide concrete examples where IDs could be useful, argue that there could be significant demand for IDs from key actors, analyze how those actors could incentivize ID adoption, explore a potential implementation of our framework for deployers of AI systems, and highlight limitations and risks. IDs seem most warranted in settings where AI systems could have a large impact upon the world, such as in making financial transactions or contacting real humans. With further study, IDs could help to manage a world where AI systems pervade society.

2024-06-16

ArXiv (preprint)

IDs for AI Systems

Noam Kolt

Peter Wills

Usman Anwar

Christian Schroeder de Witt

Lewis Hammond

David M. Krueger

Lennart Heim

Markus Anderljung

AI systems are increasingly pervasive, yet information needed to decide whether and how to engage with them may not exist or be accessible. … (see more)A user may not be able to verify whether a system has certain safety certifications. An investigator may not know whom to investigate when a system causes an incident. It may not be clear whom to contact to shut down a malfunctioning system. Across a number of domains, IDs address analogous problems by identifying particular entities (e.g., a particular Boeing 747) and providing information about other entities of the same class (e.g., some or all Boeing 747s). We propose a framework in which IDs are ascribed to instances of AI systems (e.g., a particular chat session with Claude 3), and associated information is accessible to parties seeking to interact with that system. We characterize IDs for AI systems, provide concrete examples where IDs could be useful, argue that there could be significant demand for IDs from key actors, analyze how those actors could incentivize ID adoption, explore a potential implementation of our framework for deployers of AI systems, and highlight limitations and risks. IDs seem most warranted in settings where AI systems could have a large impact upon the world, such as in making financial transactions or contacting real humans. With further study, IDs could help to manage a world where AI systems pervade society.

2024-06-16

ArXiv (preprint)

IDs for AI Systems

Noam Kolt

Peter Wills

Usman Anwar

Christian Schroeder de Witt

Lewis Hammond

David M. Krueger

Lennart Heim

Markus Anderljung

2024-06-16

ArXiv (preprint)

IDs for AI Systems

Noam Kolt

Peter Wills

Usman Anwar

Christian Schroeder de Witt

Lewis Hammond

David M. Krueger

Lennart Heim

Markus Anderljung

2024-06-16

ArXiv (preprint)

IDs for AI Systems

Noam Kolt

Peter Wills

Usman Anwar

Christian Schroeder de Witt

Lewis Hammond

David M. Krueger

Lennart Heim

Markus Anderljung

AI systems are increasingly pervasive, yet information needed to decide whether and how to engage with them may not exist or be accessible. … (see more)A user may not be able to verify whether a system has certain safety certifications. An investigator may not know whom to investigate when a system causes an incident. It may not be clear whom to contact to shut down a malfunctioning system. Across a number of domains, IDs address analogous problems by identifying particular entities (e.g., a particular Boeing 747) and providing information about other entities of the same class (e.g., some or all Boeing 747s). We propose a framework in which IDs are ascribed to instances of AI systems (e.g., a particular chat session with Claude 3), and associated information is accessible to parties seeking to interact with that system. We characterize IDs for AI systems, provide concrete examples where IDs could be useful, argue that there could be significant demand for IDs from key actors, analyze how those actors could incentivize ID adoption, explore a potential implementation of our framework for deployers of AI systems, and highlight limitations and risks. IDs seem most warranted in settings where AI systems could have a large impact upon the world, such as in making financial transactions or contacting real humans. With further study, IDs could help to manage a world where AI systems pervade society.

2024-06-16

ArXiv (preprint)