Different Horses for Different Courses: Comparing Bias Mitigation Algorithms in ML
Prakhar Ganeesh
Usman Gohar
Lu Cheng
Golnoosh Farnadi
With fairness concerns gaining significant attention in Machine Learning (ML), several bias mitigation techniques have been proposed, often … (voir plus)compared against each other to find the best method. These benchmarking efforts tend to use a common setup for evaluation under the assumption that providing a uniform environment ensures a fair comparison. However, bias mitigation techniques are sensitive to hyperparameter choices, random seeds, feature selection, etc., meaning that comparison on just one setting can unfairly favour certain algorithms. In this work, we show significant variance in fairness achieved by several algorithms and the influence of the learning pipeline on fairness scores. We highlight that most bias mitigation techniques can achieve comparable performance, given the freedom to perform hyperparameter optimization, suggesting that the choice of the evaluation parameters-rather than the mitigation technique itself-can sometimes create the perceived superiority of one method over another. We hope our work encourages future research on how various choices in the lifecycle of developing an algorithm impact fairness, and trends that guide the selection of appropriate algorithms.
2024-11-17
ArXiv (prépublication)
doi.org
arxiv.org
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach
Meera Desai
Nicholas Pangakis
A. F. Cooper
Angelina Wang
Solon Barocas
Alexandra Chouldechova
Chad Atalla
Su Lin Blodgett
Emily Corvi
P. A. Dow
Jean Garcia-Gathright
Alexandra Olteanu
Stefanie Reed
Emily Sheng
Dan Vann
Jennifer Wortman Vaughan
Matthew Vogel
Hannah Washington
Abigail Z. Jacobs … (voir 1 de plus)
Microsoft Research
Across academia, industry, and government, there is an increasing awareness that the measurement tasks involved in evaluating generative AI … (voir plus)(GenAI) systems are especially difficult. We argue that these measurement tasks are highly reminiscent of measurement tasks found throughout the social sciences. With this in mind, we present a framework, grounded in measurement theory from the social sciences, for measuring concepts related to the capabilities, impacts, opportunities, and risks of GenAI systems. The framework distinguishes between four levels: the background concept, the systematized concept, the measurement instrument(s), and the instance-level measurements themselves. This four-level approach differs from the way measurement is typically done in ML, where researchers and practitioners appear to jump straight from background concepts to measurement instruments, with little to no explicit systematization in between. As well as surfacing assumptions, thereby making it easier to understand exactly what the resulting measurements do and do not mean, this framework has two important implications for evaluating evaluations: First, it can enable stakeholders from different worlds to participate in conceptual debates, broadening the expertise involved in evaluating GenAI systems. Second, it brings rigor to operational debates by offering a set of lenses for interrogating the validity of measurement instruments and their resulting measurements.
2024-11-17
ArXiv (prépublication)
doi.org
arxiv.org
Towards AI-designed genomes using a variational autoencoder
N.K. Dudek
Doina Precup
Genomes encode elaborate networks of genes whose products must seamlessly interact to support living organisms. Humans’ capacity to unders… (voir plus)tand these biological systems is limited by their sheer size and complexity. In this work, we develop a proof of concept framework for training a machine learning algorithm to model bacterial genome composition. To achieve this, we create simplified representations of genomes in the form of binary vectors that indicate the encoded genes, henceforth referred to as genome vectors. A denoising variational autoencoder was trained to accept corrupted genome vectors, in which most genes had been masked, and reconstruct the original. The resulting model, DeepGenomeVector, effectively captures complex dependencies in genomic networks, as evaluated by both qualitative and quantitative metrics. An in-depth functional analysis of a generated genome vector shows that its encoded pathways are interconnected, near complete, and ecologically cohesive. On the test set, where the model’s ability to reconstruct uncorrupted genome vectors was evaluated, AUC and F1 scores of 0.98 and 0.83, respectively, support the model’s strong performance. This work showcases the power of machine learning approaches for synthetic biology and highlights the possibility that AI agents may one day be able to design genomes that animate carbon-based cells.
2024-11-17
bioRxiv (prépublication)
doi.org
IntentGPT: Few-shot Intent Discovery with Large Language Models
Juan A. Rodriguez
Nicholas Botzer
David Vazquez
Chris Pal
Marco Pedersoli
Issam Hadj Laradji
In today's digitally driven world, dialogue systems play a pivotal role in enhancing user interactions, from customer service to virtual ass… (voir plus)istants. In these dialogues, it is important to identify user's goals automatically to resolve their needs promptly. This has necessitated the integration of models that perform Intent Detection. However, users' intents are diverse and dynamic, making it challenging to maintain a fixed set of predefined intents. As a result, a more practical approach is to develop a model capable of identifying new intents as they emerge. We address the challenge of Intent Discovery, an area that has drawn significant attention in recent research efforts. Existing methods need to train on a substantial amount of data for correctly identifying new intents, demanding significant human effort. To overcome this, we introduce IntentGPT, a novel training-free method that effectively prompts Large Language Models (LLMs) such as GPT-4 to discover new intents with minimal labeled data. IntentGPT comprises an \textit{In-Context Prompt Generator}, which generates informative prompts for In-Context Learning, an \textit{Intent Predictor} for classifying and discovering user intents from utterances, and a \textit{Semantic Few-Shot Sampler} that selects relevant few-shot examples and a set of known intents to be injected into the prompt. Our experiments show that IntentGPT outperforms previous methods that require extensive domain-specific data and fine-tuning, in popular benchmarks, including CLINC and BANKING, among others.
2024-11-16
ArXiv (prépublication)
doi.org
arxiv.org
IntentGPT: Few-shot Intent Discovery with Large Language Models
Juan A. Rodriguez
Nicholas Botzer
David Vazquez
Chris Pal
Marco Pedersoli
Issam Hadj Laradji
In today's digitally driven world, dialogue systems play a pivotal role in enhancing user interactions, from customer service to virtual ass… (voir plus)istants. In these dialogues, it is important to identify user's goals automatically to resolve their needs promptly. This has necessitated the integration of models that perform Intent Detection. However, users' intents are diverse and dynamic, making it challenging to maintain a fixed set of predefined intents. As a result, a more practical approach is to develop a model capable of identifying new intents as they emerge. We address the challenge of Intent Discovery, an area that has drawn significant attention in recent research efforts. Existing methods need to train on a substantial amount of data for correctly identifying new intents, demanding significant human effort. To overcome this, we introduce IntentGPT, a novel training-free method that effectively prompts Large Language Models (LLMs) such as GPT-4 to discover new intents with minimal labeled data. IntentGPT comprises an \textit{In-Context Prompt Generator}, which generates informative prompts for In-Context Learning, an \textit{Intent Predictor} for classifying and discovering user intents from utterances, and a \textit{Semantic Few-Shot Sampler} that selects relevant few-shot examples and a set of known intents to be injected into the prompt. Our experiments show that IntentGPT outperforms previous methods that require extensive domain-specific data and fine-tuning, in popular benchmarks, including CLINC and BANKING, among others.
2024-11-16
ArXiv (prépublication)
arxiv.org
arxiv.org
Towards a General Recipe for Combinatorial Optimization with Multi-Filter GNNs
Frederik Wenkel
Semih Cantürk
Stefan Horoi
Michael Perlmutter
Guy Wolf
2024-11-16
logconference.io/LOG/2024/Conference (présentation orale)
openreview.net
openreview.net
UTG: Towards a Unified View of Snapshot and Event Based Models for Temporal Graphs
Shenyang Huang
Farimah Poursafaei
Reihaneh Rabbany
Guillaume Rabusseau
Emanuele Rossi
2024-11-16
logconference.io/LOG/2024/Conference (poster)
doi.org
openreview.net
EDAI Framework for Integrating Equity, Diversity, and Inclusion Throughout the Lifecycle of AI to Improve Health and Oral Health Care: Qualitative Study
Samira Abbasgholizadeh-Rahimi
Richa Shrivastava
Anita Brown-Johnson
Pascale Caidor
Claire Davies
Amal Idrissi Janati
Pascaline Kengne Talla
Sreenath Madathil
Bettina M Willie
Elham Emami
2024-11-15
Journal of Medical Internet Research (publié)
doi.org
"On the goals of linguistic theory": Revisiting Chomskyan theories in the era of AI
Eva Portelance
Masoud Jasbi
Theoretical linguistics seeks to explain what human language is, and why. Linguists and cognitive scientists have proposed different theoret… (voir plus)ical models of what language is, as well as cognitive factors that shape it, and allow humans to 'produce', 'understand', and 'acquire' natural languages. However, humans may no longer be the only ones learning to 'generate', 'parse', and 'learn' natural language: artificial intelligence (AI) models such as large language models are proving to have impressive linguistic capabilities. Many are thus questioning what role, if any, such models should play in helping theoretical linguistics reach its ultimate research goals? In this paper, we propose to answer this question, by reiterating the tenets of generative linguistics, a leading school of thought in the field, and by considering how AI models as theories of language relate to each of these important concepts. Specifically, we consider three foundational principles, finding roots in the early works of Noam Chomsky: (1) levels of theoretical adequacy; (2) procedures for linguistic theory development; (3) language learnability and Universal Grammar. In our discussions of each principle, we give special attention to two types of AI models: neural language models and neural grammar induction models. We will argue that such models, in particular neural grammar induction models, do have a role to play, but that this role is largely modulated by the stance one takes regarding each of these three guiding principles.
2024-11-15
ArXiv (prépublication)
doi.org
arxiv.org
"On the goals of linguistic theory": Revisiting Chomskyan theories in the era of AI
Eva Portelance
Masoud Jasbi
Theoretical linguistics seeks to explain what human language is, and why. Linguists and cognitive scientists have proposed different theoret… (voir plus)ical models of what language is, as well as cognitive factors that shape it, and allow humans to 'produce', 'understand', and 'acquire' natural languages. However, humans may no longer be the only ones learning to 'generate', 'parse', and 'learn' natural language: artificial intelligence (AI) models such as large language models are proving to have impressive linguistic capabilities. Many are thus questioning what role, if any, such models should play in helping theoretical linguistics reach its ultimate research goals? In this paper, we propose to answer this question, by reiterating the tenets of generative linguistics, a leading school of thought in the field, and by considering how AI models as theories of language relate to each of these important concepts. Specifically, we consider three foundational principles, finding roots in the early works of Noam Chomsky: (1) levels of theoretical adequacy; (2) procedures for linguistic theory development; (3) language learnability and Universal Grammar. In our discussions of each principle, we give special attention to two types of AI models: neural language models and neural grammar induction models. We will argue that such models, in particular neural grammar induction models, do have a role to play, but that this role is largely modulated by the stance one takes regarding each of these three guiding principles.
2024-11-15
ArXiv (prépublication)
doi.org
arxiv.org
Outcomes of guidelines from health technology assessment organizations in community-based primary care: a systematic mixed studies review
Ashkan Baradaran
Raymond Tolentino
Roland Grad
Isabelle Ganache
Genevieve Gore
Samira Abbasgholizadeh-Rahimi
Pierre Pluye
2024-11-14
International Journal of Technology Assessment in Health Care (publié)
doi.org
Canadian Spine Society
Antoine Dionne
Majeed Al-Zakri
Hubert Labelle
Julie Joncas
Baron Lonner
Ali Eren
Patrick Cahill
Peter Newton
Liisa Jaakkimainen
Teresa To
Maryse Bouchard
Sarah Hardy
Dilani Thevarajah
Rajendra Sakhrekar
Ayesha Hadi
Andrea Doria
Aya Mitani
Andrew Howard
Samuel Yoon
Karen Mathias … (voir 346 de plus)
Tracey Bastrom
Amer Samdani
Marjolaine Roy-Beaudry
Marie Beausejour
Rachelle Imbeault
Justin Dufresne
Stefan Parent
Jessica Romeo
Holly Livock
Kevin Smit
James Jarvis
Andrew Tice
Vivien K. Chan
Robert Cho
Selina Poon
David L. Skaggs
Geoffrey K. Shumilak
Brett Rocos
Juan P. Sardi
Anastasios Charalampidis
Jeff Gum
Peter S. Tretiakov
Oluwatobi Onafowokan
Jamshaid Mir
Ankita Das
Tyler Williamson
Pooja Dave
Bailey Imbo
Jordan Lebovic
Pawel Jankowski
Peter G. Passias
Yousef Aljamaan
Vishal P. Varshney
Ramesh Sahjpaul
Jill Osborn
Rémi Pelletier-Roy
Michael Asmussen
Manjot Birk
Taryn Ludwig
Fred Nicholls
Ariel Zohar
Janneke Loomans
Ferran Pellise
Justin Smith
So Kato
Zeeshan Sardar
Lawrence G. Lenke
Stephen J. Lewis
Aazad Abbas
Jay Toor
Gurjovan Sahi
Dusan Kovacevic
Johnathan Lex
Firoz Miyanji
Anthony V. Perruccio
Nizar Mahomed
Mayilee Canizares
Yousef Kamel
Galil Osman
Nikolaus Koegl
Brandon Herrington
Renan R. Fernandes
Jennifer C. Urquhart
Ramtin Hakimjavadi
Zachary DeVries
Noah Fine
Laura Stone
Mohit Kapoor
Alexandre Chenevert
Sonia Bédard
Julien Goulet
Jerome Couture
Bernard LaRue
Meaghan Rye
Alexa Roussac
Neda Naghdi
Luciana G. Macedo
James Elliott
Richard DeMont
Véronique Pepin
Zhi Wang
Maroun Rizkallah
Jesse Shen
Michel Alexandre Lebreton
Edisond Florial
Fidaa Alshakfa
Ghassan Boubez
Abdullah A.S.M. AlDuwaisan
Kim Phan
Sarah Nowell
Niels Wedderkopp
Michael Craig
Abdul Al-Shawwa
Kalum Ost
Saswati Tripathy
Bradley W. Jacobs
Nathan Evaniew
Chris Bailey
W. Bradley Jacobs
Andrew Nataraj
David W. Cadotte
Kenneth C. Thomas
Hamilton Hall
Eva Y. Liu
Amit R.L. Persad
Nathan Baron
Daryl Fourney
Jingyi Huang
Thamer Alfawaz
Tinghua Zhang
CSORN Investigators
Karlo M. Pedro
Mohammed Ali Alvi
Jessica C.W. Wang
Nicolas Dea
Tamir Ailon
Scott Paquette
John Street
Charlotte Dandurand
Rohail Mumtaz
Khaled Skaik
Eugene K. Wai
Alexandra Stratton
Ragavan Manoharan
Jenna Smith-Forrester
JoAnne E. Douglas
Evan Nemeth
Jacob Alant
Sean Barry
Andrew Glennie
William Oxner
Lutz M. Weise
Sabahat Saeed
Patrick Toyota
Jack Su
Braeden Newton
Nicole Coote
Maria S. Rachevits
Helen Razmjou
Susan Robarts
Albert Yee
Joel Finkelstein
Alysa Almojuela
Frederick Zeiler
Sarvesh Logsetty
Perry Dhaliwal
Mark Abdelnour
Yuxin Zhang
Stephen P. Kingwell
Philippe Phan
Taylor A. Smith
Michael Bond
Stephan Dombrowski
Gwyneth Price
Jose Manuel García-Moreno
Steven Qiu
Vithushan Surendran
Victoria Shi Emily Cheung
Sophie Ngana
Muhammad A. Qureshi
Sunjay V. Sharma
Markian Pahuta
Daipayan Guha
Ahmed Essa
Husain Shakil
James Byrne
Andrew S. Jack
Francois Mathieu
Eva Yuan
Christopher W. Smith
Erin M. Harrington
Rachel H. Jaffe
Alick P. Wang
Karim Ladha
Avery B. Nathens
Ryan V. Sandarage
Ahmad Galuta
Eve C. Tsai
Naama Rotem-Kohavi
Marcel Dvorak
Jijie Xu
Nader Fallah
Zeina Waheed
Melody Chen
Vanessa K. Noonan
Toluyemi Malomo
Charles G. Fisher
Rachael Jaffe
Peter Coyte
Brian Chan
Armaan Malhotra
Rebecca Hancock-Howard
Jefferson R. Wilson
Christopher D. Witiw
Newton Cho
Jordan Squair
Viviana Aureli
Nicholas James
Lea Bole-Feysot
Inssia Dewany
Nicolas Hankov
Laetitia Baud
Anna Leonhartsberger
Kristina Sveistyte
Michael Skinnider
Matthieu Gautier
Katia Galan
Maged Goubran
Jimmy Ravier
Frederic Merlos
Laura Batti
Stéphane Pagès
Nadia Bérard
Nadine Intering
Camille Varescon
Stefano Carda
Kay Bartholdi
Thomas Hutson
Claudia Kathe
Michael Hodara
Mark Anderson
Bogdan Draganski
Robin Demesmaeker
Leonie Asboth
Quentin Barraud
Jocelyne Bloch
Gregoire Courtine
Sean D. Christie
Ryan Greene
Mustafa Nadi
Bill Oxner
Lisa Julien
Clara Lownie
Cumhur F.C. Öner
Alexander Joeris
Klaus Schnake
Mark Phillips
Alexander R. Vaccaro
Richard Bransford
Eugen Cezar Popescu
Mohammed El-Sharkawi
Shanmuganathan Rajasekaran
Lorin M. Benneker
Greg D. Schroeder
Jin W. Tee
John France
Jérôme Paquet
Richard Allen
William F. Lavelle
Emiliano Vialle
David Magnuson
Andréane Richard-Denis
Yvan Petit
Francis Bernard
Dorothy Barthélemy
Lukas Grassner
Daniel Garcia-Ovejero
Evelyn Beyerer
Orpheus Mach
Iris Leister
Doris Maier
Ludwig Aigner
Angel Arevalo-Martin
Mark Alexander MacLean
Antoinette Charles
Raphaële Charest-Morin
Rory Goodwin
Michael H. Weber
Emile Brouillard
Ismail Laassassy
Paul Khoueir
Étienne Bourassa-Moreau
Gilles Maurais
Jean-Marc Mac-Thiong
Julien Francisco Zaldivar-Jolissaint
Aysha Allard Brown
Kitty So
Neda Manouchehri
Megan Webster
Jay Ethridge
Audrey Warner
Avril Billingsley
Rochelle Newsome
Kirsten Bale
Andrew Yung
Mehara Seneviratne
Jimmy Cheng
Jing Wang
Shenani Basnayake
Femke Streijger
Manraj Heran
Piotr Kozlowski
Brian K. Kwon
Jeff D. Golan
Lior M. Elkaim
Qais Alrashidi
Miltiadis Georgiopoulos
Oliver Lasry
Drew A. Bednar
Alyson Love
Soroush Nedaie
Pranjan Gandhi
Prarthan C. Amin
Christopher J. Neilsen
Amanda Vandewint
Y. Raja Rampersaud
Jeffrey Hebert
Eden Richardson
Jillian Kearney
Raja Rampersaud
Aditya Raj
Nanadan Marathe
Greg McIntosh
Manmeet Dhiman
Taylor J. Bader
David Hart
Ganesh Swamy
Neil Duncan
Dragana Ponjevic
John R. Matyas
Connor P. O’Brien
Erin Bigney
Edward Abraham
Neil Manson
Najmedden Attabib
Chris Small
Luke LaRochelle
Gabriella Rivas
James Lawrence
Robert Ravinsky
Lily S. Switzer
David E. Lebel
Chanelle Montpetit
Nicolas Vaillancourt
Emma Nadler
Jennifer A. Dermott
Dorothy J. Kim
Brent Rosenstein
Daniel Wolfe
Geoffrey Dover
Mathieu Boily
Maryse Fortin
Jetan Badhiwala
Vishu Karthikeyan
Yingshi He
Michael G. Fehlings
2024-11-13
Canadian journal of surgery. Journal canadien de chirurgie (publié)
doi.org