Publications

End-to-end Conditional Robust Optimization
Abhilash Reddy Chenreddy
Erick Delage
2024-09-11
Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence (published)
proceedings.mlr.press
proceedings.mlr.press
Neural Active Learning Meets the Partial Monitoring Framework
Maxime Heuillet
Ola Ahmad
Audrey Durand
2024-09-11
Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence (published)
doi.org
proceedings.mlr.press
Shedding Light on Large Generative Networks: Estimating Epistemic Uncertainty in Diffusion Models
Lucas Berry
Axel Brando
David Meger
Generative diffusion models, notable for their large parameter count (exceeding 100 million) and operation within high-dimensional image spa… (see more)ces, pose significant challenges for traditional uncertainty estimation methods due to computational demands. In this work, we introduce an innovative framework, Diffusion Ensembles for Capturing Uncertainty (DECU), designed for estimating epistemic uncertainty for diffusion models. The DECU framework introduces a novel method that efficiently trains ensembles of conditional diffusion models by incorporating a static set of pre-trained parameters, drastically reducing the computational burden and the number of parameters that require training. Additionally, DECU employs Pairwise-Distance Estimators (PaiDEs) to accurately measure epistemic uncertainty by evaluating the mutual information between model outputs and weights in high-dimensional spaces. The effectiveness of this framework is demonstrated through experiments on the ImageNet dataset, highlighting its capability to capture epistemic uncertainty, specifically in under-sampled image classes.
2024-09-11
Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence (published)
doi.org
proceedings.mlr.press
Feedback-guided Data Synthesis for Imbalanced Classification
Reyhane Askari Hemmat
Mohammad Pezeshki
Florian Bordes
Michal Drozdzal
Adriana Romero
Current status quo in machine learning is to use static datasets of real images for training, which often come from long-tailed distribution… (see more)s. With the recent advances in generative models, researchers have started augmenting these static datasets with synthetic data, reporting moderate performance improvements on classification tasks. We hypothesize that these performance gains are limited by the lack of feedback from the classifier to the generative model, which would promote the usefulness of the generated samples to improve the classifier's performance. In this work, we introduce a framework for augmenting static datasets with useful synthetic samples, which leverages one-shot feedback from the classifier to drive the sampling of the generative model. In order for the framework to be effective, we find that the samples must be close to the support of the real data of the task at hand, and be sufficiently diverse. We validate three feedback criteria on a long-tailed dataset (ImageNet-LT, Places-LT) as well as a group-imbalanced dataset (NICO++). On ImageNet-LT, we achieve state-of-the-art results, with over
2024-09-10
TMLR (accepted)
doi.org
openreview.net
Quality issues in Machine Learning Software Systems
Pierre-Olivier Côté
Amin Nikanjam
Rached Bouchoucha
Ilan Basta
Mouna Abidi
Foutse Khomh
Context: An increasing demand is observed in various domains to employ Machine Learning (ML) for solving complex problems. ML models are imp… (see more)lemented as software components and deployed in Machine Learning Software Systems (MLSSs). Problem: There is a strong need for ensuring the serving quality of MLSSs. False or poor decisions of such systems can lead to malfunction of other systems, significant financial losses, or even threat to human life. The quality assurance of MLSSs is considered as a challenging task and currently is a hot research topic. Moreover, it is important to cover all various aspects of the quality in MLSSs. Objective: This paper aims to investigate the characteristics of real quality issues in MLSSs from the viewpoint of practitioners. This empirical study aims to identify a catalog of bad-practices related to poor quality in MLSSs. Method: We plan to conduct a set of interviews with practitioners/experts, believing that interviews are the best method to retrieve their experience and practices when dealing with quality issues. We expect that the catalog of issues developed at this step will also help us later to identify the severity, root causes, and possible remedy for quality issues of MLSSs, allowing us to develop efficient quality assurance tools for ML models and MLSSs.
2024-09-10
Empirical Software Engineering (published)
doi.org
arxiv.org
Are Heterophily-Specific GNNs and Homophily Metrics Really Effective? Evaluation Pitfalls and New Benchmarks
Sitao Luan
Qincheng Lu
Chenqing Hua
Xinyu Wang
Jiaqi Zhu
Xiao-Wen Chang
Guy Wolf
Jian Tang
Over the past decade, Graph Neural Networks (GNNs) have achieved great success on machine learning tasks with relational data. However, rece… (see more)nt studies have found that heterophily can cause significant performance degradation of GNNs, especially on node-level tasks. Numerous heterophilic benchmark datasets have been put forward to validate the efficacy of heterophily-specific GNNs and various homophily metrics have been designed to help people recognize these malignant datasets. Nevertheless, there still exist multiple pitfalls that severely hinder the proper evaluation of new models and metrics. In this paper, we point out three most serious pitfalls: 1) a lack of hyperparameter tuning; 2) insufficient model evaluation on the real challenging heterophilic datasets; 3) missing quantitative evaluation benchmark for homophily metrics on synthetic graphs. To overcome these challenges, we first train and fine-tune baseline models on
2024-09-08
ArXiv (preprint)
doi.org
arxiv.org
Correction: Economic evaluation of the effect of needle and syringe programs on skin, soft tissue, and vascular infections in people who inject drugs: a microsimulation modelling approach
Jihoon Lim
W Alton Russell
Mariam El-Sheikh
David L Buckeridge
Dimitra Panagiotoglou
2024-09-08
Harm Reduction Journal (published)
doi.org
Perspectives on virtual interviews and emerging technologies integration in family medicine residency programs: a cross-sectional survey study
Raymond Tolentino
Charo Rodriguez
Fanny Hersson-Edery
Julie Lane
Samira Abbasgholizadeh Rahimi
During the coronavirus disease of 2019 (COVID-19) pandemic, in-person interviews for the recruitment of family medicine residents shifted to… (see more) online (virtual) interviews. The purpose of this study was twofold: (1) to gather the ideas about virtual interviews of family medicine applicants (interviewees), and faculty and staff who interviewed these applicants (interviewers), and (2) to describe interviewers’ and interviewees’ opinions of use of emerging technologies such as artificial intelligence (AI) and virtual reality (VR) in the recruitment process as well as during clinical practice. This was a cross-sectional survey study. Participants were both interviewers and candidates who applied to the McGill University Family Medicine Residency Program for the 2020–2021 and 2021–2022 cycles. The study population was constituted by N = 132 applicants and N = 60 interviewers. The response rate was 91.7% (55/60) for interviewers and 43.2% (57/132) for interviewees. Both interviewers (43.7%) and interviewees (68.5%) were satisfied with connecting through virtual interviews. Interviewers (43.75%) and interviewees (55.5%) would prefer for both options to be available. Both interviewers (50%) and interviewees (72%) were interested in emerging technologies. Almost all interviewees (95.8%) were interested in learning about AI and VR and its application in clinical practice with the majority (60.8%) agreeing that it should be taught within medical training. Although experience of virtual interviewing during the COVID-19 pandemic has been positive for both interviewees and interviewers, the findings of this study suggest that it will be unlikely that virtual interviews completely replace in-person interviews for selecting candidates for family medicine residency programs in the long term as participants value aspects of in-person interviews and would want a choice in format. Since incoming family medicine physicians seem to be eager to learn and utilize emerging technologies such as AI and VR, educators and institutions should consider family physicians’ needs due to the changing technological landscape in family medicine education. The online version contains supplementary material available at 10.1186/s12909-024-05874-5.
2024-09-08
BMC Medical Education (published)
doi.org
What is Your Favorite Gender, MLM? Gender Bias Evaluation in Multilingual Masked Language Models
Emily M. Bender
Jeongrok Yu
Timnit Gebru
Seong Ug Kim
Angelina McMillan-642
Jacob Choi
Jinho D. Choi
Su Lin Blodgett
Solon Barocas
Hal Daumé III
Gilsinia Lopez
A.R. Olteanu
Robert Sim
Hanna Wallach. 2021
Stereotyp-657
Bias is a disproportionate prejudice in favor of one side against another. Due to the success of transformer-based Masked Language Models (M… (see more)LMs) and their impact on many NLP tasks, a systematic evaluation of bias in these models is needed more than ever. While many studies have evaluated gender bias in English MLMs, only a few works have been conducted for the task in other languages. This paper proposes a multilingual approach to estimate gender bias in MLMs from 5 languages: Chinese, English, German, Portuguese, and Spanish. Unlike previous work, our approach does not depend on parallel corpora coupled with English to detect gender bias in other languages using multilingual lexicons. Moreover, a novel model-based method is presented to generate sentence pairs for a more robust analysis of gender bias, compared to the traditional lexicon-based method. For each language, both the lexicon-based and model-based methods are applied to create two datasets respectively, which are used to evaluate gender bias in an MLM specifically trained for that language using one existing and 3 new scoring metrics. Our results show that the previous approach is data-sensitive and not stable as it does not remove contextual dependencies irrelevant to gender. In fact, the results often flip when different scoring metrics are used on the same dataset, suggesting that gender bias should be studied on a large dataset using multiple evaluation metrics for best practice.
2024-09-06
Information (published)
doi.org
arxiv.org
Pretrainable geometric graph neural network for antibody affinity maturation
Huiyu Cai
Zuobai Zhang
Mingkai Wang
Bozitao Zhong
Yanling Wu
Tianlei Ying
Jian Tang
Quanxiao Li
Yuxuan Zhong
Increasing the binding affinity of an antibody to its target antigen is a crucial task in antibody therapeutics development. This paper pres… (see more)ents a pretrainable geometric graph neural network, GearBind, and explores its potential inin silicoaffinity maturation. Leveraging multi-relational graph construction, multi-level geometric message passing and contrastive pretraining on mass-scale, unlabeled protein structural data, GearBind outperforms previous state-of-the-art approaches on SKEMPI and an independent test set. A powerful ensemble model based on GearBind is then derived and used to successfully enhance the binding of two antibodies with distinct formats and target antigens. ELISA EC50values of the designed antibody mutants are decreased by up to 17 fold, andKDvalues by up to 6.1 fold. These promising results underscore the utility of geometric deep learning and effective pretraining in macromolecule interaction modeling tasks.
2024-09-05
Nature Communications (published)
doi.org
The Strength of Fuel Refueling Location Problem Formulations
Nagisa Sugishita
Margarida Carvalho
Ribal Atallah
2024-09-05
ArXiv (preprint)
arxiv.org
arxiv.org
CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learning
Luke Rowe
Roger Girgis
Anthony Gosselin
Bruno Carrez
Florian Golemo
Felix Heide
Liam Paull
Christopher Pal
Evaluating autonomous vehicle stacks (AVs) in simulation typically involves replaying driving logs from real-world recorded traffic. However… (see more), agents replayed from offline data do not react to the actions of the AV, and their behaviour cannot be easily controlled to simulate counterfactual scenarios. Existing approaches have attempted to address these shortcomings by proposing methods that rely on heuristics or learned generative models of real-world data but these approaches either lack realism or necessitate costly iterative sampling procedures to control the generated behaviours. In this work, we take an alternative approach and propose CtRL-Sim, a method that leverages return-conditioned offline reinforcement learning within a physics-enhanced Nocturne simulator to efficiently generate reactive and controllable traffic agents. Specifically, we process real-world driving data through the Nocturne simulator to generate a diverse offline reinforcement learning dataset, annotated with various reward terms. With this dataset, we train a return-conditioned multi-agent behaviour model that allows for fine-grained manipulation of agent behaviours by modifying the desired returns for the various reward components. This capability enables the generation of a wide range of driving behaviours beyond the scope of the initial dataset, including those representing adversarial behaviours. We demonstrate that CtRL-Sim can efficiently generate diverse and realistic safety-critical scenarios while providing fine-grained control over agent behaviours. Further, we show that fine-tuning our model on simulated safety-critical scenarios generated by our model enhances this controllability.
2024-09-04
Conference on Robot Learning (accepted)
doi.org
proceedings.mlr.press