Portrait de Olivier Lefebvre n'est pas disponible

Olivier Lefebvre

Alumni

Publications

Diagnosing Low-Confidence Patient Profiles via Localized Harmful Covariate Shift Detection
Olivier Lefebvre
Hakima Laribi
Dan Poenaru
Martin Vallières
Before deploying clinical prediction models, their reliability must be assessed to ensure safe use. Errors may arise from distribution shift… (voir plus)s between training and deployment or from heterogeneous performance where the model systematically underperforms for certain patient subgroups. Yet most post-hoc diagnostic methods evaluate reliability only at the population level, checking whether the deployment data resemble the training data and whether the model performs well on average, without accounting for heterogeneity that may cause some patients to be disproportionately affected. We propose a unified diagnostic pipeline that combines both perspectives. MED3pa first discovers interpretable low-confidence patient profiles, and Detectron is then applied within each profile to test whether its degradation is associated with a detectable harmful covariate shift. Across three clinical prediction tasks and three injected subgroup shifts in MIMIC-IV, we find that shift detectability itself is informative. An injected age shift, undetectable at the population level, was nonetheless localized to specific degraded profiles, whereas sex- and race-related shifts were detected broadly and therefore offered little guidance for identifying the most affected patients.
2026-10-12
arXiv (prépublication)
doi.org
arxiv.org
Predictive Performance Precision Analysis in Medicine: Identification of low-confidence predictions at patient and profile levels (MED3pa I)
Olivier Lefebvre
Félix Camirand Lemyre
Jean-François Ethier
Lyna Hiba Chikouche
Ludmila Amriou
Dan Poenaru
Martin Vallières
Artificial Intelligence models are increasingly used in healthcare, yet global performance metrics can mask variations in reliability across… (voir plus) individual patients or subgroups with shared attributes, called patient profiles . This study introduces MED3pa, a method that identifies when models are less reliable, allowing clinicians to better assess model limitations. We propose a framework that estimates predictive confidence using three combined approaches: Individualized (IPC), Aggregated (APC), and Mixed Predictive Confidence (MPC). IPC estimates confidence for each patient, APC assesses it across profiles, and MPC combines both. We evaluate our method on four datasets: one simulated, two public, and one private clinical dataset. Metrics by Declaration Rate (MDR) curves show how performance changes when retaining only the most confident predictions, while interpretable decision trees reveal profiles with higher or lower model confidence. We demonstrate our method in internal, temporal, and external validation settings, as well as through a clinical example. In internal validation, limiting predictions to the 93% most confident cases improved sensitivity by 14.3% and the AUC by 5.1%. In the clinical example, MED3pa identified a patient profile with high misclassification risk, demonstrating its potential for safer deployment. By identifying low-confidence predictions, our framework improves model reliability in clinical settings. It can be integrated into decision support systems to help clinicians make more informed decisions. Confidence thresholds help balance model performance with the proportion of patients for whom predictions are considered reliable. Better leveraging confidence in model predictions could improve reliability and trustworthiness, supporting safer and more effective use in healthcare.
2025-08-25
medRxiv (prépublication)
doi.org