Publications

Distilling Privileged Multimodal Information for Expression Recognition using Optimal Transport
Muhammad Haseeb Aslam
Muhammad Osama Zeeshan
Soufiane Belharbi
Marco Pedersoli
Alessandro Lameiras Koerich
Simon Bacon
Eric Granger
Deep learning models for multimodal expression recognition have reached remarkable performance in controlled laboratory environments because… (voir plus) of their ability to learn complementary and redundant semantic information. However, these models struggle in the wild, mainly because of the unavailability and quality of modalities used for training. In practice, only a subset of the training-time modalities may be available at test time. Learning with privileged information enables models to exploit data from additional modalities that are only available during training. State-of-the-art knowledge distillation (KD) methods have been proposed to distill information from multiple teacher models (each trained on a modality) to a common student model. These privileged KD methods typically utilize point-to-point matching, yet have no explicit mechanism to capture the structural information in the teacher representation space formed by introducing the privileged modality. We argue that encoding this same structure in the student space may lead to enhanced student performance. This paper introduces a new structural KD mechanism based on optimal transport (OT), where entropy-regularized OT distills the structural dark knowledge. Our privileged KD with OT (PKDOT) method captures the local structures in the multimodal teacher representation by calculating a cosine similarity matrix and selecting the top-k anchors to allow for sparse OT solutions, resulting in a more stable distillation process. Experiments1 were performed on two challenging problems - pain estimation on the Biovid dataset (ordinal classification) and arousal-valance prediction on the Affwild2 dataset (regression). Results show that our proposed method can outperform state-of-the-art privileged KD methods on these problems. The diversity among modalities and fusion architectures indicates that PKDOT is modality-and model-agnostic.
2024-05-26
2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG) (publié)
doi.org
arxiv.org
Estimating Expectations without Sampling: Neural Stein Estimation
Mohsin Hasan
Dinghuai Zhang
Cheikh Ahmed
Awa Khouna
Yoshua Bengio
We propose a method for estimating the expected value of a given function …
2024-05-26
approximateinference.org/AABI/2024/Symposium (accepté)
openreview.net
openreview.net
Guided Interpretable Facial Expression Recognition via Spatial Action Unit Cues
Soufiane Belharbi
Marco Pedersoli
Alessandro Lameiras Koerich
Simon Bacon
Eric Granger
Although state-of-the-art classifiers for facial expression recognition (FER) can achieve a high level of accuracy, they lack interpretabili… (voir plus)ty, an important feature for end-users. Experts typically associate spatial action units (AUs) from a codebook to facial regions for the visual interpretation of expressions. In this paper, the same expert steps are followed. A new learning strategy is proposed to explicitly incorporate AU cues into classifier training, allowing to train deep interpretable models. During training, this AU codebook is used, along with the input image expression label, and facial landmarks, to construct a AU heatmap that indicates the most discriminative image regions of interest w.r.t the facial expression. This valuable spatial cue is leveraged to train a deep interpretable classifier for FER. This is achieved by constraining the spatial layer features of a classifier to be correlated with AU heatmaps. Using a composite loss, the classifier is trained to correctly classify an image while yielding interpretable visual layer-wise attention correlated with AU maps, simulating the expert decision process. Our strategy only relies on image class expression for supervision, without additional manual annotations. Our new strategy is generic, and can be applied to any deep CNN - or transformer-based classifier without requiring any architectural change or significant additional training time. Our extensive evaluation11Our code is available at:https://github.com/sbelharbi/interpretable-fer-aus. on two public benchmarks RAF-DB, and AffectNet datasets shows that our proposed strategy can improve layer-wise interpretability without degrading classification performance. In addition, we explore a common type of interpretable classifiers that rely on class activation mapping (CAM) methods, and show that our approach can also improve CAM interpretability.
2024-05-26
2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG) (publié)
doi.org
arxiv.org
An Introduction to Vision-Language Modeling
Florian Bordes
Richard Yuanzhe Pang
Anurag Ajay
Alexander C. Li
Adrien Bardes
Suzanne Petryk
Oscar Mañas
Zhiqiu Lin
Anas Mahmoud
Bargav Jayaraman
Mark Ibrahim
Melissa Hall
Yunyang Xiong
Jonathan Lebensold
Candace Ross
Srihari Jayakumar
Chuan Guo
Diane Bouchacourt
Haider Al-Tahan
Karthik Padthe … (voir 22 de plus)
Vasu Sharma
Huijuan Xu 0001
Hu Xu
Xiaoqing Ellen Tan
Megan Richards
Samuel Lavoie
Pietro Astolfi
Reyhane Askari Hemmat
Jun Chen
Kushal Tirumala
Rim Assouel
Mazda Moayeri
Arjang Talattof
Kamalika Chaudhuri
Zechun Liu
Xilun Chen
Quentin Garrido
Karen Ullrich
Aishwarya Agrawal
Kate Saenko
Asli Celikyilmaz
Vikas Chandra
2024-05-26
arXiv (prépublication)
doi.org
arxiv.org
Neuro-GPT: Towards A Foundation Model for EEG
Wenhui Cui
Woojae Jeong
Philipp Thölke
Takfarinas Medani
Karim Jerbi CoCo Lab
Anand A. Joshi
Richard M. Leahy
To handle the scarcity and heterogeneity of electroencephalography (EEG) data for Brain-Computer Interface (BCI) tasks, and to harness the p… (voir plus)ower of large publicly available data sets, we propose Neuro-GPT, a foundation model consisting of an EEG encoder and a GPT model. The foundation model is pre-trained on a large-scale data set using a self-supervised task that learns how to reconstruct masked EEG segments. We then fine-tune the model on a Motor Imagery Classification task to validate its performance in a low-data regime (9 subjects). Our experiments demonstrate that applying a foundation model can significantly improve classification performance compared to a model trained from scratch, which provides evidence for the generalizability of the foundation model and its ability to address challenges of data scarcity and heterogeneity in EEG. The code is publicly available at github.com/wenhui0206/NeuroGPT.
2024-05-26
2024 IEEE International Symposium on Biomedical Imaging (ISBI) (publié)
doi.org
arxiv.org
Phenospectral similarity as an index of ecological integrity
Patrick Osei Darko
Étienne Laliberté
Margaret Kalácska
J. Pablo Arroyo-Mora
Andrew Gonzalez
Juan Zuloaga
In collaboration with the International Union for the Conservation of Nature (IUCN) Taskforce on Biodiversity and Protected Areas, countries… (voir plus) worldwide are working to develop a new systematic approach to inform the Key Biodiversity Areas (KBAs) initiative. The goal is to map KBAs from the national to global scales with a baseline international standard in support of biodiversity conservation efforts. According to the IUCN standard, one of the five criteria used to identify potential KBAs, is the Ecological Integrity (EI) of the ecosystem. Sites identified with respect to EI must have an intact ecological community and be characterized by minimal anthropogenic disturbance. In this study, a new EI metric, phenospectral similarity (PSpecM), has been developed and implemented in Google Earth Engine to identify potential forest stands of high EI from a large set of candidate stands. The implementation of PSpecM requires a network of known reference sites of high EI and target ecological units of the same land cover type for comparison to help identify potential sites of high EI. Here, we tested PSpecM on a ∼12,000 km2 study area in the Laurentian region, Quebec, Canada, using Sentinel-2 and PlanetScope (Dove) satellite imagery. Considering the phenological effect on reflectance, we found a 2,700 km2 spatial extent, equivalent to approximately 22% of the study area, commonly delineated as potential areas of high EI by both PlanetScope (Dove) and Sentinel-2. Without consideration of phenology, the total area delineated as potential areas of high EI increased to 5,505 km2, equivalent to around 45% of the study area. Our results show that PSpecM can be computed for rapid assessments of forest stands to identify potential areas of high EI on a large geographic scale and serve as an additional conservation tool that can be applied to the ongoing global and national identification of KBAs.
2024-05-26
Frontiers in Environmental Science (publié)
doi.org
Subject-Based Domain Adaptation for Facial Expression Recognition
Muhammad Osama Zeeshan
Muhammad Haseeb Aslam
Soufiane Belharbi
Alessandro Lameiras Koerich
Marco Pedersoli
Simon Bacon
Eric Granger
Adapting a deep learning model to a specific target individual is a challenging facial expression recognition (FER) task that may be achieve… (voir plus)d using unsupervised domain adaptation (UDA) methods. Although several UDA methods have been proposed to adapt deep FER models across source and target data sets, multiple subject-specific source domains are needed to accurately represent the intra-and inter-person variability in subject-based adaption. This paper considers the setting where domains correspond to individuals, not entire datasets. Unlike UDA, multi-source domain adaptation (MSDA) methods can leverage multiple source datasets to improve the accuracy and robustness of the target model. However, previous methods for MSDA adapt image classification models across datasets and do not scale well to a more significant number of source domains. This paper introduces a new MSDA method for subject-based domain adaptation in FER. It efficiently leverages information from multiple source subjects (labeled source domain data) to adapt a deep FER model to a single target individual (unlabeled target domain data). During adaptation, our subject-based MSDA first computes a between-source discrepancy loss to mitigate the domain shift among data from several source subjects. Then, a new strategy is employed to generate augmented confident pseudo-labels for the target subject, allowing a reduction in the domain shift between source and target subjects. Experiments1 performed on the challenging BioVid heat and pain dataset with 87 subjects and the UNBC-McMaster shoulder pain dataset with 25 subjects show that our subject-based MSDA can outperform state-of-the-art methods yet scale well to multiple subject-based source domains.
2024-05-26
2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG) (publié)
doi.org
arxiv.org
Reproducibility Study on Adversarial Attacks Against Robust Transformer Trackers
Fatemeh Nourilenjan Nokabadi
Jean-François Lalonde
Christian Gagné
New transformer networks have been integrated into object tracking pipelines and have demonstrated strong performance on the latest benchmar… (voir plus)ks. This paper focuses on understanding how transformer trackers behave under adversarial attacks and how different attacks perform on tracking datasets as their parameters change. We conducted a series of experiments to evaluate the effectiveness of existing adversarial attacks on object trackers with transformer and non-transformer backbones. We experimented on 7 different trackers, including 3 that are transformer-based, and 4 which leverage other architectures. These trackers are tested against 4 recent attack methods to assess their performance and robustness on VOT2022ST, UAV123 and GOT10k datasets. Our empirical study focuses on evaluating adversarial robustness of object trackers based on bounding box versus binary mask predictions, and attack methods at different levels of perturbations. Interestingly, our study found that altering the perturbation level may not significantly affect the overall object tracking results after the attack. Similarly, the sparsity and imperceptibility of the attack perturbations may remain stable against perturbation level shifts. By applying a specific attack on all transformer trackers, we show that new transformer trackers having a stronger cross-attention modeling achieve a greater adversarial robustness on tracking datasets, such as VOT2022ST and GOT10k. Our results also indicate the necessity for new attack methods to effectively tackle the latest types of transformer trackers. The codes necessary to reproduce this study are available at https://github.com/fatemehN/ReproducibilityStudy.
2024-05-25
TMLR (accepté)
doi.org
openreview.net
An Improved Neuro-Symbolic Architecture to Fine-Tune Generative AI Systems
Chao Yin
Quentin Cappart
Gilles Pesant
2024-05-24
Integration of Constraint Programming, Artificial Intelligence, and Operations Research (publié)
doi.org
Towards a Generic Representation of Combinatorial Problems for Learning-Based Approaches
Léo Boisvert
Hélène Verhaeghe
Quentin Cappart
2024-05-24
Integration of Constraint Programming, Artificial Intelligence, and Operations Research (publié)
doi.org
arxiv.org
Manifold Metric: A Loss Landscape Approach for Predicting Model Performance
Pranshu Malviya
Jerry Huang
Quentin Fournier
A. Chandar
Determining the optimal model for a given task often requires training multiple models from scratch, which becomes impractical as dataset an… (voir plus)d model sizes grow. A more efficient alternative is to expand smaller pre-trained models, but this approach is underutilized due to a limited understanding of its impact on the training dynamics. Existing methods for quantifying this impact have notable limitations, including computation cost. To address this, we introduce a new perspective based on the loss landscape, which has been shown to contain a manifold of linearly connected minima. Specifically, we propose a metric that estimates the size of this manifold to study the impact of model expansion. Our experiments reveal a strong correlation between performance gains and our manifold metric, enabling more informed model comparison and offering a first step toward a geometry-driven approach for reliable model expansion. Notably, our metric outperforms other baselines, even when different types of expansion with equivalent number of parameters are applied to a model.
2024-05-23
ArXiv (prépublication)
arxiv.org
arxiv.org
Neural computations in prosopagnosia
Simon Faghel-Soubeyrand
Anne-Raphaelle Richoz
Delphine Waeber
Jessica Woodhams
Frédéric Gosselin
Roberto Caldara
Ian Charest
We aimed to identify neural computations underlying the loss of face identification ability by modelling the brain activity of brain-lesione… (voir plus)d patient PS, a well-documented case of acquired pure prosopagnosia. We collected a large dataset of high-density electrophysiological (EEG) recordings from PS and neurotypicals while they completed a one-back task on a stream of face, object, animal and scene images. We found reduced neural decoding of face identity around the N170 window in PS, and conjointly revealed normal non-face identification in this patient. We used Representational Similarity Analysis (RSA) to correlate human EEG representations with those of deep neural network (DNN) models of vision and caption-level semantics, offering a window into the neural computations at play in patient PS’s deficits. Brain representational dissimilarity matrices (RDMs) were computed for each participant at 4 ms steps using cross-validated classifiers. PS’s brain RDMs showed significant reliability across sessions, indicating meaningful measurements of brain representations with RSA even in the presence of significant lesions. Crucially, computational analyses were able to reveal PS’s representational deficits in high-level visual and semantic brain computations. Such multi-modal data-driven characterisations of prosopagnosia highlight the complex nature of processes contributing to face recognition in the human brain. Highlights We assess the neural computations in the prosopagnosic patient PS using EEG, RSA, and deep neural networks Neural dynamics of brain-lesioned PS are reliably captured using RSA Neural decoding shows normal evidence for non-face individuation in PS Neural decoding shows abnormal neural evidence for face individuation in PS PS shows impaired high-level visual and semantic neural computations
2024-05-23
Cerebral Cortex (New York, NY) (publié)
doi.org