Publications

Hessian Aware Low-Rank Perturbation for Order-Robust Continual Learning
Jiaqi Li
Rui Wang
Yuanhao Lai
Changjian Shui
Sabyasachi Sahoo
Charles X. Ling
Shichun Yang
Boyu Wang
Christian Gagné
Fan Zhou
Continual learning aims to learn a series of tasks sequentially without forgetting the knowledge acquired from the previous ones. In this wo… (see more)rk, we propose the Hessian Aware Low-Rank Perturbation algorithm for continual learning. By modeling the parameter transitions along the sequential tasks with the weight matrix transformation, we propose to apply the low-rank approximation on the task-adaptive parameters in each layer of the neural networks. Specifically, we theoretically demonstrate the quantitative relationship between the Hessian and the proposed low-rank approximation. The approximation ranks are then globally determined according to the marginal increment of the empirical loss estimated by the layer-specific gradient and low-rank approximation error. Furthermore, we control the model capacity by pruning less important parameters to diminish the parameter growth. We conduct extensive experiments on various benchmarks, including a dataset with large-scale tasks, and compare our method against some recent state-of-the-art methods to demonstrate the effectiveness and scalability of our proposed method. Empirical results show that our method performs better on different benchmarks, especially in achieving task order robustness and handling the forgetting issue. The source code is at https://github.com/lijiaqi/HALRP.
2023-11-25
ArXiv (preprint)
doi.org
arxiv.org
Low Compute Unlearning via Sparse Representations
Vedant Shah
Frederik Träuble
Ashish Malik
Hugo Larochelle
Michael Curtis Mozer
Sanjeev Arora
Yoshua Bengio
Anirudh Goyal
Machine unlearning, which involves erasing knowledge about a \emph{forget set} from a trained model, can prove to be costly and infeasible … (see more)using existing techniques. We propose a low-compute unlearning technique based on a discrete representational bottleneck. We show that the proposed technique efficiently unlearns the forget set and incurs negligible damage to the model's performance on the rest of the dataset. We evaluate the proposed technique on the problem of class unlearning using four datasets: CIFAR-10, CIFAR-100, LACUNA-100 and ImageNet-1k. We compare the proposed technique to SCRUB, a state-of-the-art approach which uses knowledge distillation for unlearning. Across all four datasets, the proposed technique performs as well as, if not better than SCRUB while incurring almost no computational cost.
2023-11-25
arXiv (preprint)
doi.org
openreview.net
Gaining Biological Insights through Supervised Data Visualization
Jake S. Rhodes
Adrien Aumon
Sacha Morin
Marc Girard
Catherine Larochelle
Boaz Lahav
Elsa Brunet-Ratnasingham
Amélie Pagliuzza
Lorie Marchitto
Wei Zhang
Adele Cutler
Francois Grand’Maison
Anhong Zhou
Andrés Finzi
Nicolas Chomont
Daniel E. Kaufmann
Stephanie Zandee
Alexandre Prat
Guy Wolf
Kevin R. Moon
Dimensionality reduction-based data visualization is pivotal in comprehending complex biological data. The most common methods, such as PHAT… (see more)E, t-SNE, and UMAP, are unsupervised and therefore reflect the dominant structure in the data, which may be independent of expert-provided labels. Here we introduce a supervised data visualization method called RF-PHATE, which integrates expert knowledge for further exploration of the data. RF-PHATE leverages random forests to capture intricate featurelabel relationships. Extracting information from the forest, RF-PHATE generates low-dimensional visualizations that highlight relevant data relationships while disregarding extraneous features. This approach scales to large datasets and applies to classification and regression. We illustrate RF-PHATE’s prowess through three case studies. In a multiple sclerosis study using longitudinal clinical and imaging data, RF-PHATE unveils a sub-group of patients with non-benign relapsingremitting Multiple Sclerosis, demonstrating its aptitude for time-series data. In the context of Raman spectral data, RF-PHATE effectively showcases the impact of antioxidants on diesel exhaust-exposed lung cells, highlighting its proficiency in noisy environments. Furthermore, RF-PHATE aligns established geometric structures with COVID-19 patient outcomes, enriching interpretability in a hierarchical manner. RF-PHATE bridges expert insights and visualizations, promising knowledge generation. Its adaptability, scalability, and noise tolerance underscore its potential for widespread adoption.
2023-11-22
bioRxiv (preprint)
doi.org
Mitigating Shortcut Learning with Diffusion Counterfactuals and Diverse Ensembles
Luca Scimeca
Alexander Rubinstein
Damien Teney
Seong Joon Oh
Armand Mihai Nicolicioiu
Yoshua Bengio
Spurious correlations in the data, where multiple cues are predictive of the target labels, often lead to a phenomenon known as shortcut lea… (see more)rning, where a model relies on erroneous, easy-to-learn cues while ignoring reliable ones. In this work, we propose
2023-11-22
ArXiv (preprint)
openreview.net
openreview.net
Propositional Logics for the Lawvere Quantale
Giorgio Bacci
Radu Mardare
Prakash Panangaden
Gordon Plotkin
2023-11-22
Electronic Notes in Theoretical Informatics and Computer Science (published)
doi.org
arxiv.org
scSniper: Single-cell Deep Neural Network-based Identification of Prominent Biomarkers
Mingyang Li
Yanshuo Chen
Jun Ding
2023-11-22
bioRxiv (preprint)
doi.org
Unveiling the Impact of Arsenic Toxicity on Immune Cells in Atherosclerotic Plaques: Insights from Single-Cell Multi-Omics Profiling
Kiran Makhani
Xiuhui Yang
France Dierick
Nivetha Subramaniam
Natascha Gagnon
Talin Ebrahimian
Hao Wu
Jun Ding
Koren K. Mann
2023-11-22
bioRxiv (preprint)
doi.org
Deep Learning Model for Multi-Step Ahead Prediction of Solar Irradiance: Case of Study of Morocco
Saad Benbrahim
Loubna Benabbou
Hanane Dagdougui
Ismail Belhaj
Abdelali Djdiaa
Hicham Bouzekri
Abdelaziz Berrado
Accurate solar irradiance forecasting is crucial for managing energy generation and consumption in the rapidly evolving landscape of renewab… (see more)le energy. It enables renewable energy operators to make informed decisions and maximize their output. This study employs deep learning-based forecasting models to predict the Global Horizontal Irradiance (GHI) of the R&D platform situated in Ouarzazate, Morocco. A sensitivity analysis was conducted on multiple scenarios for a one day-ahead horizon. Moreover, a forecasting technique that encompasses numerous horizons, ranging from one day to three days in advance, was evaluated. The study's findings suggest that the encoder-decoder model we proposed exhibited superior performance compared to the other models tested and produced dependable predictions.
2023-11-21
2023 14th International Conference on Intelligent Systems: Theories and Applications (SITA) (published)
doi.org
Towards an Effective Electrical Market Design: Identifying and Defining Key Criteria for Decision-Making
Souhaila Chiguer
Loubna Benabbou
Hanane Dagdougui
Ismail Belhaj
Abdelali Djdiaa
Hicham Bouzekri
Abdelaziz Berrado
In our changing energy landscape, electricity is taking a major role in achieving decarbonization goals. Electricity can be a clean and effi… (see more)cient source of energy, and it is well-suited to help countries meet their climate goals. However, the electrical market is complex and constantly evolving, and it is important to carefully choose the design elements of the market to ensure that it is meeting its objectives. In this context, evaluating an electrical market's effectiveness requires a multifaceted approach that takes into account a range of elements, from environmental impact to economic viability. This paper provides an overview of several evaluation methods for different objectives to finally select the key criteria to consider in assisting decision-makers, regulators, and stakeholders in developing an electricity market that is not only effective but also reliable and sustainable.
2023-11-21
2023 14th International Conference on Intelligent Systems: Theories and Applications (SITA) (published)
doi.org
Responses to pattern-violating visual stimuli evolve differently over days in somata and distal apical dendrites
Colleen J. Gillon
Jason E. Pina
Jérôme A. Lecoq
Ruweida Ahmed
Yazan N. Billeh
Shiella Caldejon
Peter Groblewski
Timothy M. Henley
India Kato
Eric Lee
Jennifer Luviano
Kyla Mace
Chelsea Nayan
Thuyanh V. Nguyen
Kat North
Jed Perkins
Sam Seid
Matthew T. Valley
Ali Williford
Yoshua Bengio … (see 3 more)
Timothy P. Lillicrap
Blake A. Richards
Joel Zylberberg
Scientists have long conjectured that the neocortex learns patterns in sensory data to generate top-down predictions of upcoming stimuli. In… (see more) line with this conjecture, different responses to pattern-matching vs pattern-violating visual stimuli have been observed in both spiking and somatic calcium imaging data. However, it remains unknown whether these pattern-violation signals are different between the distal apical dendrites, which are heavily targeted by top-down signals, and the somata, where bottom-up information is primarily integrated. Furthermore, it is unknown how responses to pattern-violating stimuli evolve over time as an animal gains more experience with them. Here, we address these unanswered questions by analyzing responses of individual somata and dendritic branches of layer 2/3 and layer 5 pyramidal neurons tracked over multiple days in primary visual cortex of awake, behaving female and male mice. We use sequences of Gabor patches with patterns in their orientations to create pattern-matching and pattern-violating stimuli, and two-photon calcium imaging to record neuronal responses. Many neurons in both layers show large differences between their responses to pattern-matching and pattern-violating stimuli. Interestingly, these responses evolve in opposite directions in the somata and distal apical dendrites, with somata becoming less sensitive to pattern-violating stimuli and distal apical dendrites more sensitive. These differences between the somata and distal apical dendrites may be important for hierarchical computation of sensory predictions and learning, since these two compartments tend to receive bottom-up and top-down information, respectively.
2023-11-20
Journal of Neuroscience (published)
doi.org
A Study of Human-Robot Handover through Human-Human Object Transfer
Charlotte Morissette
Bobak H. Baghi
Francois Hogan
Gregory Dudek
In this preliminary study, we investigate changes in handover behaviour when transferring hazardous objects with the help of a high-resoluti… (see more)on touch sensor. Participants were asked to hand over a safe and hazardous object (a full cup and an empty cup) while instrumented with a modified STS sensor. Our data shows a clear distinction in the length of handover for the full cup vs the empty one, with the former being slower. Sensor data further suggests a change in tactile behaviour dependent on the object's risk factor. The results of this paper motivate a deeper study of tactile factors which could characterize a risky handover, allowing for safer human-robot interactions in the future.
2023-11-20
ArXiv (preprint)
doi.org
arxiv.org
Challenging Common Assumptions About Catastrophic Forgetting and Knowledge Accumulation
Timothee LESORT
Oleksiy Ostapenko
Pau Rodríguez
Diganta Misra
Md Rifat Arefin
Laurent Charlin
Irina Rish
Building learning agents that can progressively learn and accumulate knowledge is the core goal of the continual learning (CL) research fiel… (see more)d. Unfortunately, training a model on new data usually compromises the performance on past data. In the CL literature, this effect is referred to as catastrophic forgetting (CF). CF has been largely studied, and a plethora of methods have been proposed to address it on short sequences of non-overlapping tasks. In such setups, CF always leads to a quick and significant drop in performance in past tasks. Nevertheless, despite CF, recent work showed that SGD training on linear models accumulates knowledge in a CL regression setup. This phenomenon becomes especially visible when tasks reoccur. We might then wonder if DNNs trained with SGD or any standard gradient-based optimization accumulate knowledge in such a way. Such phenomena would have interesting consequences for applying DNNs to real continual scenarios. Indeed, standard gradient-based optimization methods are significantly less computationally expensive than existing CL algorithms. In this paper, we study the progressive knowledge accumulation (KA) in DNNs trained with gradient-based algorithms in long sequences of tasks with data re-occurrence. We propose a new framework, SCoLe (Scaling Continual Learning), to investigate KA and discover that catastrophic forgetting has a limited effect on DNNs trained with SGD. When trained on long sequences with data sparsely re-occurring, the overall accuracy improves, which might be counter-intuitive given the CF phenomenon. We empirically investigate KA in DNNs under various data occurrence frequencies and propose simple and scalable strategies to increase knowledge accumulation in DNNs.
2023-11-19
Proceedings of The 2nd Conference on Lifelong Learning Agents (published)
doi.org
proceedings.mlr.press