Automatic Pruning of Fine-tuning Datasets for Transformer-based Language Models
Sayed Mohammadreza Tayaranian Hosseini
Seyyed Hasan Mozafari
Brett H. Meyer
James J. Clark
Warren Gross
Transformer-based language models have shown state-of-the-art performance on a variety of natural language understanding tasks. To achieve t… (voir plus)his performance, these models are first pre-trained on general corpus and then fine-tuned on downstream tasks. Previous work studied the effect of pruning the training set of the downstream tasks on the performance of the model on its evaluation set. In this work, we propose an automatic dataset pruning method for the training set of fine-tuning tasks. Our method is based on the model’s success rate in correctly classifying each training data point. Unlike previous work which relies on user feedback to determine subset size, our method automatically extracts training subsets that are adapted for each pair of model and fine-tuning task. Our method provides multiple subsets for use in dataset pruning that navigate the trade-off between subset size and evaluation accuracy. Our largest subset, which we also refer to as the winning ticket subset, is on average
2025-02-17
Proceedings of The 3rd Conference on Lifelong Learning Agents (publié)
proceedings.mlr.press
BRIGHTER: BRIdging the Gap in Human-Annotated Textual Emotion Recognition Datasets for 28 Languages
Shamsuddeen Hassan Muhammad
Nedjma OUSIDHOUM
Idris Abdulmumin
Jan Philip Wahle
Terry Lima Ruas
Meriem Beloucif
Christine de Kock
Nirmal Surange
Daniela Teodorescu
Ibrahim Ahmad
David Ifeoluwa Adelani
Alham Fikri Aji
Felermino Ali
Ilseyar Alimova
Vladimir Araujo
Nikolay Babakov
Naomi Baes
Ana-Maria Bucur
Andiswa Bukula
Guanqun Cao … (voir 28 de plus)
Rodrigo Tufino Cardenas
Rendi Chevi
Chiamaka Ijeoma Chukwuneke
Alexandra Ciobotaru
Daryna Dementieva
Murja Sani Gadanya
Robert Geislinger
Bela Gipp
Oumaima Hourrane
Oana Ignat
Falalu Lawan
Rooweither Mabuya
Rahmad Mahendra
Vukosi Marivate
Andrew Piper
Alexander Panchenko
Charles Henrique Porto Ferreira
Vitaly Protasov
Samuel Rutunda
Manish Shrivastava
Aura Cristina Udrea
Lilian D. A. Wanzare
Sophie Wu
Florian Valentin Wunderlich
Hanif Muhammad Zhafran
Tianhui Zhang
Yi Zhou
Saif M. Mohammad
2025-02-17
ArXiv (prépublication)
doi.org
arxiv.org
BRIGHTER: BRIdging the Gap in Human-Annotated Textual Emotion Recognition Datasets for 28 Languages
Shamsuddeen Hassan Muhammad
Nedjma OUSIDHOUM
Idris Abdulmumin
Jan Philip Wahle
Terry Lima Ruas
Meriem Beloucif
Christine de Kock
Nirmal Surange
Daniela Teodorescu
Ibrahim Ahmad
David Ifeoluwa Adelani
Alham Fikri Aji
Felermino Ali
Ilseyar Alimova
Vladimir Araujo
Nikolay Babakov
Naomi Baes
Ana-Maria Bucur
Andiswa Bukula
Guanqun Cao … (voir 28 de plus)
Rodrigo Tufino Cardenas
Rendi Chevi
Chiamaka Ijeoma Chukwuneke
Alexandra Ciobotaru
Daryna Dementieva
Murja Sani Gadanya
Robert Geislinger
Bela Gipp
Oumaima Hourrane
Oana Ignat
Falalu Lawan
Rooweither Mabuya
Rahmad Mahendra
Vukosi Marivate
Andrew Piper
Alexander Panchenko
Charles Henrique Porto Ferreira
Vitaly Protasov
Samuel Rutunda
Manish Shrivastava
Aura Cristina Udrea
Lilian D. A. Wanzare
Sophie Wu
Florian Valentin Wunderlich
Hanif Muhammad Zhafran
Tianhui Zhang
Yi Zhou
Saif M. Mohammad
2025-02-17
ArXiv (prépublication)
doi.org
arxiv.org
Channel-Selective Normalization for Label-Shift Robust Test-Time Adaptation
Pedro Vianna
Muawiz Chaudhary
Paria Mehrbod
An Tang
Guy Cloutier
Guy Wolf
Michael Eickenberg
Eugene Belilovsky
Deep neural networks have useful applications in many different tasks, however their performance can be severely affected by changes in the … (voir plus)data distribution. For example, in the biomedical field, their performance can be affected by changes in the data (different machines, populations) between training and test datasets. To ensure robustness and generalization to real-world scenarios, test-time adaptation has been recently studied as an approach to adjust models to a new data distribution during inference. Test-time batch normalization is a simple and popular method that achieved compelling performance on domain shift benchmarks. It is implemented by recalculating batch normalization statistics on test batches. Prior work has focused on analysis with test data that has the same label distribution as the training data. However, in many practical applications this technique is vulnerable to label distribution shifts, sometimes producing catastrophic failure. This presents a risk in applying test time adaptation methods in deployment. We propose to tackle this challenge by only selectively adapting channels in a deep network, minimizing drastic adaptation that is sensitive to label shifts. Our selection scheme is based on two principles that we empirically motivate: (1) later layers of networks are more sensitive to label shift (2) individual features can be sensitive to specific classes. We apply the proposed technique to three classification tasks, including CIFAR10-C, Imagenet-C, and diagnosis of fatty liver, where we explore both covariate and label distribution shifts. We find that our method allows to bring the benefits of TTA while significantly reducing the risk of failure common in other methods, while being robust to choice in hyperparameters.
2025-02-17
Proceedings of The 3rd Conference on Lifelong Learning Agents (publié)
doi.org
arxiv.org
Characterizing co-purchased food products with soda, fresh fruits, and fresh vegetables using loyalty card purchasing data in Montréal, Canada, 2015–2017
Hiroshi Mamiya
Kody Crowell
Catherine L. Mah
Amélie Quesnel-Vallée
Aman Verma
David Buckeridge
2025-02-17
The International Journal of Behavioral Nutrition and Physical Activity (publié)
doi.org
In-Context Parametric Inference: Point or Distribution Estimators?
Sarthak Mittal
Yoshua Bengio
Nikolay Malkin
Guillaume Lajoie
Bayesian and frequentist inference are two fundamental paradigms in statistical estimation. Bayesian methods treat hypotheses as random vari… (voir plus)ables, incorporating priors and updating beliefs via Bayes' theorem, whereas frequentist methods assume fixed but unknown hypotheses, relying on estimators like maximum likelihood. While extensive research has compared these approaches, the frequentist paradigm of obtaining point estimates has become predominant in deep learning, as Bayesian inference is challenging due to the computational complexity and the approximation gap of posterior estimation methods. However, a good understanding of trade-offs between the two approaches is lacking in the regime of amortized estimators, where in-context learners are trained to estimate either point values via maximum likelihood or maximum a posteriori estimation, or full posteriors using normalizing flows, score-based diffusion samplers, or diagonal Gaussian approximations, conditioned on observations. To help resolve this, we conduct a rigorous comparative analysis spanning diverse problem settings, from linear models to shallow neural networks, with a robust evaluation framework assessing both in-distribution and out-of-distribution generalization on tractable tasks. Our experiments indicate that amortized point estimators generally outperform posterior inference, though the latter remain competitive in some low-dimensional problems, and we further discuss why this might be the case.
2025-02-17
ArXiv (prépublication)
doi.org
arxiv.org
In-Context Parametric Inference: Point or Distribution Estimators?
Sarthak Mittal
Yoshua Bengio
Nikolay Malkin
Guillaume Lajoie
Bayesian and frequentist inference are two fundamental paradigms in statistical estimation. Bayesian methods treat hypotheses as random vari… (voir plus)ables, incorporating priors and updating beliefs via Bayes' theorem, whereas frequentist methods assume fixed but unknown hypotheses, relying on estimators like maximum likelihood. While extensive research has compared these approaches, the frequentist paradigm of obtaining point estimates has become predominant in deep learning, as Bayesian inference is challenging due to the computational complexity and the approximation gap of posterior estimation methods. However, a good understanding of trade-offs between the two approaches is lacking in the regime of amortized estimators, where in-context learners are trained to estimate either point values via maximum likelihood or maximum a posteriori estimation, or full posteriors using normalizing flows, score-based diffusion samplers, or diagonal Gaussian approximations, conditioned on observations. To help resolve this, we conduct a rigorous comparative analysis spanning diverse problem settings, from linear models to shallow neural networks, with a robust evaluation framework assessing both in-distribution and out-of-distribution generalization on tractable tasks. Our experiments indicate that amortized point estimators generally outperform posterior inference, though the latter remain competitive in some low-dimensional problems, and we further discuss why this might be the case.
2025-02-17
ArXiv (prépublication)
arxiv.org
arxiv.org
Integrating Present and Past in Unsupervised Continual Learning
Yipeng Zhang
Laurent Charlin
Richard Zemel
Mengye Ren
We formulate a unifying framework for *unsupervised continual learning (UCL)*, which disentangles learning objectives that are specific to t… (voir plus)he present and the past data, encompassing *stability*, *plasticity*, and *cross-task consolidation*. The framework reveals that many existing UCL approaches overlook cross-task consolidation and try to balance plasticity and stability in a shared embedding space. This results in worse performance due to a lack of within-task data diversity and reduced effectiveness in learning the current task. Our method, *Osiris*, which explicitly optimizes all three objectives on separate embedding spaces, achieves state-of-the-art performance on all benchmarks, including two novel ones proposed in this paper featuring semantically structured task sequences. Finally, we show some preliminary evidence that continual models can benefit from such more realistic learning scenarios.
2025-02-17
Proceedings of The 3rd Conference on Lifelong Learning Agents (publié)
proceedings.mlr.press
Intuitive physics understanding emerges from self-supervised pretraining on natural videos
Quentin Garrido
Nicolas Ballas
Mahmoud Assran
Adrien Bardes
Laurent Najman
Michael Rabbat
Emmanuel Dupoux
Yann LeCun
We investigate the emergence of intuitive physics understanding in general-purpose deep neural network models trained to predict masked regi… (voir plus)ons in natural videos. Leveraging the violation-of-expectation framework, we find that video prediction models trained to predict outcomes in a learned representation space demonstrate an understanding of various intuitive physics properties, such as object permanence and shape consistency. In contrast, video prediction in pixel space and multimodal large language models, which reason through text, achieve performance closer to chance. Our comparisons of these architectures reveal that jointly learning an abstract representation space while predicting missing parts of sensory input, akin to predictive coding, is sufficient to acquire an understanding of intuitive physics, and that even models trained on one week of unique video achieve above chance performance. This challenges the idea that core knowledge -- a set of innate systems to help understand the world -- needs to be hardwired to develop an understanding of intuitive physics.
2025-02-17
ArXiv (prépublication)
doi.org
arxiv.org
Intuitive physics understanding emerges from self-supervised pretraining on natural videos
Quentin Garrido
Nicolas Ballas
Mahmoud Assran
Adrien Bardes
Laurent Najman
Michael Rabbat
Emmanuel Dupoux
Yann LeCun
We investigate the emergence of intuitive physics understanding in general-purpose deep neural network models trained to predict masked regi… (voir plus)ons in natural videos. Leveraging the violation-of-expectation framework, we find that video prediction models trained to predict outcomes in a learned representation space demonstrate an understanding of various intuitive physics properties, such as object permanence and shape consistency. In contrast, video prediction in pixel space and multimodal large language models, which reason through text, achieve performance closer to chance. Our comparisons of these architectures reveal that jointly learning an abstract representation space while predicting missing parts of sensory input, akin to predictive coding, is sufficient to acquire an understanding of intuitive physics, and that even models trained on one week of unique video achieve above chance performance. This challenges the idea that core knowledge -- a set of innate systems to help understand the world -- needs to be hardwired to develop an understanding of intuitive physics.
2025-02-17
ArXiv (prépublication)
arxiv.org
arxiv.org
Meta-Analysis with Untrusted Data
Shiva Kaul
Geoff Gordon
Meta-analyses are usually conducted on small amounts of “trusted” data, ideally from randomized, controlled trials. Excluding untrusted … (voir plus)(observational) data — such as medical records and related scientific literature — avoids potential confounding and ensures unbiased conclusions. Unfortunately, this exclusion can reduce predictive accuracy to the point of clinical irrelevance, especially when trials are heterogeneous. This paper shows how untrusted data can be safely incorporated into meta-analysis, improving predictions without sacrificing rigor or introducing unproven assumptions. Our approach, called conformal meta-analysis, consists of (1) learning a (potentially flawed) prior distribution from the untrusted data, (2) using the prior and trusted data to derive a simple, fully-conformal prediction interval for the observed trial effect, and (3) analytically extracting an interval for the true (unobserved) effect. In multiple experiments on healthcare datasets, our algorithms deliver tighter, sounder intervals than traditional ones. This paper conceptually realigns meta-analysis as a foundation for evidence-based medicine, embracing heterogeneity and untrusted data for more nuanced, precise predictions.
2025-02-17
Proceedings of the 4th Machine Learning for Health Symposium (publié)
proceedings.mlr.press
Partial Models for Building Adaptive Model-Based Reinforcement Learning Agents
Safa Alver
Ali Rahimi-Kalahroudi
Doina Precup
In neuroscience, one of the key behavioral tests for determining whether a subject of study exhibits model-based behavior is to study its ad… (voir plus)aptiveness to local changes in the environment. In reinforcement learning, however, recent studies have shown that modern model-based agents display poor adaptivity to such changes. The main reason for this is that modern agents are typically designed to improve sample efficiency in single task settings and thus do not take into account the challenges that can arise in other settings. In local adaptation settings, one particularly important challenge is in quickly building and maintaining a sufficiently accurate model after a local change. This is challenging for deep model-based agents as their models and replay buffers are monolithic structures lacking distribution shift handling capabilities. In this study, we show that the conceptually simple idea of partial models can allow deep model-based agents to overcome this challenge and thus allow for building locally adaptive model-based agents. By modeling the different parts of the state space through different models, the agent can not only maintain a model that is accurate across the state space, but it can also quickly adapt it in the presence of a local change in the environment. We demonstrate this by showing that the use of partial models in agents such as deep Dyna-Q, PlaNet and Dreamer can allow for them to effectively adapt to the local changes in their environments.
2025-02-17
Proceedings of The 3rd Conference on Lifelong Learning Agents (publié)
doi.org
arxiv.org