Publications

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Hugo Laurençon
Lucile Saulnier
Thomas Wang
Christopher Akiki
Albert Villanova del Moral
Teven Le Scao
Leandro Von Werra
Chenghao Mou
Eduardo González Ponferrada
Huu Nguyen
Jörg Frohberg
Mario Šaško
Quentin Lhoest
Angelina McMillan-Major
Gérard Dupont
Stella Biderman
Anna Rogers
Loubna Ben allal
Francesco De Toni
Giada Pistilli … (voir 34 de plus)
Olivier Nguyen
Somaieh Nikpoor
Maraim Masoud
Pierre Colombo
Javier de la Rosa
Paulo Villegas
Tristan Thrush
Shayne Longpre
Sebastian Nagel
Leon Weber
Manuel Romero Muñoz
Jian Zhu
Daniel Van Strien
Zaid Alyafeai
Khalid Almubarak
Vu Minh Chien
Itziar Gonzalez-Dios
Aitor Soroa
Kyle Lo
Manan Dey
Pedro Ortiz Suarez
Aaron Gokaslan
Shamik Bose
David Ifeoluwa Adelani
Long Phan
Hieu Tran
Ian Yu
Suhas Pai
Jenny Chim
Violette Lepercq
Suzana Ilic
Margaret Mitchell
Sasha Luccioni
Yacine Jernite
As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multili… (voir plus)ngual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.
2021-12-31
Advances in Neural Information Processing Systems 35 (NeurIPS 2022) (publié)
doi.org
openreview.net
On the Convergence of Stochastic Extragradient for Bilinear Games with Restarted Iteration Averaging
Chris Junchi Li
Yaodong Yu
Nicolas Loizou
Gauthier Gidel
Yitong Ma
Nicolas Roux
Michael I. Jordan
We study the stochastic bilinear minimax optimization problem, presenting an analysis of the same-sample Stochastic ExtraGradient (SEG) meth… (voir plus)od with constant step size, and presenting variations of the method that yield favorable convergence. In sharp contrasts with the basic SEG method whose last iterate only contracts to a fixed neighborhood of the Nash equilibrium, SEG augmented with iteration averaging provably converges to the Nash equilibrium under the same standard settings, and such a rate is further improved by incorporating a scheduled restarting procedure. In the interpolation setting where noise vanishes at the Nash equilibrium, we achieve an optimal convergence rate up to tight constants. We present numerical experiments that validate our theoretical findings and demonstrate the effectiveness of the SEG method when equipped with iteration averaging and restarting.
2021-12-31
AISTATS (publié)
arxiv.org
arxiv.org
The Curious Case of Absolute Position Embeddings
Koustuv Sinha
Amirhossein Kazemnejad
Siva Reddy
Joelle Pineau
Dieuwke Hupkes
Adina Williams
Transformer language models encode the notion of word order using positional information. Most commonly, this positional information is repr… (voir plus)esented by absolute position embeddings (APEs), that are learned from the pretraining data. However, in natural language, it is not absolute position that matters, but relative position, and the extent to which APEs can capture this type of information has not been investigated. In this work, we observe that models trained with APE over-rely on positional information to the point that they break-down when subjected to sentences with shifted position information. Specifically, when models are subjected to sentences starting from a non-zero position (excluding the effect of priming), they exhibit noticeably degraded performance on zero to full-shot tasks, across a range of model families and model sizes. Our findings raise questions about the efficacy of APEs to model the relativity of position information, and invite further introspection on the sentence and word order processing strategies employed by these models.
2021-12-31
EMNLP (Findings) (publié)
doi.org
arxiv.org
On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?
Nouha Dziri
Sivan Milton
Mo Yu
Osmar Zaiane
Siva Reddy
Knowledge-grounded conversational models are known to suffer from producing factually invalid statements, a phenomenon commonly called hallu… (voir plus)cination. In this work, we investigate the underlying causes of this phenomenon: is hallucination due to the training data, or to the models? We conduct a comprehensive human study on both existing knowledge-grounded conversational benchmarks and several state-of-the-art models. Our study reveals that the standard benchmarks consist of >60% hallucinated responses, leading to models that not only hallucinate but even amplify hallucinations. Our findings raise important questions on the quality of existing datasets and models trained using them. We make our annotations publicly available for future research.
2021-12-31
arXiv (prépublication)
doi.org
arxiv.org
The Role of Robotics in Achieving the United Nations Sustainable Development Goals - The Experts' Meeting at the 2021 IEEE/RSJ IROS Workshop [Industry Activities].
Vincent Mai
Bram Vanderborght
Tamás Haidegger
Alaa M. Khamis
Niraj Bhargava
Dominik B. O. Boesl
Katleen Gabriels
An Jacobs
AJung Moon
Robin R. Murphy
Yasushi Nakauchi
Edson Prestes
Bram Vanderborght
Ricardo Vinuesa
Carl-Maria Mörch
2021-12-31
IEEE Robotics Autom. Mag. (publié)
doi.org
The Secret to Better AI and Better Software (Is Requirements Engineering)
Nelly Bencomo
Jin L.C. Guo
Rachel Harrison
Hans-Martin Heyn
Tim Menzies
Recently, practitioners and researchers met to discuss the role of requirements, and AI and SE. We offer here notes on that fascinating disc… (voir plus)ussion. Also, have you considered writing for this column? This “SE for AI” column publishes commentaries on the growing field of SE for AI. Submissions are welcomed and encouraged (1,000–2,400 words, each figure and table counts as 250 words, try to use fewer than 12 references, and keep the discussion practitioner focused). Please submit your ideas to me at timm@ieee.org.—Tim Menzies
2021-12-31
IEEE Software (publié)
doi.org
There is no fundamental trade-off between prediction accuracy and feature importance reliability
Jianzhong Chen
L.Q.R. Ooi
Jingwei Li
L. Christopher
Asplund
Simon B. Eickhoff
Danilo Bzdok
Avram J. Holmes
Blake T. Thomas
Yeo
There is significant interest in using neuroimaging data to predict behavior. The predictive models are often interpreted by the computation… (voir plus) of feature importance, which quantifies the predictive relevance of an imaging feature. Tian and Zalesky (2021) suggest that feature importance estimates exhibit low test-retest reliability, pointing to a potential trade-off between prediction accuracy and feature importance reliability. This trade-off is counter-intuitive because both prediction accuracy and test-retest reliability reflect the reliability of brain-behavior relationships across independent samples. Here, we revisit the relationship between prediction accuracy and feature importance reliability in a large well-powered dataset across a wide range of behavioral measures. We demonstrate that, with a sufficient sample size, feature importance (operationalized as Haufe-transformed weights) can achieve fair to excellent test-retest reliability. More specifically, with a sample size of about 2600 participants, Haufe-transformed weights achieve average intra-class correlation coefficients of 0.75, 0.57 and 0.53 for cognitive, personality and mental health measures respectively. Haufe-transformed weights are much more reliable than original regression weights and univariate FC-behavior correlations. Intriguingly, feature importance reliability is strongly positively correlated with prediction accuracy across phenotypes. Within a particular behavioral domain, there was no clear relationship between prediction performance and feature importance reliability across regression algorithms. Finally, we show mathematically that feature importance reliability is necessary, but not sufficient, for low feature importance error. In the case of linear models, lower feature importance error leads to lower prediction error (up to a scaling by the feature covariance matrix). Overall, we find no fundamental trade-off between feature importance reliability and prediction accuracy.
2021-12-31
(publié)
www.semanticscholar.org
TIML: Task-Informed Meta-Learning for crop type mapping
Gabriel Tseng
Hannah Kerner
David Rolnick
Labeled datasets for agriculture are extremely spatially imbalanced. When developing algorithms for data-sparse regions, a previously explor… (voir plus)ed approach is to use transfer learning from data-rich regions. While standard transfer learning approaches typically leverage only direct inputs and outputs, geospatial imagery and agricultural data is rich in metadata that can inform transfer learning algorithms, such as the spatial coordinates of data-points. We build on previous work exploring use of meta-learning to crop type mapping in data-sparse regions and introduce task-informed meta-learning (TIML), an augmentation to model-agnostic meta-learning which takes advantage of this metadata. We apply TIML to the CropHarvest dataset, a global dataset of agricultural class labels paired with remote sensing data. In addition, we introduce the concept of forgetfulness when training meta-learning models on many similar tasks to mitigate memorization of training tasks. We find that TIML significantly improves average performance across the CropHarvest evaluation tasks compared to a range of benchmark models, measured using AUC ROC and F1 scores.
2021-12-31
AAAI.org/2022/Workshop/AIAFS (publié)
openreview.net
openreview.net
Title: Functional architecture of the aging brain
Roni Setton
Laetitia Mwilambwe-Tshilobo
Manesh Girn
Amber W. Lockrow
Giulia Baracchini
Alexander J. Lowe
Benjamin N. Cassidy
Jian Li
Wen-Ming Luh
Danilo Bzdok
Richard M. Leahy
Tian Ge
Daniel S. Margulies
Bratislav Misic
Boris C Bernhardt
W. Dale Stevens
Felipe De Brigard
Prantik Kundu
Richard S. Gary
Gary R. Turner … (voir 1 de plus)
R. Nathan Spreng
The intrinsic functional connectome can reveal how a lifetime of learning and lived experience is represented in the functional architecture… (voir plus) of the aging brain. We investigated whether network dedifferentiation, a hallmark of brain aging, reflects a global shift in network dynamics, or comprises network-specific changes that reflect the changing landscape of aging cognition. We implemented a novel multi-faceted strategy involving multi-echo fMRI acquisition and de-noising, individualized cortical parcellation, and multivariate (gradient and edge-level) functional connectivity methods. Twenty minutes of resting-state fMRI data and cognitive assessments were collected in younger (n=181) and older (n=120) adults. Dimensionality in the BOLD signal was lower for older adults, consistent with global network dedifferentiation. Functional connectivity gradients were largely age-invariant. In contrast, edge-level connectivity showed widespread changes with age, revealing discrete, network-specific dedifferentiation patterns. Visual and somatosensory regions were more integrated within the functional connectome; default and frontoparietal regions showed greater coupling; and the dorsal attention network was less differentiated from transmodal regions. Associations with cognition suggest that the formation and preservation of integrated, large-scale brain networks supports complex cognitive abilities. However, into older adulthood, the connectome is dominated by large-scale network disintegration, global dedifferentiation and network-specific dedifferentiation associated with age-related cognitive change.
2021-12-31
(publié)
www.semanticscholar.org
Titre: Title: Comparison of Myelin Imaging Techniques in Ex Vivo Spinal Cord Auteur:
Nikola Stikov
Julien Cohen-Adad
Manh-Tung Vuong
Vuong Manh Tung
Myelin is a dielectric material that wraps around the axons of nerve fibers to enable fast conduction of signals throughout the nervous syst… (voir plus)em. Loss of myelin can cause anywhere from minor interruption to complete disruption of nerve impulses in a range of neurodegenerative diseases such as multiple sclerosis and Parkinson’s disease. There is an ongoing debate in the myelin imaging community about which biomarker based on Magnetic Resonance Imaging (MRI) is more correlated with myelin. In this work, we implemented and compared several MRI-based myelin imaging techniques (quantitative magnetization transfer imaging, myelin water imaging, and proton density imaging) by evaluating their repeatability and their relation to large-scale histology in the ex vivo spinal cords of a rat, a dog, and a human. While there are studies investigating the relationship between pairs of them as well as with histology, to the best of our knowledge, this is the first study that implemented and compared all those methods at the same time to evaluate their reproducibility and their correlation with myelin. Qualitatively the contrasts were similar, and all techniques had comparable scan-rescan and correlations with histology. Surprisingly, the voxel-wise correlations between the various myelin measures were almost as high as the scan-rescan correlations. The correlations decreased when only white matter was considered, which could be due to the small dynamic range of the measurement, or due to artifacts related to the preparation and panoramic scanning of the tissue. We conclude that the myelin imaging techniques explored in this thesis exhibit similar specificity to myelin, yet the histological correlations suggest that more work is needed to determine the optimal myelin imaging protocol. The study also pointed out some potential miscalibrations during acquisitions as well as data processing that may lead to anywhere from minor to major impact on the accuracy of the results. These include B1 mapping, insufficient spoiling and variation of the predelay time. We have also standardized the data processing routines by upgrading qMTLab to qMRLab which adds several quantitative MR methods to the toolbox, such as standard T1 mapping and field mapping. In addition, the data of the dog spinal cord in this study will be published together with the analysis scripts to help the interested reader to reproduce the findings from this thesis.
2021-12-31
(publié)
www.semanticscholar.org
Toward Reliable Neural Specifications
Chuqin Geng
Nham Le
Xiaojie Xu
Zhaoyue Wang
Arie Gurfinkel
Xujie Si
We propose a new family of specifications called neural as specification , which uses the intrinsic information of neural networks — neu… (voir plus)ral activation patterns (NAP), rather than input data to specify the correctness and/or robustness of neural network predictions. We present a simple statistical approach to mining dominant neural activation patterns. We analyze NAPs from a statistical point of view and find that a single can cover a large number of training and testing data points whereas ad hoc data-as-specification only covers the given reference data point. To show the effectiveness of discovered NAPs, we formally important properties, as various types of misclassifications happen for a and is no-ambiguity between different We show that by using we can verify the prediction of the space , of the we is a and for abstract the state of each neuron to only activated and deactivated by leveraging NAPs. We would like to explore more refined abstractions such as { ( −∞ ] , (0 , 1] , (1 , ∞ ] } in future work.
2021-12-31
arXiv.org (prépublication)
doi.org
openreview.net
Towards Better Evaluation for Dynamic Link Prediction
Farimah Poursafaei
Andy Huang
Shenyang Huang
Kellin Pelrine
Reihaneh Rabbany
Despite the prevalence of recent success in learning from static graphs, learning from time-evolving graphs remains an open challenge. In th… (voir plus)is work, we design new, more stringent evaluation procedures for link prediction specific to dynamic graphs, which reflect real-world considerations, to better compare the strengths and weaknesses of methods. First, we create two visualization techniques to understand the reoccurring patterns of edges over time and show that many edges reoccur at later time steps. Based on this observation, we propose a pure memorization-based baseline called EdgeBank. EdgeBank achieves surprisingly strong performance across multiple settings which highlights that the negative edges used in the current evaluation are easy. To sample more challenging negative edges, we introduce two novel negative sampling strategies that improve robustness and better match real-world applications. Lastly, we introduce six new dynamic graph datasets from a diverse set of domains missing from current benchmarks, providing new challenges and opportunities for future research. Our code repository is accessible at https://github.com/fpour/DGB.git.
2021-12-31
Advances in Neural Information Processing Systems 35 (NeurIPS 2022) (publié)
doi.org
openreview.net