Publications

Predicting the Impact of Model Expansion through the Minima Manifold: A Loss Landscape Perspective
Pranshu Malviya
Jerry Huang
Aristide Baratin
Quentin Fournier
Sarath Chandar
The optimal model for a given task is often challenging to determine, requiring training multiple models from scratch which becomes prohibit… (voir plus)ive as dataset and model sizes grow. A more efficient alternative is to reuse smaller pre-trained models by expanding them, however, this is not widely adopted as how this impacts training dynamics remains poorly understood. While prior works have introduced statistics to measure these effects, they remain flawed. To rectify this, we offer a new approach for understanding and quantifying the impact of expansion through the lens of the loss landscape, which has been shown to contain a manifold of linearly connected minima. Building on this new perspective, we propose a metric to study the impact of expansion by estimating the size of the manifold. Experimental results show a clear relationship between gains in performance and manifold size, enabling the comparison of candidate models and presenting a first step towards expanding models more reliably based on geometric properties of the loss landscape.
2024-05-23
arXiv (prépublication)
doi.org
arxiv.org
Assortment Optimization with Visibility Constraints
Théo Barré
Omar El Housni
Andrea Lodi
2024-05-21
Integer Programming and Combinatorial Optimization (publié)
doi.org
arxiv.org
Attention as an RNN
Leo Feng
Frederick Tung
Hossein Hajimirsadeghi
Mohamed Osama Ahmed
Yoshua Bengio
Greg Mori
The advent of Transformers marked a significant breakthrough in sequence modelling, providing a highly performant architecture capable of le… (voir plus)veraging GPU parallelism. However, Transformers are computationally expensive at inference time, limiting their applications, particularly in low-resource settings (e.g., mobile and embedded devices). Addressing this, we (1) begin by showing that attention can be viewed as a special Recurrent Neural Network (RNN) with the ability to compute its \textit{many-to-one} RNN output efficiently. We then (2) show that popular attention-based models such as Transformers can be viewed as RNN variants. However, unlike traditional RNNs (e.g., LSTMs), these models cannot be updated efficiently with new tokens, an important property in sequence modelling. Tackling this, we (3) introduce a new efficient method of computing attention's \textit{many-to-many} RNN output based on the parallel prefix scan algorithm. Building on the new attention formulation, we (4) introduce \textbf{Aaren}, an attention-based module that can not only (i) be trained in parallel (like Transformers) but also (ii) be updated efficiently with new tokens, requiring only constant memory for inferences (like traditional RNNs). Empirically, we show Aarens achieve comparable performance to Transformers on
2024-05-21
ArXiv (prépublication)
doi.org
arxiv.org
High-effect gene-coding variants impact cognition, mental well-being, and neighborhood safety substrates in brain morphology
Jakub Kopal
Guillaume Huguet
Justin Marotta
Shambhavi Aggarwal
Nicole Osayande
Kuldeep Kumar
Zohra Saci
Martineau Jean-Louis
Xiaoqian J. Chai
Tian Ge
B.T. Thomas Yeo
Paul M. Thompson
Carrie E. Bearden
Ole A. Andreassen
Sébastien Jacquemont
Danilo Bzdok
Our genetic makeup, together with environmental and social influences, shape our brain's development. Yet, the imaging genetics field has st… (voir plus)ruggled to integrate all these modalities to investigate the interplay between genetic blueprint, environment, human health, daily living skills and outcomes. Hence, we interrogated the Adolescent Brain Cognitive Development (ABCD) cohort to outline the effects of rare high-effect genetic variants on brain architecture and corresponding implications on cognitive, behavioral, psychosocial, and socioeconomic traits. Specifically, we designed a holistic pattern-learning algorithm that quantitatively dissects the impacts of copy number variations (CNVs) on brain structure and 962 behavioral variables spanning 20 categories in 7,657 adolescents. Our results reveal associations between genetic alterations, higher-order brain networks, and specific parameters of the family well-being (increased parental and child stress, anxiety and depression) or neighborhood dynamics (decreased safety); effects extending beyond the impairment of cognitive ability or language capacity, dominantly reported in the CNV literature. Our investigation thus spotlights a far-reaching interplay between genetic variation and subjective life quality in adolescents and their families.
2024-05-21
medRxiv (prépublication)
doi.org
High-effect gene-coding variants impact cognition, mental well-being, and neighborhood safety substrates in brain morphology
Jakub Kopal
Guillaume Huguet
Justin Marotta
Shambhavi Aggarwal
Nicole Osayande
Kuldeep Kumar
Zohra Saci
Martineau Jean-Louis
Xiaoqian J. Chai
Tian Ge
B.T. Thomas Yeo
Paul M. Thompson
Carrie E. Bearden
Ole A. Andreassen
Sébastien Jacquemont
Danilo Bzdok
Our genetic makeup, together with environmental and social influences, shape our brain's development. Yet, the imaging genetics field has st… (voir plus)ruggled to integrate all these modalities to investigate the interplay between genetic blueprint, environment, human health, daily living skills and outcomes. Hence, we interrogated the Adolescent Brain Cognitive Development (ABCD) cohort to outline the effects of rare high-effect genetic variants on brain architecture and corresponding implications on cognitive, behavioral, psychosocial, and socioeconomic traits. Specifically, we designed a holistic pattern-learning algorithm that quantitatively dissects the impacts of copy number variations (CNVs) on brain structure and 962 behavioral variables spanning 20 categories in 7,657 adolescents. Our results reveal associations between genetic alterations, higher-order brain networks, and specific parameters of the family well-being (increased parental and child stress, anxiety and depression) or neighborhood dynamics (decreased safety); effects extending beyond the impairment of cognitive ability or language capacity, dominantly reported in the CNV literature. Our investigation thus spotlights a far-reaching interplay between genetic variation and subjective life quality in adolescents and their families.
2024-05-21
medRxiv (prépublication)
doi.org
Static graph approximations of dynamic contact networks for epidemic forecasting
Razieh Shirzadkhani
Shenyang Huang
Abby Leung
Reihaneh Rabbany
Epidemic modeling is essential in understanding the spread of infectious diseases like COVID-19 and devising effective intervention strategi… (voir plus)es to control them. Recently, network-based disease models have integrated traditional compartment-based modeling with real-world contact graphs and shown promising results. However, in an ongoing epidemic, future contact network patterns are not observed yet. To address this, we use aggregated static networks to approximate future contacts for disease modeling. The standard method in the literature concatenates all edges from a dynamic graph into one collapsed graph, called the full static graph. However, the full static graph often leads to severe overestimation of key epidemic characteristics. Therefore, we propose two novel static network approximation methods, DegMST and EdgeMST, designed to preserve the sparsity of real world contact network while remaining connected. DegMST and EdgeMST use the frequency of temporal edges and the node degrees respectively to preserve sparsity. Our analysis show that our models more closely resemble the network characteristics of the dynamic graph compared to the full static ones. Moreover, our analysis on seven real-world contact networks suggests EdgeMST yield more accurate estimations of disease dynamics for epidemic forecasting when compared to the standard full static method.
2024-05-21
Scientific Reports (publié)
doi.org
Uncovering the essence of diverse media biases from the semantic embedding space
Hong Huang
Hua Zhu
Wenshi Liu
Hua Gao
Hai Jin
Bang Liu
Media bias widely exists in the articles published by news media, influencing their readers’ perceptions, and bringing prejudice or injust… (voir plus)ice to society. However, current analysis methods usually rely on human efforts or only focus on a specific type of bias, which cannot capture the varying magnitudes, connections, and dynamics of multiple biases, thus remaining insufficient to provide a deep insight into media bias. Inspired by the Cognitive Miser and Semantic Differential theories in psychology, and leveraging embedding techniques in the field of natural language processing, this study proposes a general media bias analysis framework that can uncover biased information in the semantic embedding space on a large scale and objectively quantify it on diverse topics. More than 8 million event records and 1.2 million news articles are collected to conduct this study. The findings indicate that media bias is highly regional and sensitive to popular events at the time, such as the Russia-Ukraine conflict. Furthermore, the results reveal some notable phenomena of media bias among multiple U.S. news outlets. While they exhibit diverse biases on different topics, some stereotypes are common, such as gender bias. This framework will be instrumental in helping people have a clearer insight into media bias and then fight against it to create a more fair and objective news environment.
2024-05-21
Humanities and Social Sciences Communications (publié)
doi.org
Learning the Game: Decoding the Differences between Novice and Expert Players in a Citizen Science Game with Millions of Players.
Eddie Cai
Roman Sarrazin-Gendron
Renata Mutalova
Parham Ghasemloo Gheidari
Alexander Butyaev
Gabriel Richard
Sébastien Caisse
Rob Knight
Mathieu Blanchette
Attila Szantner
Jérôme Waldispühl
In recent years, video games have surged in popularity, attracting millions of players across platforms. Citizen science games (CSGs) levera… (voir plus)ge the processing power of gamers to solve computational and scientific problems. Borderlands Science (BLS) is a mini-game within the mass market game Borderlands 3 that turns multiple sequence alignment (MSA) problems into puzzles. Parallel research demonstrated that BLS players outperformed classical approaches solving small sequence alignment tasks. This study aims to analyze the strategical differences in player solutions in BLS as they gain experience. Through the many collected player solutions from players of different experience level, we gained insights into players’ strategies, differences between expert and non-expert players, and how strategies evolve. We developed a Markov chain trained on solutions from players of different experience levels to understand their actions and outcomes. Results indicate that expert players utilize more gaps and achieve more matches, gradually improving and converging toward unique strategies. Our findings reveal distinct and evolving player strategies. For future citizen science projects, it will be important to consider the identification of player strategies and their evolution over time to improve the game design and data processing.
2024-05-20
Proceedings of the 19th International Conference on the Foundations of Digital Games (publié)
doi.org
Penalty weight tuning in high dose rate brachytherapy using multi-objective Bayesian optimization
Hossein Jafarzadeh
Majd Antaki
Ximeng Mao
Marie Duclos
Farhard Maleki
Shirin A Enger
Objective. Treatment plan optimization in high dose rate brachytherapy often requires manual fine-tuning of penalty weights for each objecti… (voir plus)ve, which can be time-consuming and dependent on the planner's experience. To automate this process, this study used a multi-criteria approach called multi-objective Bayesian optimization with q-noisy expected hypervolume improvement as its acquisition function (MOBO-qNEHVI). Approach. The treatment plans of 13 prostate cancer patients were retrospectively imported to a research treatment planning system, RapidBrachyMTPS, where fast mixed integer optimization (FMIO) performs dwell time optimization given a set of penalty weights to deliver 15 Gy to the target volume. MOBO-qNEHVI was used to find patient-specific Pareto optimal penalty weight vectors that yield clinically acceptable dose volume histogram metrics. The relationship between the number of MOBO-qNEHVI iterations and the number of clinically acceptable plans per patient (acceptance rate) was investigated. The performance time was obtained for various parameter configurations. Main results. MOBO-qNEHVI found clinically acceptable treatment plans for all patients. With increasing the number of MOBO-qNEHVI iterations, the acceptance rate grew logarithmically while the performance time grew exponentially. Fixing the penalty weight of the tumour volume to maximum value, adding the target dose as a parameter, initiating MOBO-qNEHVI with 25 parallel sampling of FMIO, and running 6 MOBO-qNEHVI iterations found solutions that delivered 15 Gy to the hottest 95% of the clinical target volume while respecting the dose constraints to the organs at risk. The average acceptance rate for each patient was 89.74% ± 8.11%, and performance time was 66.6 ± 12.6 s. The initiation took 22.47 ± 7.57 s, and each iteration took 7.35 ± 2.45 s to find one Pareto solution.Significance. MOBO-qNEHVI combined with FMIO can automatically explore the trade-offs between treatment plan objectives in a patient specific manner within a minute. This approach can reduce the dependency of plan quality on planner’s experience and reduce dose to the organs at risk.
2024-05-20
Physics in Medicine & Biology (publié)
doi.org
Investigation of dosimetric characteristics of radiochromic film in response to alpha particles emitted from Americium-241.
Victor D. Diaz‐Martinez
Mélodie Cyr
S. Devic
Nada Tomic
David F. Lewis
S. Enger
BACKGROUND In radiotherapy, it is essential to deliver prescribed doses to tumors while minimizing damage to surrounding healthy tissue. Acc… (voir plus)urate measurements of absorbed dose are required for this purpose. Gafchromic® external beam therapy (EBT) radiochromic films have been widely used in radiotherapy. While the dosimetric characteristics of the EBT3 model film have been extensively studied for photon and charged particle beams (protons, electrons, and carbon ions), little research has been done on α
2024-05-19
Medical Physics (Lancaster) (publié)
doi.org
BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization
Milos Nikolic
Ghouthi Boukli hacene
Ciaran Bannon
Alberto Delmas Lascorz
Matthieu Courbariaux
Omar Mohamed Awad
Yoshua Bengio
Isak Edo Vivancos
Vincent Gripon
Andreas Moshovos
Neural networks have demonstrably achieved state-of-the art accuracy using low-bitlength integer quantization, yielding both execution time … (voir plus)and energy benefits on existing hardware designs that support short bitlengths. However, the question of finding the minimum bitlength for a desired accuracy remains open. We introduce a training method for minimizing inference bitlength at any granularity while maintaining accuracy. Namely, we propose a regularizer that penalizes large bitlength representations throughout the architecture and show how it can be modified to minimize other quantifiable criteria, such as number of operations or memory footprint. We demonstrate that our method learns thrifty representations while maintaining accuracy. With ImageNet, the method produces an average per layer bitlength of 4.13, 3.76 and 4.36 bits on AlexNet, ResNet18 and MobileNet V2 respectively, remaining within 2.0%, 0.5% and 0.5% of the base TOP-1 accuracy.
2024-05-18
2024 IEEE International Symposium on Circuits and Systems (ISCAS) (publié)
doi.org
arxiv.org
Coordination among leaf and fine-root traits along a strong natural soil fertility gradient
Xavier Guilbeault-Mayers
Hans Lambers
Étienne Laliberté
2024-05-17
Plant and Soil (publié)
doi.org