Publications

Neural manifolds and learning regimes in neural-interface tasks
Alexandre Payeur
Amy L. Orsborn
Guillaume Lajoie
Neural activity tends to reside on manifolds whose dimension is lower than the dimension of the whole neural state space. Experiments using … (voir plus)brain-computer interfaces (BCIs) with microelectrode arrays implanted in the motor cortex of nonhuman primates have provided ways to test whether neural manifolds influence learning-related neural computations. Starting from a learned BCI-controlled motor task, these experiments explored the effect of changing the BCI decoder to implement perturbations that were either “aligned” or not with the pre-existing neural manifold. In a series of studies, researchers found that within-manifold perturbations (WMPs) evoked fast reassociations of existing neural patterns for rapid adaptation, while outside-manifold perturbations (OMPs) triggered a slower adaptation process that led to the emergence of new neural patterns. Together, these findings have been interpreted as suggesting that these different rates of adaptation might be associated with distinct learning mechanisms. Here, we investigated whether gradient-descent learning could alone explain these differences. Using an idealized model that captures the fixed-point dynamics of recurrent neural networks, we uncovered gradient-based learning dynamics consistent with experimental findings. Crucially, this experimental match arose only when the network was initialized in a lazier learning regime, a concept inherited from deep learning theory. A lazy learning regime—in contrast with a rich regime—implies small changes on synaptic strengths throughout learning. For OMPs, these small changes were less effective at increasing performance and could lead to unstable adaptation with a heightened sensitivity to learning rates. For WMPs, they helped reproduce the reassociation mechanism on short adaptation time scales, especially with large input variances. Since gradient descent has many biologically plausible variants, our findings establish lazy gradient-based learning as a plausible mechanism for adaptation under network-level constraints and unify several experimental results from the literature.
2023-03-11
bioRxiv (prépublication)
doi.org
A Systematic Study of Joint Representation Learning on Protein Sequences and Structures
Zuobai Zhang
Chuanrui Wang
Minghao Xu
Vijil Chenthamarakshan
Aurelie Lozano
Payel Das
Jian Tang
Learning effective protein representations is critical in a variety of tasks in biology such as predicting protein functions. Recent sequenc… (voir plus)e representation learning methods based on Protein Language Models (PLMs) excel in sequence-based tasks, but their direct adaptation to tasks involving protein structures remains a challenge. In contrast, structure-based methods leverage 3D structural information with graph neural networks and geometric pre-training methods show potential in function prediction tasks, but still suffers from the limited number of available structures. To bridge this gap, our study undertakes a comprehensive exploration of joint protein representation learning by integrating a state-of-the-art PLM (ESM-2) with distinct structure encoders (GVP, GearNet, CDConv). We introduce three representation fusion strategies and explore different pre-training techniques. Our method achieves significant improvements over existing sequence- and structure-based methods, setting new state-of-the-art for function annotation. This study underscores several important design choices for fusing protein sequence and structure information. Our implementation is available at https://github.com/DeepGraphLearning/ESM-GearNet.
2023-03-10
ArXiv (prépublication)
arxiv.org
arxiv.org
Molecular pathology of acute respiratory distress syndrome, mechanical ventilation and abnormal coagulation in severe COVID-19
Antoine Soulé
William Ma
Katelyn Yixiu Liu
Catherine Allard
Salman Qureshi
Karine Tremblay
Amin Emad
Simon Rousseau
Systemic inflammation in critically ill patients can lead to serious consequences such as acute respiratory distress syndrome (ARDS), a cond… (voir plus)ition characterized by the presence of lung inflammation, edema, and impaired gas exchange, associated with poor survival. Understanding molecular pathobiology is essential to improve critical care of these patients. To this end, we use multimodal profiles of SARS-CoV-2 infected hospitalized participants to the Biobanque Québécoise de la COVID-19 (BQC-19) to characterize endophenotypes associated with different degrees of disease severity. Proteomic, metabolomic, and genomic characterization supported a role for neutrophil-associated procoagulant activity in severe COVID-19 ARDS that is inversely correlated with sphinghosine-1 phosphate plasma levels. Fibroblast Growth Factor Receptor (FGFR) and SH2-containing transforming protein 4 (SHC4) signaling were identified as molecular features associated with endophenotype 6 (EP6). Mechanical ventilation in EP6 was associated with alterations in lipoprotein metabolism. These findings help define the molecular mechanisms related to specific severe outcomes, that can be used to identify early unfavorable clinical trajectories and treatable traits to improve the survival of critically ill patients.
2023-03-09
medRxiv (prépublication)
doi.org
The Critical Node Game
Gabriele Dragotto
Amine Boukhtouta
Andrea Lodi
Mehdi Taobane
Cloud networks are the backbone of the modern distributed internet infrastructure as they provision most of the on-demand resources organiza… (voir plus)tions and individuals use daily. However, any abrupt cyber-attack could disrupt the provisioning of some of the cloud resources fulfilling the needs of customers, industries, and governments. In this work, we introduce a game-theoretic model that assesses the cyber-security risk of cloud networks and informs security experts on the optimal security strategies. Our approach combines game theory, combinatorial optimization, and cyber-security and aims at minimizing the unexpected network disruptions caused by malicious cyber-attacks under uncertainty. Methodologically, our approach consists of a simultaneous and non-cooperative attacker-defender game where each player solves a combinatorial optimization problem parametrized in the variables of the other player. Practically, our approach enables security experts to (i.) assess the security posture of the cloud network, and (ii.) dynamically adapt the level of cyber-protection deployed on the network. We provide a detailed analysis of a real-world cloud network and demonstrate the efficacy of our approach through extensive computational tests.
2023-03-09
ArXiv (prépublication)
doi.org
arxiv.org
Transfer Entropy Bottleneck: Learning Sequence to Sequence Information Transfer
Damjan Kalajdzievski
Ximeng Mao
Pascal Fortier-Poisson
Guillaume Lajoie
Blake A. Richards
When presented with a data stream of two statistically dependent variables, predicting the future of one of the variables (the target stream… (voir plus)) can benefit from information about both its history and the history of the other variable (the source stream). For example, fluctuations in temperature at a weather station can be predicted using both temperatures and barometric readings. However, a challenge when modelling such data is that it is easy for a neural network to rely on the greatest joint correlations within the target stream, which may ignore a crucial but small information transfer from the source to the target stream. As well, there are often situations where the target stream may have previously been modelled independently and it would be useful to use that model to inform a new joint model. Here, we develop an information bottleneck approach for conditional learning on two dependent streams of data. Our method, which we call Transfer Entropy Bottleneck (TEB), allows one to learn a model that bottlenecks the directed information transferred from the source variable to the target variable, while quantifying this information transfer within the model. As such, TEB provides a useful new information bottleneck approach for modelling two statistically dependent streams of data in order to make predictions about one of them.
2023-03-07
Transactions on Machine Learning Research (accepté)
doi.org
openreview.net
A cognitive fingerprint in human random number generation
Marc-Andre Schulz
Sebastian Baier
Benjamin Timmermann
Danilo Bzdok
Karsten Witt

Most work in the neurosciences collapses data from multiple subjects to obtain robust statistical results. This research agenda ignores t… (voir plus)hat even in healthy subjects brain structure and function are known to be highly variable. Recently, Finn and colleagues showed that the brain's functional organisation is unique to each individual and can yield human-specific connectome fingerprints. This raises the question whether unique functional brain architecture may reflect a unique implementation of cognitive processes and problem solving - i.e. "Can we identify single individuals based on how they think?". The present study addresses the general question of interindividual differences in the specific context of human random number generation. We analyzed the deployment of recurrent patterns in the pseudorandom sequences to develop an identification scheme based on subject-specific volatility patterns. We demonstrate that individuals can be reliably identified based on how they how they generate randomness patterns alone. We moreover show that this phenomenon is driven by individual preference and inhibition of patterns, together forming a cognitive fingerprint.

2023-03-06
Charité - Universitätsmedizin Berlin (publié)
doi.org
Enhancing Protein Language Model with Structure-based Encoder and Pre-training
Zuobai Zhang
Minghao Xu
Aurelie Lozano
Vijil Chenthamarakshan
Payel Das
Jian Tang
Protein language models (PLMs) pre-trained on large-scale protein sequence corpora have achieved impressive performance on various downstrea… (voir plus)m protein understanding tasks. Despite the ability to implicitly capture inter-residue contact information, transformer-based PLMs cannot encode protein structures explicitly for better structure-aware protein representations. Besides, the power of pre-training on available protein structures has not been explored for improving these PLMs, though structures are important to determine functions. To tackle these limitations, in this work, we enhance the PLM with structure-based encoder and pre-training. We first explore feasible model architectures to combine the advantages of a state-of-the-art PLM (i.e., ESM-1b) and a state-of-the-art protein structure encoder (i.e., GearNet). We empirically verify the ESM-GearNet that connects two encoders in a series way as the most effective combination model. To further improve the effectiveness of ESM-GearNet, we pre-train it on massive unlabeled protein structures with contrastive learning, which aligns representations of co-occurring subsequences so as to capture their biological correlation. Extensive experiments on EC and GO protein function prediction benchmarks demonstrate the superiority of ESM-GearNet over previous PLMs and structure encoders, and clear performance gains are further achieved by structure-based pre-training upon ESM-GearNet. The source code will be made public upon acceptance.
2023-03-05
ICLR.cc/2023/Workshop/MLDD (poster)
openreview.net
openreview.net
EurNet: Efficient Multi-Range Relational Modeling of Protein Structure
Minghao Xu
Yuanfan Guo
Yi Xu
Jian Tang
Xinlei Chen
Yuandong Tian
Modeling the 3D structures of proteins is critical for obtaining effective protein structure representations, which further boosts protein f… (voir plus)unction understanding. Existing protein structure encoders mainly focus on modeling short-range interactions within protein structures, while they neglect modeling the interactions at multiple length scales that are actually complete interactive patterns in protein structures. To attain complete interaction modeling with efficient computation, we introduce the EurNet for Efficient multi-range relational modeling. In EurNet, we represent the protein structure as a multi-relational residue-level graph with different types of edges for modeling short-range, medium-range and long-range interactions. To efficiently process these different interactive relations, we propose a novel modeling layer, called Gated Relational Message Passing (GRMP), as the basic building block of EurNet. GRMP can capture multiple interactive relations in protein structures with little extra computational cost. We verify the state-of-the-art performance of EurNet on EC and GO protein function prediction benchmarks, and the proposed GRMP layer is proved to achieve better efficiency-performance trade-off than the widely-used relational graph convolution.
2023-03-05
ICLR.cc/2023/Workshop/MLDD (poster)
openreview.net
openreview.net
Privacy-Preserving Fair Item Ranking
Jiajun Sun
Sikha Pentyala
Martine De Cock
Golnoosh Farnadi
Users worldwide access massive amounts of curated data in the form of rankings on a daily basis. The societal impact of this ease of access … (voir plus)has been studied and work has been done to propose and enforce various notions of fairness in rankings. Current computational methods for fair item ranking rely on disclosing user data to a centralized server, which gives rise to privacy concerns for the users. This work is the first to advance research at the conjunction of producer (item) fairness and consumer (user) privacy in rankings by exploring the incorporation of privacy-preserving techniques; specifically, differential privacy and secure multi-party computation. Our work extends the equity of amortized attention ranking mechanism to be privacy-preserving, and we evaluate its effects with respect to privacy, fairness, and ranking quality. Our results using real-world datasets show that we are able to effectively preserve the privacy of users and mitigate unfairness of items without making additional sacrifices to the quality of rankings in comparison to the ranking mechanism in the clear.
2023-03-05
ArXiv (prépublication)
doi.org
arxiv.org
Task-Agnostic Graph Neural Network Evaluation via Adversarial Collaboration
Xiangyu Zhao
Hannes Stärk
Dominique Beaini
Pietro Lio
Yiren Zhao
2023-03-05
ICLR.cc/2023/Workshop/MLDD (poster)
doi.org
openreview.net
Improved Robustness Against Adaptive Attacks With Ensembles and Error-Correcting Output Codes
Thomas Philippon
Christian Gagné
Neural network ensembles have been studied extensively in the context of adversarial robustness and most ensemble-based approaches remain vu… (voir plus)lnerable to adaptive attacks. In this paper, we investigate the robustness of Error-Correcting Output Codes (ECOC) ensembles through architectural improvements and ensemble diversity promotion. We perform a comprehensive robustness assessment against adaptive attacks and investigate the relationship between ensemble diversity and robustness. Our results demonstrate the benefits of ECOC ensembles for adversarial robustness compared to regular ensembles of convolutional neural networks (CNNs) and show why the robustness of previous implementations is limited. We also propose an adversarial training method specific to ECOC ensembles that allows to further improve robustness to adaptive attacks.
2023-03-03
ArXiv (prépublication)
doi.org
arxiv.org
Objectives Matter: Understanding the Impact of Self-Supervised Objectives on Vision Transformer Representations
Shashank Shekhar
Florian Bordes
P Vincent
Ari S. Morcos
Joint-embedding based learning (e.g., SimCLR, MoCo, DINO) and reconstruction-based learning (e.g., BEiT, SimMIM, MAE) are the two leading pa… (voir plus)radigms for self-supervised learning of vision transformers, but they differ substantially in their transfer performance. Here, we aim to explain these differences by analyzing the impact of these objectives on the structure and transferability of their representations. Our analysis reveals that reconstruction-based learning features are significantly dissimilar to joint-embedding based learning features and that models trained with similar objectives learn similar features even across architectures. These differences arise early in the network, primarily driven by attention and normalization layers. We find that joint-embedding features yield better linear probe transfer for classification because the different objectives drive different distributions of information and invariances in the representation. These differences explain opposite trends in transfer performance for downstream tasks that require spatial specificity in features. Finally, we address how fine-tuning changes reconstructive representations to enable better transfer, showing that it re-organizes the information to be more similar to pre-trained joint embedding models.
2023-03-03
ICLR.cc/2023/Workshop/ME-FoMo (spotlight)
doi.org
openreview.net