Publications

Smoothness-Adaptive Sharpness-Aware Minimization for Finding Flatter Minima
Hiroki Naganuma
Junhyung Lyle Kim
Anastasios Kyrillidis
Ioannis Mitliagkas
The sharpness-aware minimization (SAM) procedure recently gained increasing attention due to its favorable generalization ability to unseen … (voir plus)data. SAM aims to find flatter (local) minima, utilizing a minimax objective. An immediate challenge in the application of SAM is the adjustment of two pivotal step sizes, which significantly influence its effectiveness. We introduce a novel, straightforward approach for adjusting step sizes that adapts to the smoothness of the objective function, thereby reducing the necessity for manual tuning. This method, termed Smoothness-Adaptive SAM (SA-SAM), not only simplifies the optimization process but also promotes the method's inherent tendency to converge towards flatter minima, enhancing performance in specific models.
2024-03-04
ICLR.cc/2024/Workshop/PML4LRS (poster)
openreview.net
openreview.net
F$^3$low: Frame-to-Frame Coarse-grained Molecular Dynamics with SE(3) Guided Flow Matching
Shaoning Li
Yusong Wang
Mingyu Li
Bin Shao
Nanning Zheng
Zhang Jian
Jian Tang
2024-03-03
ICLR.cc/2024/Workshop/GEM (poster)
openreview.net
openreview.net
Enhancing and Evaluating Logical Reasoning Abilities of Large Language Models
Shujie Deng
Honghua Dong
Xujie Si
2024-03-03
ICLR.cc/2024/Workshop/SeT_LLM (publié)
openreview.net
openreview.net
Fusing Neural and Physical: Augment Protein Conformation Sampling with Tractable Simulations
Jiarui Lu
Zuobai Zhang
Bozitao Zhong
Chence Shi
Jian Tang
The protein dynamics are common and important for their biological functions and properties, the study of which usually involves time-consum… (voir plus)ing molecular dynamics (MD) simulations *in silico*. Recently, generative models has been leveraged as a surrogate sampler to obtain conformation ensembles with orders of magnitude faster and without requiring any simulation data (a "zero-shot" inference). However, being agnostic of the underlying energy landscape, the accuracy of such generative model may still be limited. In this work, we explore the few-shot setting of such pre-trained generative sampler which incorporates MD simulations in a tractable manner. Specifically, given a target protein of interest, we first acquire some seeding conformations from the pre-trained sampler followed by a number of physical simulations in parallel starting from these seeding samples. Then we fine-tuned the generative model using the simulation trajectories above to become a target-specific sampler. Experimental results demonstrated the superior performance of such few-shot conformation sampler at a tractable computational cost.
2024-03-03
ICLR.cc/2024/Workshop/GEM (poster)
openreview.net
openreview.net
Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok
Tikeng Notsawo Pascal Junior
Pascal Notsawo
Hattie Zhou
Mohammad Pezeshki
Irina Rish
Guillaume Dumas
2024-03-03
ICLR.cc/2024/Workshop/ME-FoMo (poster)
doi.org
openreview.net
Self-evaluation and self-prompting to improve the reliability of LLMs
Alexandre Piché
Aristides Milios
Dzmitry Bahdanau
Christopher Pal
In order to safely deploy Large Language Models (LLMs), they must be capable of dynamically adapting their behavior based on their level of … (voir plus)knowledge and uncertainty associated with specific topics. This adaptive behavior, which we refer to as self-restraint, is non-trivial to teach since it depends on the internal knowledge of an LLM. By default, LLMs are trained to maximize the next token likelihood which does not teach the model to modulate its answer based on its level of uncertainty. In order to learn self-restraint, we devise a simple objective that can encourage the model to produce generation that the model is confident in. To optimize this objective, we introduce ReSearch, an iterative search algorithm based on self-evaluation and self-prompting. Our method results in fewer hallucinations overall, both for known and unknown topics, as the model learns to selectively restrain itself. In addition, our method elegantly incorporates the ability to decline, when the model assesses that it cannot provide a response without a high proportion of hallucination.
2024-03-03
ICLR.cc/2024/Workshop/SeT_LLM (publié)
openreview.net
openreview.net
Structure-Informed Protein Language Model
Zuobai Zhang
Jiarui Lu
Vijil Chenthamarakshan
Aurelie Lozano
Payel Das
Jian Tang
Protein language models are a powerful tool for learning protein representations through pre-training on vast protein sequence datasets. Ho… (voir plus)wever, traditional protein language models lack explicit structural supervision, despite its relevance to protein function. To address this issue, we introduce the integration of remote homology detection to distill structural information into protein language models without requiring explicit protein structures as input. We evaluate the impact of this structure-informed training on downstream protein function prediction tasks. Experimental results reveal consistent improvements in function annotation accuracy for EC number and GO term prediction. Performance on mutant datasets, however, varies based on the relationship between targeted properties and protein structures. This underscores the importance of considering this relationship when applying structure-aware training to protein function prediction tasks. Code and model weights will be made available upon acceptance.
2024-03-03
ICLR.cc/2024/Workshop/GEM (poster)
openreview.net
openreview.net
Towards DNA-Encoded Library Generation with GFlowNets
Michał Koziarski
Mohammed Abukalam
Vedant Shah
Louis Vaillancourt
Doris Alexandra Schuetz
Moksh Jain
Almer Van Der Sloot
Mathieu Bourgey
Anne Marinier
Yoshua Bengio
2024-03-03
GEM @ International Conference on Learning Representations (poster)
doi.org
openreview.net
Distinct social behavior and inter-brain connectivity in Dyads with autistic individuals
Quentin Moreau
Florence Brun
Anaël Ayrolles
Jacqueline Nadel
Guillaume Dumas
Autism Spectrum Disorder (ASD) is defined by distinctive socio-cognitive behaviors that deviate from typical patterns. Notably, social imita… (voir plus)tion skills appear to be particularly impacted, manifesting early on in development. This paper compared the behavior and inter-brain dynamics of dyads made up of two typically developing (TD) participants with mixed dyads made up of ASD and TD participants during social imitation tasks. By combining kinematics and EEG-hyperscanning, we show that individuals with ASD exhibited a preference for the follower rather than the lead role in imitating scenarios. Moreover, the study revealed inter-brain synchrony differences, with low-alpha inter-brain synchrony differentiating control and mixed dyads. The study’s findings suggest the importance of studying interpersonal phenomena in dynamic and ecological settings and using hyperscanning methods to capture inter-brain dynamics during actual social interactions.
2024-03-02
Social Neuroscience (publié)
doi.org
Communicating Study Design Trade-offs in Software Engineering
Martin P. Robillard
Deeksha M. Arya
Neil Ernst
Jin L.C. Guo
Maxime Lamothe
Mathieu Nassif
Nicole Novielli
Alexander Serebrenik
Igor Steinmacher
Klaas-Jan Stol
2024-03-01
ACM Transactions on Software Engineering and Methodology (publié)
doi.org
A Compositional Typed Semantics for Universal Dependencies
Laurestine Bradford
Timothy John O'donnell
Siva Reddy
2024-03-01
ArXiv (prépublication)
doi.org
arxiv.org
Latent Idiom Recognition for a Minimalist Functional Array Language Using Equality Saturation
Jonathan Van der Cruysse
Christophe Dubach
Accelerating programs is typically done by recognizing code idioms matching high-performance libraries or hardware interfaces. However, reco… (voir plus)gnizing such idioms automatically is challenging. The idiom recognition machinery is difficult to write and requires expert knowledge. In addition, slight variations in the input program might hide the idiom and defeat the recognizer. This paper advocates for the use of a minimalist functional array language supporting a small, but expressive, set of operators. The minimalist design leads to a tiny sets of rewrite rules, which encode the language semantics. Crucially, the same minimalist language is also used to encode idioms. This removes the need for hand-crafted analysis passes, or for having to learn a complex domain-specific language to define the idioms. Coupled with equality saturation, this approach is able to match the core functions from the BLAS and PyTorch libraries on a set of computational kernels. Compared to reference C kernel implementations, the approach produces a geometric mean speedup of 1.46× for C programs using BLAS, when generating such programs from the high-level minimalist language.
2024-03-01
2024 IEEE/ACM International Symposium on Code Generation and Optimization (CGO) (publié)
doi.org
arxiv.org