Publications

Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset

Yash More

Jackie CK Cheung

In an effort to mitigate the harms of large language models (LLMs), learning from human feedback (LHF) has been used to steer LLMs towards o… (voir plus)utputs that are intended to be both less harmful and more helpful. Despite the widespread adoption of LHF in practice, the quality of this feedback and its effectiveness as a safety mitigation technique remain unclear. This study addresses these issues by auditing the widely-used Helpful and Harmless (HH) dataset by Anthropic. Our work includes: (1) a thorough investigation of the dataset's content through both manual and automated evaluation; (2) experiments demonstrating the dataset's impact on models' safety; and (3) an analysis of the 100 most influential papers citing this dataset. Through our audit, we showcase how conceptualization failures and quality issues identified in the HH dataset can create additional harms by leading to disparate safety behaviors across demographic groups. Our findings highlight the need for more nuanced, context-sensitive approaches to safety mitigation in LLMs.

2024-11-11

ArXiv (prépublication)

doi.org

arxiv.org

Fault Localization in Deep Learning-based Software: A System-level Approach

Mohammad Mehdi Morovati

Amin Nikanjam

Foutse Khomh

2024-11-11

ArXiv (prépublication)

doi.org

arxiv.org

Feature learning as alignment: a structural property of gradient descent in non-linear neural networks

Daniel Beaglehole

Ioannis Mitliagkas

Atish Agarwala

Understanding the mechanisms through which neural networks extract statistics from input-label pairs through feature learning is one of the … (voir plus)most important unsolved problems in supervised learning. Prior works demonstrated that the gram matrices of the weights (the neural feature matrices, NFM) and the average gradient outer products (AGOP) become correlated during training, in a statement known as the neural feature ansatz (NFA). Through the NFA, the authors introduce mapping with the AGOP as a general mechanism for neural feature learning. However, these works do not provide a theoretical explanation for this correlation or its origins. In this work, we further clarify the nature of this correlation, and explain its emergence. We show that this correlation is equivalent to alignment between the left singular structure of the weight matrices and the newly defined pre-activation tangent features at each layer. We further establish that the alignment is driven by the interaction of weight changes induced by SGD with the pre-activation features, and analyze the resulting dynamics analytically at early times in terms of simple statistics of the inputs and labels. We prove the derivative alignment occurs with high probability in specific high dimensional settings. Finally, motivated by the observation that the NFA is driven by this centered correlation, we introduce a simple optimization rule that dramatically increases the NFA correlations at any given layer and improves the quality of features learned.

2024-11-10

TMLR (accepté)

doi.org

openreview.net

Impact of LLM-based Review Comment Generation in Practice: A Mixed Open-/Closed-source User Study

Doriane Olewicki

Leuson Da Silva

Suhaib Mujahid

Arezou Amini

Benjamin Mah

Marco Castelluccio

Sarra Habchi

Foutse Khomh

Bram Adams

2024-11-10

ArXiv (prépublication)

doi.org

arxiv.org

Specific inhibition and disinhibition in the higher-order structure of a cortical connectome

Michael W. Reimann

Daniela Egas Santander

András Ecker

Eilif B. Muller

Neurons are thought to act as parts of assemblies with strong internal excitatory connectivity. Conversely, inhibition is often reduced to b… (voir plus)lanket inhibition with no targeting specificity. We analyzed the structure of excitation and inhibition in the MICrONS mm 3 dataset, an electron microscopic reconstruction of a piece of cortical tissue. We found that excitation was structured around a feed-forward flow in large non-random neuron motifs with a structure of information flow from a small number of sources to a larger number of potential targets. Inhibitory neurons connected with neurons in specific sequential positions of these motifs, implementing targeted and symmetrical competition between them. None of these trends are detectable in only pairwise connectivity, demonstrating that inhibition is structured by these large motifs. While descriptions of inhibition in cortical circuits range from non-specific blanket-inhibition to targeted, our results describe a form of targeting specificity existing in the higher-order structure of the connectome. These findings have important implications for the role of inhibition in learning and synaptic plasticity.

2024-11-10

Cerebral Cortex (New York, NY) (publié)

doi.org

Reaction-conditioned De Novo Enzyme Design with GENzyme

Chenqing Hua

Jiarui Lu

Yang Liu

Odin Zhang

Jian Tang

Rex Ying

Wengong Jin

Guy Wolf

Doina Precup

Shuangjia Zheng

The introduction of models like RFDiffusionAA, AlphaFold3, AlphaProteo, and Chai1 has revolutionized protein structure modeling and interact… (voir plus)ion prediction, primarily from a binding perspective, focusing on creating ideal lock-and-key models. However, these methods can fall short for enzyme-substrate interactions, where perfect binding models are rare, and induced fit states are more common. To address this, we shift to a functional perspective for enzyme design, where the enzyme function is defined by the reaction it catalyzes. Here, we introduce \textsc{GENzyme}, a \textit{de novo} enzyme design model that takes a catalytic reaction as input and generates the catalytic pocket, full enzyme structure, and enzyme-substrate binding complex. \textsc{GENzyme} is an end-to-end, three-staged model that integrates (1) a catalytic pocket generation and sequence co-design module, (2) a pocket inpainting and enzyme inverse folding module, and (3) a binding and screening module to optimize and predict enzyme-substrate complexes. The entire design process is driven by the catalytic reaction being targeted. This reaction-first approach allows for more accurate and biologically relevant enzyme design, potentially surpassing structure-based and binding-focused models in creating enzymes capable of catalyzing specific reactions. We provide \textsc{GENzyme} code at https://github.com/WillHua127/GENzyme.

2024-11-09

ArXiv (prépublication)

doi.org

arxiv.org

A new species of Hoplostethus from Sumatra, eastern Indian Ocean, with comments on its most similar congeners (Trachichthyiformes: Trachichthyidae).

Yo Su

Alexander N. Kotlyar

Hsiu-Chin Lin

Toshio Kawai

HSUAN-CHING HO

2024-11-07

Journal of Fish Biology (publié)

doi.org

Assemblies, synapse clustering, and network topology interact with plasticity to explain structure-function relationships of the cortical connectome

András Ecker

Daniela Egas Santander

Marwan Abdellah

Jorge Blanco Alonso

Sirio Bolaños-Puchet

Giuseppe Chindemi

Dhuruva Priyan Gowri Mariyappan

James B Isbister

James King

Pramod Kumbhar

Ioannis Magkanaris

Eilif B Muller

Michael W Reimann

Abstract Synaptic plasticity underlies the brain’s ability to learn and adapt. While experiments in brain slices have reve… (voir plus)aled mechanisms and protocols for the induction of plasticity between pairs of neurons, how these synaptic changes are coordinated in biological neuronal networks to ensure the emergence of learning remains poorly understood. Simulation and modeling have emerged as important tools to study learning in plastic networks, but have yet to achieve a scale that incorporates realistic network structure, active dendrites, and multi-synapse interactions, key determinants of synaptic plasticity. To rise to this challenge, we endowed an existing large-scale cortical network model, incorporating data-constrained dendritic processing and multi-synaptic connections, with a calcium-based model of functional plasticity that captures the diversity of excitatory connections extrapolated to in vivo-like conditions. This allowed us to study how dendrites and network structure interact with plasticity to shape stimulus representations at the microcircuit level. In our simulations, plasticity acted sparsely and specifically, firing rates and weight distributions remained stable without additional homeostatic mechanisms. At the circuit level, we found plasticity was driven by co-firing stimulus-evoked functional assemblies, spatial clustering of synapses on dendrites, and the topology of the network connectivity. As a result of the plastic changes, the network became more reliable with more stimulus-specific responses. We confirmed our testable predictions in the MICrONS datasets, an openly available electron microscopic reconstruction of a large volume of cortical tissue. Our results quantify at a large scale how the dendritic architecture and higher-order structure of cortical microcircuits play a central role in functional plasticity and provide a foundation for elucidating their role in learning.

2024-11-06

eLife (publié)

doi.org

Follow the CSF flow: probing multiciliated ependymal cells in brain pathology

Adam M.R. Groh

L. M. Hodgson

Danilo Bzdok

Jo Anne Stratton

2024-11-06

Trends in Molecular Medicine (publié)

doi.org

Unlearning in- vs. out-of-distribution data in LLMs under gradient-based method

Teodora Baluta

Pascal Lamblin

Daniel Tarlow

Fabian Pedregosa

Gintare Karolina Dziugaite

Machine unlearning aims to solve the problem of removing the influence of selected training examples from a learned model. Despite the incre… (voir plus)asing attention to this problem, it remains an open research question how to evaluate unlearning in large language models (LLMs), and what are the critical properties of the data to be unlearned that affect the quality and efficiency of unlearning. This work formalizes a metric to evaluate unlearning quality in generative models, and uses it to assess the trade-offs between unlearning quality and performance. We demonstrate that unlearning out-of-distribution examples requires more unlearning steps but overall presents a better trade-off overall. For in-distribution examples, however, we observe a rapid decay in performance as unlearning progresses. We further evaluate how example's memorization and difficulty affect unlearning under a classical gradient ascent-based approach.

2024-11-06

ArXiv (prépublication)

doi.org

arxiv.org

CSGraph2Vec: Distributed Graph-Based Representation Learning for Assembly Functions

Wael J. Alhashemi

Benjamin C. M. Fung

Adel Abusitta

Claude Fachkha

2024-11-05

2024 IEEE International Conference on Recent Advances in Systems Science and Engineering (RASSE) (publié)

doi.org