Publications

Universal algorithm for transforming Hamiltonian eigenvalues

Tatsuki Odake

Philip Taranto

Mio Murao

Manipulating Hamiltonians governing physical systems has found a broad range of applications, from quantum chemistry to semiconductor design… (voir plus). In this work, we provide a new way of manipulating Hamiltonians, by transforming their eigenvalues while keeping their eigenstates unchanged. We develop a universal algorithm that deterministically implements any desired (suitably differentiable) function on the eigenvalues of any unknown Hamiltonian, whose positive-time and negative-time dynamics are given as a black box. Our algorithm uses correlated randomness to efficiently combine two subroutines -- namely controlization and Fourier series simulation -- exemplifying a general compilation procedure that we develop. The time complexity of our algorithm is significantly reduced via said compilation technique compared to a na{ï}ve concatenation of the subroutines and outperforms similar methods based on the quantum singular value transformation.

2025-03-30

Physical Review Research (publié)

doi.org

arxiv.org

Steering CLIP's vision transformer with sparse autoencoders

Sonia Joseph

Praneet Suresh

Ethan Goldfarb

Lorenz Hufe

Yossi Gandelsman

Robert Graham

Danilo Bzdok

Wojciech Samek

Blake Aaron Richards

While vision models are highly capable, their internal mechanisms remain poorly understood-- a challenge which sparse autoencoders (SAEs) ha… (voir plus)ve helped address in language, but which remains underexplored in vision. We address this gap by training SAEs on CLIP's vision transformer and uncover key differences between vision and language processing, including distinct sparsity patterns for SAEs trained across layers and token types. We then provide the first systematic analysis of the steerability of CLIP's vision transformer by introducing metrics to quantify how precisely SAE features can be steered to affect the model's output. We find that 10-15% of neurons and features are steerable, with SAEs providing thousands more steerable features than the base model. Through targeted suppression of SAE features, we then demonstrate improved performance on three vision disentanglement tasks (CelebA, Waterbirds, and typographic attacks), finding optimal disentanglement in middle model layers, and achieving state-of-the-art performance on defense against typographic attacks. We release our CLIP SAE models and code to support future research in vision transformer interpretability.

2025-03-29

MIV @ IEEE/CVF Conference on Computer Vision and Pattern Recognition (poster)

doi.org

openreview.net

Bridging biodiversity and ecosystem services through useful plant species

Nina Obiar

Isaac Eckert

Janelle Baker

Daniel Moerman

Laura J. Pollock

2025-03-27

Plants, People, Planet (publié)

doi.org

Optimal neural summarization for full-field weak lensing cosmological implicit inference

Denise Lanzieri

Justine Zeghal

T. Lucas Makinen

Alexandre Boucaud

Jean-Luc Starck

François Lanusse

Context. Traditionally, weak lensing cosmological surveys have been analyzed using summary statistics that were either motivated by their an… (voir plus)alytically tractable likelihoods (e.g., power spectrum) or by their ability to access some higher-order information (e.g., peak counts), but at the cost of requiring a simulation-based inference approach. In both cases, even if the statistics can be very informative, they are not designed nor guaranteed to be statistically sufficient (i.e., to capture all the cosmological information content of the data). With the rise of deep learning, however, it has becomes possible to create summary statistics that are specifically optimized to extract the full cosmological information content of the data. Yet, a fairly wide range of loss functions have been used in practice in the weak lensing literature to train such neural networks, leading to the natural question of whether a given loss should be preferred and whether sufficient statistics can be achieved in theory and in practice under these different choices. Aims. We compare different neural summarization strategies that have been proposed in the literature to identify the loss function that leads to theoretically optimal summary statistics for performing full-field cosmological inference. In doing so, we aim to provide guidelines and insights to the community to help guide future neural network-based cosmological inference analyses. Methods. We designed an experimental setup that allows us to isolate the specific impact of the loss function used to train neural summary statistics on weak lensing data at fixed neural architecture and simulation-based inference pipeline. To achieve this, we developed the sbi_lens JAX package, which implements an automatically differentiable lognormal weak lensing simulator and the tools needed to perform explicit full-field inference with a Hamiltonian Monte Carlo (HMC) sampler over this model. Using sbi_lens , we simulated a w CDM LSST Year 10 weak lensing analysis scenario in which the full-field posterior obtained by HMC sampling gives us a ground truth that can be compared to different neural summarization strategies. Results. We provide theoretical insight into the different loss functions being used in the literature, including mean squared error (MSE) regression, and show that some do not necessarily lead to sufficient statistics, while those motivated by information theory, in particular variational mutual information maximization (VMIM), can in principle lead to sufficient statistics. Our numerical experiments confirm these insights, and we show on our simulated w CDM scenario that the figure of merit (FoM) of an analysis using neural summary statistics optimized under VMIM achieves 100% of the reference Ω c − σ 8 full-field FoM, while an analysis using summary statistics trained under simple MSE achieves only 81% of the same reference FoM.

2025-03-27

Astronomy & Astrophysics (publié)

doi.org

debug-gym: A Text-Based Environment for Interactive Debugging

Xingdi Yuan

Morgane M Moss

Charbel Feghali

Chinmay Singh

Darya Moldavskaya

Drew MacPhee

Lucas Caccia

Matheus Pereira

Minseon Kim

Alessandro Sordoni

Marc-Alexandre Côté

2025-03-26

ArXiv (prépublication)

doi.org

arxiv.org

How do language models learn facts? Dynamics, curricula and hallucinations

Nicolas Zucchet

Jörg Bornschein

Stephanie Chan

Andrew Lampinen

Razvan Pascanu

Soham De

2025-03-26

ArXiv (prépublication)

doi.org

arxiv.org

PRISM: High-Resolution & Precise Counterfactual Medical Image Generation using Language-guided Stable Diffusion

Developing reliable and generalizable deep learning systems for medical imaging faces significant obstacles due to spurious correlations, da… (voir plus)ta imbalances, and limited text annotations in datasets. Addressing these challenges requires architectures robust to the unique complexities posed by medical imaging data. The rapid advancements in vision-language foundation models within the natural image domain prompt the question of how they can be adapted for medical imaging tasks. In this work, we present PRISM, a framework that leverages foundation models to generate high-resolution, language-guided medical image counterfactuals using Stable Diffusion. Our approach demonstrates unprecedented precision in selectively modifying spurious correlations (the medical devices) and disease features, enabling the removal and addition of specific attributes while preserving other image characteristics. Through extensive evaluation, we show how PRISM advances counterfactual generation and enables the development of more robust downstream classifiers for clinically deployable solutions. To facilitate broader adoption and research, we make our code publicly available at https://github.com/Amarkr1/PRISM.

2025-03-26

MIDL.io/2025/Conference (présentation orale)

doi.org

openreview.net

StarFlow: Generating Structured Workflow Outputs From Sketch Images

Patrice Béchard

Chao Wang

Amirhossein Abaskohi

Juan A. Rodriguez

Christopher Pal

David Vázquez

Spandana Gella

Sai Rajeswar

Perouz Taslakian

2025-03-26

ArXiv (prépublication)

doi.org

arxiv.org

A Text-guided Protein Design Framework

Shengchao Liu

Yutao Zhu

Yanjing Li

Zhuoxinran Li

Jiarui Lu

Zhao Xu

Weili Nie

Anthony Gitter

Chaowei Xiao

Jian Tang

Arvind Ramanathan

Hongyu Guo

Anima Anandkumar

Current AI-assisted protein design mainly utilizes protein sequential and structural information. Meanwhile, there exists tremendous knowled… (voir plus)ge curated by humans in the text format describing proteins' high-level functionalities. Yet, whether the incorporation of such text data can help protein design tasks has not been explored. To bridge this gap, we propose ProteinDT, a multi-modal framework that leverages textual descriptions for protein design. ProteinDT consists of three subsequent steps: ProteinCLAP which aligns the representation of two modalities, a facilitator that generates the protein representation from the text modality, and a decoder that creates the protein sequences from the representation. To train ProteinDT, we construct a large dataset, SwissProtCLAP, with 441K text and protein pairs. We quantitatively verify the effectiveness of ProteinDT on three challenging tasks: (1) over 90% accuracy for text-guided protein generation; (2) best hit ratio on 12 zero-shot text-guided protein editing tasks; (3) superior performance on four out of six protein property prediction benchmarks.

2025-03-26

Nature Machine Intelligence (publié)

doi.org

arxiv.org

Assessing SAM for Tree Crown Instance Segmentation from Drone Imagery

Mélisande Teng

2025-03-25