Publications

VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Text
Tianyu Zhang
Suyuchen Wang
Lu Li
Ge Zhang
Perouz Taslakian
Sai Rajeswar
Jie Fu
Bang Liu
Yoshua Bengio
We introduce Visual Caption Restoration (VCR), a novel vision-language task that challenges models to accurately restore partially obscured … (see more)texts using pixel-level hints within images. This task stems from the observation that text embedded in images is intrinsically different from common visual elements and natural language due to the need to align the modalities of vision, text, and text embedded in images. While numerous works have integrated text embedded in images into visual question-answering tasks, approaches to these tasks generally rely on optical character recognition or masked language modeling, thus reducing the task to mainly text-based processing. However, text-based processing becomes ineffective in VCR as accurate text restoration depends on the combined information from provided images, context, and subtle cues from the tiny exposed areas of masked texts. We develop a pipeline to generate synthetic images for the VCR task using image-caption pairs, with adjustable caption visibility to control the task difficulty. With this pipeline, we construct a dataset for VCR called VCR-Wiki using images with captions from Wikipedia, comprising 2.11M English and 346K Chinese entities in both easy and hard split variants. Our results reveal that current vision language models significantly lag behind human performance in the VCR task, and merely fine-tuning the models on our dataset does not lead to notable improvements. We release VCR-Wiki and the data construction code to facilitate future research.
2025-04-22
International Conference on Learning Representations (Accept (Poster))
doi.org
openreview.net
LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities
Thomas Schmied
Jörg Bornschein
Jordi Grau-Moya
Markus Wulfmeier
Razvan Pascanu
2025-04-21
ArXiv (preprint)
doi.org
arxiv.org
Neural Kinematic Bases for Fluids
Yibo Liu
Paul Kry
Kenny Erleben
Noam Aigerman
Sune Darkner
Teseo Schneider
2025-04-21
ArXiv (preprint)
doi.org
arxiv.org
Cortical differences across psychiatric disorders and associated common and rare genetic variants
Kuldeep Kumar
Zhijie Liao
Jakub Kopal
Clara Moreau
Christopher R. K. Ching
Claudia Modenato
Will Snyder
Sayeh Kazem
Charles-Olivier Martin
C.O. Martin
Anne-Marie Bélanger
Valérie K. Fontaine
Khadije Jizi
Rune Boen
Guillaume Huguet
Zohra Saci
Leila Kushan
Ana I. Silva
Marianne B.M. van den Bree
David E.J. Linden … (see 16 more)
Michael J. Owen
Jeremy Hall
Sarah Lippé
Guillaume Dumas
Bogdan Draganski
Laura Almasy
Sophia I. Thomopoulos
Neda Jahanshad
Ida E. Sønderby
Ole A. Andreassen
David C. Glahn
Armin Raznahan
Carrie Bearden
Tomáš Paus
Paul M. Thompson
Sébastien Jacquemont
2025-04-18
medRxiv (preprint)
doi.org
Deep Learning Unlocks the True Potential of Organ Donation after Circulatory Death with Accurate Prediction of Time-to-Death
Xingzhi Sun
Edward De Brouwer
Chen Liu
Smita Krishnaswamy
Ramesh Batra
𝟏
Increasing the number of organ donations after circulatory death (DCD) has been identified as one of the most important ways of addressing t… (see more)he ongoing organ shortage. While recent technological advances in organ transplantation have increased their success rate, a substantial challenge in increasing the number of DCD donations resides in the uncertainty regarding the timing of cardiac death after terminal extubation, impacting the risk of prolonged ischemic organ injury, and negatively affecting post-transplant outcomes. In this study, we trained and externally validated an ODE-RNN model, which combines recurrent neural network with neural ordinary equations and excels in processing irregularly-sampled time series data. The model is designed to predict time-to-death following terminal extubation in the intensive care unit (ICU) using the last 24 hours of clinical observations. Our model was trained on a cohort of 3,238 patients from Yale New Haven Hospital, and validated on an external cohort of 1,908 patients from six hospitals across Connecticut. The model achieved accuracies of 95.3 {+/-} 1.0% and 95.4 {+/-} 0.7% for predicting whether death would occur in the first 30 and 60 minutes, respectively, with a calibration error of 0.024 {+/-} 0.009. Heart rate, respiratory rate, mean arterial blood pressure (MAP), oxygen saturation (SpO2), and Glasgow Coma Scale (GCS) scores were identified as the most important predictors. Surpassing existing clinical scores, our model sets the stage for reduced organ acquisition costs and improved post-transplant outcomes.
2025-04-18
Scientific Reports (published)
doi.org
Impact of Reducing Time Lived With Colostomies on Social Stigma Affecting Children With Anorectal Malformations in Southwestern Uganda.
Felix Oyania
Caroline Q. Stephens
Sarah Ullrich
Meera Kotagal
Amy M. Shui
Caleb Tuhumwire
Godfrey Zari Rukundo
Joseph Ngonzi
Ava Yap
Francis Bajunirwe
Dan Poenaru
Doruk Ozgediz
BACKGROUND The social stigma of families of children living with colostomies due to anorectal malformation (ARM) is significant in low-incom… (see more)e countries (LICs). Improved access to pediatric surgery has resulted in more 1-stage ARM procedures in Southwestern Uganda, avoiding colostomy creation, but the impact on social stigma experienced by families is unknown. We hypothesized that this change would decrease the social stigma experienced by families. METHODS A single-center mixed retrospective and prospective cohort study with combined qualitative data of families of children with ARM who underwent corrective surgery compared the stigma experienced by those with colostomies to those without. The Kilifi Stigma Scale of Epilepsy (KSSE) was used to assess social stigma. Multivariable regression analysis assessed differences in the stigma experienced, controlling for age at diagnosis, rurality, distance traveled, sex, and parental education. Subgroup analysis assessed the impact of colostomy duration on stigma, stratified over parental education. RESULTS Patient/family dyads with 238 ARM were included; 177 (74%) received a colostomy. Most patients were male (51%), lived in rural areas (71%), and had parents with primary school education (65%). For those without a colostomy, the median KSSE was 0 (Q1-Q3 0-0), compared to 11 (Q1-Q3 3-20) for colostomy. On multivariable analysis, after controlling for age at diagnosis, rurality, distance traveled, sex, and parental education attainment, families of patients with ARM who received a colostomy had a median KSSE score 7.8 points higher than those who did not receive a colostomy (coefficient 7.78, 95% 3.14-12.43, and p = 0.001). When the duration of colostomy (in years) was examined, the median KSSE score increased by 1.58 points for each additional year for a patient who had a colostomy (IRR 1.58, 95% CI: 0.76-2.40, and p  0.001). CONCLUSION Adopting a 1-stage ARM repair for the select types, which avoids colostomy creation, significantly reduces the exper
2025-04-18
World Journal of Surgery (published)
doi.org
Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation
Muhammad Haseeb Aslam
Clara Martinez
Marco Pedersoli
Alessandro Lameiras Koerich
Ali Etemad
Eric Granger
Advances in self-distillation have shown that when knowledge is distilled from a teacher to a student using the same deep learning (DL) arch… (see more)itecture, the student performance can surpass the teacher particularly when the network is overparameterized and the teacher is trained with early stopping. Alternatively, ensemble learning also improves performance, although training, storing, and deploying multiple models becomes impractical as the number of models grows. Even distilling an ensemble to a single student model or weight averaging methods first requires training of multiple teacher models and does not fully leverage the inherent stochasticity for generating and distilling diversity in DL models. These constraints are particularly prohibitive in resource-constrained or latency-sensitive applications such as wearable devices. This paper proposes to train only one model and generate multiple diverse teacher representations using distillation-time dropout. However, generating these representations stochastically leads to noisy representations that are misaligned with the learned task. To overcome this problem, a novel stochastic self-distillation (SSD) training strategy is introduced for filtering and weighting teacher representation to distill from task-relevant representations only, using student-guided knowledge distillation (SGKD). The student representation at each distillation step is used as authority to guide the distillation process. Experimental results on real-world affective computing, wearable/biosignal datasets from the UCR Archive, the HAR dataset, and image classification datasets show that the proposed SSD method can outperform state-of-the-art methods without increasing the model size at both training and testing time, and incurs negligible computational complexity compared to state-of-the-art ensemble learning and weight averaging methods.
2025-04-18
ArXiv (preprint)
doi.org
arxiv.org
Online HD-tRNS over the right temporoparietal junction modulates social inference but not motor coordination
Quentin Moreau
Vincent Chamberland
Lisane Moses
Gabriela Milanova
Guillaume Dumas
Social interactions are fundamental to human cognition, with the right temporoparietal junction (rTPJ) playing a key role in integrating mot… (see more)or coordination and social inference. While transcranial random noise stimulation (tRNS) is a promising technique for modulating cortical excitability in real time, its effect on dynamic social processes remains largely unexplored. This study applied high-definition tRNS (HD-tRNS) over the rTPJ during an interactive task to modulate motor coordination and social inference. Eighty neurotypical adults (49 female) were equally distributed across two experiments: Experiment 1, a block design with randomized active and sham stimulation blocks; or Experiment 2, a trial-by-trial design with intermixed stimulation protocols. Participants performed a coordination task with a covert virtual partner programmed to behave cooperatively or competitively. Kinematic data and self-reported attributions of humanness and cooperativeness were analyzed. The results showed that HD-tRNS over the rTPJ did not affect motor coordination or overall task performance in either experiment. However, in Experiment 1, active stimulation progressively reduced attributed humanness and cooperativeness toward the competitive virtual partner, suggesting enhanced detection of antagonistic intent. This gradual modulation of social inference was absent in Experiment 2, where frequent protocol switching likely disrupted the buildup of stimulation effects. Together, these findings highlight the rTPJ's causal role in self–other distinction, underscore the importance of stimulation protocol design in shaping social cognition, and support the exploration of targeted neuromodulation in clinical and developmental populations with atypical social cognition.
2025-04-18
eNeuro (published)
doi.org
DTractor enhances cell type deconvolution in spatial transcriptomics by integrating deep neural networks, transfer learning, and matrix factorization
Yong Jin Kweon
Chenyu Liu
Gregory Fonseca
Jun Ding
Spatial transcriptomics (ST) captures gene expression with spatial context but lacks single-cell resolution. Single-cell RNA sequencing (scR… (see more)NA-seq) offers high-resolution profiles without spatial information. Accurate spot-level decomposition requires effective integration of both. We present DTractor, a deep learning-based framework that improves cell-type deconvolution in ST data through spatial constraints and transfer learning. DTractor achieves dual utilization of scRNA-seq reference data by incorporating both a cell-type-specific gene expression matrix and learned latent embeddings into a unified matrix factorization model. This joint modeling enables accurate estimation of cell-type proportions and cell-type-resolved gene expression within each spatial spot, while preserving biological and spatial coherence. DTractor further applies spatial regularization to maintain local tissue structure. Across multiple ST platforms and tissue types, DTractor demonstrates improved decomposition accuracy, robustness, and interpretability compared to existing methods. The results from DTractor support downstream applications such as spatial domain analysis and the study of spatially organized cellular behaviors.
2025-04-17
bioRxiv (preprint)
doi.org
GeoLS: an Intensity-based, Geodesic Soft Labeling for Image Segmentation
Sukesh Adiga Vasudeva
Jose Dolz
Hervé Lombaert
Soft-label assignments have emerged as prominent strategies in training dense prediction problems, such as image segmentation. These approac… (see more)hes mitigate the limitations of hard labels, such as inter-class relationships in the data and spatial relationships between a given pixel and its neighbors. Nevertheless, most existing methods rely only on ground-truth masks and ignore the underlying image context associated with each label. For instance, image intensities convey information that could potentially clear ambiguities in the annotation. This paper, therefore, proposes a Geodesic Label Smoothing (GeoLS) approach that incorporates image intensity information within the soft labeling process. Specifically, we leverage the geodesic distance transform to capture the intensity variations between pixels. The generated maps geodesically modify the hard labels to obtain new intensity-based soft labels. The resulting geodesic soft labels better model spatial and class-wise relationships as they capture the variations of image gradients across classes and anatomy. The benefits of our intensity-based geodesic soft labels are assessed on three diverse sets of publicly accessible segmentation datasets. Our experimental results show that the proposed method consistently improves the segmentation accuracy compared to state-of-the-art soft-labeling techniques in terms of the Dice similarity and Hausdorff distance.
2025-04-17
Machine Learning for Biomedical Imaging (published)
doi.org
Multimodal population study reveals the neurobiological underpinnings of chronotype
Le Zhou
Karin Saltoun
Julie Carrier
Kai-Florian Storch
Robin I. M. Dunbar
Danilo Bzdok
2025-04-16
Nature Human Behaviour (unknown)
doi.org
Sliced-Wasserstein Distance-based Data Selection
Julien Pallage
Antoine Lesage-Landry
We propose a new unsupervised anomaly detection method based on the sliced-Wasserstein distance for training data selection in machine learn… (see more)ing approaches. Our filtering technique is interesting for decision-making pipelines deploying machine learning models in critical sectors, e.g., power systems, as it offers a conservative data selection and an optimal transport interpretation. To ensure the scalability of our method, we provide two efficient approximations. The first approximation processes reduced-cardinality representations of the datasets concurrently. The second makes use of a computationally light Euclidian distance approximation. Additionally, we open the first dataset showcasing localized critical peak rebate demand response in a northern climate. We present the filtering patterns of our method on synthetic datasets and numerically benchmark our method for training data selection. Finally, we employ our method as part of a first forecasting benchmark for our open-source dataset.
2025-04-16
ArXiv (preprint)
doi.org
arxiv.org