Pouya Bashivan

2024-09-25

NeurIPS.cc/2024/Conference (poster)

Burst firing optimizes invariant coding of natural communication signals by electrosensory neural populations

Michael G. Metzen

Amin Akhshi

Anmar Khadra

Maurice J. Chacron

2024-09-13

bioRxiv (preprint)

Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques

Rishika Bhagwatkar

Shravan Nayak

Reza Bayat

Alexis Roger

Daniel Z Kaplan

Irina Rish

Vision-Language Models (VLMs) have witnessed a surge in both research and real-world applications. However, as they becoming increasingly pr… (see more)evalent, ensuring their robustness against adversarial attacks is paramount. This work systematically investigates the impact of model design choices on the adversarial robustness of VLMs against image-based attacks. Additionally, we introduce novel, cost-effective approaches to enhance robustness through prompt formatting. By rephrasing questions and suggesting potential adversarial perturbations, we demonstrate substantial improvements in model robustness against strong image-based attacks such as Auto-PGD. Our findings provide important guidelines for developing more robust VLMs, particularly for deployment in safety-critical environments.

2024-06-28

ICML.cc/2024/Workshop/NextGenAISafety (poster)

iWISDM: Assessing instruction following in multimodal models at scale

Xiaoxuan Lei

Lucas Gomez

Hao Yuan Bai

The ability to perform complex tasks from detailed instructions is a key to the remarkable achievements of our species. As humans, we are no… (see more)t only capable of performing a wide variety of tasks but also very complex ones that may entail hundreds or thousands of steps to complete. Large language models and their more recent multimodal counterparts that integrate textual and visual inputs have achieved unprecedented success in performing complex tasks. Yet, most existing benchmarks are largely confined to single-modality inputs — either text or vision — and thus, narrowing the scope of multimodal integration assessments, particularly for instruction-following in multimodal contexts. To bridge this gap, we introduce the instructed-Virtual VISual Decision Making (iWISDM) environment engineered to generate a limitless array of vision-language tasks of varying complexity. Using iWISDM, we compiled three distinct benchmarks of instruction following visual tasks across varying complexity levels and evaluated several newly developed multimodal models on these benchmarks. Our findings establish iWISDM as a robust benchmark for assessing the instructional adherence of both existing and emergent multimodal models and highlight a large gap in these models’ ability to precisely follow instructions.

2024-06-17

ICML.cc/2024/Workshop/LLMs_and_Cognition (poster)

Local lateral connectivity is sufficient for replicating cortex-like topographical organization in deep neural networks

Xinyu Qian

Amirozhan Dehghani

Asa Borzabadi Farahani

Across the primate cortex, neurons that perform similar functions tend to be spatially grouped together. In high-level visual cortex, this w… (see more)idely observed biological rule manifests itself as a modular organization of neuronal clusters, each tuned to a specific object category. The tendency toward short connections is one of the most widely accepted views of why such an organization exists in the brains of many animals. Yet, how such a feat is implemented at the neural level remains unclear. Here, using artificial deep neural networks as test beds, we demonstrate that a topographical organization similar to that in the primary, intermediate, and high-level human visual cortex emerges when units in these models are laterally connected and their weight parameters are tuned by top-down credit assignment. Importantly, the emergence of the modular organization in the absence of explicit topography-inducing learning rules and objectives questions their necessity and suggests that local lateral connectivity alone may be sufficient for the formation of the topographic organization across the cortex.

2024-06-16

ICML.cc/2024/Workshop/AI4Science (poster)

A Hybrid CNN-Transformer Approach for Continuous Fine Finger Motion Decoding from sEMG Signals

Zihan Weng

Xiabing Zhang

Yufeng Mou

Chanlin Yi

Fali Li

Peng Xu

This work presents a novel approach that synergistically integrates convolutional neural networks (CNNs) and Transformer models for decoding… (see more) continuous fine finger motions from surface electromyography (sEMG) signals. This integration capitalizes on CNNs’ proficiency in extracting rich temporal and spatial features from multichannel sEMG data and the Transformer’s superior capability in recognizing complex patterns and long-range dependencies. A significant advancement in this field is the use of a custom-developed Epidermal Electrode Array Sleeve (EEAS) for capturing high-fidelity sEMG signals, enabling more accurate and reliable signal acquisition than traditional methods. The decoded joint angles could be used in seamless and intuitive human-machine interaction in various applications, such as virtual reality, augmented reality, robotic control, and prosthetic control. Evaluations demonstrate the superior performance of the proposed CNN-Transformer hybrid architecture in decoding continuous fine finger motions, outperforming individual CNN and Transformer models. The synergistic integration of CNNs and Transformers presents a powerful framework for sEMG decoding, offering exciting opportunities for naturalistic and intuitive human-machine interaction applications. Its robustness and efficiency make it an ideal choice for real-world applications, promising to enhance the interface between humans and machines significantly. The implications of this research extend to advancing the understanding of human neuromuscular signals and their application in computing interfaces.

2024-06-14

2024 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA) (published)

How well do models of visual cortex generalize to out of distribution samples?

Yifei Ren

2024-01-01

PLoS Comput. Biol. (published)

Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design.

Rishika Bhagwatkar

Shravan Nayak

Irina Rish

2024-01-01

EMNLP (Findings) (published)

The feature landscape of visual cortex

Rudi Tong

Ronan da Silva

Dongyan Lin

Arna Ghosh

James Wilsenach

Erica Cianfarano

Blake Richards

Stuart Trenholm

Understanding computations in the visual system requires a characterization of the distinct feature preferences of neurons in different visu… (see more)al cortical areas. However, we know little about how feature preferences of neurons within a given area relate to that area’s role within the global organization of visual cortex. To address this, we recorded from thousands of neurons across six visual cortical areas in mouse and leveraged generative AI methods combined with closed-loop neuronal recordings to identify each neuron’s visual feature preference. First, we discovered that the mouse’s visual system is globally organized to encode features in a manner invariant to the types of image transformations induced by self-motion. Second, we found differences in the visual feature preferences of each area and that these differences generalized across animals. Finally, we observed that a given area’s collection of preferred stimuli (‘own-stimuli’) drive neurons from the same area more effectively through their dynamic range compared to preferred stimuli from other areas (‘other-stimuli’). As a result, feature preferences of neurons within an area are organized to maximally encode differences among own-stimuli while remaining insensitive to differences among other-stimuli. These results reveal how visual areas work together to efficiently encode information about the external world.

2023-11-05

bioRxiv (preprint)

Using modular connectome-based predictive modeling to reveal brain-behavior relationships of individual differences in working memory

Huayi Yang

Junjun Zhang

Zhenlan Jin

Ling Li

2023-06-22

Brain Structure and Function (published)

Towards Out-of-Distribution Adversarial Robustness

Adam Ibrahim

Charles Guille-Escuret

Adversarial robustness continues to be a major challenge for deep learning. A core issue is that robustness to one type of attack often fail… (see more)s to transfer to other attacks. While prior work establishes a theoretical trade-off in robustness against different

2023-06-20

ICML.cc/2023/Workshop/AdvML-Frontiers (published)

Learning Robust Kernel Ensembles with Kernel Average Pooling

Adam Ibrahim

Amirozhan Dehghani

Yifei Ren

2022-09-30

ArXiv (preprint)