Publications

Benchmarking Vision Language Models for Cultural Understanding

Sjoerd van Steenkiste

Lisa Anne Hendricks

Karolina Stanczak

Foundation models and vision-language pre-training have notably advanced Vision Language Models (VLMs), enabling multimodal processing of vi… (see more)sual and linguistic data. However, their performance has been typically assessed on general scene understanding - recognizing objects, attributes, and actions - rather than cultural comprehension. This study introduces CulturalVQA, a visual question-answering benchmark aimed at assessing VLM's geo-diverse cultural understanding. We curate a collection of 2,378 image-question pairs with 1-5 answers per question representing cultures from 11 countries across 5 continents. The questions probe understanding of various facets of culture such as clothing, food, drinks, rituals, and traditions. Benchmarking VLMs on CulturalVQA, including GPT-4V and Gemini, reveals disparity in their level of cultural understanding across regions, with strong cultural understanding capabilities for North America while significantly lower performance for Africa. We observe disparity in their performance across cultural facets too, with clothing, rituals, and traditions seeing higher performances than food and drink. These disparities help us identify areas where VLMs lack cultural understanding and demonstrate the potential of CulturalVQA as a comprehensive evaluation set for gauging VLM progress in understanding diverse cultures.

2023-12-31

EMNLP (published)

doi.org

arxiv.org

Bio-Mechanical Poet: An Immersive Audiovisual Playground for Brain Signals and Generative AI.

Philipp Thölke

Antoine Bellemare‐Pepin

Yann Harel

François Lespinasse

Karim Jerbi CoCo Lab

2023-12-31

ICCC (published)

dblp.uni-trier.de

blocks: Version 0.2.0

A Theano framework for building and training neural networks

2023-12-31

International Conference on Machine Learning (published)

doi.org

CATRO: Channel Pruning via Class-Aware Trace Ratio Optimization

Wenzheng Hu

Ning Liu

Zhengping Che

Mingyang Li

Jian Tang

Changshui Zhang

Jianqiang Wang

Deep convolutional neural networks are shown to be overkill with high parametric and computational redundancy in many application scenarios,… (see more) and an increasing number of works have explored model pruning to obtain lightweight and efficient networks. However, most existing pruning approaches are driven by empirical heuristics and rarely consider the joint impact of channels, leading to unguaranteed and suboptimal performance. In this article, we propose a novel channel pruning method via class-aware trace ratio optimization (CATRO) to reduce the computational burden and accelerate the model inference. Utilizing class information from a few samples, CATRO measures the joint impact of multiple channels by feature space discriminations and consolidates the layerwise impact of preserved channels. By formulating channel pruning as a submodular set function maximization problem, CATRO solves it efficiently via a two-stage greedy iterative optimization procedure. More importantly, we present theoretical justifications on convergence of CATRO and performance of pruned networks. Experimental results demonstrate that CATRO achieves higher accuracy with similar computation cost or lower computation cost with similar accuracy than other state-of-the-art channel pruning algorithms. In addition, because of its class-aware property, CATRO is suitable to prune efficient networks adaptively for various classification subtasks, enhancing handy deployment and usage of deep networks in real-world applications.

2023-12-31

IEEE Trans. Neural Networks Learn. Syst. (published)

doi.org

arxiv.org

ChatGPT: What Every Pediatric Surgeon Should Know About Its Potential Uses and Pitfalls

Raquel González

Dan Poenaru

Russell Woo

A Francois Trappey

Stewart Carter

David Darcy

Ellen Encisco

Brian Gulack

Doug Miniati

Edzhem Tombash

Eunice Y. Huang

2023-12-31

Journal of Pediatric Surgery (published)

doi.org

CL-MASR: A Continual Learning Benchmark for Multilingual ASR

Luca Della Libera

Pooneh Mousavi

Salah Zaiem

Yusuf Cem Sübakan

Mirco Ravanaelli

2023-12-31

IEEE ACM Trans. Audio Speech Lang. Process. (published)

doi.org

arxiv.org

CODA: an open-source platform for federated analysis and machine learning on distributed healthcare data

Louis Mullie

Jonathan Afilalo

Patrick Archambault

Rima Bouchakri

Kip Brown

David L. Buckeridge

Yiorgos Alexandros Cavayas

Alexis F. Turgeon

Denis Martineau

François Lamontagne

Martine Lebrasseur

Renald Lemieux

Jeffrey Li

Michaël Sauthier

Pascal St-Onge

An Tang

William Witteman

Michaël Chassé

Distributed computations facilitate multi-institutional data analysis while avoiding the costs and complexity of data pooling. Existing appr… (see more)oaches lack crucial features, such as built-in medical standards and terminologies, no-code data visualizations, explicit disclosure control mechanisms, and support for basic statistical computations, in addition to gradient-based optimization capabilities. We describe the development of the Collaborative Data Analysis (CODA) platform, and the design choices undertaken to address the key needs identified during our survey of stakeholders. We use a public dataset (MIMIC-IV) to demonstrate end-to-end multi-modal FL using CODA. We assessed the technical feasibility of deploying the CODA platform at 9 hospitals in Canada, describe implementation challenges, and evaluate its scalability on large patient populations. The CODA platform was designed, developed, and deployed between January 2020 and January 2023. Software code, documentation, and technical documents were released under an open-source license. Multi-modal federated averaging is illustrated using the MIMIC-IV and MIMIC-CXR datasets. To date, 8 out of the 9 participating sites have successfully deployed the platform, with a total enrolment of >1M patients. Mapping data from legacy systems to FHIR was the biggest barrier to implementation. The CODA platform was developed and successfully deployed in a public healthcare setting in Canada, with heterogeneous information technology systems and capabilities. Ongoing efforts will use the platform to develop and prospectively validate models for risk assessment, proactive monitoring, and resource usage. Further work will also make tools available to facilitate migration from legacy formats to FHIR and DICOM.

2023-12-31

Journal of the American Medical Informatics Association (published)

doi.org

Common Challenges of Deep Reinforcement Learning Applications Development: An Empirical Study

Mohammad Mehdi Morovati

Florian Tambon

Mina Taraghi

Amin Nikanjam

Foutse Khomh

Machine Learning (ML) is increasingly being adopted in different industries. Deep Reinforcement Learning (DRL) is a subdomain of ML used to … (see more)produce intelligent agents. Despite recent developments in DRL technology, the main challenges that developers face in the development of DRL applications are still unknown. To fill this gap, in this paper, we conduct a large-scale empirical study of 927 DRL-related posts extracted from Stack Overflow, the most popular Q&A platform in the software community. Through the process of labeling and categorizing extracted posts, we created a taxonomy of common challenges encountered in the development of DRL applications, along with their corresponding popularity levels. This taxonomy has been validated through a survey involving 65 DRL developers. Results show that at least 45% of developers experienced 18 of the 21 challenges identified in the taxonomy. The most frequent source of difficulty during the development of DRL applications are Comprehension, API usage, and Design problems, while Parallel processing, and DRL libraries/frameworks are classified as the most difficult challenges to address, with respect to the time required to receive an accepted answer. We hope that the research community will leverage this taxonomy to develop efficient strategies to address the identified challenges and improve the quality of DRL applications.

2023-12-31

Empir. Softw. Eng. (published)

doi.org

arxiv.org

Cone-Traced Supersampling with Subpixel Edge Reconstruction.

Andrei Chubarau

Yangyang Zhao

Ruby Rao

D. Nowrouzezahrai

Paul Kry

While signed distance fields (SDFs) in theory offer infinite level of detail, they are typically rendered using the sphere tracing algorithm… (see more) at finite resolutions, which causes the common rasterized image synthesis problem of aliasing. Most existing optimized antialiasing solutions rely on polygon mesh representations; SDF-based geometry can only be directly antialiased with the computationally expensive supersampling or with post-processing filters that may produce undesirable blurriness and ghosting. In this work, we present cone-traced supersampling (CTSS), an efficient and robust spatial antialiasing solution that naturally complements the sphere tracing algorithm, does not require casting additional rays per pixel or offline prefiltering, and can be easily implemented in existing real-time SDF renderers. CTSS performs supersampling along the traced ray near surfaces with partial visibility – object contours – identified by evaluating cone intersections within a pixel's view frustum. We further introduce subpixel edge reconstruction (SER), a technique that extends CTSS to locate and resolve complex pixels with geometric edges in relatively flat regions, which are otherwise undetected by cone intersections. Our combined solution relies on a specialized sampling strategy to minimize the number of shading computations and correlates sample visibility to aggregate the samples. With comparable antialiasing quality at significantly lower computational cost, CTSS is a reliable practical alternative to conventional supersampling.

2023-12-31

IEEE Trans. Vis. Comput. Graph. (published)

doi.org

Connecting Weighted Automata, Tensor Networks and Recurrent Neural Networks through Spectral Learning

Tianyu Li

Doina Precup

Guillaume Rabusseau

In this paper, we present connections between three models used in different research fields: weighted finite automata~(WFA) from formal lan… (see more)guages and linguistics, recurrent neural networks used in machine learning, and tensor networks which encompasses a set of optimization techniques for high-order tensors used in quantum physics and numerical analysis. We first present an intrinsic relation between WFA and the tensor train decomposition, a particular form of tensor network. This relation allows us to exhibit a novel low rank structure of the Hankel matrix of a function computed by a WFA and to design an efficient spectral learning algorithm leveraging this structure to scale the algorithm up to very large Hankel matrices.We then unravel a fundamental connection between WFA and second-orderrecurrent neural networks~(2-RNN): in the case of sequences of discrete symbols, WFA and 2-RNN with linear activationfunctions are expressively equivalent. Leveraging this equivalence result combined with the classical spectral learning algorithm for weighted automata, we introduce the first provable learning algorithm for linear 2-RNN defined over sequences of continuous input vectors.This algorithm relies on estimating low rank sub-blocks of the Hankel tensor, from which the parameters of a linear 2-RNN can be provably recovered. The performances of the proposed learning algorithm are assessed in a simulation study on both synthetic and real-world data.

2023-12-31

Mach. Learn. (published)

doi.org

arxiv.org

Consolidating Separate Degradations Model via Weights Fusion and Distillation

Dinesh Daultani

Hugo Larochelle

Real-world images prevalently contain different varieties of degradation, such as motion blur and luminance noise. Computer vision recogniti… (see more)on models trained on clean images perform poorly on degraded images. Previously, several works have explored how to perform image classification of degraded images while training a single model for each degradation. Nevertheless, it becomes challenging to host several degradation models for each degradation on limited hardware applications and to estimate degradation parameters correctly at the run-time. This work proposes a method for effectively combining several models trained separately on different degradations into a single model to classify images with different types of degradations. Our proposed method is four-fold: (1) train a base model on clean images, (2) fine-tune the base model in-dividually for all given image degradations, (3) perform a fusion of weights given the fine-tuned models for individual degradations, (4) perform fine-tuning on given task using distillation and cross-entropy loss. Our proposed method can outperform previous state-of-the-art methods of pretraining in out-of-distribution generalization based on degradations such as JPEG compression, salt-and-pepper noise, Gaussian blur, and additive white Gaussian noise by 2.5% on CIFAR-100 dataset and by 1.3% on CIFAR-10 dataset. Moreover, our proposed method can handle degra-dation used for training without any explicit information about degradation at the inference time. Code will be available at https://github.com/dineshdaultani/FusionDistill.

2023-12-31

2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) (published)

doi.org

Context-Aware Assistant Selection for Improved Inference Acceleration with Large Language Models

Jerry Huang

Prasanna Parthasarathi

Mehdi Rezagholizadeh

A. Chandar

2023-12-31

EMNLP (published)

doi.org

arxiv.org