Publications

Periodic agent-state based Q-learning for POMDPs
Amit Sinha
Matthieu Geist
Aditya Mahajan
The standard approach for Partially Observable Markov Decision Processes (POMDPs) is to convert them to a fully observed belief-state MDP. H… (see more)owever, the belief state depends on the system model and is therefore not viable in reinforcement learning (RL) settings. A widely used alternative is to use an agent state, which is a model-free, recursively updateable function of the observation history. Examples include frame stacking and recurrent neural networks. Since the agent state is model-free, it is used to adapt standard RL algorithms to POMDPs. However, standard RL algorithms like Q-learning learn a stationary policy. Our main thesis that we illustrate via examples is that because the agent state does not satisfy the Markov property, non-stationary agent-state based policies can outperform stationary ones. To leverage this feature, we propose PASQL (periodic agent-state based Q-learning), which is a variant of agent-state-based Q-learning that learns periodic policies. By combining ideas from periodic Markov chains and stochastic approximation, we rigorously establish that PASQL converges to a cyclic limit and characterize the approximation error of the converged periodic policy. Finally, we present a numerical experiment to highlight the salient features of PASQL and demonstrate the benefit of learning periodic policies over stationary policies.
2024-08-01
EWRL/2024/Workshop (accepted)
openreview.net
openreview.net
scHiCyclePred: a deep learning framework for predicting cell cycle phases from single-cell Hi-C data using multi-scale interaction information
Yingfu Wu
Zhenqi Shi
Xiangfei Zhou
Pengyu Zhang
Xiuhui Yang
Jun Ding
Hao Wu
2024-07-31
Communications Biology (published)
doi.org
Assessing Programming Task Difficulty for Efficient Evaluation of Large Language Models
Florian Tambon
Amin Nikanjam
Foutse Khomh
Giuliano Antoniol
2024-07-30
ArXiv (preprint)
doi.org
arxiv.org
Strong gravitational lensing as a probe of dark matter
Simona Vegetti
Simon Birrer
Giulia Despali
C. Fassnacht
Daniel A. Gilman
Yashar Hezaveh
L.
Laurence Perreault-Levasseur
J. McKean
D. Powell
Conor M. O'riordan
G.
Vernardos
Dark matter structures within strong gravitational lens galaxies and along their line of sight leave a gravitational imprint on the multiple… (see more) images of lensed sources. Strong gravitational lensing provides, therefore, a key test of different dark matter models in a way that is independent of the baryonic content of matter structures on subgalactic scales. In this chapter, we describe how galaxy-scale strong gravitational lensing observations are sensitive to the physical nature of dark matter. We provide a historical perspective of the field, and review its current status. We discuss the challenges and advances in terms of data, treatment of systematic errors and theoretical predictions, that will enable one to deliver a stringent and robust test of different dark matter models in the near future. With the advent of the next generation of sky surveys, the number of known strong gravitational lens systems is expected to increase by several orders of magnitude. Coupled with high-resolution follow-up observations, these data will provide a key opportunity to constrain the properties of dark matter with strong gravitational lensing.
2024-07-30
Space Science Reviews (published)
doi.org
arxiv.org
AAPM task group report 288: Recommendations for guiding radiotherapy event narratives
Bruce Thomadsen
Ajay Kapur
Bette Blankenship
Barrett Caldwell
Lindsey Claps
Joanne Cunningham
Jennifer Elee
Suzanne Evans
Eric Ford
Debbie Gilley
Sandra Hayden
Kathleen Hintenlang
Rishabh Kapoor
John Kildea
Linda Kroger
Ksenija Kujundzic
Qing Liang
Sasa Mutic
Anita O'Donovan
Michael O'Hara … (see 6 more)
Zoubir Ouhib
Jatinder Palta
Todd Pawlicki
William Salter
Stacey Schmidt
Sugata Tripathi
2024-07-29
Medical Physics (Lancaster) (published)
doi.org
Implicitly Bayesian Prediction Rules in Deep Learning
Bruno Mlodozeniec
David Scott Krueger
Richard Turner
The Bayesian approach leads to coherent updates of predictions under new data, which makes adhering to Bayesian principles appealing in deci… (see more)sion-making contexts. Traditionally, integrating Bayesian principles into models like deep neural networks involves setting priors on parameters and approximating posteriors. This is done despite the fact that, typically, priors on parameters reflect any prior beliefs only insofar as they dictate function space behaviour. In this paper, we rethink this approach and consider what properties characterise a prediction rule as being Bayesian. Algorithms meeting such criteria can be deemed implicitly Bayesian — they make the same predictions as some Bayesian model, without explicitly manifesting priors and posteriors. We argue this might be a more fruitful approach towards integrating Bayesian principles into deep learning. In this paper, we propose how to measure how close a general prediction rule is to being implicitly Bayesian, and empirically evaluate multiple prediction strategies using our approach. We also show theoretically that agents relying on non-implicitly Bayesian prediction rules can be easily exploited in adversarial betting settings.
2024-07-29
Proceedings of the 6th Symposium on Advances in Approximate Bayesian Inference (published)
proceedings.mlr.press
Long-term plasticity induces sparse and specific synaptic changes in a biophysically detailed cortical model
András Ecker
Daniela Egas Santander
Marwan Abdellah
Jorge Blanco Alonso
Sirio Bolaños-Puchet
Giuseppe Chindemi
Dhuruva Priyan Gowri Mariyappan
James B. Isbister
James King
Pramod Kumbhar
Ioannis Magkanaris
Eilif Benjamin Muller
Michael W. Reimann
2024-07-29
bioRxiv (preprint)
doi.org
scCross: A Deep Generative Model for Unifying Single-cell Multi-omics with Seamless Integration, Cross-modal Generation, and In-silico Exploration
Xiuhui Yang
Koren K. Mann
Hao Wu
Jun Ding
Single-cell multi-omics illuminate intricate cellular states, yielding transformative insights into cellular dynamics and disease. Yet, whil… (see more)e the potential of this technology is vast, the integration of its multifaceted data presents challenges. Some modalities have not reached the robustness or clarity of established scRNA-seq. Coupled with data scarcity for newer modalities and integration intricacies, these challenges limit our ability to maximize single-cell omics benefits. We introduce scCross: a tool adeptly engineered using variational autoencoder, generative adversarial network principles, and the Mutual Nearest Neighbors (MNN) technique for modality alignment. This synergy ensures seamless integration of varied single-cell multi-omics data. Beyond its foundational prowess in multi-omics data integration, scCross excels in single-cell cross-modal data generation, multi-omics data simulation, and profound in-silico cellular perturbations. Armed with these capabilities, scCross is set to transform the field of single-cell research, establishing itself in the nuanced integration, generation, and simulation of complex multi-omics data.
2024-07-29
Genome Biology (published)
doi.org
Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatment
Aamer Abdul Rahman
Pranav Agarwal
Rita Noumeir
Philippe Jouvet
Vincent Michalski
Samira Ebrahimi Kahou
Offline reinforcement learning has shown promise for solving tasks in safety-critical settings, such as clinical decision support. Its appli… (see more)cation, however, has been limited by the lack of interpretability and interactivity for clinicians. To address these challenges, we propose the medical decision transformer (MeDT), a novel and versatile framework based on the goal-conditioned reinforcement learning paradigm for sepsis treatment recommendation. MeDT uses the decision transformer architecture to learn a policy for drug dosage recommendation. During offline training, MeDT utilizes collected treatment trajectories to predict administered treatments for each time step, incorporating known treatment outcomes, target acuity scores, past treatment decisions, and current and past medical states. This analysis enables MeDT to capture complex dependencies among a patient's medical history, treatment decisions, outcomes, and short-term effects on stability. Our proposed conditioning uses acuity scores to address sparse reward issues and to facilitate clinician-model interactions, enhancing decision-making. Following training, MeDT can generate tailored treatment recommendations by conditioning on the desired positive outcome (survival) and user-specified short-term stability improvements. We carry out rigorous experiments on data from the MIMIC-III dataset and use off-policy evaluation to demonstrate that MeDT recommends interventions that outperform or are competitive with existing offline reinforcement learning methods while enabling a more interpretable, personalized and clinician-directed approach.
2024-07-28
ArXiv (preprint)
doi.org
arxiv.org
Reinforcement Learning Informed Evolutionary Search for Autonomous Systems Testing
Dmytro Humeniuk
Foutse Khomh
Giuliano Antoniol
Evolutionary search-based techniques are commonly used for testing autonomous robotic systems. However, these approaches often rely on compu… (see more)tationally expensive simulator-based models for test scenario evaluation. To improve the computational efficiency of the search-based testing, we propose augmenting the evolutionary search (ES) with a reinforcement learning (RL) agent trained using surrogate rewards derived from domain knowledge. In our approach, known as RIGAA (Reinforcement learning Informed Genetic Algorithm for Autonomous systems testing), we first train an RL agent to learn useful constraints of the problem and then use it to produce a certain part of the initial population of the search algorithm. By incorporating an RL agent into the search process, we aim to guide the algorithm towards promising regions of the search space from the start, enabling more efficient exploration of the solution space. We evaluate RIGAA on two case studies: maze generation for an autonomous ant robot and road topology generation for an autonomous vehicle lane keeping assist system. In both case studies, RIGAA converges faster to fitter solutions and produces a better test suite (in terms of average test scenario fitness and diversity). RIGAA also outperforms the state-of-the-art tools for vehicle lane keeping assist system testing, such as AmbieGen and Frenetic.
2024-07-27
ACM Transactions on Software Engineering and Methodology (published)
doi.org
arxiv.org
Canada's Provincial Covid-19 Pandemic Modelling Efforts: A Review of Mathematical Models and Their Impacts on the Responses
Yiqing Xia
Jorge Luis Flores Anato
Caroline Colijin
Naveed Janjua
Michael Otterstatter
Mike Irvine
Tyler Williamson
Marie B. Varughese
Michael Li
Nathaniel Osgood
David J. D. Earn
Beate Sander
Lauren E. Cipriano
Kumar Murty
Fanyu Xiu
Arnaud Godin
David Buckeridge
Amy Hurford
Sharmistha Mishra
Mathieu Maheu-Giroux
2024-07-25
Canadian Journal of Public Health (published)
doi.org
Development of Error Passing Network for Optimizing the Prediction of VO$_2$ peak in Childhood Acute Leukemia Survivors
Nicolas Raymond
Hakima Laribi
Maxime Caru
Mehdi Mitiche
Valerie Marcil
Maja Krajinovic
Daniel Curnier
Daniel Sinnett
Martin Vallières
Approximately two-thirds of survivors of childhood acute lymphoblastic leukemia (ALL) cancer develop late adverse effects post-treatment. Pr… (see more)ior studies explored prediction models for personalized follow-up, but none integrated the usage of neural networks to date. In this work, we propose the Error Passing Network (EPN), a graph-based method that leverages relationships between samples to propagate residuals and adjust predictions of any machine learning model. We tested our approach to estimate patients’ \vo peak, a reliable indicator of their cardiac health. We used the EPN in conjunction with several baseline models and observed up to 12.16% improvement in the mean average percentage error compared to the last established equation predicting \vo peak in childhood ALL survivors. Along with this performance improvement, our final model is more efficient considering that it relies only on clinical variables that can be self-reported by patients, therefore removing the previous need of executing a resource-consuming physical test.
2024-07-24
Proceedings of the fifth Conference on Health, Inference, and Learning (published)
proceedings.mlr.press