Foutse Khomh

Biography

Foutse Khomh is a full professor of software engineering at Polytechnique Montréal, a Canada CIFAR AI Chair – Trustworthy Machine Learning Software Systems, and an FRQ-IVADO Research Chair in Software Quality Assurance for Machine Learning Applications. Khomh completed a PhD in software engineering at Université de Montréal in 2011, for which he received an Award of Excellence. He was also awarded a CS-Can/Info-Can Outstanding Young Computer Science Researcher Prize in 2019.

His research interests include software maintenance and evolution, machine learning systems engineering, cloud engineering, and dependable and trustworthy ML/AI. His work has received four Ten-year Most Influential Paper (MIP) awards, and six Best/Distinguished Paper Awards. He has served on the steering committee of numerous organizations in software engineering, including SANER (chair), MSR, PROMISE, ICPC (chair), and ICSME (vice-chair). He initiated and co-organized Polytechnique Montréal‘s Software Engineering for Machine Learning Applications (SEMLA) symposium and the RELENG (release engineering) workshop series.

Khomh co-founded the NSERC CREATE SE4AI: A Training Program on the Development, Deployment and Servicing of Artificial Intelligence-based Software Systems, and is a principal investigator for the DEpendable Explainable Learning (DEEL) project.

He also co-founded Confiance IA, a Quebec consortium focused on building trustworthy AI, and is on the editorial board of multiple international software engineering journals, including IEEE Software, EMSE and JSEP. He is a senior member of IEEE.

Current Students

Nanda Assobjio Brice Yvan

Collaborating Alumni - Polytechnique Montréal

Gabriel Laberge

PhD - Polytechnique Montréal

Github

forough majidi

PhD - Polytechnique Montréal

Website

Mo Malekpour

Master's Research - Polytechnique Montréal

Website

Github

Elnathan Tiokou Tiokou Fangang

Mohamed Amine Merzouk

Postdoctorate - Polytechnique Montréal

Co-supervisor :

Master's Research - Polytechnique Montréal

Master's Research - Polytechnique Montréal

Ben Braiek Yasmine

Master's Research - Polytechnique Montréal

Publications

Towards Optimizing SQL Generation via LLM Routing

Mohammadhossein Malekpour

Nour Shaheen

Amine Mhedhbi

Text-to-SQL enables users to interact with databases through natural language, simplifying access to structured data. Although highly capabl… (see more)e large language models (LLMs) achieve strong accuracy for complex queries, they incur unnecessary latency and dollar cost for simpler ones. In this paper, we introduce the first LLM routing approach for Text-to-SQL, which dynamically selects the most cost-effective LLM capable of generating accurate SQL for each query. We present two routing strategies (score- and classification-based) that achieve accuracy comparable to the most capable LLM while reducing costs. We design the routers for ease of training and efficient inference. In our experiments, we highlight a practical and explainable accuracy-cost trade-off on the BIRD dataset.

2024-11-06

ArXiv (preprint)

Impact of LLM-based Review Comment Generation in Practice: A Mixed Open-/Closed-source User Study

Doriane Olewicki

Leuson Da Silva

Suhaib Mujahid

Arezou Amini

Benjamin Mah

Marco Castelluccio

Sarra Habchi

Bram Adams

2024-11-01

arXiv (published)

Tracing Optimization for Performance Modeling and Regression Detection

Kaveh Shahedi

Heng Li

Maxime Lamothe

Software performance modeling plays a crucial role in developing and maintaining software systems. A performance model analytically describe… (see more)s the relationship between the performance of a system and its runtime activities. This process typically examines various aspects of a system's runtime behavior, such as the execution frequency of functions or methods, to forecast performance metrics like program execution time. By using performance models, developers can predict expected performance and thereby effectively identify and address unexpected performance regressions when actual performance deviates from the model's predictions. One common and precise method for capturing performance behavior is software tracing, which involves instrumenting the execution of a program, either at the kernel level (e.g., system calls) or application level (e.g., function calls). However, due to the nature of tracing, it can be highly resource-intensive, making it impractical for production environments where resources are limited. In this work, we propose statistical approaches to reduce tracing overhead by identifying and excluding performance-insensitive code regions, particularly application-level functions, from tracing while still building accurate performance models that can capture performance degradations. By selecting an optimal set of functions to be traced, we can construct optimized performance models that achieve an R-2 score of up to 99% and, sometimes, outperform full tracing models (models using non-optimized tracing data), while significantly reducing the tracing overhead by more than 80% in most cases. Our optimized performance models can also capture performance regressions in our studied programs effectively, demonstrating their usefulness in real-world scenarios. Our approach is fully automated, making it ready to be used in production environments with minimal human effort.

2024-11-01

arXiv (published)

Doctoral Symposium Committee

Anthony Cleve

Christian Lange

Silvia Breu

Manar H. Alalfi

Mario Luca Bernardi

Cornelia Boldyreff

Marco D'Ambros

Simon Denier

Natalia Dragan

Ekwa Duala-Ekoko

Fausto Fasano

Adnane Ghannem

Carmine Gravino

Maen Hammad

Imed Hammouda

Salima Hassaine

Yue Jia

Zhen Ming (Jack) Jiang

Adam Kiezun … (see 11 more)

Jay Kothari

Jonathan Memaitre

Naouel Moha

Rocco Oliveto

Denys Poshyvanyk

Michele Risi

Giuseppe Scanniello

Bonita Sharif

Andrew Sutton

Anis Yousefi

Eugenio Zimeo

Manar H. Alalfi Mario Luca Bernardi Cornelia Boldyreff Anthony Cleve Marco D'Ambros Simon Denier Natalia Dragan Ekwa Duala-Ekoko Fausto Fasa… (see more)no Adnane Ghannem Carmine Gravino Maen Hammad Imed Hammouda Salima Hassaine Yue Jia Zhen Ming Jiang Foutse Khomh Adam Kiezun Jay Kothari Jonathan Memaitre Naouel Moha Rocco Oliveto Denys Poshyvanyk Michele Risi Giuseppe Scanniello Bonita Sharif Andrew Sutton Anis Yousefi Eugenio Zimeo

2024-10-28

2024 IEEE 35th International Symposium on Software Reliability Engineering Workshops (ISSREW) (published)

Doctoral Symposium Committee

Anthony Cleve

Christian Lange

Silvia Breu

Manar H. Alalfi

Mario Luca Bernardi

Cornelia Boldyreff

Marco D'Ambros

Simon Denier

Natalia Dragan

Ekwa Duala-Ekoko

Fausto Fasano

Adnane Ghannem

Carmine Gravino

Maen Hammad

Imed Hammouda

Salima Hassaine

Yue Jia

Zhen Ming Jiang

Adam Kiezun … (see 11 more)

Jay Kothari

Jonathan Memaitre

Naouel Moha

Rocco Oliveto

Denys Poshyvanyk

Michele Risi

Giuseppe Scanniello

Bonita Sharif

Andrew Sutton

Anis Yousefi

Eugenio Zimeo

2024-10-28

2024 IEEE 35th International Symposium on Software Reliability Engineering Workshops (ISSREW) (published)

In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic Manipulators

Dmytro Humeniuk

Houssem Ben Braiek

Thomas Reid

2024-10-27

Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (published)

LIBS-Raman Multimodal Architecture for Automated Lunar Prospecting

Jérôme Pigeon

Richard Boudreault

Ahmed Ashraf

Pooneh Maghoul

2024-10-10

Earth and Space 2024 (published)

LIBS-Raman Multimodal Architecture for Automated Lunar Prospecting

Jérôme Pigeon

Richard Boudreault

Ahmed Ashraf

P. Maghoul

2024-10-10

Earth and Space 2024 (published)

Toward Debugging Deep Reinforcement Learning Programs with RLExplorer

Rached Bouchoucha

Ahmed Haj Yahmed

Darshan Patil

Janarthanan Rajendran

Amin Nikanjam

Sarath Chandar

Deep reinforcement learning (DRL) has shown success in diverse domains such as robotics, computer games, and recommendation systems. However… (see more), like any other software system, DRL-based software systems are susceptible to faults that pose unique challenges for debugging and diagnosing. These faults often result in unexpected behavior without explicit failures and error messages, making debugging difficult and time-consuming. Therefore, automating the monitoring and diagnosis of DRL systems is crucial to alleviate the burden on developers. In this paper, we propose RLExplorer, the first fault diagnosis approach for DRL-based software systems. RLExplorer automatically monitors training traces and runs diagnosis routines based on properties of the DRL learning dynamics to detect the occurrence of DRL-specific faults. It then logs the results of these diagnoses as warnings that cover theoretical concepts, recommended practices, and potential solutions to the identified faults. We conducted two sets of evaluations to assess RLExplorer. Our first evaluation of faulty DRL samples from Stack Overflow revealed that our approach can effectively diagnose real faults in 83% of the cases. Our second evaluation of RLExplorer with 15 DRL experts/developers showed that (1) RLExplorer could identify 3.6 times more defects than manual debugging and (2) RLExplorer is easily integrated into DRL applications.

2024-10-06

2024 IEEE International Conference on Software Maintenance and Evolution (ICSME) (published)

Toward Debugging Deep Reinforcement Learning Programs with RLExplorer

Rached Bouchoucha

Ahmed Haj Yahmed

Darshan Patil

Janarthanan Rajendran

Amin Nikanjam

Sarath Chandar

2024-10-06

ArXiv (preprint)

Understanding Web Application Workloads and Their Applications: Systematic Literature Review and Characterization

Roozbeh Aghili

Qiaolin Qin

Heng Li

2024-10-06

2024 IEEE International Conference on Software Maintenance and Evolution (ICSME) (published)

What Information Contributes to Log-based Anomaly Detection? Insights from a Configurable Transformer-Based Approach

Xingfang Wu

Heng Li

Log data are generated from logging statements in the source code, providing insights into the execution processes of software applications … (see more)and systems. State-of-the-art log-based anomaly detection approaches typically leverage deep learning models to capture the semantic or sequential information in the log data and detect anomalous runtime behaviors. However, the impacts of these different types of information are not clear. In addition, existing approaches have not captured the timestamps in the log data, which can potentially provide more fine-grained temporal information than sequential information. In this work, we propose a configurable transformer-based anomaly detection model that can capture the semantic, sequential, and temporal information in the log data and allows us to configure the different types of information as the model's features. Additionally, we train and evaluate the proposed model using log sequences of different lengths, thus overcoming the constraint of existing methods that rely on fixed-length or time-windowed log sequences as inputs. With the proposed model, we conduct a series of experiments with different combinations of input features to evaluate the roles of different types of information in anomaly detection. When presented with log sequences of varying lengths, the model can attain competitive and consistently stable performance compared to the baselines. The results indicate that the event occurrence information plays a key role in identifying anomalies, while the impact of the sequential and temporal information is not significant for anomaly detection in the studied public datasets. On the other hand, the findings also reveal the simplicity of the studied public datasets and highlight the importance of constructing new datasets that contain different types of anomalies to better evaluate the performance of anomaly detection models.

2024-09-30

ArXiv (preprint)