Foutse Khomh

Biography

Foutse Khomh is a full professor of software engineering at Polytechnique Montréal, a Canada CIFAR AI Chair – Trustworthy Machine Learning Software Systems, and an FRQ-IVADO Research Chair in Software Quality Assurance for Machine Learning Applications. Khomh completed a PhD in software engineering at Université de Montréal in 2011, for which he received an Award of Excellence. He was also awarded a CS-Can/Info-Can Outstanding Young Computer Science Researcher Prize in 2019.

His research interests include software maintenance and evolution, machine learning systems engineering, cloud engineering, and dependable and trustworthy ML/AI. His work has received four Ten-year Most Influential Paper (MIP) awards, and six Best/Distinguished Paper Awards. He has served on the steering committee of numerous organizations in software engineering, including SANER (chair), MSR, PROMISE, ICPC (chair), and ICSME (vice-chair). He initiated and co-organized Polytechnique Montréal‘s Software Engineering for Machine Learning Applications (SEMLA) symposium and the RELENG (release engineering) workshop series.

Khomh co-founded the NSERC CREATE SE4AI: A Training Program on the Development, Deployment and Servicing of Artificial Intelligence-based Software Systems, and is a principal investigator for the DEpendable Explainable Learning (DEEL) project.

He also co-founded Confiance IA, a Quebec consortium focused on building trustworthy AI, and is on the editorial board of multiple international software engineering journals, including IEEE Software, EMSE and JSEP. He is a senior member of IEEE.

Current Students

Houssem Ben Braiek

Postdoctorate - Polytechnique Montréal

PhD - Polytechnique Montréal

Github

forough majidi

PhD - Polytechnique Montréal

Website

Khouloud Oueslati

Master's Research - Polytechnique Montréal

Arian Qazvini

Master's Research - Polytechnique Montréal

Website

Github

Elnathan Tiokou Tiokou Fangang

Master's Research - Polytechnique Montréal

Ben Braiek Yasmine

Master's Research - Polytechnique Montréal

Publications

Bugs in Large Language Models Generated Code: An Empirical Study

Florian Tambon

Arghavan Moradi Dakhel

Amin Nikanjam

Michel C. Desmarais

Giuliano Antoniol

Large Language Models (LLMs) for code have gained significant attention recently. They can generate code in different programming languages … (see more)based on provided prompts, fulfilling a long-lasting dream in Software Engineering (SE), i.e., automatic code generation. Similar to human-written code, LLM-generated code is prone to bugs, and these bugs have not yet been thoroughly examined by the community. Given the increasing adoption of LLM-based code generation tools (e.g., GitHub Copilot) in SE activities, it is critical to understand the characteristics of bugs contained in code generated by LLMs. This paper examines a sample of 333 bugs collected from code generated using three leading LLMs (i.e., CodeGen, PanGu-Coder, and Codex) and identifies the following 10 distinctive bug patterns: Misinterpretations, Syntax Error, Silly Mistake, Prompt-biased code, Missing Corner Case, Wrong Input Type, Hallucinated Object, Wrong Attribute, Incomplete Generation, and Non-Prompted Consideration. The bug patterns are presented in the form of a taxonomy. The identified bug patterns are validated using an online survey with 34 LLM practitioners and researchers. The surveyed participants generally asserted the significance and prevalence of the bug patterns. Researchers and practitioners can leverage these findings to develop effective quality assurance techniques for LLM-generated code. This study sheds light on the distinctive characteristics of LLM-generated code.

2025-02-12

Empirical Software Engineering (published)

Application of deep reinforcement learning for intrusion detection in Internet of Things: A systematic review

Saeid Jamshidi

Amin Nikanjam

Kawser Wazed Nafi

Rasoul Rasta

2025-01-31

Internet of Things (published)

Anti-patterns and Code Smells for Multi-language Systems

Mouna Abidi

Manel Grichi

Yann‐Gaël Guéhéneuc

2024-12-31

Transactions on Pattern Languages of Programming (published)

Automated UML Visualization of Software Ecosystems: Tracking Versions, Dependencies, and Security Updates

Vanessa Kan

M. P. Lnu

Solomon Berhe

C. El Kari

Marc Maynard

2024-12-31

ANT/EDI40 (published)

Correction to: Assessing the adoption of security policies by developers in terraform across different cloud providers

Alexandre Verdet

Mohammad Hamdaqa

Leuson Da Silva

2024-12-31

Empir. Softw. Eng. (published)

Diffusion-Based Adversarial Purification for Intrusion Detection

Mohamed Amine Merzouk

Erwan Beurier

Reda Yaich

N. Cuppens-Boulahia

Frédéric Cuppens

2024-12-31

Database Security (published)

Evaluating machine learning-driven intrusion detection systems in IoT: Performance and energy consumption

Saeid Jamshidi

Kawser Wazed Nafi

Amin Nikanjam

2024-12-31

Comput. Ind. Eng. (published)

MLOps, LLMOps, FMOps, and Beyond

Chakkrit Kla Tantithamthavorn

Fabio Palomba

Joselito Joey Chua

2024-12-31

IEEE Software (published)

Trained Without My Consent: Detecting Code Inclusion In Language Models Trained on Code

Vahid Majdinasab

Amin Nikanjam

Code auditing ensures that the developed code adheres to standards, regulations, and copyright protection by verifying that it does not cont… (see more)ain code from protected sources. The recent advent of Large Language Models (LLMs) as coding assistants in the software development process poses new challenges for code auditing. The dataset for training these models is mainly collected from publicly available sources. This raises the issue of intellectual property infringement as developers' codes are already included in the dataset. Therefore, auditing code developed using LLMs is challenging, as it is difficult to reliably assert if an LLM used during development has been trained on specific copyrighted codes, given that we do not have access to the training datasets of these models. Given the non-disclosure of the training datasets, traditional approaches such as code clone detection are insufficient for asserting copyright infringement. To address this challenge, we propose a new approach, TraWiC; a model-agnostic and interpretable method based on membership inference for detecting code inclusion in an LLM's training dataset. We extract syntactic and semantic identifiers unique to each program to train a classifier for detecting code inclusion. In our experiments, we observe that TraWiC is capable of detecting 83.87% of codes that were used to train an LLM. In comparison, the prevalent clone detection tool NiCad is only capable of detecting 47.64%. In addition to its remarkable performance, TraWiC has low resource overhead in contrast to pair-wise clone detection that is conducted during the auditing process of tools like CodeWhisperer reference tracker, across thousands of code snippets.

2024-12-31

ACM Trans. Softw. Eng. Methodol. (published)

Editorial: Special Issue on Software Engineering and AI for Data Quality

Andreas Metzger

Phu Nguyen

Sagar Sen

This editorial summarizes the content of the Special Issue on Software Engineering and AI for Data Quality of the Journal of Data and Inform… (see more)ation Quality (JDIQ).

2024-12-18

Journal of Data and Information Quality (published)

Leveraging Data Characteristics for Bug Localization in Deep Learning Programs

Ruchira Manke

Mohammad Wardat

Hridesh Rajan

Deep Learning (DL) is a class of machine learning algorithms that are used in a wide variety of applications. Like any software system, DL p… (see more)rograms can have bugs. To support bug localization in DL programs, several tools have been proposed in the past. As most of the bugs that occur due to improper model structure known as structural bugs lead to inadequate performance during training, it is challenging for developers to identify the root cause and address these bugs. To support bug detection and localization in DL programs, in this paper, we propose Theia, which detects and localizes structural bugs in DL programs. Unlike the previous works, Theia considers the training dataset characteristics to automatically detect bugs in DL programs developed using two deep learning libraries, Keras and PyTorch . Since training the DL models is a time-consuming process, Theia detects these bugs at the beginning of the training process and alerts the developer with informative messages containing the bug's location and actionable fixes which will help them to improve the structure of the model. We evaluated Theia on a benchmark of 40 real-world buggy DL programs obtained from Stack Overflow . Our results show that Theia successfully localizes 57/75 structural bugs in 40 buggy programs, whereas NeuraLint, a state-of-the-art approach capable of localizing structural bugs before training localizes 17/75 bugs.

2024-12-15

ACM Transactions on Software Engineering and Methodology (published)

Harnessing Pre-trained Generalist Agents for Software Engineering Tasks

Paulina Stevia Nouwou Mindom

Amin Nikanjam

Nowadays, we are witnessing an increasing adoption of Artificial Intelligence (AI) to develop techniques aimed at improving the reliability,… (see more) effectiveness, and overall quality of software systems. Deep reinforcement learning (DRL) has recently been successfully used for automation in complex tasks such as game testing and solving the job-shop scheduling problem. However, these specialized DRL agents, trained from scratch on specific tasks, suffer from a lack of generalizability to other tasks and they need substantial time to be developed and re-trained effectively. Recently, DRL researchers have begun to develop generalist agents, able to learn a policy from various environments and capable of achieving performances similar to or better than specialist agents in new tasks. In the Natural Language Processing or Computer Vision domain, these generalist agents are showing promising adaptation capabilities to never-before-seen tasks after a light fine-tuning phase and achieving high performance. This paper investigates the potential of generalist agents for solving SE tasks. Specifically, we conduct an empirical study aimed at assessing the performance of two generalist agents on two important SE tasks: the detection of bugs in games (for two games) and the minimization of makespan in a scheduling task, to solve the job-shop scheduling problem (for two instances). Our results show that the generalist agents outperform the specialist agents with very little effort for fine-tuning, achieving a 20% reduction of the makespan over specialized agent performance on task-based scheduling. In the context of game testing, some generalist agent configurations detect 85% more bugs than the specialist agents. Building on our analysis, we provide recommendations for researchers and practitioners looking to select generalist agents for SE tasks, to ensure that they perform effectively.

2024-12-10

Empirical Software Engineering (published)