Foutse Khomh

houssem.ben-braiek@polymtl.ca

Biography

Foutse Khomh is a full professor of software engineering at Polytechnique Montréal, a Canada CIFAR AI Chair – Trustworthy Machine Learning Software Systems, and an FRQ-IVADO Research Chair in Software Quality Assurance for Machine Learning Applications. Khomh completed a PhD in software engineering at Université de Montréal in 2011, for which he received an Award of Excellence. He was also awarded a CS-Can/Info-Can Outstanding Young Computer Science Researcher Prize in 2019.

His research interests include software maintenance and evolution, machine learning systems engineering, cloud engineering, and dependable and trustworthy ML/AI. His work has received four Ten-year Most Influential Paper (MIP) awards, and six Best/Distinguished Paper Awards. He has served on the steering committee of numerous organizations in software engineering, including SANER (chair), MSR, PROMISE, ICPC (chair), and ICSME (vice-chair). He initiated and co-organized Polytechnique Montréal‘s Software Engineering for Machine Learning Applications (SEMLA) symposium and the RELENG (release engineering) workshop series.

Khomh co-founded the NSERC CREATE SE4AI: A Training Program on the Development, Deployment and Servicing of Artificial Intelligence-based Software Systems, and is a principal investigator for the DEpendable Explainable Learning (DEEL) project.

He also co-founded Confiance IA, a Quebec consortium focused on building trustworthy AI, and is on the editorial board of multiple international software engineering journals, including IEEE Software, EMSE and JSEP. He is a senior member of IEEE.

Current Students

Houssem Ben Braiek

Postdoctorate - Polytechnique Montréal

PhD - Polytechnique Montréal

forough.majidi@mila.quebec

Website

anthonia-oluchukwu.njoku@mila.quebec

Anthonia Oluchukwu Njoku

PhD - Polytechnique Montréal

khouloud.oueslati@mila.quebec

Khouloud Oueslati

Master's Research - Polytechnique Montréal

Arian Qazvini

Master's Research - Polytechnique Montréal

arian.qazvini@mila.quebec

Website

Github

Elnathan Tiokou Tiokou Fangang

Master's Research - Polytechnique Montréal

elnathan.tiokou@mila.quebec

Ben Braiek Yasmine

Master's Research - Polytechnique Montréal

yasmine.ben-braiek@mila.quebec

Publications

Anti-patterns and Code Smells for Multi-language Systems

Mouna Abidi

Manel Grichi

Yann‐Gaël Guéhéneuc

2025-03-20

Lecture Notes in Computer Science (published)

Understanding the impact of IoT security patterns on CPU usage and energy consumption: a dynamic approach for selecting patterns with deep reinforcement learning

Saeid Jamshidi

Amin Nikanjam

Kawser Wazed Nafi

2025-03-09

International Journal of Information Security (published)

Self-adaptive cyber defense for sustainable IoT: A DRL-based IDS optimizing security and energy efficiency

Saeid Jamshidi

Ashkan Amirnia

Amin Nikanjam

Kawser Wazed Nafi

Samira Keivanpour

2025-02-28

Journal of Network and Computer Applications (published)

Assessing the adoption of security policies by developers in terraform across different cloud providers

Alexandre Verdet

Mohammad Hamdaqa

Leuson Da Silva

Cloud computing has become popular thanks to the widespread use of Infrastructure as Code (IaC) tools, allowing the community to manage and … (see more)configure cloud infrastructure using scripts. However, the scripting process does not automatically prevent practitioners from introducing misconfigurations, vulnerabilities, or privacy risks. As a result, ensuring security relies on practitioners’ understanding and the adoption of explicit policies. To understand how practitioners deal with this problem, we perform an empirical study analyzing the adoption of scripted security best practices present in Terraform files, applied on AWS, Azure, and Google Cloud. We assess the adoption of these practices by analyzing a sample of 812 open-source GitHub projects. We scan each project’s configuration files, looking for policy implementation through static analysis (Checkov and Tfsec). The category Access policy emerges as the most widely adopted in all providers, while Encryption at rest presents the most neglected policies. Regarding the cloud providers, we observe that AWS and Azure present similar behavior regarding attended and neglected policies. Finally, we provide guidelines for cloud practitioners to limit infrastructure vulnerability and discuss further aspects associated with policies that have yet to be extensively embraced within the industry.

2025-02-26

Empirical Software Engineering (published)

AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons

Shaona Ghosh

Heather Frase

Adina Williams

Sarah Luger

Paul Rottger

Fazl Barez

Sean McGregor

Kenneth Fricklas

Mala Kumar

Quentin Feuillade--Montixi

Kurt Bollacker

Felix Friedrich

Ryan Tsang

Bertie Vidgen

Alicia Parrish

Chris Knotz

Eleonora Presani

Jonathan Bennion

Marisa Ferrara Boston

Mike Kuniavsky … (see 81 more)

Wiebke Hutiri

James Ezick

Malek Ben Salem

Rajat Sahay

Sujata Goswami

Usman Gohar

Ben Huang

Supheakmungkol Sarin

Elie Alhajjar

Canyu Chen

Roman Eng

K. Manjusha

Virendra Mehta

Eileen Peters Long

Murali Krishna Emani

Natan Vidra

Benjamin Rukundo

Abolfazl Shahbazi

Kongtao Chen

Rajat Ghosh

Vithursan Thangarasa

Pierre Peign'e

Abhinav Singh

Max Bartolo

Satyapriya Krishna

Mubashara Akhtar

Rafael Gold

Cody Coleman

Luis Oala

Vassil Tashev

Joseph Marvin Imperial

Amy Russ

Sasidhar Kunapuli

Nicolas Miailhe

Julien Delaunay

Bhaktipriya Radharapu

Rajat Shinde

Tuesday

Debojyoti Dutta

Declan Grabb

Ananya Gangavarapu

Saurav Sahay

Agasthya Gangavarapu

Patrick Schramowski

Stephen Singam

Tom David

Xudong Han

Priyanka Mary Mammen

Tarunima Prabhakar

Venelin Kovatchev

Ahmed M. Ahmed

Kelvin Manyeki

Sandeep Madireddy

Fedor Zhdanov

Joachim Baumann

N. Vasan

Xianjun Yang

Carlos Mougn

Jibin Rajan Varghese

Hussain Chinoy

Seshakrishna Jitendar

Manil Maskey

Claire V. Hardgrove

Tianhao Li

Aakash Gupta

Emil Joswin

Yifan Mai

Shachi H. Kumar

Çigdem Patlak

Kevin Lu

Vincent Alessi

Sree Bhargavi Balija

Chenhe Gu

Robert Sullivan

James Gealy

Matt Lavrisa

James Goel

Peter Mattson

Percy Liang

Joaquin Vanschoren

2025-02-18

ArXiv (preprint)

Bugs in Large Language Models Generated Code: An Empirical Study

Florian Tambon

Arghavan Moradi Dakhel

Amin Nikanjam

Michel C. Desmarais

Giuliano Antoniol

Large Language Models (LLMs) for code have gained significant attention recently. They can generate code in different programming languages … (see more)based on provided prompts, fulfilling a long-lasting dream in Software Engineering (SE), i.e., automatic code generation. Similar to human-written code, LLM-generated code is prone to bugs, and these bugs have not yet been thoroughly examined by the community. Given the increasing adoption of LLM-based code generation tools (e.g., GitHub Copilot) in SE activities, it is critical to understand the characteristics of bugs contained in code generated by LLMs. This paper examines a sample of 333 bugs collected from code generated using three leading LLMs (i.e., CodeGen, PanGu-Coder, and Codex) and identifies the following 10 distinctive bug patterns: Misinterpretations, Syntax Error, Silly Mistake, Prompt-biased code, Missing Corner Case, Wrong Input Type, Hallucinated Object, Wrong Attribute, Incomplete Generation, and Non-Prompted Consideration. The bug patterns are presented in the form of a taxonomy. The identified bug patterns are validated using an online survey with 34 LLM practitioners and researchers. The surveyed participants generally asserted the significance and prevalence of the bug patterns. Researchers and practitioners can leverage these findings to develop effective quality assurance techniques for LLM-generated code. This study sheds light on the distinctive characteristics of LLM-generated code.

2025-02-12

Empirical Software Engineering (published)

Application of deep reinforcement learning for intrusion detection in Internet of Things: A systematic review

Saeid Jamshidi

Amin Nikanjam

Kawser Wazed Nafi

Rasoul Rasta

2025-01-31

Internet of Things (published)

Automated UML Visualization of Software Ecosystems: Tracking Versions, Dependencies, and Security Updates

Vanessa Kan

M. P. Lnu

Solomon Berhe

C. El Kari

Marc Maynard

2024-12-31

ANT/EDI40 (published)

An empirical study of testing machine learning in the wild

Moses Openja

Armstrong Foundjem

Zhen Ming

Mouna Abidi

Ahmed E. Hassan

Recently, machine and deep learning (ML/DL) algorithms have been increasingly adopted in many software systems. Due to their inductive natur… (see more)e, ensuring the quality of these systems remains a significant challenge for the research community. Unlike traditional software built deductively by writing explicit rules, ML/DL systems infer rules from training data. Recent research in ML/DL quality assurance has adapted concepts from traditional software testing, such as mutation testing, to improve reliability. However, it is unclear if these proposed testing techniques are adopted in practice, or if new testing strategies have emerged from real-world ML deployments. There is little empirical evidence about the testing strategies. To fill this gap, we perform the first fine-grained empirical study on ML testing in the wild to identify the ML properties being tested, the testing strategies, and their implementation throughout the ML workflow. We conducted a mixed-methods study to understand ML software testing practices. We analyzed test files and cases from 11 open-source ML/DL projects on GitHub. Using open coding, we manually examined the testing strategies, tested ML properties, and implemented testing methods to understand their practical application in building and releasing ML/DL software systems. Our findings reveal several key insights: 1.) The most common testing strategies, accounting for less than 40%, are Grey-box and White-box methods, such as Negative Testing, Oracle Approximation and Statistical Testing. 2.) A wide range of 17 ML properties are tested, out of which only 20% to 30% are frequently tested, including Consistency, Correctness}, and Efficiency. 3.) Bias and Fairness is more tested in Recommendation, while Security & Privacy is tested in Computer Vision (CV) systems, Application Platforms, and Natural Language Processing (NLP) systems.

2024-12-31

ACM Trans. Softw. Eng. Methodol. (published)

Harnessing Pre-trained Generalist Agents for Software Engineering Tasks

Paulina Stevia Nouwou Mindom

Amin Nikanjam

Nowadays, we are witnessing an increasing adoption of Artificial Intelligence (AI) to develop techniques aimed at improving the reliability,… (see more) effectiveness, and overall quality of software systems. Deep reinforcement learning (DRL) has recently been successfully used for automation in complex tasks such as game testing and solving the job-shop scheduling problem. However, these specialized DRL agents, trained from scratch on specific tasks, suffer from a lack of generalizability to other tasks and they need substantial time to be developed and re-trained effectively. Recently, DRL researchers have begun to develop generalist agents, able to learn a policy from various environments and capable of achieving performances similar to or better than specialist agents in new tasks. In the Natural Language Processing or Computer Vision domain, these generalist agents are showing promising adaptation capabilities to never-before-seen tasks after a light fine-tuning phase and achieving high performance. This paper investigates the potential of generalist agents for solving SE tasks. Specifically, we conduct an empirical study aimed at assessing the performance of two generalist agents on two important SE tasks: the detection of bugs in games (for two games) and the minimization of makespan in a scheduling task, to solve the job-shop scheduling problem (for two instances). Our results show that the generalist agents outperform the specialist agents with very little effort for fine-tuning, achieving a 20% reduction of the makespan over specialized agent performance on task-based scheduling. In the context of game testing, some generalist agent configurations detect 85% more bugs than the specialist agents. Building on our analysis, we provide recommendations for researchers and practitioners looking to select generalist agents for SE tasks, to ensure that they perform effectively.

2024-12-31

Empir. Softw. Eng. (published)

MLOps, LLMOps, FMOps, and Beyond

Chakkrit Kla Tantithamthavorn

Fabio Palomba

Joselito Joey Chua

2024-12-31

IEEE Software (published)

Reputation Gaming in Crowd Technical Knowledge Sharing

Iren Mazloomzadeh

Gias Uddin

Ashkan Sami

Stack Overflow incentive system awards users with reputation scores to ensure quality. The decentralized nature of the forum may make the in… (see more)centive system prone to manipulation. This paper offers, for the first time, a comprehensive study of the reported types of reputation manipulation scenarios that might be exercised in Stack Overflow and the prevalence of such reputation gamers by a qualitative study of 1,697 posts from meta Stack Exchange sites. We found four different types of reputation fraud scenarios, such as voting rings where communities form to upvote each other repeatedly on similar posts. We developed algorithms that enable platform managers to automatically identify these suspicious reputation gaming scenarios for review. The first algorithm identifies isolated/semi-isolated communities where probable reputation frauds may occur mostly by collaborating with each other. The second algorithm looks for sudden unusual big jumps in the reputation scores of users. We evaluated the performance of our algorithms by examining the reputation history dashboard of Stack Overflow users from the Stack Overflow website. We observed that around 60-80% of users flagged as suspicious by our algorithms experienced reductions in their reputation scores by Stack Overflow.

2024-12-31

ACM Trans. Softw. Eng. Methodol. (published)