Foutse Khomh

Biography

Foutse Khomh is a full professor of software engineering at Polytechnique Montréal, a Canada CIFAR AI Chair – Trustworthy Machine Learning Software Systems, and an FRQ-IVADO Research Chair in Software Quality Assurance for Machine Learning Applications. Khomh completed a PhD in software engineering at Université de Montréal in 2011, for which he received an Award of Excellence. He was also awarded a CS-Can/Info-Can Outstanding Young Computer Science Researcher Prize in 2019.

His research interests include software maintenance and evolution, machine learning systems engineering, cloud engineering, and dependable and trustworthy ML/AI. His work has received four Ten-year Most Influential Paper (MIP) awards, and six Best/Distinguished Paper Awards. He has served on the steering committee of numerous organizations in software engineering, including SANER (chair), MSR, PROMISE, ICPC (chair), and ICSME (vice-chair). He initiated and co-organized Polytechnique Montréal‘s Software Engineering for Machine Learning Applications (SEMLA) symposium and the RELENG (release engineering) workshop series.

Khomh co-founded the NSERC CREATE SE4AI: A Training Program on the Development, Deployment and Servicing of Artificial Intelligence-based Software Systems, and is a principal investigator for the DEpendable Explainable Learning (DEEL) project.

He also co-founded Confiance IA, a Quebec consortium focused on building trustworthy AI, and is on the editorial board of multiple international software engineering journals, including IEEE Software, EMSE and JSEP. He is a senior member of IEEE.

Current Students

Houssem Ben Braiek

Postdoctorate - Polytechnique Montréal

PhD - Polytechnique Montréal

Github

forough majidi

PhD - Polytechnique Montréal

Website

Khouloud Oueslati

Master's Research - Polytechnique Montréal

Arian Qazvini

Master's Research - Polytechnique Montréal

Website

Github

Elnathan Tiokou Tiokou Fangang

Master's Research - Polytechnique Montréal

Ben Braiek Yasmine

Master's Research - Polytechnique Montréal

Publications

Chain of Targeted Verification Questions to Improve the Reliability of Code Generated by LLMs

Sylvain Kouemo Ngassom

Arghavan Moradi Dakhel

Florian Tambon

LLM-based assistants, such as GitHub Copilot and ChatGPT, have the potential to generate code that fulfills a programming task described in … (see more)a natural language description, referred to as a prompt. The widespread accessibility of these assistants enables users with diverse backgrounds to generate code and integrate it into software projects. However, studies show that code generated by LLMs is prone to bugs and may miss various corner cases in task specifications. Presenting such buggy code to users can impact their reliability and trust in LLM-based assistants. Moreover, significant efforts are required by the user to detect and repair any bug present in the code, especially if no test cases are available. In this study, we propose a self-refinement method aimed at improving the reliability of code generated by LLMs by minimizing the number of bugs before execution, without human intervention, and in the absence of test cases. Our approach is based on targeted Verification Questions (VQs) to identify potential bugs within the initial code. These VQs target various nodes within the Abstract Syntax Tree (AST) of the initial code, which have the potential to trigger specific types of bug patterns commonly found in LLM-generated code. Finally, our method attempts to repair these potential bugs by re-prompting the LLM with the targeted VQs and the initial code. Our evaluation, based on programming tasks in the CoderEval dataset, demonstrates that our proposed method outperforms state-of-the-art methods by decreasing the number of targeted errors in the code between 21% to 62% and improving the number of executable code instances to 13%.

2024-07-09

Proceedings of the 1st ACM International Conference on AI-Powered Software (published)

Design smells in multi-language systems and bug-proneness: a survival analysis

Mouna Abidi

Md Saidur Rahman

Moses Openja

2024-07-02

Empirical Software Engineering (published)

A Context-Driven Approach for Co-Auditing Smart Contracts with The Support of GPT-4 code interpreter

Mohamed Salah Bouafif

Chen Zheng

Ilham Qasse

Ed Zulkoski

Mohammad Hamdaqa

The surge in the adoption of smart contracts necessitates rigorous auditing to ensure their security and reliability. Manual auditing, altho… (see more)ugh comprehensive, is time-consuming and heavily reliant on the auditor's expertise. With the rise of Large Language Models (LLMs), there is growing interest in leveraging them to assist auditors in the auditing process (co-auditing). However, the effectiveness of LLMs in smart contract co-auditing is contingent upon the design of the input prompts, especially in terms of context description and code length. This paper introduces a novel context-driven prompting technique for smart contract co-auditing. Our approach employs three techniques for context scoping and augmentation, encompassing code scoping to chunk long code into self-contained code segments based on code inter-dependencies, assessment scoping to enhance context description based on the target assessment goal, thereby limiting the search space, and reporting scoping to force a specific format for the generated response. Through empirical evaluations on publicly available vulnerable contracts, our method demonstrated a detection rate of 96\% for vulnerable functions, outperforming the native prompting approach, which detected only 53\%. To assess the reliability of our prompting approach, manual analysis of the results was conducted by expert auditors from our partner, Quantstamp, a world-leading smart contract auditing company. The experts' analysis indicates that, in unlabeled datasets, our proposed approach enhances the proficiency of the GPT-4 code interpreter in detecting vulnerabilities.

2024-06-25

ArXiv (preprint)

PathOCL: Path-Based Prompt Augmentation for OCL Generation with GPT-4

Seif Abukhalaf

Mohammad Hamdaqa

The rapid progress of AI-powered programming assistants, such as GitHub Copilot, has facilitated the development of software applications. T… (see more)hese assistants rely on large language models (LLMs), which are foundation models (FMs) that support a wide range of tasks related to understanding and generating language. LLMs have demonstrated their ability to express UML model specifications using formal languages like the Object Constraint Language (OCL). However, the context size of the prompt is limited by the number of tokens an LLM can process. This limitation becomes significant as the size of UML class models increases. In this study, we introduce PathOCL, a novel path-based prompt augmentation technique designed to facilitate OCL generation. PathOCL addresses the limitations of LLMs, specifically their token processing limit and the challenges posed by large UML class models. PathOCL is based on the concept of chunking, which selectively augments the prompts with a subset of UML classes relevant to the English specification. Our findings demonstrate that PathOCL, compared to augmenting the complete UML class model (UML-Augmentation), generates a higher number of valid and correct OCL constraints using the GPT-4 model. Moreover, the average prompt size crafted using PathOCL significantly decreases when scaling the size of the UML class models.

2024-06-11

Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering (published)

Data Cleaning and Machine Learning: A Systematic Literature Review

Pierre-Olivier Côté

Amin Nikanjam

Nafisa Ahmed

Dmytro Humeniuk

Context: Machine Learning (ML) is integrated into a growing number of systems for various applications. Because the performance of an ML mod… (see more)el is highly dependent on the quality of the data it has been trained on, there is a growing interest in approaches to detect and repair data errors (i.e., data cleaning). Researchers are also exploring how ML can be used for data cleaning; hence creating a dual relationship between ML and data cleaning. To the best of our knowledge, there is no study that comprehensively reviews this relationship. Objective: This paper's objectives are twofold. First, it aims to summarize the latest approaches for data cleaning for ML and ML for data cleaning. Second, it provides future work recommendations. Method: We conduct a systematic literature review of the papers published between 2016 and 2022 inclusively. We identify different types of data cleaning activities with and for ML: feature cleaning, label cleaning, entity matching, outlier detection, imputation, and holistic data cleaning. Results: We summarize the content of 101 papers covering various data cleaning activities and provide 24 future work recommendations. Our review highlights many promising data cleaning techniques that can be further extended. Conclusion: We believe that our review of the literature will help the community develop better approaches to clean data.

2024-06-10

Automated Software Engineering (published)

Characterizing and Classifying Developer Forum Posts with their Intentions

Xingfang Wu

Eric Laufer

Heng Li

Santhosh Srinivasan

Jayden Luo

With the rapid growth of the developer community, the amount of posts on online technical forums has been growing rapidly, which poses diffi… (see more)culties for users to filter useful posts and find important information. Tags provide a concise feature dimension for users to locate their interested posts and for search engines to index the most relevant posts according to the queries. However, most tags are only focused on the technical perspective (e.g., program language, platform, tool). In most cases, forum posts in online developer communities reveal the author's intentions to solve a problem, ask for advice, share information, etc. The modeling of the intentions of posts can provide an extra dimension to the current tag taxonomy. By referencing previous studies and learning from industrial perspectives, we create a refined taxonomy for the intentions of technical forum posts. Through manual labeling and analysis on a sampled post dataset extracted from online forums, we understand the relevance between the constitution of posts (code, error messages) and their intentions. Furthermore, inspired by our manual study, we design a pre-trained transformer-based model to automatically predict post intentions. The best variant of our intention prediction framework, which achieves a Micro F1-score of 0.589, Top 1-3 accuracy of 62.6% to 87.8%, and an average AUC of 0.787, outperforms the state-of-the-art baseline approach. Our characterization and automated classification of forum posts regarding their intentions may help forum maintainers or third-party tool developers improve the organization and retrieval of posts on technical forums. We have released our annotated dataset and codes in our supplementary material package.

2024-06-04

Empirical Software Engineering (published)

Towards a Reliable French Speech Recognition Tool for an Automated Diagnosis of Learning Disabilities

Jihene Rezgui

Félix Jobin

Younes Kechout

Chritine Turgeon

Dyslexia, characterized by severe challenges in reading and spelling acquisition, presents a substantial barrier to proficient literacy, res… (see more)ulting in significantly reduced reading speed (2 to 3 times slower) and diminished text comprehension. With a prevalence ranging from 5G to 10% in the population, early intervention by speech and language pathologists (SLPs) can mitigate dyslexia's effects, but the diagnosis bottleneck impedes timely support. To address this, we propose leveraging machine learning tools to expedite the diagnosis process, focusing on automating phonetic transcription, a critical step in dyslexia assessment. We investigated the practicality of two model configurations utilizing Google's speech-to-text API with children speech in evaluation scenarios and compared their results against transcriptions crafted by experts. The first configuration focuses on Google API's speech-to-text while the second integrates Phonemizer, a text-to-phonemes tool based on a dictionary. Results analysis indicate that our Google-Phonemizer model yields reading accuracies comparable to those computed from human-made transcriptions, offering promise for clinical application. These findings underscore the potential of AI-driven solutions to enhance dyslexia diagnosis efficiency, paving the way for improved accessibility to vital SLP services.

2024-05-27

2024 International Conference on Smart Applications, Communications and Networking (SmartNets) (published)

Generative AI in Software Engineering Must Be Human-Centered: The Copenhagen Manifesto

Daniel Russo

Sebastian Baltes

Niels van Berkel

Paris Avgeriou

Fabio Calefato

Beatriz Cabrero-Daniel

Gemma Catolino

Jürgen Cito

Neil Ernst

Thomas Fritz

Hideaki Hata

Reid Holmes

Maliheh Izadi

Mikkel Baun Kjærgaard

Grischa Liebel

Alberto Lluch Lafuente

Stefano Lambiase

Walid Maalej

Gail Murphy … (see 15 more)

Nils Brede Moe

Gabrielle O'Brien

Elda Paja

Mauro Pezzè

John Stouby Persson

Rafael Prikladnicki

Paul Ralph

Martin P. Robillard

Thiago Rocha Silva

Klaas-Jan Stol

Margaret-Anne Storey

Viktoria Stray

Paolo Tell

Christoph Treude

Bogdan Vasilescu

2024-04-30

Journal of Systems and Software (published)

Introducing v0.5 of the AI Safety Benchmark from MLCommons

Bertie Vidgen

Adarsh Agrawal

Ahmed M. Ahmed

Victor Akinwande

Namir Al-Nuaimi

Najla Alfaraj

Elie Alhajjar

Lora Aroyo

Trupti Bavalatti

Borhane Blili-Hamelin

K. Bollacker

Rishi Bomassani

Marisa Ferrara Boston

Sim'eon Campos

Kal Chakra

Canyu Chen

Cody Coleman

Zacharie Delpierre Coudert

Leon Strømberg Derczynski

Debojyoti Dutta … (see 77 more)

Ian Eisenberg

James R. Ezick

Heather Frase

Brian Fuller

Ram Gandikota

Agasthya Gangavarapu

Ananya Gangavarapu

James Gealy

Rajat Ghosh

James Goel

Usman Gohar

Sujata Goswami

Scott A. Hale

Wiebke Hutiri

Joseph Marvin Imperial

Surgan Jandial

Nicholas C. Judd

Felix Juefei-Xu

Bhavya Kailkhura

Hannah Rose Kirk

Kevin Klyman

Chris Knotz

Michael Kuchnik

Shachi H. Kumar

Chris Lengerich

Bin Li

Zeyi Liao

Eileen Peters Long

Victor Lu

Yifan Mai

Priyanka Mary Mammen

Kelvin Manyeki

Sean McGregor

Virendra Mehta

Shafee Mohammed

Emanuel Moss

Lama Nachman

Dinesh Jinenhally Naganna

Amin Nikanjam

Besmira Nushi

Luis Oala

Iftach Orr

Alicia Parrish

Çigdem Patlak

William Pietri

Forough Poursabzi-Sangdeh

Eleonora Presani

Fabrizio Puletti

Paul Rottger

Saurav Sahay

Tim Santos

Nino Scherrer

Alice Schoenauer Sebag

Patrick Schramowski

Abolfazl Shahbazi

Vin Sharma

Xudong Shen

Vamsi Sistla

Leonard Tang

Davide Testuggine

Vithursan Thangarasa

Elizabeth A Watkins

Rebecca Weiss

Christoper A. Welty

Tyler Wilbers

Adina Williams

Carole-Jean Wu

Poonam Yadav

Xianjun Yang

Yi Zeng

Wenhui Zhang

Fedor Zhdanov

Jiacheng Zhu

Percy Liang

Peter Mattson

Joaquin Vanschoren

2024-04-17

ArXiv (preprint)

Tackling the XAI Disagreement Problem with Regional Explanations

Gabriel Laberge

Yann Batiste Pequignot

Mario Marchand

2024-04-17

Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (published)

proceedings.mlr.press

Machine Learning Robustness: A Primer

Houssem Ben Braiek

This chapter explores the foundational concept of robustness in Machine Learning (ML) and its integral role in establishing trustworthiness … (see more)in Artificial Intelligence (AI) systems. The discussion begins with a detailed definition of robustness, portraying it as the ability of ML models to maintain stable performance across varied and unexpected environmental conditions. ML robustness is dissected through several lenses: its complementarity with generalizability; its status as a requirement for trustworthy AI; its adversarial vs non-adversarial aspects; its quantitative metrics; and its indicators such as reproducibility and explainability. The chapter delves into the factors that impede robustness, such as data bias, model complexity, and the pitfalls of underspecified ML pipelines. It surveys key techniques for robustness assessment from a broad perspective, including adversarial attacks, encompassing both digital and physical realms. It covers non-adversarial data shifts and nuances of Deep Learning (DL) software testing methodologies. The discussion progresses to explore amelioration strategies for bolstering robustness, starting with data-centric approaches like debiasing and augmentation. Further examination includes a variety of model-centric methods such as transfer learning, adversarial training, and randomized smoothing. Lastly, post-training methods are discussed, including ensemble techniques, pruning, and model repairs, emerging as cost-effective strategies to make models more resilient against the unpredictable. This chapter underscores the ongoing challenges and limitations in estimating and achieving ML robustness by existing approaches. It offers insights and directions for future research on this crucial concept, as a prerequisite for trustworthy AI systems.

2024-03-31

ArXiv (preprint)

Assessing the Security of GitHub Copilot Generated Code - A Targeted Replication Study

Vahid Majdinasab

Michael Joshua Bishop

Shawn Rasheed

Arghavan Moradi Dakhel

Amjed Tahir

2024-03-11

2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) (published)