Foutse Khomh

2023-10-09

Empirical Software Engineering (published)

Data Cleaning and Machine Learning: A Systematic Literature Review

Pierre-Olivier Cot'e

Amin Nikanjam

Nafisa Ahmed

Dmytro Humeniuk

Context: Machine Learning (ML) is integrated into a growing number of systems for various applications. Because the performance of an ML mod… (see more)el is highly dependent on the quality of the data it has been trained on, there is a growing interest in approaches to detect and repair data errors (i.e., data cleaning). Researchers are also exploring how ML can be used for data cleaning; hence creating a dual relationship between ML and data cleaning. To the best of our knowledge, there is no study that comprehensively reviews this relationship. Objective: This paper's objectives are twofold. First, it aims to summarize the latest approaches for data cleaning for ML and ML for data cleaning. Second, it provides future work recommendations. Method: We conduct a systematic literature review of the papers published between 2016 and 2022 inclusively. We identify different types of data cleaning activities with and for ML: feature cleaning, label cleaning, entity matching, outlier detection, imputation, and holistic data cleaning. Results: We summarize the content of 101 papers covering various data cleaning activities and provide 24 future work recommendations. Our review highlights many promising data cleaning techniques that can be further extended. Conclusion: We believe that our review of the literature will help the community develop better approaches to clean data.

2023-10-03

ArXiv (preprint)

Deploying Deep Reinforcement Learning Systems: A Taxonomy of Challenges

Ahmed Haj Yahmed

Altaf Allah Abbassi

Amin Nikanjam

Heng Li

Deep reinforcement learning (DRL), leveraging Deep Learning (DL) in reinforcement learning, has shown significant potential in achieving hum… (see more)an-level autonomy in a wide range of domains, including robotics, computer vision, and computer games. This potential justifies the enthusiasm and growing interest in DRL in both academia and industry. However, the community currently focuses mostly on the development phase of DRL systems, with little attention devoted to DRL deployment. In this paper, we propose an empirical study on Stack Overflow (SO), the most popular Q&A forum for developers, to uncover and understand the challenges practitioners faced when deploying DRL systems. Specifically, we categorized relevant SO posts by deployment platforms: server/cloud, mobile/embedded system, browser, and game engine. After filtering and manual analysis, we examined 357 SO posts about DRL deployment, investigated the current state, and identified the challenges related to deploying DRL systems. Then, we investigate the prevalence and difficulty of these challenges. Results show that the general interest in DRL deployment is growing, confirming the study’s relevance and importance. Results also show that DRL deployment is more difficult than other DRL issues. Additionally, we built a taxonomy of 31 unique challenges in deploying DRL to different platforms. On all platforms, RL environment-related challenges are the most popular, and communication-related challenges are the most difficult among practitioners. We hope our study inspires future research and helps the community overcome the most common and difficult challenges practitioners face when deploying DRL systems.

2023-10-01

2023 IEEE International Conference on Software Maintenance and Evolution (ICSME) (published)

An Empirical Study on Bugs Inside PyTorch: A Replication Study

Vahid Majdinasab

Sharon Chee Yin Ho

Mohayeminul Islam

Diego Elias Costa

Emad Shihab

Sarah Nadi

Muhammad Raza

Software systems are increasingly relying on deep learning components, due to their remarkable capability of identifying complex data patter… (see more)ns and powering intelligent behaviour. A core enabler of this change in software development is the availability of easy-to-use deep learning libraries. Libraries like PyTorch and TensorFlow empower a large variety of intelligent systems, offering a multitude of algorithms and configuration options, applicable to numerous domains of systems. However, bugs in those popular deep learning libraries also may have dire consequences for the quality of systems they enable; thus, it is important to understand how bugs are identified and fixed in those libraries.Inspired by a study of Jia et al., which investigates the bug identification and fixing process at TensorFlow, we characterize bugs in the PyTorch library, a very popular deep learning framework. We investigate the causes and symptoms of bugs identified during PyTorch’s development, and assess their locality within the project, and extract patterns of bug fixes. Our results highlight that PyTorch bugs are more like traditional software projects bugs, than related to deep learning characteristics. Finally, we also compare our results with the study on TensorFlow, highlighting similarities and differences across the bug identification and fixing process.

2023-10-01

2023 IEEE International Conference on Software Maintenance and Evolution (ICSME) (published)

Robustness assessment of hyperspectral image CNNs using metamorphic testing

Rached Bouchoucha

Houssem Ben Braiek

Sonia Bouzidi

Rania Zaatour

2023-10-01

Information and Software Technology (published)

Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing

Arghavan Moradi Dakhel

Amin Nikanjam

Vahid Majdinasab

Michel C. Desmarais

One of the critical phases in software development is software testing. Testing helps with identifying potential bugs and reducing maintenan… (see more)ce costs. The goal of automated test generation tools is to ease the development of tests by suggesting efficient bug-revealing tests. Recently, researchers have leveraged Large Language Models (LLMs) of code to generate unit tests. While the code coverage of generated tests was usually assessed, the literature has acknowledged that the coverage is weakly correlated with the efficiency of tests in bug detection. To improve over this limitation, in this paper, we introduce MuTAP for improving the effectiveness of test cases generated by LLMs in terms of revealing bugs by leveraging mutation testing. Our goal is achieved by augmenting prompts with surviving mutants, as those mutants highlight the limitations of test cases in detecting bugs. MuTAP is capable of generating effective test cases in the absence of natural language descriptions of the Program Under Test (PUTs). We employ different LLMs within MuTAP and evaluate their performance on different benchmarks. Our results show that our proposed method is able to detect up to 28% more faulty human-written code snippets. Among these, 17% remained undetected by both the current state-of-the-art fully automated test generation tool (i.e., Pynguin) and zero-shot/few-shot learning approaches on LLMs. Furthermore, MuTAP achieves a Mutation Score (MS) of 93.57% on synthetic buggy code, outperforming all other approaches in our evaluation. Our findings suggest that although LLMs can serve as a useful tool to generate test cases, they require specific post-processing steps to enhance the effectiveness of the generated test cases which may suffer from syntactic or functional errors and may be ineffective in detecting certain types of bugs and testing corner cases PUTs.

2023-08-31

ArXiv (preprint)

A comparison of reinforcement learning frameworks for software testing tasks

Paulina Stevia Nouwou Mindom

Amin Nikanjam

2023-08-24

Empirical Software Engineering (published)

Reinforcement Learning Informed Evolutionary Search for Autonomous Systems Testing

Dmytro Humeniuk

Giuliano Antoniol

Evolutionary search-based techniques are commonly used for testing autonomous robotic systems. However, these approaches often rely on compu… (see more)tationally expensive simulator-based models for test scenario evaluation. To improve the computational efficiency of the search-based testing, we propose augmenting the evolutionary search (ES) with a reinforcement learning (RL) agent trained using surrogate rewards derived from domain knowledge. In our approach, known as RIGAA (Reinforcement learning Informed Genetic Algorithm for Autonomous systems testing), we first train an RL agent to learn useful constraints of the problem and then use it to produce a certain part of the initial population of the search algorithm. By incorporating an RL agent into the search process, we aim to guide the algorithm towards promising regions of the search space from the start, enabling more efficient exploration of the solution space. We evaluate RIGAA on two case studies: maze generation for an autonomous ant robot and road topology generation for an autonomous vehicle lane keeping assist system. In both case studies, RIGAA converges faster to fitter solutions and produces a better test suite (in terms of average test scenario fitness and diversity). RIGAA also outperforms the state-of-the-art tools for vehicle lane keeping assist system testing, such as AmbieGen and Frenetic.

2023-08-24

ArXiv (preprint)

Exploring Security Practices in Infrastructure as Code: An Empirical Study

Alexandre Verdet

Mohammad Hamdaqa

Léuson M. P. Da Silva

Cloud computing has become popular thanks to the widespread use of Infrastructure as Code (IaC) tools, allowing the community to convenientl… (see more)y manage and configure cloud infrastructure using scripts. However, the scripting process itself does not automatically prevent practitioners from introducing misconfigurations, vulnerabilities, or privacy risks. As a result, ensuring security relies on practitioners understanding and the adoption of explicit policies, guidelines, or best practices. In order to understand how practitioners deal with this problem, in this work, we perform an empirical study analyzing the adoption of IaC scripted security best practices. First, we select and categorize widely recognized Terraform security practices promulgated in the industry for popular cloud providers such as AWS, Azure, and Google Cloud. Next, we assess the adoption of these practices by each cloud provider, analyzing a sample of 812 open-source projects hosted on GitHub. For that, we scan each project configuration files, looking for policy implementation through static analysis (checkov). Additionally, we investigate GitHub measures that might be correlated with adopting these best practices. The category Access policy emerges as the most widely adopted in all providers, while Encryption in rest are the most neglected policies. Regarding GitHub measures correlated with best practice adoption, we observe a positive, strong correlation between a repository number of stars and adopting practices in its cloud infrastructure. Based on our findings, we provide guidelines for cloud practitioners to limit infrastructure vulnerability and discuss further aspects associated with policies that have yet to be extensively embraced within the industry.

2023-08-07

ArXiv (preprint)

The Different Faces of AI Ethics Across the World: A Principle-to-Practice Gap Analysis

Lionel Nganyewou Tidjon

Artificial Intelligence (AI) is transforming our daily life with many applications in healthcare, space exploration, banking, and finance. T… (see more)his rapid progress in AI has brought increasing attention to the potential impacts of AI technologies on society, with ethically questionable consequences. In recent years, several ethical principles have been released by governments, national organizations, and international organizations. These principles outline high-level precepts to guide the ethical development, deployment, and governance of AI. However, the abstract nature, diversity, and context-dependence of these principles make them difficult to implement and operationalize, resulting in gaps between principles and their execution. Most recent work analyzed and summarized existing AI principles and guidelines but did not provide findings on principle-to-practice gaps nor how to mitigate them. These findings are particularly important to ensure that AI practical guidances are aligned with ethical principles and values. In this article, we provide a contextual and global evaluation of current ethical AI principles for all continents, with the aim to identify potential principle characteristics tailored to specific countries or applicable across countries. Next, we analyze the current level of AI readiness and current practical guidances of ethical AI principles in different countries, to identify gaps in the practical guidance of AI principles and their causes. Finally, we propose recommendations to mitigate the principle-to-practice gaps.

2023-08-01

IEEE Transactions on Artificial Intelligence (published)

Intelligent Software Maintenance

Mohammad Masudur Rahman

Antoine Barbez

2023-07-20

Optimising the Software Development Process with Artificial Intelligence (published)

Chat2Code: A Chatbot for Model Specification and Code Generation, The Case of Smart Contracts

Ilham Qasse

Shailesh Mishra

Björn þór Jónsson

Mohammad Hamdaqa

The potential of automatic code generation through Model-Driven Engineering (MDE) frameworks has yet to be realized. Beyond their ability to… (see more) help software professionals write more accurate, reusable code, MDE frameworks could make programming accessible for a new class of domain experts. However, domain experts have been slow to embrace these tools, as they still need to learn how to specify their applications' requirements using the concrete syntax (i.e., textual or graphical) of the new and unified domain-specific language. Conversational interfaces (chatbots) could smooth the learning process and offer a more interactive way for domain experts to specify their application requirements and generate the desired code. If integrated with MDE frameworks, chatbots may offer domain experts with richer domain vocabulary without sacrificing the power of agnosticism that unified modelling frameworks provide. In this paper, we discuss the challenges of integrating chatbots within MDE frameworks and then examine a specific application: the auto-generation of smart contract code based on conversational syntax. We demonstrate how this can be done and evaluate our approach by conducting a user experience survey to assess the usability and functionality of the chatbot framework. The paper concludes by drawing attention to the potential benefits of leveraging Language Models (LLMs) in this context.

2023-07-01

2023 IEEE International Conference on Software Services Engineering (SSE) (published)