A survey of flaky tests

O Parry, GM Kapfhammer, M Hilton… - ACM Transactions on …, 2021 - dl.acm.org
Tests that fail inconsistently, without changes to the code under test, are described as flaky.
Flaky tests do not give a clear indication of the presence of software bugs and thus limit the …

Exploring the intersection between software maintenance and machine learning—a systematic map** study

OA Bastías, J Díaz, J López Fenner - Applied Sciences, 2023 - mdpi.com
While some areas of software engineering knowledge present great advances with respect
to the automation of processes, tools, and practices, areas such as software maintenance …

[HTML][HTML] Assessing test artifact quality—A tertiary study

HKV Tran, M Unterkalmsteiner, J Börstler… - Information and Software …, 2021 - Elsevier
Context: Modern software development increasingly relies on software testing for an ever
more frequent delivery of high quality software. This puts high demands on the quality of the …

Flakeflagger: Predicting flakiness without rerunning tests

A Alshammari, C Morris, M Hilton… - 2021 IEEE/ACM 43rd …, 2021 - ieeexplore.ieee.org
When developers make changes to their code, they typically run regression tests to detect if
their recent changes (re) introduce any bugs. However, many tests are flaky, and their …

Flakify: A black-box, language model-based predictor for flaky tests

S Fatima, TA Ghaleb, L Briand - IEEE Transactions on Software …, 2022 - ieeexplore.ieee.org
Software testing assures that code changes do not adversely affect existing functionality.
However, a test case can be flaky, ie, passing and failing across executions, even for the …

Understanding reproducibility and characteristics of flaky tests through test reruns in Java projects

W Lam, S Winter, A Astorga, V Stodden… - 2020 IEEE 31st …, 2020 - ieeexplore.ieee.org
Flaky tests are tests that can non-deterministically pass and fail. They pose a major
impediment to regression testing, because they provide an inconclusive assessment on …

Empirically evaluating flaky test detection techniques combining test case rerunning and machine learning models

O Parry, GM Kapfhammer, M Hilton… - Empirical Software …, 2023 - Springer
A flaky test is a test case whose outcome changes without modification to the code of the test
case or the program under test. These tests disrupt continuous integration, cause a loss of …

Shake it! detecting flaky tests caused by concurrency with shaker

D Silva, L Teixeira, M d'Amorim - 2020 IEEE International …, 2020 - ieeexplore.ieee.org
A test is said to be flaky when it non-deterministically passes or fails. Test flakiness
negatively affects the effectiveness of regression testing and, consequently, impacts …

A large-scale longitudinal study of flaky tests

W Lam, S Winter, A Wei, T **e, D Marinov… - Proceedings of the ACM …, 2020 - dl.acm.org
Flaky tests are tests that can non-deterministically pass or fail for the same code version.
These tests undermine regression testing efficiency, because developers cannot easily …

Flakyfix: Using large language models for predicting flaky test fix categories and test code repair

S Fatima, H Hemmati, L Briand - IEEE Transactions on …, 2024 - ieeexplore.ieee.org
Flaky tests are problematic because they non-deterministically pass or fail for the same
software version under test, causing confusion and wasting development effort. While …