Flaky tests

A “flaky” test is one that exhibits intermittent or sporadic failure, that seems to have non-deterministic behaviour. Sometimes it passes, sometimes it fails, and it’s not clear why. This page discusses pytest features that can help and other general strategies for identifying, fixing or mitigating them.

Why flaky tests are a problem

Flaky tests are particularly troublesome when a continuous integration (CI) server is being used, so that all tests must pass before a new code change can be merged. If the test result is not a reliable signal – that a test failure means the code change broke the test – developers can become mistrustful of the test results, which can lead to overlooking genuine failures. It is also a source of wasted time as developers must re-run test suites and investigate spurious failures.

Potential root causes

System state

Broadly speaking, a flaky test indicates that the test relies on some system state that is not being appropriately controlled - the test environment is not sufficiently isolated. Higher level tests are more likely to be flaky as they rely on more state.

Flaky tests sometimes appear when a test suite is run in parallel (such as use of pytest-xdist). This can indicate a test is reliant on test ordering.

  • Perhaps a different test is failing to clean up after itself and leaving behind data which causes the flaky test to fail.

  • The flaky test is reliant on data from a previous test that doesn’t clean up after itself, and in parallel runs that previous test is not always present

  • Tests that modify global state typically cannot be run in parallel.

Overly strict assertion

Overly strict assertions can cause problems with floating point comparison as well as timing issues. pytest.approx() is useful here.

Thread safety

pytest is single-threaded, executing its tests always in the same thread, sequentially, never spawning any threads itself.

Even in case of plugins which run tests in parallel, for example pytest-xdist, usually work by spawning multiple processes and running tests in batches, without using multiple threads.

It is of course possible (and common) for tests and fixtures to spawn threads themselves as part of their testing workflow (for example, a fixture that starts a server thread in the background, or a test which executes production code that spawns threads), but some care must be taken:

  • Make sure to eventually wait on any spawned threads – for example at the end of a test, or during the teardown of a fixture.

  • Avoid using primitives provided by pytest (pytest.warns(), pytest.raises(), etc) from multiple threads, as they are not thread-safe.

If your test suite uses threads and you are seeing flaky test results, do not discount the possibility that the test is implicitly using global state in pytest itself.

Other general strategies

Split up test suites

It can be common to split a single test suite into two, such as unit vs integration, and only use the unit test suite as a CI gate. This also helps keep build times manageable as high level tests tend to be slower. However, it means it does become possible for code that breaks the build to be merged, so extra vigilance is needed for monitoring the integration test results.

Video/screenshot on failure

For UI tests these are important for understanding what the state of the UI was when the test failed. pytest-splinter can be used with plugins like pytest-bdd and can save a screenshot on test failure, which can help to isolate the cause.

Delete or rewrite the test

If the functionality is covered by other tests, perhaps the test can be removed. If not, perhaps it can be rewritten at a lower level which will remove the flakiness or make its source more apparent.

Quarantine

Mark Lapierre discusses the Pros and Cons of Quarantined Tests in a post from 2018.

CI tools that rerun on failure

Azure Pipelines (the Azure cloud CI/CD tool, formerly Visual Studio Team Services or VSTS) has a feature to identify flaky tests and rerun failed tests.

Research

This is a limited list, please submit an issue or pull request to expand it!

  • Gao, Zebao, Yalan Liang, Myra B. Cohen, Atif M. Memon, and Zhen Wang. “Making system user interactive tests repeatable: When and what should we control?.” In Software Engineering (ICSE), 2015 IEEE/ACM 37th IEEE International Conference on, vol. 1, pp. 55-65. IEEE, 2015. PDF

  • Palomba, Fabio, and Andy Zaidman. “Does refactoring of test smells induce fixing flaky tests?.” In Software Maintenance and Evolution (ICSME), 2017 IEEE International Conference on, pp. 1-12. IEEE, 2017. PDF in Google Drive

  • Bell, Jonathan, Owolabi Legunsen, Michael Hilton, Lamyaa Eloussi, Tifany Yung, and Darko Marinov. “DeFlaker: Automatically detecting flaky tests.” In Proceedings of the 2018 International Conference on Software Engineering. 2018. PDF

  • Dutta, Saikat and Shi, August and Choudhary, Rutvik and Zhang, Zhekun and Jain, Aryaman and Misailovic, Sasa. “Detecting flaky tests in probabilistic and machine learning applications.” In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), pp. 211-224. ACM, 2020. PDF

  • Habchi, Sarra and Haben, Guillaume and Sohn, Jeongju and Franci, Adriano and Papadakis, Mike and Cordy, Maxime and Le Traon, Yves. “What Made This Test Flake? Pinpointing Classes Responsible for Test Flakiness.” In Proceedings of the 38th IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, 2022. PDF

  • Lamprou, Sokrates. “Non-deterministic tests and where to find them: Empirically investigating the relationship between flaky tests and test smells by examining test order dependency.” Bachelor thesis, Department of Computer and Information Science, Linköping University, 2022. LIU-IDA/LITH-EX-G–19/056–SE. PDF

  • Leinen, Fabian and Elsner, Daniel and Pretschner, Alexander and Stahlbauer, Andreas and Sailer, Michael and Jürgens, Elmar. “Cost of Flaky Tests in Continuous Integration: An Industrial Case Study.” Technical University of Munich and CQSE GmbH, Munich, Germany, 2023. PDF

  • Parry, Owain and Kapfhammer, Gregory M. and Hilton, Michael and McMinn, Phil. “A Survey of Flaky Tests.” In Transactions on Software Engineering and Methodology (TOSEM). vol. 31, issue 1. ACM, 2021. PDF

  • Parry, Owain and Kapfhammer, Gregory M. and Hilton, Michael and McMinn, Phil. “Evaluating Features for Machine Learning Detection of Order- and Non-Order-Dependent Flaky Tests.” In 2022 IEEE Conference on Software Testing, Verification and Validation (ICST). 2022. PDF

  • Parry, Owain and Kapfhammer, Gregory M. and Hilton, Michael and McMinn, Phil. “What do developer-repaired Flaky tests tell us about the effectiveness of automated Flaky test detection?” In Proceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test (AST). 2022. PDF

  • Parry, Owain and Kapfhammer, Gregory M. and Hilton, Michael and McMinn, Phil. “Surveying the developer experience of flaky tests.” In Proceedings of the 44th International Conference on Software Engineering (ICSE). 2022. PDF

  • Parry, Owain and Kapfhammer, Gregory M. and Hilton, Michael and McMinn, Phil. “Empirically evaluating flaky test detection techniques combining test case rerunning and machine learning models.” In Empirical Software Engineering. vol. 28, issue 3. Springer, 2023. PDF

  • Gruber, Martin and Roslan, Muhammad Firhard and Parry, Owain and Scharnb"{o}ck, Fabian and McMinn, Phil and Fraser, Gordon. “Do Automatic Test Generation Tools Generate Flaky Tests?” In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICSE). 2024. PDF

  • Parry, Owain and Kapfhammer, Gregory and Hilton, Michael and McMinn, Phil. “Systemic Flakiness: An Empirical Analysis of Co-Occurring Flaky Test Failures.” In Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering (EASE). 2025. PDF

  • Parry, Owain and Kapfhammer, Gregory M. and Hilton, Michael and McMinn, Phil. “Test flimsiness: Characterizing flakiness induced by mutation to the code under test.” In Proceedings of 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE). 2026. PDF

Resources