Flaky and Negative Tests
Explore the roles of negative and flaky tests in machine learning projects. Learn to design effective negative tests that catch errors early and understand how to avoid flaky tests that cause inconsistent results, enhancing your ML system's reliability and robustness.
We'll cover the following...
Negative tests
Negative tests, also known as exception tests, are essential to any machine learning project. They ensure the code can handle incorrect input or unexpected scenarios and raises an error appropriately. This is important because it helps to identify and prevent potential issues that could lead to incorrect or unexpected results.
When positive tests check that everything is good (optimistic scenario), negative tests check various (mostly graceful) potential failures. The more experienced the engineer is, the more negative tests they usually build.
“Developer tests tend to be ‘clean tests.’ Developers tend to test whether the code works (clean tests), rather than all the ways the code breaks (dirty tests). Immature testing organizations have about five clean tests for every dirty test. Mature testing organizations tend to have five dirty tests for every clean test. This ratio is not reversed ...