Design Article
Tell us What You Think
We want to know what you thought about this Design. Let us know by adding a comment.
Regression testing with random tests cannot identify regressions
Daniel Hansson, CEO Verifyter
10/15/2012 10:47 AM EDT
Backtracking is the way forward
In order to draw conclusions why a random test failed we must retest the very same test on older revisions. This means rerunning the failing test, using the same seed, on older revisions, in order to identify when the problems started to arise. This is the only way to be able to compare the test results on older revisions with the test results on the latest revision. Once you have rerun the same test on an old revision then you will be able to do the same comparison as you would with a directed test. If the same test with the same seed passes in an older revision then you are able to identify that a regression has occurred. If the same test and seed has always failed then you know this is a new test. This new test may in turn either be catching a new corner case, or alternatively it may be an illegal test. Either way you are now able to distinguish between new tests and regression in quality.
Figure 2 shows how PinDown operates on the flow of random test failures. The stream of random failures are split into regressions and new tests, where the regressions are diagnosed down to the exact revision that caused the problem and a bug report is sent to the person who committed each the error. This allows regression errors to be fixed fast and thus allows the device and testbench to maintain high quality.
The other category is new tests, i.e. tests that have always failed and are consequently covering a new test scenario. These are not failing due to a sudden regression in quality, which may lead to panic and holding the release, but is new test coverage which is overall positive news.
This setup solves the problem with using random tests in regression testing. It allows you to keep running random testing with the upside of getting good coverage without the downside of not being able to identify regressions.
|
|
|
|
|
