Design Article
Tell us What You Think
We want to know what you thought about this Design. Let us know by adding a comment.
Driving Blindfolded – Solving The Triage Challenge
Daniel Hansson - Verifyter
9/10/2012 10:25 AM EDT
Result noise
A common problem when running large regression test suites is the intermittent instability of the computers and networks used for testing. This may lead to builds or tests failing, or never completing. These results, called result noise, do not reflect the state of the device under test and should be filtered out.
It is important to be able to filter out result noise from real design results in order to reach the correct conclusions. This is done by retesting results which look suspicious or which do not behave as suspected. Another mechanism is to verify that the test is up and running by checking the results of tests that verify the flow and not the device under test.
Reporting
When it has been concluded in which revision a bug was introduced it is possible to provide information about the bug from three different sources:
Next: Continuous integration
A common problem when running large regression test suites is the intermittent instability of the computers and networks used for testing. This may lead to builds or tests failing, or never completing. These results, called result noise, do not reflect the state of the device under test and should be filtered out.
It is important to be able to filter out result noise from real design results in order to reach the correct conclusions. This is done by retesting results which look suspicious or which do not behave as suspected. Another mechanism is to verify that the test is up and running by checking the results of tests that verify the flow and not the device under test.
Reporting
When it has been concluded in which revision a bug was introduced it is possible to provide information about the bug from three different sources:
- The VCS contains key data about the revision. This includes the username of the committer, the time of the commit and the lines that were modified. This information is required in order to create a bug report with all the necessary information and to assign the bug report to the correct person, i.e. the committer of the faulty revision.
- A comparison between test results of the faulty revision with test results from a previous revision for which the same tests pass provides information about the unique characteristics of how each bug manifests itself. This method is applied to the sometimes lengthy build logs and test logs in order to distinctly point to the section in the logs where the error started to manifest itself.
- A comparison between the test results for different build configurations, all based on the same faulty revision, provides information about the build configuration characteristics where the bug manifests itself.
Next: Continuous integration
|
|
|
|
|
Navigate to related information