A common problem when
running large regression test suites is the intermittent instability of
the computers and networks used for testing. This may lead to builds or
tests failing, or never completing. These results, called result noise,
do not reflect the state of the device under test and should be filtered
It is important to be able to filter out result noise from
real design results in order to reach the correct conclusions. This is
done by retesting results which look suspicious or which do not behave
as suspected. Another mechanism is to verify that the test is up and
running by checking the results of tests that verify the flow and not
the device under test.
When it has
been concluded in which revision a bug was introduced it is possible to
provide information about the bug from three different sources:
VCS contains key data about the revision. This includes the username of
the committer, the time of the commit and the lines that were modified.
This information is required in order to create a bug report with all
the necessary information and to assign the bug report to the correct
person, i.e. the committer of the faulty revision.
- A comparison
between test results of the faulty revision with test results from a
previous revision for which the same tests pass provides information
about the unique characteristics of how each bug manifests itself. This
method is applied to the sometimes lengthy build logs and test logs in
order to distinctly point to the section in the logs where the error
started to manifest itself.
- A comparison between the test
results for different build configurations, all based on the same faulty
revision, provides information about the build configuration
characteristics where the bug manifests itself.