Mapping test results to revision information
Version Control System (VCS) contains the code of the device under
test, the test bench, the tests and all the changes made to all of its
files. The VCS contains information about each revision of each file or
module such as: the committer of the revision, the time of the commit
and the code lines that were updated. This is the information needed for
triaging: who (committer), what (revision) and when (time of commit).
What is needed is a tool that can map test failures to a revision in the
VCS in order to extract all the information needed for triaging,
including both creating a bug report and assigning it to the right
Figure 1. Automatic Triage Tool in System
order to take advantage of the information available in the VCS an
automatic triage tool consequently needs to interface against both the
VCS and the regression test system in order to map test results to
revision information (see Figure 1). The automatic triage tool also
needs to be able to communicate the result of its diagnosis
automatically to the bug tracking database, email and files on the web
and on the normal file system. The automatic triage tool also needs to
save the test results and its diagnosis in a test result database to
allow for speedy analysis and to be able to amass historical test
results information that can be used in its analysis.
the revision at which point a bug was introduced that caused certain
tests to fail is done by rerunning the failing tests on older revisions,
until the first revision is found for where these tests fail. If tests
passed on the previous revision and then started to fail on this
revision then we know that the changes made in this revision contained a
bug which caused the tests to fail. The revision that contains the bug
is called the faulty revision.
Regression testing is a type of
verification where the purpose is to ensure that the device under test
does not regress in quality. The regression tests should normally pass
and if they don’t, it is possible to trace back through the revisions
until the earliest revision is found for which the test(s) do not pass.
This is the faulty revision. However, if a test has never passed, e.g. a
new test that was just introduced, then the result may be
always-failed. In order to avoid going back too far a limit must be set
at which point a test is concluded to be always-failed.
a test failure to the faulty revision (or less common, concluding that
the test has always failed) is a very robust method. The result is
deterministic (“this test fails since this faulty revision”), which is
different from many other diagnosis methods which lists the most likely
sources of an error. For triage it is important to deterministically be
able to point out the committer in order to assign the bug report to the
correct engineer automatically. The committer of the faulty revision
may not have introduced an actual error. Instead the committer may have
done an incomplete update, where the changes made were appropriate but
the committer forgot to update associated tests or configuration files.
It may not even be the committers responsibility to update the other
files necessary to make the update complete. However, even in this
scenario the committer is the best person to assign the bug report to as
this person can quickly notice what is missing to make the update
complete and assign the bug report to the appropriate person and add the
required changes to the bug report. Automatic triaging is not about
blame, but about finding who is the person best suited to analyze a
failure in order to fix a bug as soon as possible.