Driving Blindfolded – Solving The Triage Challenge

When regression tests fail, an engineer must decide which failing tests should be further analyzed and by whom - a process called triage. The challenge is the dearth of information about the bug and its creator at this early stage. This session outlines a novel method that addresses the issue by driving the testing to gather enough information to do fully automated triaging. This paper explains how this innovative approach solves the triage challenge, thus accelerating the bug fixing process by 30%.

Introduction
Regression testing is performed regularly through-out a project in order to ensure that the quality never dips, regresses. Regression testing is normally automated in terms of kicking off the tests and putting together the test results. But the analysis of the test results is not automated. The first thing that needs to be done is to look at the test failures, decide which are real errors and which are just intermittent computer or network issues and then assign the bug to the appropriate engineer. This process is called triaging. As it is impossible to know the exact nature of a bug, before it has been thoroughly analyzed, triaging has to be done while having very little information available at this early stage. The methods used are ad hoc, based on experience, association or static assignment, where an engineer is assigned for each test to first look at the failure. The triaging process takes time as the test results needs to be translated into bug reports and if the results are sent to the wrong person(s) then it will take longer before the issues are fixed.
 
What is needed is a tool that provides automatic triage which immediately sends the bug reports to the person who committed the faulty revision in order to accelerate the bug fixing process. The earlier the correct engineer gets the bug report the sooner the bug will be fixed. In order for this to work, automatic triage needs to be very robust so that it always sends the bug reports the right way and must be fast to run and use limited amount of resources so that it is efficient to use in large systems. This paper looks at the technology behind a new tool called PinDown, which has a novel approach to automatic triage that addresses these issues.