IBM's Watson computer beats humans at Jeopardy
R Colin Johnson
2/3/2011 7:26 PM EST
However, rather than merely load the entire content of books and encyclopedia's directly, the information was edited to include only those items that are relevant to Jeopardy clues, such as famous-quotes, -people, -places and -things. In action, Watson does not "scan" this entire corpus after each clue, but has already pre-processed a body of semantic indices to enable rapid, direct access to relevant chunks of content.
"There is an enormous amount of computation every time Watson answers
a single question," said Ferrucci "There is natural language
processing, there is machine learning, there is knowledge
representation and reasoning, there is deep analytics, and it all
happens in just three seconds."
In trial Jeopardy matches with champions Ken Jennings and Brad Rutter, Watson (center) won beat the humans.
The first thing Watson does when provided with a Jeopardy clue is question analysis, which determines what type of response the question is asking for—called its lexical answer type (LAT). IBM determined from a study of more than 20,000 clues, from past
Jeopardy games, that were about 2500 different LATs, but, of course, just as there are an infinite number of possible clues there are an infinite number of possible LATs.
This question analysis algorithm was optimized to be fast enough run on a single core, but once the LAT is determined, Watson goes massively parallel, decomposing the query into hundreds of hypothetical candidate answers which are narrowed down by a software filter based on machine learning. Roughly 100 hypotheses are allowed to pass the software filter's threshold to undergo a more rigorous evaluation involving gathering evidence sources for each candidate answer which can be used to evaluate it along several different dimensions, including taxonomic, geospatial (location), temporal, source reliability, gender, name consistency, relational, passage support and theory consistency.
Next, Watson runs analytics on potentially hundreds of thousands of scores to estimate its confidence level for each candidate answer and identify the single best-supported hypothesis given the evidence. For this step, Watson goes sequential again, merging, weighing and combining using a hierarchy of machine learning and statistical modeling techniques. This final step, optimized to run on a single core, searches results in the list of candidate answers, which are then ranked by confidence levels. A monitor, visible by the Jeopardy
audience but not the other contestants, shows Watson’s top five candidate response to a clue, with a bar
graph beside each indicating its confidence level.