News & Analysis
Comment
KingofthePaupers
Jct: "Doing this ends inflation of money?" is one question I'm the only person ...
goafrit
You are correct, probably. Sometimes we miss that simple answer. Thank you.
IBM's Watson computer beats humans at Jeopardy
R Colin Johnson
2/3/2011 7:26 PM EST
Enormous computation
However, rather than merely load the entire content of books and encyclopedia's directly, the information was edited to include only those items that are relevant to Jeopardy clues, such as famous-quotes, -people, -places and -things. In action, Watson does not "scan" this entire corpus after each clue, but has already pre-processed a body of semantic indices to enable rapid, direct access to relevant chunks of content.
"There is an enormous amount of computation every time Watson answers a single question," said Ferrucci "There is natural language processing, there is machine learning, there is knowledge representation and reasoning, there is deep analytics, and it all happens in just three seconds."
The first thing Watson does when provided with a Jeopardy clue is question analysis, which determines what type of response the question is asking for—called its lexical answer type (LAT). IBM determined from a study of more than 20,000 clues, from past Jeopardy games, that were about 2500 different LATs, but, of course, just as there are an infinite number of possible clues there are an infinite number of possible LATs.
This question analysis algorithm was optimized to be fast enough run on a single core, but once the LAT is determined, Watson goes massively parallel, decomposing the query into hundreds of hypothetical candidate answers which are narrowed down by a software filter based on machine learning. Roughly 100 hypotheses are allowed to pass the software filter's threshold to undergo a more rigorous evaluation involving gathering evidence sources for each candidate answer which can be used to evaluate it along several different dimensions, including taxonomic, geospatial (location), temporal, source reliability, gender, name consistency, relational, passage support and theory consistency.
Next, Watson runs analytics on potentially hundreds of thousands of scores to estimate its confidence level for each candidate answer and identify the single best-supported hypothesis given the evidence. For this step, Watson goes sequential again, merging, weighing and combining using a hierarchy of machine learning and statistical modeling techniques. This final step, optimized to run on a single core, searches results in the list of candidate answers, which are then ranked by confidence levels. A monitor, visible by the Jeopardy audience but not the other contestants, shows Watson’s top five candidate response to a clue, with a bar graph beside each indicating its confidence level.
However, rather than merely load the entire content of books and encyclopedia's directly, the information was edited to include only those items that are relevant to Jeopardy clues, such as famous-quotes, -people, -places and -things. In action, Watson does not "scan" this entire corpus after each clue, but has already pre-processed a body of semantic indices to enable rapid, direct access to relevant chunks of content.
"There is an enormous amount of computation every time Watson answers a single question," said Ferrucci "There is natural language processing, there is machine learning, there is knowledge representation and reasoning, there is deep analytics, and it all happens in just three seconds."
In trial Jeopardy matches with champions Ken Jennings and Brad Rutter, Watson (center) won beat the humans.
The first thing Watson does when provided with a Jeopardy clue is question analysis, which determines what type of response the question is asking for—called its lexical answer type (LAT). IBM determined from a study of more than 20,000 clues, from past Jeopardy games, that were about 2500 different LATs, but, of course, just as there are an infinite number of possible clues there are an infinite number of possible LATs.
This question analysis algorithm was optimized to be fast enough run on a single core, but once the LAT is determined, Watson goes massively parallel, decomposing the query into hundreds of hypothetical candidate answers which are narrowed down by a software filter based on machine learning. Roughly 100 hypotheses are allowed to pass the software filter's threshold to undergo a more rigorous evaluation involving gathering evidence sources for each candidate answer which can be used to evaluate it along several different dimensions, including taxonomic, geospatial (location), temporal, source reliability, gender, name consistency, relational, passage support and theory consistency.
Next, Watson runs analytics on potentially hundreds of thousands of scores to estimate its confidence level for each candidate answer and identify the single best-supported hypothesis given the evidence. For this step, Watson goes sequential again, merging, weighing and combining using a hierarchy of machine learning and statistical modeling techniques. This final step, optimized to run on a single core, searches results in the list of candidate answers, which are then ranked by confidence levels. A monitor, visible by the Jeopardy audience but not the other contestants, shows Watson’s top five candidate response to a clue, with a bar graph beside each indicating its confidence level.
|
|
|
|
|
Navigate to related information
R_Colin_Johnson
2/3/2011 9:29 PM EST
By running on a configuration of off-the-shelf cluster computers, IBM will be able to quickly target business applications to solve using the same DeepQA architecture, just with a different knowledge base plugged in. Here are a few applications IBM told me they may target in the future with Watson-like cluster computers:
Healthcare: For medical applications Watson will be able to act as an expert consultant for doctors, simultaneously comparing your symptoms to those of millions of patients worldwide, along with the diagnoses that cured them and all the latest information from medical journals, supplying a ranked list of possible diagnoses.
Retail: Watson can integrate queries from retailers that simultaneously consider past buying patterns, inventory, order management and supply chain issues to supply customer relationship management (CRM) answers that are highly targeted, perhaps offer personalized "sales" while you are still standing at the register.
Financial Services: Watson's realtime analytics will enable financial institutions to what-if scenarios that include market data, current events, the opinion of analysts and a thousand other unstructured information sources that are difficult to encapsulate in to conventional algorithms--plus its machine learning capabilities will enable Watson to hone its predictive abilities even finer over time.
Government: Corporations today must deal with a dizzying array of laws and regulations that could be simultaneously considered by Watson, instead of occupying weeks of a human experts time, allowing businesses to optimize their profits in realtime while still meeting all their regulatory obligations.
What Watson Can't Do: Because the resources of the entire cluster computer are dedicated to answering each question--one at a time--Watson will not be directly amenable to traditional Internet searches, which must serve thousands of amateur users making ad-hoc queries simultaneously.
Sign in to Reply
prabhakar_deosthali
2/4/2011 12:45 AM EST
This is a great information. One question to be answered at a time is fine. I am just curious to know what is the expected response time for such a response in a single user mode. If it is a matter of seconds or even a minute, still it worth it. Because such high performance machine will be used only to get answers for some critical questions which humans are unable to find in a massive data archive.
Sign in to Reply
R_Colin_Johnson
2/4/2011 12:37 PM EST
The Jeopardy applications requires about three seconds of cluster computer time, but my guess is that financial applications might require slightly more time and retail apps may require less. Also I am sure IBM is working on way to queue up these queries for more efficiency.
Sign in to Reply
selinz
2/4/2011 12:55 PM EST
This is definitely a completely different challenge than chess. This is not "computable" so much as reliant on massive amounts of data. It will be interesting to see how it responds to some of the more abstract clues...
Sign in to Reply
yalanand
2/5/2011 6:40 AM EST
Wow, cant imagine how much effort has gone into to build this massive system. Kudos to IBM for building interesting and challenging systems.
@R_Colin_Johnson By the can we see the game online ?
Sign in to Reply
gatorsrule
2/5/2011 12:16 PM EST
From a science standpoint, I'm missing the "wow" factor here. They've taken a very well known data set (the list of all previous questions) and applied search algorithms to extrapolate that data to the same setting. "Wow" in a sense of they really spent a lot of money on doing this, but certainly not "wow" in terms of any innovation or technical breakthroughs.
Sign in to Reply
ReneCardenas
2/9/2011 1:58 PM EST
gatorsrule, ditto with the unimpressive premise to see who can retrive data faster. I would like to gauge how that system implemetation captures and responds to abstract concepts.
Sign in to Reply
Ratzo
2/6/2011 11:03 AM EST
Won't it be funny if Watson gets every answer correct, but hasn't been programmed to answer in the form of a question?
Sign in to Reply
goafrit
2/6/2011 1:34 PM EST
This is awesome. I will like to challenge this beast on how to save the world from malnutrition and famine. Maybe, it can churn out the secret formula that we have longed for ages in Africa.
I cannot wait for watch Jeoparday for this classico
Sign in to Reply
David Ashton
2/6/2011 5:11 PM EST
Goafrit, the formula to sort out Africa's problems is not a secret. Get rid of dictators like Robert Mugabe and you'll be more than half way there.
It's not rocket science......
Sign in to Reply
goafrit
2/18/2011 2:37 PM EST
You are correct, probably. Sometimes we miss that simple answer. Thank you.
Sign in to Reply
KB3001
2/6/2011 1:42 PM EST
Good publicity for IBM. The kind of research behind it is however being done in many research labs world-wide.
Sign in to Reply
Ratzo
2/6/2011 2:38 PM EST
I can't wait to see if the computer addresses the game's host by name.
"I'll take 'Sixties Sitcoms' for $100, Alex."
Sign in to Reply
CamilleK
2/6/2011 3:31 PM EST
I would love to see a '2001 Space Odyssey' category with 'who is HAL' being the answer followed by some joke about shutting down and expressing emotions. Seriously now, IBM deserves accolades for thinking outside the box and for pushing the frontiers of computing, semantics, human thought and all the underpinnings of meaning, understanding and knowledge. Jacques Lacan once said (in a different context) "language points to a lack". This is a case where a
'lack (of sentient being) will point to language' to prevail. We need smart shows on TV and I am certainly planning to watch the series.
Sign in to Reply
p_g
2/6/2011 6:04 PM EST
Its intersting to see, human brain putting all its inteligence to develop something more inteligent than brain himself.
I guess the only place computer beat humans and data churning.
Sign in to Reply
Neo1
2/6/2011 9:45 PM EST
p_g I don't think they are trying to make one above human brain, if at all they can. The challenge is to make it do any task which humans do with ease but these tasks have been shown to have enormous computational requirement which makes it daunting.
This is a great breakthrough and would sure go a long way in understanding the analytics of human cognition.
Will the game be telecast live? Hope someone puts it on youtube for all users.
Sign in to Reply
Duane Benson
2/7/2011 11:51 AM EST
I also fail to see the "wow factor" in this. But, that's part of the beauty of it in my mind. A very complex problem broken down to a massive amount of very simple problems. Isn't that what any computer is? Simply a massive amount of very basic logic gates.
Quite a number of years ago, I was told that the color television was the most complex device ever created. I'm sure there is some hyperbole in that statement, but it was an extremely complex design. Taking that functionality and replicating it on a generic computing platform, to a certain extent, has the same lack of wow as using a massive computer to win at Jeopardy, but as I see it, the "wow factor" is in the ability to simplify and use more or less generic hardware to solve those problems.
Regardless of how it is done, at some point, such a machine will be indistinguishable from a human. When that is the case, does really it matter whether it's brute force or a system designed to truly emulate the human brain?
Sign in to Reply
iniewski
2/7/2011 12:53 PM EST
I agree with Duane and others: where is the "wow" factor here? Show me something that requires intelligence not some massive brute force computations!.... dr Kris
Sign in to Reply
Ratzo
2/7/2011 1:12 PM EST
Would you be WOWed if the computer engaged in witty repartee with Alex in the "getting to know you" part of the show?
Alex: So, Watson, I understand you come from a non-traditional family.
Watson: That's right, Alex. My two dads are David Ferrucci and John E. Kelly III. They may be nerds, but they are the Best Dads Ever! In fact, when I was driving them to the show today...
Alex: Just a minute there, Watson! YOU drove THEM to the show?
Watson: Yes, Alex, but it wasn't easy. You might say it was a hard drive! Ha! Ha!
Sign in to Reply
Ratzo
2/7/2011 1:24 PM EST
... Worst case, I guess, would be if the witty repartee went something like this...
Alex: Watson, do you come from a nontraditional family?
Watson: What is the first question that Alex asked Watson today?
Alex: What?
Watson: What is the unit of power, as defined by the Scottish engineer James Watt?
Alex: Ay Jesus.
Watson: What were the last words of Charles V, King of France?
Sign in to Reply
Sheetal.Pandey
2/7/2011 4:33 PM EST
well humans have made computers and now computers will compete against humans. Great idea! It sounds great to hear as long as its a game or something for fun. IBM deserves appreciation for keeping on doing something to prove they are masters in this field.
Sign in to Reply
Warren
2/8/2011 8:28 PM EST
"Well humans made computers" and, imagining advances in gene therapy and other genetic modifications and methodologies, one can imagine computers making humans [better]. Or worse.
Sign in to Reply
R_Colin_Johnson
2/9/2011 12:40 PM EST
Just remember that these AIs are one-trick-ponies and are only as good as their specialized algorithms, which so far, are highly application specific.
Sign in to Reply
ReneCardenas
2/9/2011 1:52 PM EST
R_Colin, earlier attempts to crack a close information system using AI failed to consider the complexity of human communication (with cues and noise source in the form of modulation, accent, etc).
Were the logistics described to you, I did not get the details form this piece of that competition.
Is the question prefetched in the system by human? Speech or vision recognition?
I wonder if a system like this would be able to capture effectively all the relevant details for an effective query.
For example, in the medical case, only if the device is able to capture the essence of all relevant data, can I see that the diagnosis to be effective.
My thought is the GIGO principle can limit the effectiveness of this effort in real commercial applications.
Sign in to Reply
R_Colin_Johnson
2/9/2011 3:42 PM EST
Watson does not perform speech recognition, rather the clues are delivered in text form.
Sign in to Reply
DrQuine
2/9/2011 8:13 AM EST
The extraordinary task that Watson is taking on is contextual analysis and problem solving in essentially random domains. Anyone can design a machine to grind flour (2 rocks work) but the raw ingredients must be presented just right and the machine doesn't do anything else useful. In this case, Watson is taking on a "knowledge" problem (not just a brute force mathematical calculation or the very constrained game of chess) that challenges the most skilled humans in the world. Kudos!
Sign in to Reply
DrQuine
2/9/2011 12:49 PM EST
I'd say the nature of the Jeopardy application positions it for expansion to a wide range of applications (not just a one-trick-pony). It is mining and combining information across a wide range of human knowledge and also managing linguistic features. That is hard work. Even educated people with access to reference materials can't necessarily answer some of these questions.
Sign in to Reply
R_Colin_Johnson
2/9/2011 3:47 PM EST
IBM has all sorts of plans to expand Watson's reach, using the lessons they learned in founding the Open Advancement of Question Answering (OAQA) systems initiative. By following OAQA principles, it should be possible to house nearly any knowledge domain in a searchable format, enabling natural language queries for all sorts of applications. The only major limitation is the computing horsepower required--several seconds of supercomputer time per query--which will restrict users to those that can afford it.
Sign in to Reply
Ratzo
2/9/2011 1:29 PM EST
Watson a one-trick-pony? Nah, I think he'd be very competitive on "Wheel of Fortune". Admittedly, he wouldn't last long on "Survivor", unless he's programmed to betray his allies.
Sign in to Reply
ReneCardenas
2/9/2011 5:21 PM EST
Very clever Ratzo, very clever... I cracked a loud laughter.
Thanks for the humor.
Sign in to Reply
RWasserman
2/9/2011 6:09 PM EST
You might be onto something, Ratso.
Betraying your allies is the most human behavior.
Sign in to Reply
R0ckstar
2/10/2011 3:28 PM EST
I must be too easily impressed, because this is really impressive. A milestone even. From here, it's just a few short years before Watson's kids are answering their own questions recursively from being loaded up with all manner of highly specialized, obscure, & complex research material in medical, materials science and physics, and spitting out unified field theory and transparent aluminum formulas. WOW.
Sign in to Reply
KingofthePaupers
2/19/2011 4:41 PM EST
Jct: "Doing this ends inflation of money?" is one question I'm the only person claims to have solved and I'd bet Watson cannot.
Sign in to Reply