Has a computer program passed the Turing Test? Don't believe the hype
Reports that a chatbot named 'Eugene Goostman' has passed the Turing Test have been widely exaggerated.
Tue, Jun 10, 2014 at 08:41 PM
If you believe everything you read on the Internet, then you've probably been fooled by recent reports that a computer program named Eugene Goostman has passed the notorious Turing Test, an experiment proposed by Alan Turing in 1950 that is designed to test whether machines can think.
A news release from the University of Reading originally set off the firestorm. Since then, reports from sources as far ranging as the Washington Post, The Independent and Computerworld have publicized exaggerated versions. Some headlines (such as those from The Independent and Computerworld) have gone so far as to suggest that Eugene Goostman is a "supercomputer." If you've been taken in by the hype, you might even believe that the robot apocalypse is right around the corner.
First of all, let's be clear on exactly what Eugene Goostman is: it's a chatbot. It's not a supercomputer; it's not even a computer. It's a computer script, with nothing particularly super about it. You may have encountered other chatbots while engaging with tech support or troubleshooting on the Internet. They are computer programs designed to simulate (repeat: simulate) an intelligent conversation with one or more human users, usually via text chatting. Unlike chatbots that aim to provide you with tech support, however, Eugene Goostman is intended to simply engage in small talk. More specifically, Goostman is designed to simulate conversation of a 13-year-old boy.
Original reports claimed that Goostman was the first program to ever pass the Turing Test, which if you're a Turing Test believer means that Goostman is intelligent. But critics have since piled on, raising doubts about whether the experiment qualifies as a proper Turing Test at all.
"It's nonsense, complete nonsense," Stevan Harnad, professor of cognitive sciences at the University of Quebec in Montreal, told the Guardian. "We have not passed the Turing test. We are not even close."
Alan Turing originally suggested that once computer programs could fool people into believing they are human around 30 percent of the time, that they could be classified as intelligent. The exact parameters of a proper Turing Test, however, have never been firmly established, and have been hotly debated ever since Turing originally suggested his version. Turing also never made clear why 30 percent ought to be the magic number.
If you believe that Eugene Goostman passed the Turing Test, then you must also believe in a Turing Test with exceptionally loose parameters. Just 10 out of 30 judges were fooled by the 13-year-old impersonator, barely passing the 30 percent threshold. But are 30 judges enough, and were they a representative sample? The panel included several celebrity judges, such as actor Robert Llewellyn of the British sci-fi sitcom "Red Dwarf."
Several critics (such as those at Wired and Techdirt), have scrutinized whether the choice of judges was adequate. It has also been pointed out that Eugene Goostman contains a built-in 'cheat,' namely that it pretends to be a 13-year-old boy from Ukraine. This sets up judges to make excuses for odd responses. A judge might speculate: "But it's just a 13-year-old boy, and English is not even his first language."
Furthermore, if you want to grant such loose test parameters, then you can't claim Eugene Goostman as the first program to pass the Turing Test. Several other programs have passed similarly loose tests, and with better results than Goostman. For instance, a program named Cleverbot convinced a whopping 59 percent of judges it was human back in 2011. But few seem to recall those headlines today.
Regardless of whether Eugene Goostman can be said to have passed some sort of test threshold, there are few experts who would ascribe the program with intelligence. For instance, Marvin Minsky, one of the most revered names in artificial intelligence, told the Guardian: "Nothing is learned from poorly designed 'experiments'. Ask the program if you can push a car with a string. And, if not, then, why not?"
In other words, whatever Goostman's accomplishments establish, they don't convincingly establish intelligence. And since the question of intelligence is supposed to be the whole point, Goostman's accomplishments don't appear to tell us much at all.
Related on MNN: