Just in time for the 60th anniversary of Alan Turing‘s death, a group of researchers claim to have developed a program that has passed the test Turing invented for artificial intelligence for the very first time. However, those claims might have been overstated, as many disagree that the Turing test has actually been passed.
At a time when computers were little more than abacuses Turing foresaw, at least dimly, the developments that would come from his pioneering work. Noting that it was very difficult to define what thinking really is, Turing suggested the important test was whether a robot could imitate a human. He proposed that a machine could be defined as intelligent if it could engage in a conversation with someone who could not tell if they were interacting with a human or computer.
Presumably the spambots assuring recipients they are horny women just dying to get naked for them fool some people, so a little more definition is required. Turing predicted that by the year 2000, “an average interrogator will not have more than 70% chance of making the right identification” between a human and computer over the course of five minutes.
However, the 30% pass rate over five minutes was just Turing’s prediction for what would be possible at the time, not a requirement of the test itself. That is one benchmark being used to make the claim of passing this test, which is slightly problematic.
By 2012 researchers were doing well enough that systematic tests were required. At the time, none of the computer programs being tested were able to fool 30% of the judges over the course of a five minute chat. One of them however, “Eugene Goostman”, a chatbot pretending to be a 13 year old boy from Odessa, got very close.
Two years later, Goostman made the grade, although it is unclear to what extent this is a result of improvements in the program, or just a slightly different sample of judges. Moving from just under 29% to the 33% Goostman scored this time doesn’t suggest its Russian computer programmers have been bounding along in the meantime – unless the judges are also getting smarter or publicity from Goostman’s near success last time altered some judges to his identity.
Critics of this announcement have said that such a small sample size of judges isn’t enough to accurately depict the “average investigator” that Turing required. Goostman is also described as a chatbot, not a “supercomputer” as some have reported, which adds to the skepticism.
In the trials, conducted by the Royal Society, judges engage in conversation via text with a mix of human and computer respondents and have to pick which category they think each falls into. Four other computers failed.
Arguably, pretending to be a 13-year-old whose first language is not English is cheating. “Our main idea was that he can claim that he knows anything, but his age also makes it perfectly reasonable that he doesn’t know everything,” said Vladimir Veselov, one of the creators of the program. “We spent a lot of time developing a character with a believable personality.”
Nevertheless, some commentators have suggested this is a good time to think about the dangers of tireless computers capable of talking to millions of people at once when it comes to fraud. “Having a computer that can trick a human into thinking that someone, or even something, is a person we trust is a wake-up call to cybercrime,” said Kevin Warwick of the University of Reading, the organizers of the test.
Claims of success have been made before, but Warwick says, “this event involved the most simultaneous comparison tests than ever before, was independently verified and, crucially, the conversations were unrestricted. A true Turing Test does not set the questions or topics prior to the conversations. We are therefore proud to declare that Alan Turing’s Test was passed for the first time on Saturday.” Still, not everyone is convinced.
Anyone wanting to chat with Goostman, or congratulate him on his success, can do so at http://www.princetonai.com/bot/bot.jsp. Despite telling The Independent, “I feel about beating the Turing test in quite convenient way. Nothing original,” Goostman appears to be a little overwhelmed by all the interest – his site was down when we tried it.