Don't believe the science hype – we haven't created true AI yet

http://www.theguardian.com/commentisfree/2014/jun/11/ai-eugene-goostman-artificial-intelligence

Version 0 of 1.

At the Royal Society last week Kevin Warwick, professor of cybernetics at Reading University, organised a version of what he called the Turing test – intended to decide if you're talking to a human or a machine.

Warwick claimed that a program designed by the Amazon software developer Vladimir Veselov had passed the test by convincing 30% of the judges that it was a 13-year-old Ukrainian boy called Eugene Goostman.

This event has mightily stirred up the artificial intelligence research community, with a Guardian comment after Ian Sample's article on 9 June capturing a typical view: "The problem is that this is a PR stunt for a mediocre university, not a useful measure." Other comments recalled the history of Warwick's eye-catching gestures – he once had a chip implanted in his arm so as to communicate directly with his partner.

So has some AI milestone been reached? Is Eugene Goostman the best conversation program ever created? Turing argued in 1950 that if a computer could be substituted into the old guessing game of "Am I getting notes from a man or a woman?" and no one noticed, then we should just admit computers could think and stop worrying. But Turing was not proposing a test of any sort, and in his setup, the question "Is this a machine or a person?" is never asked, because one thinks one is answering a question about sex. Once you pose the human-or-machine question directly, things start to get very odd.

The long-running Loebner competition has a similar format except that judges are not asked to say whether each candidate is human, but to rate them numerically. At the end, the scores are added and the performances are ranked. So far the "people" have always come out on top. The Loebner competition is not what Turing had in mind either, but it is at least systematic: it avoids the explicit question "Is this a machine?" with its ranking system, and publishes the transcripts of the best systems on its website, which Warwick declined to do.

All this makes it rather unlikely that Goostman was any better than the refined and experienced chatbots in the Loebner competitions; indeed, it seems Goostman competed there in 2012 and didn't do well. A defect of the Loebner setup is that there is no methodology for comparing the winner one year with any other, so it's not clear if the winners are getting better.

A team I led at the University of Sheffield won in 1997, and my hunch is that the winners now are much the same in terms of quality, but we cannot know for sure. What we can know is that there is no reason to believe Goostman is better than the Loebner winners.

The chatbot world, many of whose products are now being customised to replace call centres and perform in sex chatrooms, is a largely amateur field, The winners are not usually academic AI researchers, who want to bring to bear big- data analyses, theories of computer learning and linguistics. The winners are often just very laborious handcrafted systems that have programmed in thousands of possible replies to whatever is said to them. AI theoreticians do not like that, and believe machines must be rational – as they think we are – and will have to learn to talk as we do. The problem is that sophisticated theoretical approaches have not yet produced the kind of sheer performance level that amateur chatbots often do. And the game has been changed by the appearance of Apple's Siri, which answers questions on iPhones. Although this was developed out of a large US research project, its success comes from the very large number of hand-programmed special responses, such as the comic answer to the question: "How do I bury a body?" (Try it.)

Siri combines aspects of research-based systems and chatbots; certainly there must be some intelligent mechanisms to allow a participant to change subjects in conversation and go back to early unfinished topics, as people can, but chatbots can do none of these. There will be a huge market for such devices if we can get the mixture right – for digital companions that chat to lonely old people in their homes as well as for replacing soul-destroying human work in call centres. But all this is a long, hard slog.

We have had nearly 60 years of computer conversationalists since Colby's Parry system, which mimicked a paranoid patient in an asylum and could hold people enthralled and amazed for long periods of time. Yet it was Weizenbaum's Eliza, a naive and trivial program, that got all the publicity at the time. Hype really does exist in science; the answer must be the right combination of patience, cynicism and stamina.