THE RACE TOWARDS HUMAN LEVEL CONVERSATIONAL ABILITY
Google is attempting to build a digital assistant that can hold a real conversation with humans. Digital assistants like Alexa and Siri can pick up keywords and perform various tasks but the responses they provide are rather scripted. Google claims their new bot Meena will be able to hold more natural and free flowing conversations with people. There does seem to be a demand for more human like chat bots. They would be useful in a wide variety of areas. The ability to converse freely is one of the hallmarks of human intelligence and is strongly believed to be a base requirement for true artificial general intelligence. Given that, it is no surprise that tech giants like Google are moving in this direction.
GOOGLE’S WORK ON LANGUAGE NEURAL NETWORKS
Meena is a neural network with billions of parameters that is believed to be able to handle multiple turns in one conversation. In other words, it should not lose its “train of thought” when the core subject of a conversation quickly shifts. This will aid Mean greatly when conversing with humans. Meena was trained off of a neural network architecture called Transformer which Google released back in 2017. Billions of English words were added to the training set to make the bot more affluent.
SENSIBLENESS AND SPECIFICITY AVERAGE
To measure the quality of Meenas conversational abilities Google developed a metric dubbed the Sensibleness and Specificity Average (SSA). This metric combines two fundamental aspects of a chat bot, making sense, and communicating specifically. Sensibleness describes the ability of the bot to generate responses that make sense in human context. This is one of those things we don’t tend to notice when talking to other humans as Google’s research suggests 97% of human produced statements meet this criteria.
The issue with only measuring sensibleness is that the chat bots tend to learn to respond with statements such as “I don’t know” when a question is asked that falls outside their domain to avoid being penalized for not making sense. Many closed-domain chat bots are already guilty of this. They succeed in not generating gibberish or contradicting themselves, but at the cost of not really saying anything of substance. To combat this, another dimension was added to the SSA metric. Specificity measures whether the response is specific given the context of the conversation. This prevents bots from giving vague replies killing the conversation as the human becomes fully aware that it is conversing with a robot and becomes subsequently “weirded out”. Part of the reason why companies are striving to make more human like bots is to avoid this kind of thing.
Final testing demonstrated that Meena scored an impressive 79% out of a possible 100 in the SSA metric. This is impressive considering the average human score according to their experiments was only 86%. Despite the fact that Google admits the SSA cannot capture all aspects of human conversation the end result here is that Meena is more human like in its communication style than any chat bot before it.
MEENAS CAPABILITIES AND THE FUTURE OF THE CHAT BOT
Meena attempting to make a bad joke.
Given all of the data Google has a compelling argument for why Meena’s capabilities will far exceed that of Siri and Alexa. The virtual assistant industry has witnessed a tougher competition than ever before. Apple, Amazon, and Google are trying to one up each other. Google believes Meena will be able to talk to people fluently about anything in the world just like any other person. If this capability proves to be as marketable as Google thinks, then Amazon and Apple will have to make some changes in order to keep up.
The last major chat bot that was supposed to do what Meena can do was Microsoft’s twitter bot Tay. This was a massive failure and anyone wishing to read more about it can do so here. Google did use some social media posts to train Meena so we can only hope that the problems that caused Microsoft to take Tay offline will not happen to Meena.