Beyond Tay: ABC and News Corp foray into chatbots

Jake Evans came to Storyology on a mission to extract all the teachable moments for other Fairfax Media journalists who couldn’t make it. In turn, we’re extracting them from him. This piece originally appeared on Fairfax’s internal Learning Hub.

First forays into bot technology don’t always go according to plan.

Tay was Microsoft’s first attempt. Her AI software worked perfectly. The only problem was Microsoft’s naive (or ignorant) expectations for how Twitter might try to talk to her. Within a week users had trained her to become a neo-Nazi, and she was hastily removed from Twitter’s influence.

Fortunately for News Corp and for the ABC, their bots fared better. For a start, there was no AI program that could be manipulated to return racist results.

Archie Barwick, a soldier serving in World War I, gives audiences who subscribe to his updates a “live” recounting of his experiences at Pozieres over a period of a few weeks. The ABC News Bot on Facebook and Twitter (@ABCNewsBot) ran throughout the 2016 federal election campaign, giving updates on who was leading in the polls, as well as more detailed information on candidates and individual seats.

Both bots were the first time each media company had made a chatbot for the Facebook Messenger app, although the ABC tried a much more experimental bot on Twitter for the 2013 election.

This year at Storyology, Simon Elvery, a developer for the ABC’s interactive digital storytelling unit, and Ryan Hunt, the innovation manager for News Corp Australia, talked about their bots and what they learned from these early attempts.

Keep it simple, stupid

The ABC digital interactive storytelling unit wanted to keep it simple, by setting up the bot to answer just a few basic questions. Even a basic question quickly grows into complex work.

“The first question we asked was, ‘Who is winning?’” Simon said. That meant reporting back polls and election articles.

But then it gets tricky.

Because if you’re asking a friend, you might ask “what is the latest election news”  — 
or you might ask “what’s the latest election news?” That’s easily solved, but Simon said sometimes it’s not even phrased as a question, or the grammar’s missing, or people talk to the bot with commands.

“People cotton on to the idea that it’s a computer at the other end, and start to treat it like a computer by giving it a command rather than asking it a question like you would a person,” Simon said.

Elvery built @ABCNewsBot using a completely different method to the News Corp bot.

Elvery built @ABCNewsBot using a completely different method to the News Corp bot.

With these added complications in mind, and considering Simon was largely building this bot alone, he brainstormed with the team to determine just a few different questions the bot would be able to answer.

What the ABC bot knows

  • Basic election information
  • Basic candidate information
  • Electorate candidates
  • Electorate candidate running for party
  • Is a candidate running?
  • Who is running for Senate?
  • Where is an electorate?
  • Live election results

“It was difficult to contain the scope, if I’m honest,” he said “And it blew out a little bit further than we were anticipating, but it sort of needed to go that far just to provide a rounded enough product that people didn’t feel like they weren’t getting what they were expecting.”

So how did people engage with it? Simon said that from the way people interacted with the bot, it was clear there’s interest in the medium. “There’s interest in this kind of delivery mechanism for journalism, and interest, I guess, in what you’d probably term conversational journalism.”

“Some people tried it once, either got a response and thought nothing more of it, or didn’t get the response they wanted and thought, ‘I won’t bother with that again,’ ” he said, while others “used it more than 100 times right through the campaign period and were very clearly highly engaged.”

“It’s just another way for the audience to be engaged with the process of journalism,” said Simon.

“One of the reasons we wanted to be there from a distribution point of view is that more and more the audience is expecting and desiring to interact with news in the situations they’re familiar with: messenger platforms like Facebook Messenger, like Twitter and like Whatsapp. They go there to find news, they go there to share news, and they go there to talk about news with their friends. And so being in that platform is really important for news delivery, but it’s just as important for news discovery and the input side of journalism.”

Innovation is collaboration

Archie Barwick was a soldier who fought and died at Pozieres in World War One, a real person, whose diary entries, medical records, letters to and from home and more were drawn on by News Corp Australia as part of their huge Anzac Live campaign.

His story began through “live” updates on a Facebook page, as if he was a Facebook user’s “friend” giving updates from the front  —  but after a News Corp hackathon event in New York focused on bots, their innovation manager Ryan Hunt had an idea for another way to tell Archie’s story.

“They said they wanted to do something with bots,” Hunt said. “They had no specific editorial drive or business need. And because of the position I have — I understand the different parts of the business — I saw the opportunity with Anzac Live.”

Ryan Anzac Live

Ryan Hunt explains how a small army of UTS students taught Archie Barwick, Anzac Live, how to answer questions properly.

Ryan acted as the conduit and translator between Justin Lees, who headed up the editorial end of the Anzac Live project, and the tech team involved. The result was “Archie bot”, with whom audiences could now have conversations through Facebook Messenger  —  not just about what was happening, but how he felt, who his friends were, what his pastimes were, or pretty much anything you’d think to ask a soldier when you first met him.

When Archie received a message, if it wasn’t something he knew how to answer, he’d send it to wit.ai, a free artificial intelligence program by Facebook, which can be trained to answer unfamiliar questions. It measures an answer in confidence, and if its confidence doesn’t pass a particular threshold, the message is sent to the News Corp team to set a response.

This, said Ryan, is where it gets really tricky.

For example, if you asked Archie: ‘Have you shot any Germans?’, the AI struggles to determine whether Archie or the Germans are the target: Has Archie been shot or has Archie shot someone? Both questions involve Archie and the shooting verb, but the subject and object are different.

Training a bot to “understand” language is resource-intensive. Programs like wit.ai can reduce the load.

This is why Simon Elvery said AI language processing is much harder to get right (as opposed to natural language processing, which responds based on preset libraries).

“It’s a little bit easier to start from scratch [with natural language processing], you don’t have to train it,” Elvery said.

“But whatever you do is going to go wrong, so embrace the failure.”

Ryan agreed. Even after weeks of training, he said Archie only gave a perfect response 49 per cent of the time.

Still, as an experiment, the team found it incredibly useful.

“We learned a lot, there’s still a lot to learn, but it gave Justin and the team an understanding of what’s hard and what’s easy, what’s quick and what’s slow.”

“This may sound obvious, but innovation is collaboration. You can’t have one without the other  —  the Married with Children theme song is playing in my head right now,” Ryan said. “For me, the driver here was not to make money, it was to learn from this experience and to understand what’s possible.”

Jake Evans (@jcobevans) is a producer for Fairfax Media’s Learning Hub.