The birth of a fact-checking operation

While Australia has seen just a couple of short-lived political fact-checking operations, in the United States the genre has gone mainstream. Especially with this election. But fact-checking is still pretty new, and there’s a lot to experiment with. What do you check? How do you change people’s minds? In the Midwest U.S., University of Wisconsin-Madison professors Lucas Graves and Michael Wagner jumped out of their research zone and into the media fray to launch a new fact-checking site called The Observatory. Graves was among the first researchers to study fact-checking operations, while Wagner studies political communication and is a frequent news commentator.

The just-launched Observatory is part academic class, in which student reporters learn to check politicians’ claims, and part collaboration with the Wisconsin Center for Investigative Journalism, a small, independent nonprofit news organisation that’s housed at UW-Madison.

The Walkley Foundation’s own Kate Golden was previously the multimedia director for WCIJ, and spent six years working down the hall from these guys. So she called a Skype reunion to check in on how the new project was going — starting with a nice and easy icebreaker.

Kate Golden, Walkleys: How can fact-checking units have any hope of spreading truth in this post-fact journo-apocalypse?

Graves: I really take issue with the whole post-fact or post-truth argument. I mean, I understand why we’ve seen all of this agonising about it. But at the same time, though (Donald) Trump is an extraordinarily fact-free candidate, it’s never been the case that our politics were driven primarily by careful, rational collective decision-making. If anything it’s a question of degree, not of a fundamental shift in American politics.

The more fact-checking there is, the better — because it becomes harder for politicians to consistently repeat something that multiple fact-checkers regularly say is false.

Wagner: I agree with Lucas that we’re not in a post-fact apocalypse. I think it’s difficult for fact-checkers to spread truth to those who don’t want to hear it — there’s some evidence that people just don’t want to hear things that cause them to change points of view they hold deeply. But there’s a little bit of work that shows fact-checks can have a positive long-term effect on how people come to understand what’s true. The more fact-checking there is, the better — because it becomes harder for politicians to consistently repeat something that multiple fact-checkers regularly say is false.

Graves: And there’s two ways that fact checks can make a difference. One is by informing people, convincing them to let go of false beliefs. But the other way is by discouraging politicians from lying. It’s obviously not completely effective in either one of those axes. But I’d agree with Mike — it’s better than nothing.

Wagner: A third way fact-checking might matter is that it might persuade journalists who aren’t fact-checkers to stop repeating claims that have been debunked.

So are you hoping for a utopian media future where all journalists consider themselves fact-checkers and we don’t have to call this stuff by a special term to make it sound extra authoritative?

Wagner: No. I don’t think we all need to hold hands and reduce all newsworthy events to fact-checks. I think traditional reporting, deadline reporting, breaking news reporting, deep feature reporting will all hopefully continue to thrive. And if those forms of journalism can be influenced by fact-checking with respect to maybe giving a little extra scrutiny to amazing claims, or not repeating things fact-checkers have shown as false, I think that’s all to the good.

Graves: We’ve seen more fact-checks being inserted into straight news reports. We see more cases this election cycle of an article that mentions a candidate’s claim in passing, and then also notes that it’s false — which is not something you saw in the past. And that’s all to the good. But absolutely there’s still a place for straight news reporting and other kinds of investigations. And even though he-said she-said reporting is really vilified, and I think it’s good that we’re seeing a broad cultural shift away from that, there is still something to the basic principle that when in doubt, at least, journalists should try to interview people on all sides of a question. The problem was always when that principle was abused, and used as an excuse for lazy reporting.

Wagner: And fact-checks take a lot of time. I don’t know that it would be a way for most news organisations to sustain themselves if they reduced their work to a model where everything operated like fact-checks operate.

Were you champing at the bit to get this thing up and running given the barrage of falsehoods in this presidential election, or is this more about Wisconsin?

Wagner: Yes. Eight weeks ago we had 16 students who had more or less never done a fact-check and were maybe passingly familiar with Politifact. And now we have a website with more than a dozen stories. … We certainly wanted to get it up and running in this election cycle. … It fits the semester schedule. But it was also a way to launch us in an environment where there were going to be lots of claims to check, from people running for state office and of course those running for the Senate. And of course the two major candidates running for president.

One of the major fact-checks we’ll have up later this week (Eds.: this story) is actually a mostly true claim from Trump, which he hasn’t had a lot of, but I think that’ll be kind of a fun story.

In one of the debates, Trump said that since Obama has been president, almost 4,000 people have been shot in Chicago. The actual number’s in the 3,600 range. So on the one hand, he’s pretty close. On the other, the implication of his claim is that things have gotten worse under Obama. But in fact overall shootings in Chicago have dropped precipitously over the past several decades. So we tell the story about how that number’s pretty close, but overall it is not an accurate representation of violence in Chicago.

It reminds me how he said that the Trump Tower was 68 floors instead of 58. It’s almost true, but the fact that it’s not quite true is also telling.

Graves: What’s the name that he came up with for that in his ghostwritten The Art of the Deal? Truthful hyperbole.

Fact-checking class is such a great way to introduce students to the kinds of sources and the kinds of analysis that they should be doing in their straight news articles.

Wagner: Take something that sounds true or is close to being true, and make it a little better.

Graves: If someone’s going to go on to do straight reporting, especially political reporting, I think fact-checking class is such a great way to introduce students to the kinds of sources and the kinds of analysis that they should be doing in their straight news articles.

That leads me to my next question — you’re working with students, whose critical thinking skills — and accuracy — are still very much in progress. Which is true for all of us, but more so. How do you fact-check the fact-checkers?

Wagner: We go through a multi-stage process. They pitch the claim they want to check. We work with them to try to find claims we think will be checkable. Or if they’re not checkable, they’ll still be able to produce an interesting story about why. And then they write a draft of the story. In the grading process of course there are some corrections for AP style and a lot of writing advice, and then there’s also a lot of “How did you know that?” comments. They write a second draft, we go through it a little more quickly so we can then bump it up to the Wisconsin Center for Investigative Journalism. And there the managing editor Dee Hall and others on staff go through the WCIJ fact-checking process, where every single fact in the story is numbered, and the journalist proves to the editor how they know that’s true.

I’m very familiar with that form of torture.

Graves: Just one of the reasons we’re happy to be working with WCIJ.

Wagner: Yeah, if we had to do that, we might have one story published.

So what advice would you give to others who want to start their own weird collaborations between journalism and academia — advice about effectively joining forces from those very different worlds?

Wagner: One important thing to do is to marry well. We’re very lucky to have partnered with WCIJ. There are things I would not have thought about but for Lucas when it comes to how to organise the class or how to teach people how to find a checkable claim, or what our rating system should be — Should it be a scale? Should it not be a scale? If it is a scale, should it have a Half True option or not, which we don’t have?

You want enough cooks in the kitchen to make sure you have thought about why you’re adding the ingredients you’re adding, but not so many that you can’t make quick decisions to get things done. There were a few moments along the way where if one more voice came in with feedback about the logo or something like that, I might have lost my mind. But I think we had just the right number of collaborators to keep us moving forward but avoid groupthink.

In this particular semester, it was marketing the course over the spring and summer to students I really knew would be interested and would do a good job.

Graves: Another thing is that because this collaboration is backed by a dedicated Baldwin (Idea) Grant, that really makes the marriage a lot happier. No one’s being asked to do a lot of extra work, except Mike, for no pay. I’m serious actually — I think that’s an important part of a successful classroom-news outlet collaboration.

Marry well. And rich.

Graves: Exactly.

Wagner: If we were saying to WCIJ, spend 30 to 120 minutes with 16 students every week to fact-check a story for nothing, we’d still hear them laughing from when we asked in August.

You two are experts on what communication strategies are effective at changing people’s minds, correcting them or helping facts stick. What research insights like that helped you develop The Observatory?

Wagner: One bit of research that I am particularly fond of shows that people are able to overcome misperceptions they have about groups with whom they disagree. And that research suggests that visual evidence can be more impactful than just reading and words on a page. And so I’ve encouraged, in some situations, fact-checks that our students have produced that could be visual to be visual. … But it doesn’t apply to every fact-check. Not everything has that visual element to it.

Graves: There’s also evidence that people are more willing to accept information they would otherwise reject if it comes from a trusted source. I think in general, whenever possible in these fact-checks, they’re making the effort to quote sources that the broadest range of Wisconsinites will consider trustworthy and acceptable.

Wagner: Wisconsin has a pretty long tradition of a strong and competent bureaucracy, which is not something most states can say. And a lot of the work our students have done has been dealing with people who were running the department of transportation, or assembling the state budget, or are in charge of county highway spending. So people who don’t necessarily have an obvious political axe to grind and are also public servants that Wisconsinites at least have a long tradition of trusting. Truth be told, I think there’s been a little bit of a breakdown in that over the past six or so years. But I think that’s something that’s a unique advantage to us doing this in Wisconsin, compared to some other states where state government has a less positive reputation.

I’m curious about format and style decisions about what will come off as authoritative or useful. The rating scale, headlines, teasers, how you write the articles. To what extent has research influenced that stuff versus just regular good journalistic judgment?

Graves: Most of the headlines start with “You won’t believe …”, but then we change them.

Wagner: Yeah, mostly they’re alliterated — “State senator … click here to see more.”

I don’t know that research has guided the writing of a headline or the tone, I think it’s more good journalistic practice. Especially with students, some of whom get really fired up when they’re doing these fact-checks. They come to learn in some cases that a politician has lied to them, or lied to the people. And very quickly the first draft language gets a little judgy, and it needs to be reined in a little bit.

So the way I’ve thought about the way we publish stories is to just try to report the verifiable truth, and try to deal with topics and information that relate directly to the claim we’re trying to check.

Does it add extra stress to the publication process to make explicit claims about truth vs. fact?

Wagner: Yes. I thought I would be the most nervous about a false claim. But I’m equally nervous about each one. When we say something is Verified, which is how we say it’s true, the first thing I think to myself is, gee, what if someone is able to demonstrate that it isn’t? And when we make a mostly true or mostly false designation, it’s in the context of our group consciously deciding not to have a middle category, because that makes you more likely to put everything in the half true or middle category. Are we making the right decision on balance to say this is mostly true or mostly false? Those things have kept me up, here and there, with some of the decisions we’ve ultimately made.

Graves: When I was researching fact-checking and wrote a couple of fact-checks working at Politifact, the thing that struck me most is how you can’t write around things you don’t know, in the way we all sometimes would do as journalists. You really have to address the question and commit, to the extent you can, or explain that the evidence isn’t available.

Wagner: I can think of an example where a student wrote a story, and I thought this would be one of the first things we published — and we still haven’t published it. A guy running for state legislative office who had worked at a school district said he was responsible for $500,000 worth of grants to the school district. When the student followed up with the guy, he had sent some evidence of grants he’d written, and it added up to in the neighborhood of $250,000. First we thought, is he just overstating things? And then following up we found out he had also co-written other grants. When we went to the agencies that had awarded those grants, they gave us evidence that he had in fact earned some of them. But then their records only went back so far, and he said he had done some of them before the time the records went back to.

If you were writing a regular story that wasn’t a fact-check, you’d say “We found evidence for $368,000 of the claim” and move on, but that still leaves $132,000 unaccounted for — so we really feel like we have to try to track that down. … And that’s a testament to working with WCIJ. Me, teaching two courses and trying to get my research out and having graduate students on the job market, I might just say, gosh it’d be nice if we could just get this thing done. Dee (Hall, of WCIJ) is more relentless. She’s, “No no, you have six more calls you can make before we can get this thing out.” I think that’s made it a better process. And a more difficult process. But if it was easy everyone would do it.

I’m having flashbacks.

Wagner: I bet. I should have said a trigger warning.

So are people contesting your claims yet?

Wagner: Not yet.

You’ve got time. … Anything else you want to say about the project?

Graves: Coming up with this class as part of the Baldwin (grantmaking) process encouraged us to think in a really explicit way about how we could try to make a contribution to democratic discourse in Wisconsin specifically. And also how we might break away from what other fact-checkers have done, or try to add something to the established practices.

One thing Mike mentioned was a little bit whimsical, but a lot of fact-checkers who use scales have a Half True setting that ends up being where a lot of claims end up. We were both really interested in experimenting with removing that option and forcing our fact-checkers to commit. Or decide that something is unobservable, which is what we call claims when there’s not enough evidence to figure them out. That’s a small innovation.

But much more importantly, built into the project is the idea that students will write about how members of the public perceive the fact-checks — how they make them meaningful in terms of their own understanding of politics. So they’re going to be writing response stories. And that’s something I think is really innovative and important and might give us insights that feed back into how we design fact-checks and how we think about this whole endeavor.

Michael Wagner (@prowag) is associate professor and Louis A. Maier Faculty Development Fellow in the School of Journalism and Mass Communication at the University of Wisconsin-Madison. His most recent book is Political Behavior of the America Electorate, he blogs at

Lucas Graves (@gravesmatter) is an assistant professor in the same department whose book, Deciding What’s True: The Rise of Political Fact-Checking in American Journalism, is just out.

Kate Golden is the Walkley Foundation’s multimedia manager, and she spent six years as the multimedia director and a reporter at the Wisconsin Center for Investigative Journalism.