An insider’s view of the state of open data in Australia

For our Nerd Box series, digital archivist Cassie Findlay gives us a big-picture view on open data issues.

Anyone following the Walkley Magazine’s excellent Nerd Box series will have read about some of the ways that journalists are bringing data to life to reveal more about what governments and corporations are up to. People like Simon Elvery at the ABC’s Interactive Digital Storytelling Unit and the Guardian’s Nick Evershed are producing work that sheds new light on budget spending and revenues, personal privacy considerations in the collection of metadata by telcos and more.

So how do journalists get at information like this? And how well are these mechanisms working? What follows are some personal observations based on a long stint working within government, mainly as a recordkeeping adviser and digital archivist, as well as more recently as a consultant, helping government agencies with their open data initiatives. I’ll try to outline how well the push (from within government) and pull (from the public) are working, and list a few tips for getting to the data you want.

The policy context

Governments across Australia have pretty much universally adopted good, strong policies on the release of data. Rather than being legislated, these are usually statements signed off and circulated by the Premier or Chief Minister for adoption at the agency level. Most jurisdictions have open data portals like the Commonwealth’s data.gov.au or data.nsw.gov.au. Australia has (finally) signed up to the Open Government Partnership and is working on its inaugural National Action Plan. People within government who are leading open data policy regularly hold events such as workshops and hack days to foster an interest amongst agencies in releasing data, and to connect potential users of data with custodians. The Commonwealth has had some great initiatives, such as the National Map, the Open Data Toolkit and the release of usable Federal Budget data in a timely way. State governments like NSW have their own strengths, including initiatives like the NSW Parliament’s Hansard API, providing daily transcripts in XML format.

Open data advocacy organisations, groups and enthusiasts

In parallel with the work being done within governments, there is a small but vibrant community of people and organisations who are dedicated to opening up and making use of government information to make our lives better, simpler and fairer. Code for Australia funds fellowships for technologists and others to work with government to make data more available or to use it to solve problems. The Open Australia Foundation builds tools that simplify the discovery and use of public data including Planning Alerts, Right to Know and They Vote for You. Individuals like Rosie Williams (@InfoAus) or Steve Bennett (@stevage1) have mounted an impressive array of transparency and open data projects and outreach activities. Journalists with an interest in accessing and using data get together via communities like Hacks/Hackers, an international group with regular meetups in Sydney, or the virtual Slack channel community Stories with Data.

Does that mean government agencies are releasing lots of usable data, routinely?

Well … There is a lot of data publishing happening in some sections of government. In others, not so much. Transport agencies have some great data-sharing initiatives, as do environmental and scientific agencies. Naturally, agencies whose core business is statistics, such as the ABS or sector-specific statistical organisations like NSW’s BOCSAR or CESE have mature open data programs. However, there are some parts of government that remain under-represented. There are a number of reasons for this. Social services, justice and health agencies find navigating the personal privacy implications of releasing their data challenging. While there is clear policy direction, without binding legislation there are few serious consequences for non-compliance. In general, there are problems of skills, low to nonexistent resourcing and a lack of strong senior-level direction for open data initiatives. Some agency people I have spoken with simply don’t see the point of data release, or worry about releasing “imperfect” data to the world. Often, there is confusion about whether certain officers have the authority to publish data or otherwise open up government information.

Senior bureaucrats and ministers have been, by and large, more interested in pushing open data where there are commercial opportunities in its exploitation. Where the data release instead promises benefits of a less tangible nature — such as measuring the impact of policies on citizens’ lives or highlighting potential waste of taxpayers’ dollars, the interest is diminished as questions of release become politicised. Of course, that’s why we have mechanisms like Freedom of Information and whistleblower protection laws. These serve as buffers against impunity by providing the means for the public, including the media, to request information on how the government is doing the business of governing, to unearth corruption, government waste and dodgy policy, or for insiders to make such information available. These safeguards are by no means secure, however. Threats of defunding of the responsible agencies or legislative reform to water down the public’s rights come around regularly — as we saw with the Commonwealth OAIC being in limbo until quite recently.

So there are some hits and some misses. But there are a few things that can be done  to enable more data to be available and used — both on the government side and amongst key data user groups such as journalists.

If an agency isn’t publishing certain data, how can you request it?

Ask them: It can’t hurt, and you might get lucky. Plus, depending on the jurisdiction that the data you are after is kept under, proactive release provisions may exist in the applicable FOI legislation. This is the case, for example, in New South Wales, under the Government Information (Public Access) Act 2009. This Act, known as GIPA, makes the default position for government information “open”, unless there is a valid reason to withhold it, and encourages proactive release by agencies without initiating formal processes. An agency FOI officer or data custodian can simply provide the data or documents without forms or charges. There is nothing in the Act that prevents them from providing information in data form, such as an output from a database (however, see my note below regarding the making of a “new record”).

Use FOI: Despite the archaic use of “document” in the Commonwealth FOI legislation, the accompanying definition notes that this can mean “electronically stored information”. At the state and territory government level, the legislation differs from one jurisdiction to the next, but generally no restrictions on format apply. However, responses from agencies can advise that producing reports from databases constitutes making new records, and so cannot be requested under the terms of the legislation. This raises, for me, one of the main problems with the current legislation. It was written at a time when records kept by government were static end-products; the document, the letter, the set of minutes. It did not envisage a world in which the collection, management and use of data could inform policy. Databases are living repositories of records that are dynamically generated as required. The excuse that a particular rendition has not yet been generated from a dataset collected by government should not, in my opinion, mean that it may not be, if a citizen wants to see it. This is just one of many ways the legislation requires reform — some ideas on this, from me and others, are further explained in a summary of a workshop I convened during the Hawke review of FOI in 2012.

If making an FOI application, depending on the nature of your investigation, you might consider using Right to Know, so your results will be available to all (a great service, especially as some Commonwealth agencies are not publishing released information or are only putting them online for a  limited period).

Appealing decisions may involve a few steps, depending on the legislation. In the Commonwealth, appeals are considered by the Information Commissioner. You can search and view details of past decisions under this process using AUSTLII.

Use privacy legislation: You have rights to access information about yourself held by governments and some private sector organisations, under the Federal law. These were the provisions under which journalists like Ben Grubb and Will Ockenden recovered data on themselves held by Telstra showing how we are tracked and recorded when using our mobile phones.

Get a taste of what might be coming up for release: Some jurisdictions publish schedules of data to be released as a priority. These can be aspirational and may not always be met, but they can provide useful information and possibly help when approaching agencies directly. For example: the NSW Government’s Open Data Rolling Release Schedule.

Use a “Request a dataset” button: Most government open data portals have such a thing, although the response can be slow and results patchy. Hopefully as more agencies bed down and systematise internal processes for making data release decisions this will improve.

Currently many governments in Australia are spending a lot of money and effort on data sharing and analytics within their own walls (such as with the NSW Data Analytics Centre) or on engaging with the private sector to exploit government data. These things have their place of course, but we need to balance these efforts with more than token gestures around data release for transparency and accountability. Perhaps if we have a greater demonstrable demand for data, especially from journalists, some of the barriers I’ve talked about here might start to break down.

Cassie Findlay is a digital archivist and open government enthusiast who is currently working as a consultant with Recordkeeping Innovation. She lives in San Francisco. Twitter: @CassPF