Special Report: Open Public Data

Pundits Debate Open Public Data

Discussion and debate over the opportunities and challenges of open public debate have moved from the margins of political discourse in Hamilton toward the mainstream.

By Ryan McGreal
Published March 18, 2011

In a sign that the concept of open public data has moved from the margins toward the mainstream, local media pundits in Hamilton have started debating it on cable TV, social media and blogs.

What is Open Public Data?

Open public data refers to government data that has been defined as public information and made available in open, standardized formats that are findable, accessible and machine-readable so it can be processed by third-party software applications to create resources.

The purpose is to enable citizens to find, analyze, and do useful things with public information. Software written once to provide a service can then do its job over and over again, improving service delivery while saving the city resources.

Open public data is about making public data more accessible, not making private or confidential data public. Data that has privacy implications deserves to remain private.

Worldwide, a number of governments at the federal, provincial/state and municipal level have already made formal commitments to open public data and are at various stages of creating and expanding public data repositories.

The Government of Canada just announced yesterday that it has launched a federal open data pilot project with a catalogue of 782 data sets. According to the site, the goal of the project is "to improve the ability of the public to find, download and use Government of Canada data".

An early review of the federal open data project by David Eaves suggests that the project has real potential for growth and development but that the licence under which the data has been shared is far more restrictive than that of other governments.

At the municipal level, Hamilton City Manager Chris Murray recently sent a memo to council in which he called open data "part of a transformation agenda" that will require "strong leadership and a culture change" in how the city manages data and deploys resources.

According to Murray, open public data has the potential to increase transparency, improve service delivery and engage the public more actively.

Murray added, "Information is traditionally seen as power, and opening it up means sending a message within the City administration that this is the public's information and not ours to control."

A citizens group called Open Hamilton has recently formed and is advocating for a municipal commitment to open public data. They have prepared a draft motion they hope to persuade Council to adopt.

Debate on Cable 14

In a March 8, 2011 discussion item on Cable 14's On The Record, host Larry Di Ianni raised some concerns about open public data (see video below, starting at 19:02), complaining that advocates want open data "for reasons not yet invented" and "cannot articulate a coherent reason for supporting the advocacy."

Di Ianni argued that open data can leave citizens "more confused" than informed, that "opening up data is expensive" and that some advocates "want to use the data as weapons to hold the system accountable".

He concluded, "a rationale for opening up the data is a necessary precursor to the data overload."

Rebuttals

Opinionator Laura Babcock retorted, "That's total nonsense and completely insulting, Larry. I can't believe your bias has seeped in so far to your commentary," to laughter by fellow panelists Loren Lieberman and Danya Scime.

She called open data "a no-brainer" and then articulated a terse rationale: "This is information that informs us as a citizenry. I'm not going to get 'confused' by it. I can think of many applications: when students are doing research, when you're doing civic planning, when you're part of citizens' groups, when you just want to be informed about your democracy."

She concluded, "Sorry, Larry, I don't know who's been articulating it incorrectly to you, but your emphasis on the word "activist" suggests your bias from the beginning."

Lieberman expressed concern that "some subversive media or otherwise might be spooning us incorrect findings based on a skewed look at how things are."

Scime argued, "We want the accountability, we want the transparency" and expressed hope that people use the data to get educated and informed.

More Debate

The debate continued after the show ended. Babcock tweeted that she "kicked Larry Di Ianni's proverbial butt", triggering some back-and-forth on the matter. This week, she posted an entry on Powergroup's "As Seen On TV" blog, asking: "should Hamilton have open source data? Why or why not?"

On March 14, Cal DiFalco wrote a blog entry on OpenFile Hamilton in which he expressed support for open data but argued that Di Ianni and Babcock both raised valid points.

DiFalco warned, "Having open access to data and being able to interpret it correctly are two different things." He raised the prospect of "faulty analysis" that could "reach a large audience" and "do more harm than good".

He added: "in the hands of someone with more sinister intentions, data can be 'spun,' shaped or used selectively in such a way that it can, in fact, be used maliciously, or even as a political weapon".

Follow-Up

In an email response to RTH, Di Ianni clarified, "I never expressed any concern with open data. In fact I said it was inevitable."

He repeated his three specific concerns: that advocates have not done a good enough job of articulating the case for open data; that some advocates want to use open data to go on the political attack; and that the data can be confusing to the public.

He added, "Accountability is very good, but political witch-hunting is not."

Laura Babcock acknowledged in an email to RTH that Di Ianni's argument "has some merit as it is true that all of the future uses of open data are unknown. However, that concern should not prevent open data (as I articulated on the show)."

She added that people have always tried to manipulate information for narrow political or partisan purposes, but that "politicians should prepare for a new level of transparency" in response.

She concluded, "Access to more information in a democracy can result in a more engaged, better informed electorate".

Understanding Open Public Data

As an early proponent of open public data, I take Di Ianni's objections seriously and am working on a more formal argument in defence of open public data that addresses his concerns.

I will be making a public presentation on understanding open public data this coming Wednesday evening at a talk organized by the Canadian Information Procession Society - Golden Horseshoe. If you are interested in attending, please RSVP to the organizers as space is limited.

Cost

Over the past few years, I've heard from a number of City employees who are frustrated that they waste enormous amounts of time trying to get their hands on data from other parts of the city. There's very little sharing between departments, and each silo operates as a gatekeeper to the data it produces.

An open public data policy would save the city money, in at least three ways:

  1. Employees will spend less time and money converting data from accessible formats to inaccessible formats.

  2. Employees will spend less time and money trying to obtain data from other departments.

  3. Employees will spend less time and money converting data from inaccessible formats back into accessible formats.

If the data is released in an open, accessible format from the beginning, we enjoy a triple boost in employee productivity as well as a boost in the capacity for citizens to participate meaningfully in the process of governance.

To be sure, some parts of the city are trying to move in the right direction. The Public Works Department has gotten a lot better at internal team-building and cross-departmental dialogue, and City Manager Chris Murray is trying to change the culture from the top down (hence his support for open public data). But it's still very much an uphill struggle.

In some cases there may be some upfront costs associated with making public data available in usable formats, but this should not be a barrier to embracing a commitment to open data. The development and release of public data in open formats can be a progressive, incremental undertaking - it doesn't all have to happen at once.

As always, there will be opportunities to pluck low-lying fruit by starting with data that is easiest to convert.

Remember: time invested once to make data accessible is amortized by delivering savings and economies over and over again in every subsequent use.

Attacks

The relative absence of such data doesn't seem to have stopped the attack-minded. I'd argue that such attacks are easier in the absence of data, because insinuation and accusation flow in to fill the void.

Good hard data should actually reduce the effectiveness of such attacks by countering emotion with information. Public discourse leavened by solid data can waste less time turning around uninformed opinions and reasoning by fear and ignorance.

I come from a background in open source computer programming, which shares many of the same principles as open data. It operates through open public discourse informed by reproducible results - essentially, the kind of peer review that drives science forward and moves a community of learners away from theories and models that are provably wrong.

Uses

Di Iannni acknowledged in his email to me that his concern about the potential uses of open public data was partially in response to a statement I made in an interview with the Spectator's Bill Dunphy:

A programmer by occupation, McGreal says he's not sure what people would do with the kinds of data the city and its institutions could release (data on road conditions, transit location, trash collection schedules, public health, crime figures), "but I see untapped potential in it all. I'm a believer in the value of serendipity. I won’t know how to use the data until I can see it."

I realize that's a tough sell, so here is a concrete example of the use of open public data.

In 2007, I read a Globe and Mail article about development in Hamilton that made some suspicious-sounding claims about increased building permits after the city lowered business property taxes.

I tried to get my hands on the building permit data - building type, property valuation, location - and was told I couldn't have it. The closest I could get was a page of links to a stack of summary reports - in read-only PDF - that provide permit totals by facility type.

I had to spend several hours opening each PDF (one for each month) and manually copying the summary figures into a database, which I then spent several more hours converting back into a format that was accessible and sharable.

With this partial information, I was able to determine that most of the building permit value was residential and not industrial or commercial, suggesting that lower business property taxes were not responsible for the boom in permits.

Because I didn't want my efforts to be wasted on a one-shot report, I later released the data in an open, accessible format and have since maintained it as monthly building permit reports (still PDF) are released.

At the end of the year last year, we were able to determine easily that the city had exceeded $1 billion in overall building permits for 2010. We were also able to determine that more than half of the total is still residential, an important context that casts the achievement in more accurate light.

If we had more detailed, low-level data on building permits, we would be able to provide an even better understanding of what the data mean from an economic and policy perspective.

Serendipity Revisited

But the reason I keep coming back to "serendipity" is that open public data is ultimately as much about creating new resources as it is about improving existing ones.

Open public data opens up the imagination and unlocks creativity. When advocates say we don't know all the potential uses, that is not a strike against open public data. It is an acknowledgment that open public data make transformative innovations possible.

Here is a closing case in point: the Japan Quake Map, which combines quake data feeds from the United States Geological Survey with Google Maps to create a time-lapse animation showing the location, magnitude and reach of the earthquakes that struck Japan starting last Friday, March 11.

Such creative visualizations can take large, unwieldy and perhaps "confusing" data sets and make them accessible and meaningful to everyone.

Ryan McGreal, the editor of Raise the Hammer, lives in Hamilton with his family and works as a programmer, writer and consultant. Ryan volunteers with Hamilton Light Rail, a citizen group dedicated to bringing light rail transit to Hamilton. Several of his articles have been published in the Hamilton Spectator. He also maintains a personal website and has been known to post passing thoughts on Twitter @RyanMcGreal.

19 Comments

View Comments: Nested | Flat

Read Comments

[ - ]

By Jeff Reid (registered) - website | Posted March 18, 2011 at 11:33:37

An open public data policy would save the city money, in at least three ways...

I love this part; thanks for raising this to a discussion of how to save the city money by increasing productivity.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By MountainCreature (registered) | Posted March 18, 2011 at 11:58:05

Di Ianni argued that open data can leave citizens "more confused" than informed, that "opening up data is expensive" and that some advocates "want to use the data as weapons to hold the system accountable".

Dear Larry Di Ianni,

You're damned right I want to use open data to hold the system accountable. Care to tell us what the problem with that is?

There shouldn't be a problem so long as Open Data doesn't yield findings that prompt responsible citizens and press from demanding accountability. If you have nothing to hide, then there's no reason to break a sweat, is there Di Ianni?

You're right about one thing though. Open Data is inevitable, and transparency is here to stay. Establishment guys need to get used to it, or get out.

Reply | Permalink | Context

You must be logged in to vote on this comment.

By jason (registered) | Posted March 18, 2011 at 15:12:00 in reply to Comment 61162

I vote for 'get out'.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By Why Cal Why? (anonymous) | Posted March 18, 2011 at 13:22:15

Cal DiFalco's pounding the feardrums about open data too? Say it ain't so!

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By JasonAAllen (registered) - website | Posted March 18, 2011 at 14:39:11

Di Ianni's comments demonstrate a clear misunderstanding of how the 2.0 world works. Back when information was controlled by only a small group of people (an information oligarchy), then those people/groups were pretty much free to say whatever they wanted, and spin it to whatever purpose they desired.

Now, as we move towards open data (information democracy) those with access to the information will self regulate - or more precisely regulate one another. And the 'regulation' process, as anyone who has posted inaccurate information here on RTH can attest (guilty as charged), is swift and unforgiving.

When it comes down to the concern about nameless, nefarious groups spinning the data for their own purposes - that is a MUCH greater risk when only a few people have access to that info. With Open Data, @MountainCreature has hit the nail right on the head - Open Data will hold not only the system accountable, but all of the users/presenters of that Data as well.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By Larry Di Ianni (anonymous) | Posted March 18, 2011 at 18:07:43

Thanks Ryan for clarifying my comments and for contacting me about this piece. I am glad you will take my comments seriously. I can email my entire, albeit brief, editorial to anyone who wishes. But not to belabour the point, I wasn't advocating for controlling information at all; nor was I denouncing "Open Data". I was merely making two points: 1. Advocates for Open Data must be clear in their rationale for the advocacy and 2. Civility in dealing with governments and each other is a must.

That's it really.

By the way, tune in to "For the Record" especially Tuesdays when the Opinionators and I weigh in on current issues. It is often informative and always entertaining.

You too Jason and JasonAAllen. I would welcome your feedback too.

Reply | Permalink | Context

You must be logged in to vote on this comment.

By MountainCreature (registered) | Posted March 21, 2011 at 12:03:13 in reply to Comment 61185

Thanks for the response, Larry. I understand you were not explicitly denouncing Open Data. Please forgive advocates such as myself for not being specific enough in our rationale for supporting the movement. I think that lack of specificity stems from an overall feeling that the benefits of Open Data are plain to see. All one has to do is look at the effects of new-found transparency going on all around the world in the last few months. The notion that citizens can't handle Open Data, or would become confused, is indeed insulting as Laura Babcock pointed out in her response (which was bang-on, by the way). It suggests a ignorance about how web 2.0 really works in conjunction with journalism, but your presence in this forum shows me that you're willing to learn. To help you along, I'll simply clarify my rationale for supporting Open Data:

I support Open Data because I would like to see a city where citizens and media can use the data in a civil way to hold the system and elected officials accountable in regards to lies, fraud, bad policy.

Reply | Permalink | Context

You must be logged in to vote on this comment.

By jason (registered) | Posted March 19, 2011 at 09:37:35 in reply to Comment 61185

great to hear from you Larry. Hope all is well. There are certainly many uses for open data. Yesterday I was on the HSR. Man, what a brutal experience in terms of knowing when buses are coming etc.... open data would allow local techies to develop mobile apps and bring our transit system out of the 70's.
Other data has been wide open for the power brokers in town, but not for normal citizens. An informed populace is an engaged populace. And as we know, Hamilton needs much more civic engagement if we are to move this city forward. Continued good luck with the show! Cheers

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By Tummler (anonymous) | Posted March 19, 2011 at 07:13:40

Clearly the former Mayor has a bias with respect to the open public data issue. As an example a policy of this nature might not preclude the hiring of a campaign manager to a well paid municipal contract but it would certainly make the hiring a public matter--as it should be
As noted by all commentators the movement has considerable momentum and likely will be common practice in most areas rather quickly
In the case of the City of Hamilton though there will no doubt be a rear guard action to protect the status quo
Quite simply this is the nature of the city

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By rednic (registered) | Posted March 19, 2011 at 10:04:57

over the past few years, I've heard from a number of City employees who are frustrated that they waste enormous amounts of time trying to get their hands on data from other parts of the city. There's very little sharing between departments, and each silo operates as a gatekeeper to the data it produces.

well therein lies the rub ... Bureaucrats protecting their 'turf' and consequently (in their minds) their job. What virtually every government employee forgets ... Is that the public pays their saleries and the information they preside over is owned by the public as well ...

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By Brandon (registered) | Posted March 19, 2011 at 10:25:39

Imagine how much better the stadium debate could have gone had we been able to see all the costs/benefits/issues associated with each site and all involved parties.

If WH was truly the disaster that some claimed it was then that would have been apparent to all. If it wasn't...

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By Undustrial (registered) - website | Posted March 19, 2011 at 12:44:16

It's pretty clear why Larry feels this way. There is already a fair bit of accessible data provided by the city, in one form or another. One example would be campaign donations. When citizens got their hands on his, he ended up convicted of election fraud, despite an overwhelming reluctance to act on his clear-cut violations by council and courts.

That's pretty much the best argument in favour of "open data" I've seen so far.

Reply | Permalink | Context

You must be logged in to vote on this comment.

By Ryan (registered) - website | Posted March 21, 2011 at 06:39:46 in reply to Comment 61197

he ended up convicted of election fraud

I need to point out that this is incorrect. Di Ianni pleaded guilty of violating campaign finance rules, not fraud.

Election fraud is the illegal and criminal interference with the collection and counting of votes to favour a candidate.

Di Ianni's conviction, by contrast, was for the civil (non-criminal) charge of accepting campaign donations that exceeded allowable limits.

Comment edited by administrator Ryan on 2011-03-21 07:50:16

Reply | Permalink | Context

You must be logged in to vote on this comment.

By Undustrial (registered) - website | Posted March 21, 2011 at 10:11:28 in reply to Comment 61260

Fair enough, my bad. I should have checked my specific charges. Nonetheless, he was publicly exposed for elections act, and was in fact convicted - the first ever under the new law, prompting other municipalities to start laying their own.

Whatever you want to call the charge (we had been discussing fraud charges at the time against those who misrepresented the name/number of their corporations), it's pretty clear that they didn't come from people without accountants, and did come from companies and individuals with lots to gain from his victory.

http://www.raisethehammer.org/blog/275/

Furthermore, it is pretty clear that there was absolutely no interest among councilors in investigating or pressing charges. Citizens, much like those who first exposed the donations, had to press charges civilly because the government refused to do so. Whatever councilors happened to think of these donations, though, the public obviously felt differently.

If these documents had never been public, there's virtually no chance that any of this would ever have been discovered, and a decent likelihood that he would not have lost two elections since. This is exactly why public data is important.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By Clyde_Cope (registered) | Posted March 19, 2011 at 13:07:43

Thanks Ryan for this very thought-provoking article. We certainly need to hold our politicians' collective feet to the fire so they will not backslide on this issue. What amazes me is the resistance from our politicians on this issue. When this is fully implemented it will be they, the politicians that have the most to gain - their job will be made easier and more effective - a winner on all accounts.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By Limmet (anonymous) | Posted March 19, 2011 at 16:13:46

Comments with a score below -5 are hidden by default.

You can change or disable this comment score threshold by registering an RTH user account.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By WRCU2 (registered) | Posted March 19, 2011 at 22:29:30

An open public data policy would save the city money, in at least three ways...

And IT will only cost me and the missus ten bucks each plus bus fare and drinks to hear "pundits debate open (to the) public data" policy. Wow (me winks), I'm flattered by the pundit propriety!

I guess what's good for the goose ain't quite good enough for us to gander, unless we got a few extra bucks to squander. I love to hear Ryan talk and I'm also an IT guy but I thought we wished to freely open public data and here CIPS insists we must buy. Why do I have this feeling of spit in my eye?

Good luck and nice try, I have my doubts about how well this will fly.

Reply | Permalink | Context

You must be logged in to vote on this comment.

By MattM (registered) | Posted March 21, 2011 at 15:22:21 in reply to Comment 61206

Kinda unrelated but do you purposely formulate your replies in rhyme structure? It makes it incredibly hard to understand you.

Comment edited by MattM on 2011-03-21 15:33:58

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By TB (registered) - website | Posted April 03, 2011 at 14:38:33

A quote from the LIberal Party platform:

"A key element of the Liberal plan for democratic renewal is Open Government. Liberals will adopt a new approach to information, issuing government-wide direction that the default position for all departments and agencies will be to release information to the public, both proactively and responsively, after privacy and other legal requirements are met. This initiative will see as many government datasets as possible available to the public online, free of charge in an open and searchable format. Show More

Furthermore, all Access to Information requests and responses will be posted online.

This new presumption of openness will also drive a new level of accountability for public finances. We will establish a searchable, online database for grants, contributions and contracts.

All levels of government, civil society, researchers, business and the public must have access to independent and reliable statistics and demographic information to make informed decisions and develop sound public policy to benefit all Canadians. Therefore, a Liberal government will restore the mandatory long form census."

Reply | Permalink | Context

You must be logged in to vote on this comment.
View Comments: Nested | Flat

Post a Comment

Comment Anonymously
Screen Name
What do you get if you multiply 5 and 1?
Leave This Field Blank
Comment

Events Calendar

Recent Articles

Article Archives

Blog Archives

Site Tools

Feeds