Raise the Hammer
Hamilton could be a real leader, generating local expertise in collaboration, improving city business, and engaging the public more effectively in developing innovative solutions to the city's challenges.
By Ryan McGreal
Published June 30, 2009
My June 5, 2009 essay Open Source City: Making Public Data a Platform for Participation generated a lot of enthusiasm in the comments and a number of interested emails - including, I'm happy to note, some interest from the City.
So far, most of the feedback I've gotten from the city concerns efforts to make the city's website easier to navigate (yes, city staff share the public's frustration with how difficult it is to find anything). But the open city API concept is about more than just making reports easier to find. It's about:
In short, it means turning the city's website into a partnership with residents to share and study public data, and to analyze and improve the city's operations on an ongoing basis.
Still, it's one thing to say the city should adopt an open collaboration model and another thing entirely to implement such a model. This essay seeks to expand on the previous one by addressing the following questions:
Outside of the open source community, not many people have experience with open data models. Certainly most companies and corporate organizations - even many companies that have otherwise entered the digital age - do not follow such a model, sticking instead to a rigid, hierarchical, top-down information flow that provides little opportunity for collaboration across departments and leads to a silo effect.
Open collaboration is different. It is more flat than nested, and decisions are based more on consensus across the community than on decree from the top down. That model applied to data analysis and tool development has several specific properties.
First, an open data model would live in a database, not an accretion of Excel files or proprietary 'black box' applications.
It's remarkable how much of the world's business analysis and even raw data still lives inside Excel files, with critical business rules squirreled inside templates, cell functions and macros where they're hard to find, let alone understand, and hard to share widely (hence exporting report summaries into PDFs).
Granted, Excel is a very useful human-accessible tool for summarizing data and producing reports, but it's not very machine-accessible and is a very poor data storage service. (It's also proprietary, requiring its users to buy expensive, closed-source software from Microsoft.)
Second, an open data model would be accessible across the internet via an open data application programming interface (API), which Wikipedia defines as "a set of routines, data structures, object classes and/or protocols provided by libraries and/or operating system services in order to support the building of applications."
In other words, anyone who has access to the internet could use the public API methods to access the city's data in a machine-readable form. That way, citizens can perform their own analyes and develop useful community tools that make use of city data.
In this vein, an open data model would also support an ecosystem of third-party applications built by the community. There are several ways to do this, but one way would be to host those third-party applications right inside the city's web API.
This would require some kind of distributed version control system so third-party developers can fix or enhance existing methods and even create their own tools. That entails a reliable mechanism for determining which community contributions to merge into the city's official tool repository, but there are already plenty of successful models for guided community development.
Another option for the city would be simply to provide the API and let developers host their own third-party tools. This would require some mechanism for the community to find and rate the various tools.
Just as important, an open data model would not be based on top-down, ivory tower deployment but on community-led, iterative development.
An ivory tower approach vests development in the hands of professionals whose skills are proprietary (and hence closed to third party scrutiny) and whose expertise lies in developing a product that is not released publicly until it is completed, after which the public opportunities for feedback are severely limited.
An iterative approach, by contrast, starts with something small and simple (that works) and improves it in discrete steps, in public, by a community collaborating via some kind of distributed application control system.
The expertise in an iterative approach lies not in developing the product itself, which is best done via community engagement, but in fostering the collaboration that makes community development possible. Under an iterative approach, many contributors put forward enhancements and fixes, and the project managers decide, based on community feedback, which additions to merge into the core product.
If I were a city employee, the concept of an open source data API would make me very nervous! It represents a pretty dramatic break with the traditional structure and culture of most hierarchical organizations (both corporate and governmental).
It means giving up some control over city data and acknowledging that expertise is distributed widely across the community. Given that city staff currently outsource much of their policy development to third-party consultants, this aspect of the shift may be easier to accept - it essentially entails treating the public as a third-party consultant.
But it also means exposing the city to closer public oversight, including potentially identifying wasteful processes that currently pass unchallenged because either a) no one has time to notice them or b) the people who know about them don't have the power to fix them.
It means pushing the city's internal operations into open data management and workflow management systems that are better tailored to sharing and collaboration than the prevailing model of Excel templates and proprietary black-box systems. (An open data model lives in a network-accessible database, not in an Excel file and certainly not in a PDF report.)
In short, it's a scary transformation - particularly to city staff, who may feel that either their job security or professional reputations would be at risk in such an environment.
For this reason, I propose that such a change should follow these guidelines:
First, the open API should make it easier for city staff to do their jobs, freeing them from the kind of manual busywork that no one likes - repetitive cutting and pasting, trying to chase down numbers form other departments, etc. - to do analysis that adds real value. An open, collaborative data system should eliminate a lot of this so people are freed up for more value-add contributions.
No one's job should be threatened with redundancy. Instead, it should allow staffers to do the kind of analysis that machines can't replicate. The goal should be redeploying city resources where they can work more effectively at solving problems.
It's not like there's a shortage of important work to be done; the first goal of an open source data API should be to raise the productivity of city staff so the city can solve more problems in a timely fashion.
The transformation to an open data model should be iterative and incremental, not sweeping. As John Gall famously stated, A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work..
Perhaps the best way to implement this is case-by-case, as existing processes come up for review and renewal or new projects start up.
The prevailing corporate (and public) culture would have to change from one that punishes errors (and hence drives them underground) to one that celebrates openness and sees problems not as faults to be punished but as important and valuable opportunities to improve the end product.
Under today's system, no one wants to look bad so people spend too much time data hoarding and CYA and not enough time finding and fixing problems. As a result, yet more time is wasted trying to obtain and manipulate information to produce reports that are, in turn, also hard to find and manipulate.
I sincerely believe that this is the right model at the right time. Hamilton could be a real leader, generating local expertise in city/public collaboration, realizing efficiencies and improving city business, and engaging the public more effectively in developing innovative solutions to the city's challenges.
The most successful organizations today are those that have embraced the concept of openness: of deriving value from a collaboration platform, of sharing information and resources, of aggregating the contributions of a wide array of participants, and of committing to the kind of continuous improvement that comes from respectful peer review.
By highwater (registered) | Posted June 30, 2009 at 22:15:30
If the US can do it, how hard can it be for Hamilton.
www.techpresident.com/blog-entry/pdf-09-kundra-unveils-it-spending-dashboard
By Richard Stallings (anonymous) | Posted July 01, 2009 at 00:45:53
i think this is a wonderful idea
By BitHead (anonymous) | Posted July 01, 2009 at 14:34:58
Excellent article and idea. I too was creating yet more little puddles of data with cut and paste, trying to compare some city numbers. Here are three quick thoughts:
My experience is that, even before the data or the city, there needs to be a framework or taxonomy - a "place" to put the data. Like the system that allows me to find a book in any library using Dewey decimal system, or connect to millions of different email servers with the same command, there needs to be a *common* framework into which the cities can plunk their public data and then have it retrieved. Perhaps some of the taxonomy is already in place or can be borrowed/shared with existing projects?
Data - Often, the data (at least for testing purposes) is populated a few bits at a time by people like us that need some permit or expense data. In the case of the HSR, nothing is stopping us from building a mash-up (website) to show where the buses "should be" (the budget). If the real-time data becomes available (once this catches on), we can add it to the map.
so:
1. Can we form a working group to begin on this?
2. Have you seen the "Ontario Municipal CAO's Benchmarking Initiative". Perhaps they would support this, as their job is to collect information from about 15 cities in Ontario and compare performance (water, waste, rec, police, ambulance, etc). Interesting read to see where we rank. (www.ombi.ca/index.asp)
3. I also enjoyed the article "Municipal Budgeting" by Almos Tassonyi. (www.ctf.ca/pdf/ctjpdf/2002ctj1_tassonyi.pdf)
By faragol (registered) | Posted July 01, 2009 at 20:52:25
I am glad to hear that this is gaining momentum. I would be interested in participating in a working group if one is organized.
By BitHead (anonymous) | Posted July 05, 2009 at 14:23:28
...speaking of data - does anyone have a reference to online data for carbon levels (monoxide/dioxide) in the Hamilton atmosphere?
The MOE site has an interactive map leading to monitoring stations in the north end, but the data is only for more toxic pollutants. This is just another reason an open data model would be great - the MOE data is very pretty, but extremely difficult to harvest automatically with a computer (so I have to re-enter - grumble).
Thanks.
By Lurker (anonymous) | Posted July 20, 2009 at 14:30:41
An update from David Eaves about open data in the city of Vancouver.
www.eaves.ca/2009/07/16/open-data-at-the-city-of-vancouver-an-update-1672009/
Perhaps our councilors could use this City of Vancouver motion presented in May as the basis for a similar motion in Hamilton:
eaves.ca/2009/05/14/vancouver-enters-the-age-of-the-open-city/
Please enjoy responsibly.
L.
You must be logged in to comment.
There are no upcoming events right now.
Why not post one?