Open City

Starting Hamilton's Open Public Data Sets

By Joey Coleman
Published March 24, 2011

Open Hamilton is looking for information on the City of Hamilton website which can be converted into open public data.

The first two data sets are complete, thanks to the hard work of OH member Ryan McGreal and posted to the data repository on Raise The Hammer.

The first data set is a table of the locations of school crossing guards in the city. The City of Hamilton lists the locations in an HTML table on their website.

McGreal took this table and quickly converted the intersections into their geographic coordinates (latitude and longitude) using a Python script. He made a data file of the information and plotted the locations using the Google Maps API.

The data can be downloaded in JSON or CSV format, and is now also available on the Open Government Data Initative repository. You can use OGDI to further analyze the information in Bar Graph View.

Using a similar technique, McGreal has also mapped the locations of Hamilton's recreation centres using another HTML table on the City website.

The map is available on Raise The Hammer and can be downloaded in JSON or CSV format.

These data sets represent the first layers in Open Hamilton's comprehensive City of Hamilton mashup map project: Hamilton Guide. The mashup map is inspired by VanGuide and will use all open data that includes geographical location data in the Hamilton area.

VanGuide allows users to select and layer data sets available in the Vancouver region. This information includes drinking fountain locations, bus stops, and parking lot information for a private parking company which publishes rates and locations in an open data format.

These are just some of the possibilities with open public data.

These two data sets were quick to create and not costly to produce. Opponents of open public data claim that making public data accessible this way will be costly and cause an "information overload." However, seeing crossing guard data displayed on a map is easier to understand that a large table listing the locations.

Open public data is the right thing for government to do and produces results.

this blog entry was first posted on the Open Hamilton website

Joey Coleman covers Hamilton Civic Affairs.

Read more of his work at The Public Record, or follow him on Twitter @JoeyColeman.

30 Comments

View Comments: Nested | Flat

Read Comments

[ - ]

By Pxtl (registered) - website | Posted March 24, 2011 at 11:44:06

Well, it's a start.

Permalink | Context

By Ryan (registered) - website | Posted March 24, 2011 at 11:50:28 in reply to Comment 61453

You're a programmer, right? Many hands make light work. :)

Permalink | Context

By Pxtl (registered) - website | Posted March 24, 2011 at 11:53:54 in reply to Comment 61454

Yeah, now that you've posted your Python script I'm planning to re-work it for this:

http://www.cogeco.ca/web/on/en/residenti...

when I get the time... but I should probably just bug Cogeco and see if they have a better format of the data posted online.

Permalink | Context

[ - ]

By ThumbsUp (anonymous) | Posted March 24, 2011 at 11:57:18

Glad this is starting to take shape. Nice Work, Ryan!

One minor glitch: There is at least 1 misplaced crossing guard on Beckett Drive (Queen Street Hill). By the looks of things there are several pins at that location. The 'top' one is the Woodward Ave./School data point, which also happens to be the last one listed in the CSV file.

Permalink | Context

By Ryan (registered) - website | Posted March 24, 2011 at 12:33:40 in reply to Comment 61456

Thanks - good catch. There were half a dozen or so locations that all ended up with the same geocoordinates. They should be fixed now, though I wasn't able to determine one of the locations: Sir Isaac Brock School and Pedestrian Underpass. If anyone is familiar with the area and knows the exact location of this, please let me know.

I also found one crossing guard spot that was duplicated in the city's data (with slightly different spelling of the address in each), and removed the dupe.

+1 for crowdsourcing!

Permalink | Context

[ - ]

By MattM (registered) | Posted March 24, 2011 at 11:59:55

Good stuff guys, I look forward to the results of future work. I'd gladly pay to have people doing this stuff at city hall instead of paying for their lunches :)

Permalink | Context

[ - ]

By jason (registered) | Posted March 24, 2011 at 12:51:17

it's gonna be pure anarchy in this city now with this info getting out...

Permalink | Context

[ - ]

By Ryan (registered) - website | Posted March 24, 2011 at 13:02:16

By the way, the City Services page seems to be a good source of public data sets that are easy to convert to open formats. Here are a few I just found while poking around:

There's a table of Community Halls. A little URL hacking came up with a more general page of links to various community centres. I'm going to modify the Recreation Centres data to include all the types of facilities and distinguish among them.

With a more sophisticated web scraper, you could generate a data set of public library branches and locations. You'd need to follow the link for each branch to its own page to get the address and other contact info. If I were going to tackle this, I'd probably use Python and BeautifulSoup.

If someone wants to tackle the comparatively hairy job of extracting data from a PDF, there's the community health bus schedule linked from the Community Health bus page. Note: it's a three-month schedule ending in March so it might be better use of your time to wait until the April-June schedule is posted.

This coming Fall, the flu vaccination schedule will be worth converting. (Right now it contains only last year's schedule.)

Here's a list of federal, provincial and municipal service centres in Hamilton.

How about locations and times of prenatal classes?

Or waste management transfer stations?

Or city-owned properties for sale?

There's no shortage of public data. We just need to get it in a form that's actually usable. One last thing: for all of these data sets, especially PDFs, don't be afraid to contact the people responsible and ask if you can get them in a more usable format. Even Excel is better than PDF. The worst they can do is say "no".

Permalink | Context

[ - ]

By Captain Poultry (anonymous) | Posted March 24, 2011 at 14:44:44

I recently moved back to Hamilton and was searching for my closest community centre so that I could sign up for some programming. I was very frustrated at the process of having to do my own map search using the addresses listed on the city's webpage. Your map would have saved me considerable time and effort and was something I was expecting to find on the city's website in the first place. Well done and I look forward to your upcoming projects!

Permalink | Context

By z jones (registered) | Posted March 24, 2011 at 16:36:01 in reply to Comment 61473

Tell your councilor! They need to hear this.

Comment edited by z jones on 2011-03-24 16:36:11

Permalink | Context

[ - ]

By mrjanitor (registered) | Posted March 24, 2011 at 16:46:06

Is it possible to add the school locations requiring supervision for both boards on the crossing guard map?

Permalink | Context

[ - ]

By Ryan (registered) - website | Posted March 24, 2011 at 17:44:18

Update: I received an encouraging email from the city expressing interest in this initiative, and pointing out a couple of issues - one of the guards was in the wrong place and a few were missing. I've updated the data to correct the one guard and add the missing guards. One of the problems with manually converting data from read-only formats to accessible formats is that there are more opportunities for these kinds of translation errors.

Permalink | Context

[ - ]

By Nik Garkusha | OpenHalton (anonymous) | Posted March 24, 2011 at 21:37:31

Awesome work, Ryan -- love the maps!!

I was in turn inspired by your work, and the lack of easy-to-use online tools for batch geocoding. I also really wanted to mash that up with Pedestrian Accident data that
Randy uploaded here: https://groups.google.com/group/openhamilton/browse_thread/thread/a2d74b2fa50060a#ccfe3f93d8bda058

With my rusty JavaScript skillz, I merged a few different pieces of code to make this Geocoder (Beta): http://openhalton.ca/tools/

End result - map that mashes up the Street Crossing Guard info with Pedestrian Accident data:
http://maps.google.com/maps?q=http://openhalton.ca/openhamilton/xingaccidentmashup.kml

Hopefully, others can use the tool to liberate the data & easily geocode it into new CSV (for OGDI) and/or KML for maps.

Nik

Permalink | Context

[ - ]

By xx (anonymous) | Posted March 24, 2011 at 22:25:45

Braeheid Ave and Guy Brown School, Hamilton, Ontario is showing wrong location, should be in Waterdown

Permalink | Context

By Ryan (registered) - website | Posted March 24, 2011 at 23:31:17 in reply to Comment 61493

Updated. Thanks!

Permalink | Context

[ - ]

By xy (anonymous) | Posted March 24, 2011 at 22:35:40

Isaac Brock and Pedestrian underpass is
43.192147, -79.788875

Permalink | Context

By Ryan (registered) - website | Posted March 24, 2011 at 23:28:08 in reply to Comment 61494

Updated - thanks!

Permalink | Context

[ - ]

By Hopeful (registered) | Posted March 25, 2011 at 12:14:39

This rocks Ryan! Thanks for all your work on it. Now, would it be possible to tie comment options to the places you map? This would enable those of us without the skills to do the programming (like me sadly --- although I'd love to learn) to still contribute. For instance, I'd love to be able to add information like pool sizes, accurate schedules (which the City site lacks), etc., to the locations on the rec. centre map. Just an idea... P.S. As an aside, the LA Times have a great system for portraying local info geographically: http://projects.latimes.com/mapping-la/n...

Permalink | Context

By Nik Garkusha (anonymous) | Posted March 25, 2011 at 14:03:14 in reply to Comment 61516

@Hopeful:

There are few projects, including one that my team worked on for City of Vancouver that do exactly that: VanGuide.

We're in the process of taking this Open Source project and making it city-agnostic, so that you can add different "views" for any city, like Hamilton.

Joey mentioned that VanGuide mashes up the data, but more importantly it enables citizens to tag, rate & comment the data.

Here's an "ALPHA" of the project -- updated for Hamilton feeds from OH/RT.


http://cbe845bbd7444468b4de257973e68883.cloudapp.net/

change the "perspective" to Open Hamilton on the left and then feel free to rate / comment / etc. and add your own landmarks

Nik
OpenHalton.ca | Port25.ca

Permalink | Context

By Ryan (registered) - website | Posted March 25, 2011 at 12:19:01 in reply to Comment 61516

That's an excellent idea. To be honest, I scraped the crossing guard and rec centre data and plotted them on google maps mainly as a proof of concept, as I hadn't previously worked with the google maps programming interface. You're right that there are excellent opportunities to enrich it much further with more detailed information. The good news is that the City seems interested in these opportunities as well. I hope we see some real collaboration in the near future.

Permalink | Context

[ - ]

By mrgrande (registered) | Posted March 25, 2011 at 16:08:51

Hey Ryan, I scraped the city data for property sales (mostly just to play around with the Google Maps API). Do you want this data? Is there a format you'd like it in?

Currently, I've just got a Ruby script that generates a Javascript array that I copy/paste into an HTML page.

Permalink | Context

By Ryan (registered) - website | Posted March 28, 2011 at 20:17:58 in reply to Comment 61542

Do you want this data?

Yes, please!

Is there a format you'd like it in?

If it's tabular, I'd be happy with CSV or tab delimited or whatever.

I've just got a Ruby script that generates a Javascript array that I copy/paste into an HTML page.

You could always share it as a gist on github for other would-be data excavators. :)

Permalink | Context

[ - ]

By Taxman (registered) | Posted March 28, 2011 at 19:51:55

Fantastic work Ryan!

I agree the Rec centre list needs to be updated to include facilities other than those "Indoor Gyms and Clubs" on the city's page. It should include pools, arenas, and other types of facilities.

It would be great if by clicking on one of the rec centres, it brought up a list of current and upcoming programs - things like free skate times, swimming lessons, etc.

Although I'm getting the sense that most of that data is not in a publicly available database, and would have to be manually updated everytime something changes. :-(

Permalink | Context

[ - ]

By Ryan (registered) - website | Posted March 28, 2011 at 22:53:46

Update: I've added outdoor rinks, community halls, municipal golf courses, outdoor pools, and indoor pools to the recreation centres dataset and map. I still intend to add Playlots, Splash Pads and Wading Pools.

Permalink | Context

By JoeyColeman (registered) - website | Posted March 29, 2011 at 20:40:21 in reply to Comment 61709

Can't find the file, it doesn't seem to be where I thought it would be.

Sorry.

Permalink | Context

By JoeyColeman (registered) - website | Posted March 29, 2011 at 09:37:12 in reply to Comment 61709

Howard Elliott very quickly approved my downloading the dataset from their server. I'll forward the KML file to you this afternoon as soon as I can get FTP. Behind a firewall at present.

Permalink | Context

By JoeyColeman (registered) - website | Posted March 29, 2011 at 09:11:20 in reply to Comment 61709

Ryan,

There is a KML file I created of all the splash pads and wading pools for The Spectator last year.

I'll see if I can find it and send it to you.

  • Joey

Permalink | Context

[ - ]

By Momoko (anonymous) | Posted March 31, 2011 at 12:18:06

Raise the Hammer is great :) Just came across it. I live/work in Toronto, but grew up outside of Westdale.

To both Joey/Ryan — have you heard of data journalist Pete Warden releasing an extremely-easy-to-use data-scraping toolkit last week? check it out:

http://www.datasciencetoolkit.org/

Fun!

Keep up the good work!

Momoko
@buzzdata

Permalink | Context

By Pxtl (registered) - website | Posted April 01, 2011 at 09:46:48 in reply to Comment 61785

'sup Momo. Handy link.

-- Martin Z.

Permalink | Context

By Ryan (registered) - website | Posted March 31, 2011 at 19:02:35 in reply to Comment 61785

Thanks for bringing this to our attention. I just downloaded Pete Warden's client API tools and am going through his dstk server setup to see what I can use.

Permalink | Context

View Comments: Nested | Flat

Post a Comment

You must be logged in to comment.

Events Calendar

Recent Articles

Article Archives

Blog Archives

Site Tools

Feeds