Do You See Spiders? Making Government Data Truly Open

The trend towards open government data is growing, with recent developments like Data.Gov and Vancouver’s Open3 motion, but these simply do not go far enough. In addition to publishing downloadable data and open interfaces, government needs to learn from successful commercial websites and bring their “Deep Web” data to the surface.

The internet search experience is constantly evolving. In the early days it was normal to search for a single keyword, be redirected to an authoritative website, and then explore that site to find what you were really looking for. As the search engines became smarter and publishers learned to expose their database records as individual web pages, people have learned to search for more specific information. For instance, searching for the name of a book will take you to an Amazon or Wikipedia entry for that book. Searching for the name of a current release movie will get you local show times and the name of the theatre it’s playing at.

Unfortunately, government has largely failed to recognise this change, and an entrenched tendency to develop stateful applications and portals is making the problem worse. As an example, try searching for US patent number 6368227. You will likely find a few results from ad-driven private websites that were re-publishing the government data, and maybe a broken link to the official patent search. Why bother publishing your information online if you are going to do so in a way that holds it apart from the web?

The good news is that fixing this problem is not hard. The search engines already assign a lot of authority to government sites, so you’re already a step ahead of commercial sites facing the same problem. Just follow a few simple suggestions that the rest of the web has already figured out for us:

  • Publish each well-formatted record to a consistent location on the web that will not change. This allows both people and search engines to come back to these records whenever they want.
     
  • Ensure that search engine spiders have a way of following basic hyperlinks to find this information. This can be either a simple paged set of results, or a more complex hierarchical system if the data allows.
     
  • Generate SiteMaps that link to all of your records as cheap insurance to make sure that the search engines can find all of your content. Be careful to pay attention to the maximum size, and break your data up into multiple sitemaps if necessary.
     
  • Make use of clear and logical metadata such as Title, Description and Canonical tags to ensure that both search engines and your prospective searchers can make sense of the results. Nothing worse than publishing a record with an HTML title like “32432-43A”. Nobody is going to click on that!
     

Do we still need to build applications? Absolutely! Sometimes free text search across the entire web does not offer enough granularity. Do we still need to make data and services available to third parties? Definitely! There are lots of smart people out there who can use our data to help make the world a better place. However, these are secondary to the single most effective way we have of giving citizens access to the data we maintain on their behalf. Our highest level of service can be delivered by being of the web, not just on the web.

Oh, and since this is a geospatial blog: Just because your data is in a GIS, don’t think you can avoid doing something about this. Spatial search is still nowhere near as powerful as general web search, but it’s getting better all the time. Government geodata needs to be published as web pages too.

For some concrete examples of the benefits of becoming part of the web, check out a slide show that I recently published as “Moving Beyond the Desk“. Make sure to turn on the speaker notes. If you don’t feel like watching the slides, just try searching Google for Mark Bate Statue or 2323 Rosstown Rd and see if you can find the City of Nanaimo’s data in the results. For technical information on the systems behind these results, see my previous posts on the public art project and the MapGuide GeoREST extension the City is using to publish property information.

-J

P.S. This post was prompted by James’ mention of the Moving Beyond the Desk slide show. Thanks James!

OSGeo BC Local Chapter First Meeting

Today, about 24 people gathered in two offices for a videoconference to kick off the Open Source Geospatial Foundation BC Local Chapter (In-Waiting.. we still have to apply for official recognition by OSGeo)

I was really impressed by the number of people who showed up to help us get started, and by the level of energy. We had attendees from all three levels of government, private industry, and geospatial consultants. The official minutes, goals, and objectives will be posted shortly, but my impression is that we are going to be working on regional advocacy and acting as a local support group for sharing ideas and solutions around open source software and data, and open standards.

Bright shiny future, etc, etc… Come join us :)

-J

OSGeo BC – Formation and First Meeting

Just a quick note about the formative OSGeo British Columbia Local Chapter

If you are in BC and interested in the promotion or use of open source geospatial (and I haven’t already hit you with the flood of spam to my frequent hangouts), make sure you get signed up and plan to attend our first meeting on November 15 between 11:00am and 2:00pm. For this first session, we will be meeting simultaneously in Victoria and Vancouver, with videoconferencing connecting the sites. A tentative agenda has been drafted in the wiki, but if you have strong feelings one way or another, speak up on the mailing list.

I am extremely happy with the interest in this organisation from all levels of government (with a strong showing by local government – yay team!), private industry, and consulting agencies. I believe that this shows two things. First, that hosting FOSS4G in your region helps to build an awareness of the capabilities of open source geospatial software, and second, that open source geospatial is now at the level that it merits serious consideration as part of any enterprise geospatial strategy.

Even if you cannot attend the initial meeting, I would strongly encourage you to sign up for the mailing list (or its web-based shadow at Nabble) and spark up a conversation.

-J

Value proposition for open source geospatial

For geospatial consultants, the business case for open source is easy. Your clients have fixed budgets; with open source software you get a larger portion of their budgets for customisation, and they get software that is better-tailored to their business needs and has lower ongoing costs because of a lack of “maintenance” fees. These benefits are covered very well in an article by Dirk Riehle. All you have to do is sell them to your customers.

The case has not been quite as clearly documented for proprietary geospatial software development. Autodesk has talked about the value of open-source for MapGuide, but beyond that there has not been much discussion. Within the last week two articles have been published which connect the dots very well. First, Dale Lutz posted on Safe Software’s use of open source geospatial components, with some interesting follow-on commentary by Paul Ramsey on how this works even though the open source components that Safe is using are “competing” with FME. Second, the OSGeo Journal v2 includes a very clear article by Matthew Fleagle and Michael P Gerlek on how LizardTech benefits (pdf – 171KB) from both the use of and contributions towards open source geospatial projects. This is a highly-recommended read… as is the entire journal.

It is clear that there is value in open source geospatial even for proprietary companies, and those who realize this first will gain an advantage over their slower-moving competitors. I would be interested in hearing how other proprietary companies are using open source geospatial to provide greater value to their customers and to allow greater focus on their core competencies.

-J

Ten reasons you NEED to be at FOSS4G2007

We (meaning Paul) have just published a set of ten excellent presentations that have been given early acceptance through a democratic process.

It’s clear from the quality and diversity of presentations that FOSS4G is the one conference you need to attend this year if you want to understand how Open Source Geospatial is disrupting the traditional GIS marketplace and leading innovation in previously unimagined areas. Make sure you check out the great hands-on workshops and labs, and subscribe to the presentation submissions RSS feed for some interesting reading. :)

If you’re thinking about coming, make sure to register and arrange your accomodations/travel soon. Victoria has lots of rooms, but you might not get one in your preferred price range if you don’t book early.

-J