MapGuide REST Extension. Feedback wanted!

Traditionally, access to municipal geospatial data has been application-centric, either through a web interface with a steep learning curve, or through data downloads or services for use in desktop GIS.  While these are both much better than not providing access at all, neither strategy is particularly effective in getting the data to the people most interested in it: local residents.  This limitation has become painfully clear in an age when you can find almost any other information by typing a few words into a search engine.  Also, in the absence of ready access to our data, citizens are spending countless hours re-creating it, when they could instead be adding value and stimulating the economy. 

By linking to open source components such as MapGuide, FDO, POCO, CTemplate, and libkml, the MapGuide REST framework is very simple and powerful.  With absolutely no programming, a site administrator can build out customised HTML and KML representations of their data, and enable access to default GeoJSON and FDO-based XML representations.  Pair these representations with static HTML pages that expose "service metadata", and you have data that is easily crawlable by search engine spiders, and easily understood by anyone wanting to access the data for other applications.  Enough talk; let’s go over some examples from the City of Nanaimo’s live web site.

Data URIhttp://maps.nanaimo.ca/data/ 

This is where all of the City’s RESTful GeoData will be exposed.  Currently, there is a single resource type exposed for Property information, but more is on the way and will be interlinked when it makes sense.  The index page for this directory is a static HTML page pointing to various features of the Property resource type.

Property Resourceshttp://maps.nanaimo.ca/data/property/ 

This currently displays a page that allows users to search by one of three attributes by HTML forms, and contains links to OpenSearch (with draft Geo) documents that define some of the other search and representation capabilities.

OpenSearch link:

<link 
  rel="search" 
  type="application/opensearchdescription+xml" 
  title="Properties by Street" 
  href="/data/property/search_by_street.xml"
>

As well, there is a link to a Street Index which acts as a form of sitemap, providing web crawlers with an equal-length trip to every property in the City.

Example Property HTML Representationhttp://maps.nanaimo.ca/data/property/123170.html

As much as this looks like a regular web page, it is cram-packed with features that make it distinguishable as a geospatial resource.  Most visibly, it displays an image of the property’s shape in context, and provides a link to the KML representation of that property.  Digging a little bit deeper, the page header contains links to alternate representations:

<link 
  rel="alternate" 
  type="application/vnd.google-earth.kml+xml" 
  title="KML representation of 2143 AARON WAY" 
  href="http://maps.nanaimo.ca/data/property/123170.kml" >
<link 
  rel="alternate" 
  type="application/json" 
  title="GeoJSON representation of 2143 AARON WAY" 
  href="http://maps.nanaimo.ca/data/property/123170.json" >
<link 
  rel="alternate" 
  type="text/xml" 
  title="XML representation of 2143 AARON WAY" 
  href="http://maps.nanaimo.ca/data/property/123170.xml" >

and some simple geotags:

<meta name="ICBM" content="49.18882, -123.9785">
<meta name="geo.region" content="CA-BC">
<meta name="geo.placename" content="Nanaimo">
<meta name="geo.position" content="49.18882;-123.9785"> 

Finally, within the page content itself, you will find both adr and geo microformats.

<div class="adr">
  <span class="extended-address"></span>
  <span class="street-address">2143 AARON WAY</span>, 
  <span class="locality">NANAIMO</span>,
  <span class="region">BC</span>
  <span class="country-name" style="display:none;">CANADA</span>
</div>

<div class="geo">
  <span class="latitude">49.18882</span>, 
  <span class="longitude">-123.9785</span>
</div>

Example Property KML Representationhttp://maps.nanaimo.ca/data/property/123170.kml

The KML representation uses Atom Author and Link elements to define the publisher of the data, and to store the KML’s URI and that of its alternate HTML representation.  It also uses a combination of ExtendedData and BalloonStyle entities to separate data from presentation, though I have also included a link to the HTML representation in the Description entity for earlier clients.  Finally, it uses external Style and Schema definitions to reduce the size of the file and to maintain consistency across the site.  I’d encourage you to view the source of the KML file to see how these elements work together.

Example JSON Representation:  http://maps.nanaimo.ca/data/property/123170.json

Example XML Representation:  http://maps.nanaimo.ca/data/property/123170.xml

Example PNG Representation:  http://maps.nanaimo.ca/data/property/123170.png

The native GeoJSON and FDO XML representations are provided by the extension.  No customization is possible, which ensures a consistent implementation from site to site.  The PNG representation is also native (not template), but allows you to specify a MapGuide map definition and a layer for selection.

"Endpoints" (because I’m not sure what to call them):

http://maps.nanaimo.ca/data/property/index.kml

This is a static file that allows KML clients to interact with the property information dynamically through View Based Refresh.  Basically, this file tells Google Earth to request the .kmz endpoint below using a BBOX element that will be used to return only features in view, or a warning if the viewport is too large.

http://maps.nanaimo.ca/data/property/.html

http://maps.nanaimo.ca/data/property/.kmz

http://maps.nanaimo.ca/data/property/.json

http://maps.nanaimo.ca/data/property/.xml

These allow users to perform queries on the data.  There are quite a few variations possible, but they basically support standard FDO filters, bbox, count, and start GET parameters.  There are also some funky parameter patterns that allow for easy searching from HTML forms, in the format: filter_[0-9]_(and|or)_(equal|like|likeleft|likeright|lt|gt)_attributename

http://maps.nanaimo.ca/data/property/.schema

This is a special "endpoint" that allows a smart client to determine the data schema for the XML representations of the feature.

Applications

Now, providing read-only representations of these features is pretty cool on its own, and that is all that the City of Nanaimo is prepared to do at this point.  However, the extension can be configured to allow full RESTful data access:  GET, POST, PUT, DELETE.  Combined with a yet-to-be-released FDO REST provider, users of applications that implement FDO (such as AutoCAD Map 3D, Safe Software‘s FME, 1Spatial‘s MapRelate and Radius Studio, and MapGuide) will be able to read, write, update, and delete data from this extension the same way as they would from local data stores.  I was experimenting with this against an internal server using Map 3D, and it was pretty cool to be able to edit data directly from a web site.

Applications that implement GeoJSON, such as Safe Software’s FME can also take advantage of this format to read and (untested) write to the exposed data.

json001

The GeoJSON representation has a considerable upside for light-weight mobile data editing, for instance in an OpenLayers application running inside Google Chrome (or another Gears-enabled browser) and for minimal-packet data access from mobile phone applications.

All of these are really cool applications from a geospatial professional standpoint, but they don’t really address the primary goal of the original requirement:  enabling easier access to spatial information by residents.  That’s OK though, just by making these representations available and well-linked, the search engines will pick them up and do this part of the work for me. 

This site has been active for less than two weeks, and Google is already ranking some of the property reports first in the results for generic searches within British Columbia and specific searches elsewhere.  For instance, when I search from home for "881 Hunter St" (one of the properties Google has indexed) it comes up as number 1.  When Haris searches from Slovenia, that isn’t the case, but if he searches for "881 Hunter St, Nanaimo, BC" then it does.  Most municipalities have high authority for their location name, and this is a great way of taking advantage of that authority to provide a higher level of service to our residents.

Weaknesses

The current implementation has a few issues.  Unfortunately these will require a substantial refactoring to resolve, so we’re looking at another month or so before we have a public-ready build of the code. Some of these problems are as follows:

  • HTTP status codes are often not correct.  For instance, a search that returns a single feature should redirect to that representation rather than displaying the single feature under the search representation.
  • Fine control over methods is required, such as enabling/disabling specific methods, requiring SSL, requiring certain group access, etc.
  • Existing configuration system grew in odd ways as the code was modified, and needs to be de-Frankenstein-ed
  • More control over URI patterns needs to be extended to users, for instance to allow the creation of -edit representations that return pre-populated HTML forms.
  • Need revision numbering or data checksum to reduce chance of overlapping writes.
  • Need parameter/http header overrides for method (to deal with firewalls) and format (to allow selection from HTML forms)
  • Other miscellaneous issues.

Updates

Based on feedback, Haris has enabled JSONP for the .json representations. Just add the argument _callback to the URI, with the name of your desired function as the value. e.g.: http://maps.nanaimo.ca/data/property/.json?filter_1_and_equal_street=ROSSTOWN+ROAD&_callback=rosstown

Call for Feedback

If you have a chance to look at the current implementation and see any areas where it could be improved, we’re really interested in making changes required to meet REST best practices.  We’ve been trying to pay attention to current guidance, but without a real implementation it has been hard to truly understand what is needed.

Thanks!

-J

Validating against the OGC KML Schema in jEdit

The Google Earth Outreach team has a good little tutorial on validating KML as you type in jEdit.  Unfortunately, it’s a little bit out-of-date, so here are some pointers for validating against the newer OGC 2.2 schema, using the current version of jEdit.

Follow the video instructions above for setting up jEdit, with the following exception: when you go to dock Sidekick, you click on Sidekick instead of Structure Browser and then, in order to start parsing the document, you need to select XML mode:

image

Once you have loaded your KML file, replace the root kml entity with the following instead of the older Google schemas, and you will be able to validate against the OGC’s KML and partial Atom schemas as you work:

<kml 
  xmlns="http://www.opengis.net/kml/2.2" 
  xmlns:kml="http://www.opengis.net/kml/2.2" 
  xmlns:atom="http://www.w3.org/2005/Atom"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="
    http://www.opengis.net/kml/2.2 http://schemas.opengis.net/kml/2.2.0/ogckml22.xsd
    http://www.w3.org/2005/Atom http://schemas.opengis.net/kml/2.2.0/atom-author-link.xsd"
>

One odd thing that I noticed in validating my KML.  When I placed the <name> entity anywhere other than right after its parent entity (such as Document or Placemark) the parser complained that it was not valid.  I don’t know if not having the <name> as the first entity is truly invalid, or if this is just a little quirk of the schema or the parser.  The Galdos KML Validator throws the same error.

The main reason I went through setting this up is that I decided that I had to, in good conscience, ignore the part of the OGC KML written standard that constrains Atom relation types to "related".  Not specifying the correct type of relationship means that I am contributing to the downfall of society, or at least to a less meaningful GeoWeb.  See my previous post and some follow-up at Sean Gillies’ blog for more on this.  Using jEdit allows me to ensure that my documents validate against the schema, which gives me at least some level of comfort when ignoring the online validator.

-J

KML Spec Bizarro?

I’m working on some new KML resources and, just as I try to do for my HTML, I want to at least attempt to get it to validate against an official specification.  For KML, this means the OGC KML specification.

These resources will be single-feature, and will each be related to exactly one HTML representation of the same feature.  Now, it seems logical to me that I would specify an atom:link with rel="alternate" inside of these KML files, either at the Document level or perhaps at the Placemark level.

Unfortunately, from section 9.1.3.5.1 of the official OGC KML specification:

The atom:link rel attribute shall be present and its value shall be related.

What the heck?  I’m not an XML guy by any means, but I think this means that the best I can do is specify a "I think I met that guy in a bar a couple years ago" kind of relationship, when what I really want is something more like "hey, that’s me in a dress shirt."

Now, if I’m right, the question is:  do I exclude the atom:link element from my KML, do I include the element and settle for a generic relationship, or do I intentionally break compliance with the OGC specification.  Tough choice.

-J

FME 2009 – It’s the little things

(warning, this post may contain Windows user interface screen shots some may find offensive)

Safe Software’s FME Workbench has the best user interface that I have ever used.  Hands down.  Every time I have to use something with similar functionality such as Microsoft’s SSIS, or even the time I wasted a whole weekend trying to get an open source ETL tool to do what I wanted it to do, I find myself wishing I could be working in FME instead.  It was great when I started using it six years ago, and it’s been getting better with every release.

Although Safe has definitely been busy with FME 2009 (get the whole story on fmepedia’s What’s Fine in 2009 page), it almost seems tame compared to some of the major changes in the last couple releases.  However, behind the scenes there was a major UI refresh.  Three of the new UI features are among the things I think are coolest about this latest release.

First, they created a really intuitive “default settings” interface.  As far as I can remember, previous releases didn’t even have default settings for transformers, and dataset defaults were set through a dialog box hidden in the configuration UI.  It was so much trouble to use that I don’t imagine many people were setting default values unless they had something that REALLY annoyed them:

FME 2007 default value dialog

For 2009, rather than hiding these values deep in a configuration dialog somewhere, they let users set the default as they are working with the data stores and with individual transformers.  This is something I’d love to see more applications do.  Here’s an example:

image

And another of setting transformer defaults:

image

Couldn’t be easier.  I predict that many users–even ones that didn’t know this functionality was available in previous releases–will be using this.  My only suggestion would be the addition of a “Save As …” option, so we could have two or three sets of default values for each item.  For instance, I want to have the values for both my testing and production database configurations available.  And I want to have my favourite “Smoothing” and “Thinning” profiles for the Generalizer.

The second feature that I really like about the new UI is Quick Connect.  Mark describes this over at the FME Evangelist blog better than I ever could… make sure to watch his cheesy movie for some time-saving tips. 

FME 2009 Quick Connect

I’ve known that I’ve had one of the problems that this feature solves ever since I’ve been using FME.  Basically, it was very hard to connect two transformers that were so far apart that they neither fit on the screen nor were clickable when zoomed-out.  I made a feature request for this, but my potential solution was really clunky.  With this new feature, connecting two trans-workspace transformers is quick, easy, and intuitive.

The final feature that I really like is the ability to edit source feature attributes.  Now normally, source feature attributes don’t change, and even if they do FME provides an effective wizard that allows you to refresh the source feature definitions from their original source without manual intervention.  Where this really comes in handy is when you use the optional SELECT Statement parameter to join to a secondary table:

image

Now, instead of jumping through lots of hoops to have the joined attributes show up in your workspace, you can enable the option that allows you to redefine the source feature’s attributes:

image

There are many other nice UI changes in this release, and some of the non-UI changes are really great too.  Things that I may actually use include new XML features (XQuery and XSLT transformers) and new raster features such as RasterExpressionEvaluator (not quite DOCELL yet, but getting there), support for BigTIFF, and the ability to visualize rasters in the FME Viewer.

I’m sure that there’s something for everyone in the feature list; make sure to check out FME 2009 when you get a chance.

-J

p.s. I had to change my theme as part of my WP 2.7 upgrade, and while I was at it I hacked WordPress so that XMLRPC would work properly with SSL and tried out Windows Live Writer (with a twitter notify plugin). Seems to have worked OK so far…

Web GIS Job at City of Vancouver

If you’re looking for a challenging job as part of a team developing and maintaining a complex web mapping infrastructure, you could do far worse than joining up with the City of Vancouver’s award-winning GIS Sustainment Team:

http://vancouver.ca/humanresources/jobs/09-0012O2P.htm

Vancouver is a beautiful place and the people I know in that group are all good folks.

-J