Being in a precipitous mood, I decided to plunge in and get my clichés wet. The first step was to attempt to use the new provider with Jackie Ng‘s FDO Toolbox, my go-to GUI tool for examining FDO data sets. I followed these steps:

1. Downloaded the 32 bit FDO 3.5 version of the new SpatiaLite provider from the SL-King website.
2. Copied all of the files into my C:\Program Files\FDO Toolbox\FDO directory.
3. Copied the contents of the providers_kingspatialite_entry.xml file into the providers.xml file before the closing FeatureProviderRegistry tag, modifying the FDO version string to 3.5.
4. Discovered that something had a dependency on the MS Visual C++ 7.1 runtime, so I found msvcr71.dll and msvcp71.dll on another PC and copied them into the same directory

With this in place, I was able to quickly connect to the SpatiaLite demo database:

And browse and visualize the data:

Confidence buoyed, I decided to take it a step further and attempt to connect to the same data in MapGuide. The initial steps were the same as for FDO Toolbox (adding the provider to the MapGuide FDO folder).

Once configured, I was able to quickly add a connection to the test database, but I had to manually hack in a coordinate system reference and map extents. This is due to an incompatibility between SpatiaLite and MapGuide’s coordinate system handling, and I’m hoping that the WKT projection support in the next version of SpatiaLite allows Haris to fix this.

With the data connection and its spatial context override in place, creating layer and map definitions was dead easy, and allowed me to quickly get to data visualization.

Next Up? Making the SpatiaLite FDO provider work with GeoREST. Out of time for tonight though…

UPDATE (2010-09-01): I was able to get the SpatiaLite provider working with the current preview version of GeoREST with very little difficulty, and the next official build of GeoREST will probably include this provider. If you’d like to play around with it earlier, you can download a totally unsupported package of GeoREST with Spatialite from my temp files area for a limited time only :)

-J

Notes:

1. FDO is a geospatial object/relational mapping framework, allowing consistent programmatic access to dozens of GIS data formats. It is used by MapGuide Open Source, FDO Toolbox, GeoREST, Autodesk’s geospatial products like Topobase, MapGuide Enterprise and AutoCAD Map 3D, and is also included in applications like Safe Software‘s FME and 1Spatial‘s MapRelate.

2. SpatiaLite takes SQLite‘s concept of database-in-a-file, and turns it into geodatabase-in-a-file, offering a full suite of geospatial query abilities in a lightweight extension to SQLite. FDO already has a robust SQLite provider that uses its own open geometry format, metadata, and spatial functions. In contrast, SL-King‘s new provider is intended for users who prefer to use the SpatiaLite geometry format, and takes advantage of SpatiaLite’s built-in GIS capabilities.

Follow the video instructions above for setting up jEdit, with the following exception: when you go to dock Sidekick, you click on Sidekick instead of Structure Browser and then, in order to start parsing the document, you need to select XML mode:

Once you have loaded your KML file, replace the root kml entity with the following instead of the older Google schemas, and you will be able to validate against the OGC’s KML and partial Atom schemas as you work:

<kml 

  xmlns="http://www.opengis.net/kml/2.2" 

  xmlns:kml="http://www.opengis.net/kml/2.2" 

  xmlns:atom="http://www.w3.org/2005/Atom"

  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

  xsi:schemaLocation="

    http://www.opengis.net/kml/2.2 http://schemas.opengis.net/kml/2.2.0/ogckml22.xsd

    http://www.w3.org/2005/Atom http://schemas.opengis.net/kml/2.2.0/atom-author-link.xsd"

>

One odd thing that I noticed in validating my KML.  When I placed the <name> entity anywhere other than right after its parent entity (such as Document or Placemark) the parser complained that it was not valid.  I don’t know if not having the <name> as the first entity is truly invalid, or if this is just a little quirk of the schema or the parser.  The Galdos KML Validator throws the same error.

The main reason I went through setting this up is that I decided that I had to, in good conscience, ignore the part of the OGC KML written standard that constrains Atom relation types to "related".  Not specifying the correct type of relationship means that I am contributing to the downfall of society, or at least to a less meaningful GeoWeb.  See my previous post and some follow-up at Sean Gillies’ blog for more on this.  Using jEdit allows me to ensure that my documents validate against the schema, which gives me at least some level of comfort when ignoring the online validator.

-J

These resources will be single-feature, and will each be related to exactly one HTML representation of the same feature.  Now, it seems logical to me that I would specify an atom:link with rel="alternate" inside of these KML files, either at the Document level or perhaps at the Placemark level.

Unfortunately, from section 9.1.3.5.1 of the official OGC KML specification:

The atom:link rel attribute shall be present and its value shall be related.

What the heck?  I’m not an XML guy by any means, but I think this means that the best I can do is specify a "I think I met that guy in a bar a couple years ago" kind of relationship, when what I really want is something more like "hey, that’s me in a dress shirt."

Now, if I’m right, the question is:  do I exclude the atom:link element from my KML, do I include the element and settle for a generic relationship, or do I intentionally break compliance with the OGC specification.  Tough choice.

-J

Over the past month, developers at Safe Software and the author of the FDO SQLite provider have put some time into ensuring that the SQLite provider will work properly with FME 2009. Reading worked fine out of the box, but writing required a bit of effort. FME needed datastore creation and schema writing added to their generic FDO writer, and the FDO SQLite provider needed to account for the way that FME writes to multiple schemas.

Here’s how you can take advantage of this provider in FME (and in other FDO 3.3 consumers, such as MapGuide Open Source 2.0):

• Download the unofficial binaries for the SQLite provider from my site
• Open this zipfile and copy the SQLiteProvider.dll file into your FDO directory (default c:\Program Files\FME\plugins\fdo\)
• Make a backup of the providers.xml file in that directory, and then edit the original, adding the contents of the sqlite_provider_entry.xml file in an appropriate location.

Once this installed, writing to SQLite from within FME is dead easy…

1. Add new FDO Destination Dataset:

2. Go to Settings and specify OSGeo.SQLite.3.3 as the provider name:

3: Specify the filename you want to write to:

4. Optionally, set a spatial reference system, and click on OK:

That’s it; now you can start adding tables to your SQLite file as you would any other destination dataset in FME!

As far as I know, Safe will not be distributing the SQLite provider directly with FME 2009 (it’s still in beta) primarily because the provider is not officially being released for FDO 3.3, and partially because the provider is still under heavy development. Fear not, though. I am building this provider against the 3.3 branch as often as necessary, and will post binaries as I do.

The relative ease with which this format was supported by FME can be attributed to Safe’s foresight in exposing FDO directly, rather than just using it behind-the-scenes in their SDF3 writer. They also allow FME to act as an FDO provider, which enables users of products that use FDO for their data layer (such as AutoCAD Map 3D) to access the full range of formats that FME supports.

-J

The closest I’ve got to Informix is seeing it in an ESRI employee’s custom command prompt (“Informix Rules>” or something like that) at the 2001 UC, but I’m sure that this is great news for corporations that have data stored via the Spatial DataBlade. FDO increases transparency for Informix data, allowing simple web-based publishing with MapGuide Open Source, and updates and map production using desktop applications such as Autodesk Map 3D.

-J

Traian Stanev recently proposed the creation of an SQLite provider for FDO. He was quickly arm-wrestled into supporting something close to OGC’s Simple Features for SQL specification, and working with Frank Warmerdam hammered out a GIS spec for SQLite that would work for both OGR and FDO. The beauty is that it’s a single file and can be read by any of the existing SQLite tools.

Traian completed initial development of the SQLite provider a couple weeks ago and Frank expanded OGR’s SQLite support to understand this common specification (this work is in the GDAL/OGR trunk for inclusion in the 1.6 release). These implementations have different strengths. The FDO provider was written to be blazing fast, features an in-memory spatial index, and writes to the FDO binary format. The OGR driver was written for maximum portability and allows writing WKT and WKB. Both implementations will read all three geometry formats and understand the dimension and projection information stored in the OGC-derived metadata tables.

You can download a totally unofficial build of the FDO provider from my website if you want to try it out with MapGuide 2.0 or maybe even Autodesk Map 3D 2009. I have successfully tested it in MapGuide with WKT, WKB, and FGF data. Adding this provider to MapGuide is easy:

• Drop the three files in the zipfile into your Server/bin/fdo directory
• Edit your main providers.xml file to include the SQLite provider using the included XML snippet
• Restart MapGuide

You will need some data. Testing can be done with SQLite files from the OGR sample data directory, but you will eventually want to use your own. It’s fairly simple to convert SDF and SHP. Open up a command window in your Server/bin/fdo directory and type something like:

SQLiteConverter.exe c:\src.sdf c:\dest.db

When creating a new data connection to this file, the provider only takes one configuration parameter: the full path to the file. If you run into any bugs, please post them on the FDO Trac instance.

OGR users that are tracking the trunk build can also try this out. With some amazement, I recently found that the enhancements to this driver had already been documented… obviously OGR places a premium on timely docs. ogr2ogr allows you to do a similar import operation, probably something like (untested):

ogr2ogr -f "SQLite" -dsco FORMAT=WKB dest.db src.shp


You can use additional ogr2ogr arguments to ensure that destination spatial reference and geometry type are written to the metadata tables.

Interestingly enough, a common SQLite GIS specification has been kicked around for quite some time. Last year it was discussed on the OSGeo Discuss mailing list, and more recently further discussion was held on the PostGIS mailing list and a wiki page was set up to collaborate on this idea. Obviously, there is considerable interest within the community. My personal hope is that this specification helps the idea of SQLite as a GIS data store take off.

One area where it could be improved is some kind of integration with Alessandro Furieri’s SpatiaLite extension for SQLite that allows common RDBMS GIS functionality in a native SQLite interface. Unfortunately, neither Frank nor Traian had the cycles to integrate this extension’s data format into the specification or the code at this point. Maybe we’ll get lucky and Alessandro will decide to somehow support this spec, but if not I hope there will be some convergence in the long run.

I know that there was more that I wanted to say, but it’s getting late and I don’t even have time to cut the extra gunk out of this post. Happy SQLiting!

-J

Basic improvements include:

• automatic pull from our FDM Computer Aided Dispatch (CAD… a popular acronym) database, removing requirement for dispatchers to manually update this list
• publication of the incidents in multiple formats (GeoRSS, HTML… KML planned for an intermediate update)
• integration with Google Maps

This is what the query interface for incidents looks like:

This is the embedded Google Map (using the GGeoXML class to hit the GeoRSS feed) showing incidents for the selected day:

This is the basic statistics interface allowing users to see incident activity for a date range (which also has a similar Google Map embedded):

And this is what the GeoRSS feed looks like in Google Maps

And now for the technical stuff…

Chris has put together a fairly strong process for extracting and displaying this information publicly, consisting of several components:

• A monthly FME process that reads the current 9-1-1 streets shape file and places it into a normalized database structure (one-to-many between lines and coordinates). This allows easy formatting of coordinates for different output formats (GeoRSS, KML, etc) in native ASP.Net.
• A SQL Server Integration Services job that pulls initial incident data from our CAD database, as well as updated information from the RMS (records management system), generalizes it to block ranges to reduce privacy concerns, and writes it our publicly accessible webserver.
• An ASP.Net web application that transforms an XML serialized data set into GeoRSS (and eventually other formats) using XSLT.
• An ASP.Net web application that provides the rudimentary user interface, including the incident query mechanism, incident statistics, and Google Maps integration.

The current applications are accessed via IFRAMEs because although our standard for web apps on our main site has changed from ASP to ASP.Net, our web site migration is still under way. Once the web site is redeveloped, these will be standard non-encapsulated web apps.

This is definitely just a starting point for us; the framework that has been used for this application was designed so that we can add other data sources that make sense for GeoRSS syndication, such as recent business licenses, building permits, etc. This aligns with our website redevelopment, where we are using RSS/Atom as an alternate access mechanism wherever possible.

As an initial project, there are certainly limitations with this implementation. For instance, we haven’t defined an API for pulling down specific categories or date ranges from the RSS feed. Also, because the GGeoMap class doesn’t expose properties of specific features, we were unable to link the incident rows with the map (pan and pop-up). There are third-party interfaces to Google Maps (GeoXML, EGeoXML) that work around this, and of course the option of just creating the lines ourselves, but we were trying to keep coding and dependencies to a minimum. There is a ticket in the Google Maps API issue tracker for this, so hopefully it will be addressed eventually…

Ideally we’d be using a spatially-enable database (such as PostGIS) as the underlying data store for this application, but we don’t have PostGIS in place on our public webserver yet.

-J

What does it mean? Basically: KML can still act as a self-contained data exchange format, while getting rid of the nasty part of the original <Schema> tag that defined new elements on the fly.

To illustrate the changes, I’ll take you through my previous example of Sammy G Newt. Here he is in glorious colour under the new ExtendedData system:

The first part of this new system is defining the schema; you can see how to do this here:

Looks pretty basic, right? Not much has changed in the Schema tag, except that has name and id attributes, and there is now an optional displayName element for each field.

OK, now that you’ve got the schema, you want to create a BalloonStyle that takes advantage of that schema. Here’s mine:

Can I hear the ah-ha’s?

There are a couple neat things here. First, it’s pretty obvious that if you want the data, you use the format $[nodeName/fieldName], and if you want the display name you use $[nodeName/fieldName/displayName]. Look close at the <a> tag though… I am using an identifier stored in the extended data area to link to a different record in the same KML file, using its ID. This will be great for “previous” and “next” applications among other things. This could just as easily have been used to point to a different Placemark within a remote KML file.

Now that we have a schema and a style, we can create some some content that references these:

So, what’s special about this? Two things. First, I didn’t magically create any new KML tags on the fly. Second, you can see that I am referencing both the style and the schema by URL. This means that you can either store all of your schema, markup, and code in one file, or you can break them out into as many individual files as you would like. You can have a look at my completed example here:

One File:

• Combined KML
• Combined source

Three files:

• Content KML | source
• Schema KML | source
• Style KML | source

Anyway, this last-minute addition to KML 2.2 has made me pretty happy. If you have any questions or comments about it, be sure to chime in on the official thread.

-J

P.S. Another thing that I noticed in perusing the documentation is the ability to use atom tags to link from a KML entity back to its web representation. This is important because it adds another vector for Google to use when assigning relevance to KML files in spatial search. I’m not an AtomPub expert, but I would imagine that it could also be used to allow a smart client to update features on the fly?

Being the cultured guy that I am, I chose to engage is some highly artistic feature editing in this demo:

The OpenLayers demo is highly responsive. The editing tools are intuitive, and even the node insertion feature is well implemented. What is most impressive though is what is going on in the background. Every time you insert/update a feature, an AtomPub operation is triggered. The data that is stored via these operations is (of course) also available as a GeoRSS feed, allowing you to view it in any RSS browser, or even in Google Maps:

If you look close, you will see that Kilroy’s nose’s shape changed between the two screen shots. In the two minutes it took me to get back to take a screenshot of my beautiful artwork in the original interface, some vandal had come by and given him a huge proboscis. :)

I did suffer from some disappointment in my experimentation, though not with this service. I tried to pull this feed through Yahoo Pipes and do something interesting with it, and belatedly remembered how poor their GeoRSS support is. If anyone from Yahoo is listening: please add support for full GeoRSS geometries. Points don’t cut it any more.

Anyway, in my opinion this represents the future of GIS interoperability. Look at how AtomPub ties in with some of the service chaining (featureserver/spatialreference/yahoopipes/fme) stuff I mentioned yesterday. As richer client tools such as Google Earth and other proprietary and open source desktop GIS begin to play ball, I believe that we will see a revolution in social mapping. Anyone not paying attention had better smarten up. It’s no longer about ignoring the elephant; now it’s about getting out from under the steamroller.

-J

FME is usually quick to support new formats (like KML and GeoRSS) but this time Safe has surpassed themselves, getting early JSON and GeoJSON support into their betas before the GeoJSON specification has reached a release version. Tonight I took some time out to play with this new support.

The basis of JSON support in FME is provided by two new read/write formats: JSON and GeoJSON. These new formats are augmented by two new transformers: JSONExploder and JSONExtractor. To get started, I’m going to show you how to extract data from a JSON source I happen to have lying around (it’s publicly available at Yahoo Pipes), which looks like this:

When you first import this data source into FME, it is imported with the top level of attributes broken out. In this case, Pipes returns a top-level object with several attributes (link, description, etc) that you can see in my test workspace below:

Now, this isn’t much use, because my features are hidden inside the “items” attribute. In order to get them out, I first need to explode my single object into multiple features. The new JSONExploder transformer comes to the rescue here:

Now, I have a unique feature for each of my feed items, but I really want some of the nested attributes. In particular, I want the description from the root of the item, and the nested y:location["lat"] and y:location["lon"] attributes. The JSONExtractor makes it easy to pull these out into new attributes:

And once adding a couple more of these, each of my features has some nice attributes attached to it, which I could then turn into points if I wanted:

OK, so that’s kinda cool from a straight ETL standpoint. I can take in JSON, mess with it, and then pump it out into whatever format I want. But the fun stuff is when you start getting into GeoJSON. Fortunately there are a couple early adopters, Christopher Schmidt and Howard Butler, who gave me some feeds to play around with. The first of these comes from Christopher’s super-flexible FeatureServer application (check it out, it’s open source):

The features displayed on this OpenLayers map can be easily downloaded from FeatureServer in GeoJSON format (or KML, or GeoRSS, or whatever). The URL for the GeoJSON representation is:

http://featureserver.org/featureserver.cgi/scribble?format=geojson

Pulling this into FME is as simple as creating a new FME data source, and specifying the URL:

As you can see, you can then treat this data like any other spatial data source:

Now, for a final example… Howard has a GeoJSON resource collection of counties in Iowa, accessible in a pattern something like this:

http://geoservices.hobu.biz/political/json/johnson

Now we could take this feature, in its source projection of UTM Zone 15N NAD83, but Howard’s put together a really nifty (non-commercial use only, unless you want to pay Howard some $$$) JSON-based web processing projection service. Not only that, but he’s also made it smart enough to interpret projections referenced locally, but also from the oh-so-cool (and built as a collaborative effort between Christopher and Howard) SpatialReference.org. So, all you need to do is feed it the URL of your source data, the url of your source CRS (http://spatialreference.org/ref/epsg/26915/) and the url of your destination CRS (http://spatialreference.org/ref/epsg/4326/). Like so:

http://geoservices.hobu.biz/project/?url=%22http://geoservices.hobu.biz/political/json/johnson%22&inref=http://spatialreference.org/ref/epsg/26915/proj4/&outref=http://spatialreference.org/ref/epsg/4326/proj4/

And, as just another link in the dynamic web chain, FME can read this transformed JSON feature:

Now, for desktop FME users, this gives us “Pipes on Steroids”: all the mashup flexibility of Yahoo Pipes, with the huge format support and rich processing model of FME. As cool as this is, I think the real power will be seen whenever Safe integrates this functionality into their Server product. It will allow them to play well on both the “enterprise” traditional GML/WFS/etc level and on the neogeography JSON/GeoRSS/KML mashup level with a single product from a single (or multiple if you want) data source. For organisations that want turn-key interoperability solutions, FME Server is going to rock your world.

-J

P.S. I’m thinking about getting a personalized plate that says GEO JSN :)