Category Archives: Tutorial

JSON and GeoJSON in FME

Posted on by

Many of you know about JSON, an object serialization scheme that has rapidly gained acceptance in AJAX-style applications. What you may not know is that there is an effort to standardise the representation of JSON-ified spatial features, known as GeoJSON.

FME is usually quick to support new formats (like KML and GeoRSS) but this time Safe has surpassed themselves, getting early JSON and GeoJSON support into their betas before the GeoJSON specification has reached a release version. Tonight I took some time out to play with this new support.

The basis of JSON support in FME is provided by two new read/write formats: JSON and GeoJSON. These new formats are augmented by two new transformers: JSONExploder and JSONExtractor. To get started, I’m going to show you how to extract data from a JSON source I happen to have lying around (it’s publicly available at Yahoo Pipes), which looks like this:

JSON Raw

When you first import this data source into FME, it is imported with the top level of attributes broken out. In this case, Pipes returns a top-level object with several attributes (link, description, etc) that you can see in my test workspace below:

JSON Workspace

Now, this isn’t much use, because my features are hidden inside the “items” attribute. In order to get them out, I first need to explode my single object into multiple features. The new JSONExploder transformer comes to the rescue here:

JSONExploder

Now, I have a unique feature for each of my feed items, but I really want some of the nested attributes. In particular, I want the description from the root of the item, and the nested y:location["lat"] and y:location["lon"] attributes. The JSONExtractor makes it easy to pull these out into new attributes:

JSONExtractor

And once adding a couple more of these, each of my features has some nice attributes attached to it, which I could then turn into points if I wanted:

JSON Attributes

OK, so that’s kinda cool from a straight ETL standpoint. I can take in JSON, mess with it, and then pump it out into whatever format I want. But the fun stuff is when you start getting into GeoJSON. Fortunately there are a couple early adopters, Christopher Schmidt and Howard Butler, who gave me some feeds to play around with. The first of these comes from Christopher’s super-flexible FeatureServer application (check it out, it’s open source):

FeatureServer Demo in OpenLayers

The features displayed on this OpenLayers map can be easily downloaded from FeatureServer in GeoJSON format (or KML, or GeoRSS, or whatever). The URL for the GeoJSON representation is:

http://featureserver.org/featureserver.cgi/scribble?format=geojson

Pulling this into FME is as simple as creating a new FME data source, and specifying the URL:

Add GeoJSON

As you can see, you can then treat this data like any other spatial data source:

GeoJSON Visualizer

Now, for a final example… Howard has a GeoJSON resource collection of counties in Iowa, accessible in a pattern something like this:

http://geoservices.hobu.biz/political/json/johnson

Now we could take this feature, in its source projection of UTM Zone 15N NAD83, but Howard’s put together a really nifty (non-commercial use only, unless you want to pay Howard some $$$) JSON-based web processing projection service. Not only that, but he’s also made it smart enough to interpret projections referenced locally, but also from the oh-so-cool (and built as a collaborative effort between Christopher and Howard) SpatialReference.org. So, all you need to do is feed it the URL of your source data, the url of your source CRS (http://spatialreference.org/ref/epsg/26915/) and the url of your destination CRS (http://spatialreference.org/ref/epsg/4326/). Like so:

http://geoservices.hobu.biz/project/?url=%22http://geoservices.hobu.biz/political/json/johnson%22&inref=http://spatialreference.org/ref/epsg/26915/proj4/&outref=http://spatialreference.org/ref/epsg/4326/proj4/

And, as just another link in the dynamic web chain, FME can read this transformed JSON feature:

HoBu GeoJSON Example

Now, for desktop FME users, this gives us “Pipes on Steroids”: all the mashup flexibility of Yahoo Pipes, with the huge format support and rich processing model of FME. As cool as this is, I think the real power will be seen whenever Safe integrates this functionality into their Server product. It will allow them to play well on both the “enterprise” traditional GML/WFS/etc level and on the neogeography JSON/GeoRSS/KML mashup level with a single product from a single (or multiple if you want) data source. For organisations that want turn-key interoperability solutions, FME Server is going to rock your world.

-J

P.S. I’m thinking about getting a personalized plate that says GEO JSN :)

GeoRSS: FME Rocks My World Again

Posted on by

There’s a simple indicator I use to tell when I’m too busy at work; it’s called the FME scale. Any time it takes me over a month to play with a new feature in Safe Software‘s FME, I’m swamped :)

In this case, it was particularly painful. I’m really interested in GeoRSS because I see the potential that it has for increasing the level of information my municipality can deliver to its residents. For instance, we could offer insight into current housing starts with a feed of new construction building permits.

For me, the beauty of GeoRSS is that it has value even without the maps. With a single format, I can publish information that our residents can view in their favourite feed reader, while allowing more sophisticated users to benefit from the spatial information.

As you can see from the following screenshots of a GeoRSS file displayed in Google Reader and Google Maps, I’ve finally found a little time to play around with the GeoRSS support in FME. You can click on the second image to see how it works. Keep in mind that the data is static, and not guaranteed to be accurate for any use:

Example of GeoRSS feed in feed reading software

GeoRSS for Nanaimo Building Permits generated by Safe Software FME

Until now, much of the GeoRSS that I’ve had a chance to look at has been point-based, so it’s nice to have a way of generating some more complex elements in Simple and GML varieties and, now that Google Maps supports GeoRSS, an easy way to visualise it.

So anyway, how did I generate this GeoRSS file? It was pretty easy, and I’ll take you through the steps.

First, I had to set some basic parameters for the feed in FME. This image shows the basic workflow, from create a single feature with no geometry, through to writing the feature out to the “Feed” output:

FME workflow for setting feed-specific attributes.

Inside of the AttributeCreator, I’m setting some basic values for the feed as a whole:

FME AttributeCreator used to set GeoRSS feed properties

Pretty simple, eh?

Next, I pull in some data from our property records database, and merge it with an Oracle Locator spatial table which holds the City’s parcel base:

Merge spatial and attribute data

And finally, I set the same the entry-specific GeoRSS parameters, this time using attributes of the building permit records instead of static text to fill in the blanks:

Populating GeoRSS entry attributes using FME

And that’s all she wrote. I still have to find time to fine-tune some of the attributes (such as dates and link), get some approvals, and work out the logistics of getting the data out to our external server nightly. It sure is nice that the technical part is so easy though.

If you deal with spatial data on a regular basis and haven’t evaluated FME, you’re doing yourself a disservice. Pick up the latest beta from their site (for GeoRSS support), request an evaluation license, and play around a bit. Everyone’s usage pattern is different, but I was able to justify the cost of my initial license and training with the time it saved over about six months. Safe Software deserves a lot of credit. Apart from the usefulness of FME, their support is incredible. They are the only company that I have ever dealt with where I felt I was getting real value from the software maintenance plan.

And, in case you’re wondering, they didn’t buy me a nice laptop or a 3D mouse to use while writing this. I’m just a satisfied customer :)

-J

Build MapGuide Open Source from SVN

Posted on by

I guess Chris has prodded me enough times now that I actually have to make good on my promise. Also, Tom’s work on the FC4 version got me motivated.

I’ve place a very draft version of some some instructions for downloading and building MapGuide Open Source from the SVN code repository here:

http://wiki.osgeo.org/index.php/Building_MapGuide_on_Windows

It worked for me (to the best of my recollection) and it may work for you also. If you try it and run into problems, spot something that is missing, or find something unclear, please hack up the wiki so that those following in your footsteps don’t hit the same issues.

-J