There is a lot involved in making a web map perform, especially if you are not leveraging tile caching. One part of the story is hardware, and I’m lucky enough to share space on a dual quad-core machine with 4GB RAM and relatively fast disk. Another part is proper generalization of the vector data for display; no point in carrying sub-micron precision on a map that will generally be displayed at 1:500 or smaller. And of course, there’s MapGuide’s inherent speed when properly configured. This leaves out one of the most important parts though: raster data.

Raster data is big, brutish and hard to work with, and optimizing raster access is often one of the most important parts of delivering a successful web map. Users have come to expect “satellite” imagery on their web maps, and complain when it doesn’t perform as well as Google Maps. One of the best ways that I have found of flipping and folding raster data is Frank Warmerdam’s FWTools, which wraps GDAL and some other utilities in a single easy-to-use package.

My starting point consisted of:

• 79 TIFF + Worldfile images, 10cm resolution, about 1.1GB each
• 14 TIFF + Worldfile images, 30cm resolution, about 600MB each

So, I was working with about 100GB of images, none of which were optimized for web-based display, and which did not contain the spatial reference information that the FDO Raster Provider (also based on GDAL!) works best with.

The first thing I did was set up a batch process to optimize the individual images. This involved three steps:

1. Obtain a correct .prj file containing the WKT spatial reference information for my images. The easiest place for me to get this was SpatialReference.org, but you might just have one hanging around.

http://spatialreference.org/ref/epsg/26910/

2. Reprocess the image into a Tiled GeoTIFF, with no compression and a relatively large internal block size, and specifying the projection file obtained above. The caret (^) is the DOS line continuation character:

gdal_translate ^
-co "TILED=YES" ^
-co "PROFILE=GEOTIFF" ^
-co "INTERLEAVE=BAND" ^
-co "BLOCKXSIZE=512" ^
-co "BLOCKYSIZE=512" ^
-a_srs utm83-10.prj ^
infile.tif ^
outfile.tif


You can obtain more information on gdal_translate and the GeoTIFF options on the GDAL website. Depending on your source data and intended use, other values could be more appropriate, and you really should experiment.

3. Create internal pyramids in each image so that the entire image does not need to be fetched when zoomed out. This is one of the easiest performance gains you can get if you can afford the extra disk space.

gdaladdo -r gauss output.tif 2 4 8 16 32 64 128


Once this was done, I had a really decent set of fast images to work with, but these would only be appropriate to load at large scales when only one or a very few of the images need to be opened on each map view. For smaller scales, I needed to reduce the size of the images being processed, and also reduce the number of files being accessed on each fetch. I decided to go with a simple two-tier approach: Load the individual images at scales larger than some fixed value, and load a single overview image at scales smaller than that value.

The only problem was that I did not have an appropriate overview image. I wanted something that was relatively small, highly optimized, and which had white fill in its nodata areas. Fortunately GDAL and the awesome folks in the #gdal channel at freenode came to the rescue again, this time with four steps.

1. The first thing I needed to do was build a list of all of the images I wanted to have as part of the overview and feed these into the gdalbuildvrt command to build a single virtual image. You could do this manually, but I have the awesome GnuWin32 utilities installed so used these instead; they’re almost enough to make me not miss the days when I spent most of my time in Unix:

find images/ -name "*.tif" | xargs gdalbuildvrt -resolution highest all_images.vrt


2. Because I wanted a white background on my overviews, I then edited the all_images.vrt, adding a <NoDataValue/> section at the top of each of the three <VRTRasterBand /> sections:

<VRTRasterBand dataType="Byte" band="1">
<NoDataValue>255</NoDataValue>


3. The gdalinfo command gave me the dimensions of the virtual image, each of which I then divided iteratively to give me reasonable overview dimensions which I could feed into gdal_translate.

gdal_translate ^
-outsize 53120 14000 ^
-co "TILED=YES" ^
-co "PROFILE=GEOTIFF" ^
-co "INTERLEAVE=BAND" ^
-co "BLOCKXSIZE=512" ^
-co "BLOCKYSIZE=512" ^
all_images.vrt ^
all_images.tif


When this completed, I deleted the all_images.tif.aux.xml file because I did not want to carry the additional metadata that GDAL maintains in that file.

Careful with sizes here. If you’re using an application that supports it, you can specify the -CO “BIGTIFF=YES” option to generate files larger than 4GB, but you’re likely better off generating an intermediate level of aggregated and resampled tiles instead.

4. The final step was to once again generate internal pyramids to allow for better performance at small scales:

gdaladdo -r gauss all_images.tif 2 4 8 16 32 64 128


Once these two data sets were processed, I simply used MapGuide Maestro to make two raster data connections. For the first data connection, I added all of the individual TIFF images to a composite raster type, and Maestro generated a configuration document which allows MapGuide to know which image to access for a given extent. For the second layer, I just pointed to the overview GeoTiff. I then created layers for these, experimented until I found the scale where the overview image started looking pixelated, and set the layers’ view scale properties accordingly. There are some notes on working with rasters in the Maestro documentation.

More performance could probably be gained by having an intermediate level where the coverage area was aggregated into larger tiles before being combined into one large overview image, but for the initial launch this was deemed to have high enough performance.

On my production server, I’m lucky enough to have a fast, high-spindle-count RAID shelf dedicated to storing these uncompressed TIFFs, and they scream off the disk. My test server is VMWare-based, and disk performance and space are both at a premium. In this case, I still used the TIFF overview map, but at large scales I access a set of tiled MrSID files instead. This seemed like a decent compromise given the constraints, but did seem to thrash the CPU a bit.

GDAL was one of the first open source geospatial applications I tried (not counting GRASS and MOSS) and is constantly coming in handy, whether I’m reprojecting, adding spatial reference information to images, or converting between formats.

Thanks to hobu (Howard Butler), FrankW (Frank Warmerdam) and EvenR (Even Rouault) from the #gdal IRC channel on freenode for helping me work my way to this solution. Amazing support!

-J

Over the past month, developers at Safe Software and the author of the FDO SQLite provider have put some time into ensuring that the SQLite provider will work properly with FME 2009. Reading worked fine out of the box, but writing required a bit of effort. FME needed datastore creation and schema writing added to their generic FDO writer, and the FDO SQLite provider needed to account for the way that FME writes to multiple schemas.

Here’s how you can take advantage of this provider in FME (and in other FDO 3.3 consumers, such as MapGuide Open Source 2.0):

• Download the unofficial binaries for the SQLite provider from my site
• Open this zipfile and copy the SQLiteProvider.dll file into your FDO directory (default c:\Program Files\FME\plugins\fdo\)
• Make a backup of the providers.xml file in that directory, and then edit the original, adding the contents of the sqlite_provider_entry.xml file in an appropriate location.

Once this installed, writing to SQLite from within FME is dead easy…

1. Add new FDO Destination Dataset:

2. Go to Settings and specify OSGeo.SQLite.3.3 as the provider name:

3: Specify the filename you want to write to:

4. Optionally, set a spatial reference system, and click on OK:

That’s it; now you can start adding tables to your SQLite file as you would any other destination dataset in FME!

As far as I know, Safe will not be distributing the SQLite provider directly with FME 2009 (it’s still in beta) primarily because the provider is not officially being released for FDO 3.3, and partially because the provider is still under heavy development. Fear not, though. I am building this provider against the 3.3 branch as often as necessary, and will post binaries as I do.

The relative ease with which this format was supported by FME can be attributed to Safe’s foresight in exposing FDO directly, rather than just using it behind-the-scenes in their SDF3 writer. They also allow FME to act as an FDO provider, which enables users of products that use FDO for their data layer (such as AutoCAD Map 3D) to access the full range of formats that FME supports.

-J

If you’re using MapGuide Open Source 2.0 (2.0.2 recommended) and are not happy with the default blue selection colour, you can easily change it on a per-install basis by modifying the mapviewerajax template file. This is located at:

(INSTALLDIR)\WebServerExtensions\www\viewerfiles\ajaxmappane.templ

You will need to find the function called RequestMapImage, which includes a line that looks like this:


url = webAgent +
"?OPERATION=GETDYNAMICMAPOVERLAYIMAGE&FORMAT=PNG&VERSION=1.0.0&SESSION="
+ sessionId + "&MAPNAME=" + encodeComponent(mapName)
+ "&SEQ=" + Math.random();


You will need to modify this to change the VERSION to 2.0.0, and add the BEHAVIOR and SELECTIONCOLOR parameters:


url = webAgent +
"?OPERATION=GETDYNAMICMAPOVERLAYIMAGE&FORMAT=PNG&VERSION=2.0.0&SESSION="
+ sessionId + "&MAPNAME=" + encodeComponent(mapName)
+ "&SEQ=" + Math.random() + "&BEHAVIOR=7&SELECTIONCOLOR=FF5300FF";


Before:

After:

The BEHAVIOR parameter is a bitmask that controls what is rendered, and is described in RFC 38. The SELECTIONCOLOR parameter is a hex string in RGBA format, or its integer equivalent. Note for geeks: the A (opacity) portion of the string is ignored; the server code masks this out of the passed value, and then adds 200 (C8) as the line opacity and 160 (A0) as the fill opacity.

Note, the mapagent call allows you the flexibility to add more complicated behaviour to your viewer, such as changing the selection colour dynamically based on the map name, but I’ll leave that as an exercise for the reader (mostly because I’m lazy and use Fusion anyway and wouldn’t benefit from the work).

-J

Normally MapServer appears to be fairly simple to install, but there were a couple annoying complications in my scenario. The first was that pair Networks runs exclusively on FreeBSD. The second was that it is an entirely managed service: no root access allowed, no ability to run ldconfig, no changes to php.ini. The net result was that all of the dependencies had to be compiled in static mode, and that a custom PHP CGI had to be used so that the extension_dir could be overridden.

The notes that I took during installation are available if anyone is interested. These are not guaranteed to be accurate, but are pretty close. If you are on pair Networks and trust me (are you crazy?) you can see if the binaries (5 MB) work for you. They are mostly static, so there shouldn’t be any dependencies other than the standard pair libraries, but you never know until you try…

As long as I had it compiled I felt the need to play around. I grabbed some test data from the awesome OSGeo4W project (which allows Windows users to run a large part of the OSGeo stack without hassles) and put them up on my site:

• Itasca Demo
• Gmap Demo
• Gmap Fusion Demo

I did run into some problems that I didn’t take the time to solve. I couldn’t get GEOS to link into MapServer properly, and there were some issues with some of the image formats in GDAL. I’m sure that these could have been overcome in time, but it’s the end of the weekend and I have to get back to real life :)

-J

What does it mean? Basically: KML can still act as a self-contained data exchange format, while getting rid of the nasty part of the original <Schema> tag that defined new elements on the fly.

To illustrate the changes, I’ll take you through my previous example of Sammy G Newt. Here he is in glorious colour under the new ExtendedData system:

The first part of this new system is defining the schema; you can see how to do this here:

Looks pretty basic, right? Not much has changed in the Schema tag, except that has name and id attributes, and there is now an optional displayName element for each field.

OK, now that you’ve got the schema, you want to create a BalloonStyle that takes advantage of that schema. Here’s mine:

Can I hear the ah-ha’s?

There are a couple neat things here. First, it’s pretty obvious that if you want the data, you use the format $[nodeName/fieldName], and if you want the display name you use $[nodeName/fieldName/displayName]. Look close at the <a> tag though… I am using an identifier stored in the extended data area to link to a different record in the same KML file, using its ID. This will be great for “previous” and “next” applications among other things. This could just as easily have been used to point to a different Placemark within a remote KML file.

Now that we have a schema and a style, we can create some some content that references these:

So, what’s special about this? Two things. First, I didn’t magically create any new KML tags on the fly. Second, you can see that I am referencing both the style and the schema by URL. This means that you can either store all of your schema, markup, and code in one file, or you can break them out into as many individual files as you would like. You can have a look at my completed example here:

One File:

• Combined KML
• Combined source

Three files:

• Content KML | source
• Schema KML | source
• Style KML | source

Anyway, this last-minute addition to KML 2.2 has made me pretty happy. If you have any questions or comments about it, be sure to chime in on the official thread.

-J

P.S. Another thing that I noticed in perusing the documentation is the ability to use atom tags to link from a KML entity back to its web representation. This is important because it adds another vector for Google to use when assigning relevance to KML files in spatial search. I’m not an AtomPub expert, but I would imagine that it could also be used to allow a smart client to update features on the fly?

FME is usually quick to support new formats (like KML and GeoRSS) but this time Safe has surpassed themselves, getting early JSON and GeoJSON support into their betas before the GeoJSON specification has reached a release version. Tonight I took some time out to play with this new support.

The basis of JSON support in FME is provided by two new read/write formats: JSON and GeoJSON. These new formats are augmented by two new transformers: JSONExploder and JSONExtractor. To get started, I’m going to show you how to extract data from a JSON source I happen to have lying around (it’s publicly available at Yahoo Pipes), which looks like this:

When you first import this data source into FME, it is imported with the top level of attributes broken out. In this case, Pipes returns a top-level object with several attributes (link, description, etc) that you can see in my test workspace below:

Now, this isn’t much use, because my features are hidden inside the “items” attribute. In order to get them out, I first need to explode my single object into multiple features. The new JSONExploder transformer comes to the rescue here:

Now, I have a unique feature for each of my feed items, but I really want some of the nested attributes. In particular, I want the description from the root of the item, and the nested y:location["lat"] and y:location["lon"] attributes. The JSONExtractor makes it easy to pull these out into new attributes:

And once adding a couple more of these, each of my features has some nice attributes attached to it, which I could then turn into points if I wanted:

OK, so that’s kinda cool from a straight ETL standpoint. I can take in JSON, mess with it, and then pump it out into whatever format I want. But the fun stuff is when you start getting into GeoJSON. Fortunately there are a couple early adopters, Christopher Schmidt and Howard Butler, who gave me some feeds to play around with. The first of these comes from Christopher’s super-flexible FeatureServer application (check it out, it’s open source):

The features displayed on this OpenLayers map can be easily downloaded from FeatureServer in GeoJSON format (or KML, or GeoRSS, or whatever). The URL for the GeoJSON representation is:

http://featureserver.org/featureserver.cgi/scribble?format=geojson

Pulling this into FME is as simple as creating a new FME data source, and specifying the URL:

As you can see, you can then treat this data like any other spatial data source:

Now, for a final example… Howard has a GeoJSON resource collection of counties in Iowa, accessible in a pattern something like this:

http://geoservices.hobu.biz/political/json/johnson

Now we could take this feature, in its source projection of UTM Zone 15N NAD83, but Howard’s put together a really nifty (non-commercial use only, unless you want to pay Howard some $$$) JSON-based web processing projection service. Not only that, but he’s also made it smart enough to interpret projections referenced locally, but also from the oh-so-cool (and built as a collaborative effort between Christopher and Howard) SpatialReference.org. So, all you need to do is feed it the URL of your source data, the url of your source CRS (http://spatialreference.org/ref/epsg/26915/) and the url of your destination CRS (http://spatialreference.org/ref/epsg/4326/). Like so:

http://geoservices.hobu.biz/project/?url=%22http://geoservices.hobu.biz/political/json/johnson%22&inref=http://spatialreference.org/ref/epsg/26915/proj4/&outref=http://spatialreference.org/ref/epsg/4326/proj4/

And, as just another link in the dynamic web chain, FME can read this transformed JSON feature:

Now, for desktop FME users, this gives us “Pipes on Steroids”: all the mashup flexibility of Yahoo Pipes, with the huge format support and rich processing model of FME. As cool as this is, I think the real power will be seen whenever Safe integrates this functionality into their Server product. It will allow them to play well on both the “enterprise” traditional GML/WFS/etc level and on the neogeography JSON/GeoRSS/KML mashup level with a single product from a single (or multiple if you want) data source. For organisations that want turn-key interoperability solutions, FME Server is going to rock your world.

-J

P.S. I’m thinking about getting a personalized plate that says GEO JSN :)

In this case, it was particularly painful. I’m really interested in GeoRSS because I see the potential that it has for increasing the level of information my municipality can deliver to its residents. For instance, we could offer insight into current housing starts with a feed of new construction building permits.

For me, the beauty of GeoRSS is that it has value even without the maps. With a single format, I can publish information that our residents can view in their favourite feed reader, while allowing more sophisticated users to benefit from the spatial information.

As you can see from the following screenshots of a GeoRSS file displayed in Google Reader and Google Maps, I’ve finally found a little time to play around with the GeoRSS support in FME. You can click on the second image to see how it works. Keep in mind that the data is static, and not guaranteed to be accurate for any use:

Until now, much of the GeoRSS that I’ve had a chance to look at has been point-based, so it’s nice to have a way of generating some more complex elements in Simple and GML varieties and, now that Google Maps supports GeoRSS, an easy way to visualise it.

So anyway, how did I generate this GeoRSS file? It was pretty easy, and I’ll take you through the steps.

First, I had to set some basic parameters for the feed in FME. This image shows the basic workflow, from create a single feature with no geometry, through to writing the feature out to the “Feed” output:

Inside of the AttributeCreator, I’m setting some basic values for the feed as a whole:

Pretty simple, eh?

Next, I pull in some data from our property records database, and merge it with an Oracle Locator spatial table which holds the City’s parcel base:

And finally, I set the same the entry-specific GeoRSS parameters, this time using attributes of the building permit records instead of static text to fill in the blanks:

And that’s all she wrote. I still have to find time to fine-tune some of the attributes (such as dates and link), get some approvals, and work out the logistics of getting the data out to our external server nightly. It sure is nice that the technical part is so easy though.

If you deal with spatial data on a regular basis and haven’t evaluated FME, you’re doing yourself a disservice. Pick up the latest beta from their site (for GeoRSS support), request an evaluation license, and play around a bit. Everyone’s usage pattern is different, but I was able to justify the cost of my initial license and training with the time it saved over about six months. Safe Software deserves a lot of credit. Apart from the usefulness of FME, their support is incredible. They are the only company that I have ever dealt with where I felt I was getting real value from the software maintenance plan.

And, in case you’re wondering, they didn’t buy me a nice laptop or a 3D mouse to use while writing this. I’m just a satisfied customer :)

-J

I’ve place a very draft version of some some instructions for downloading and building MapGuide Open Source from the SVN code repository here:

http://wiki.osgeo.org/index.php/Building_MapGuide_on_Windows

It worked for me (to the best of my recollection) and it may work for you also. If you try it and run into problems, spot something that is missing, or find something unclear, please hack up the wiki so that those following in your footsteps don’t hit the same issues.

-J