Author Archives: Jason Birch

About Jason Birch

Hi! As you can see on this blog, I've been something of a dyed-in-the-wool GeoGeek. More recently, my interests have been shifting towards general Local Government IT issues and communication, and topics like Open Data and Open Government. To find out more about me, check out this site's About page, or track me down on Google+

Nanaimo meet OpenID. OpenID meet Nanaimo.

Posted on by

How’s that for protocol?

If you’re anything like me, you probably use the password reset function on websites more often than the login function. This is a huge problem, both for security and for user experience.

The City of Nanaimo recognized that as useful as the city’s web applications are, requiring citizens to remember yet another password is not reasonable. Early this year the city did an initial analysis of OpenID, and Jeff Jacob–one of my colleagues–took on the task of developing the infrastructure to support OpenID and one of the first applications to take advantage of it. You can read the everyman’s description of Nanaimo’s OpenID initiative along with links to the OpenID-enabled services.

While the majority of users probably have an OpenID account already, it would not be responsible to require citizens to sign up for an external login service. A mix of forms-based and OpenID login capabilities may have been easier, but it just made more sense for Jeff to implement a city-specific OpenID provider using the DotNetOpenID open source library. This allows Nanaimo’s application login class to be more streamlined while presenting a consistent user experience, but more importantly it allows the city to act as a provider for third party / COTS web applications as these start supporting OpenID. Eventually Nanaimo citizens will be able to log into all city services using a single ID of their choice.

It is gratifying to see that during the City’s implementation phase many other organisations, such as the US Federal Government, have been embracing OpenID. Allowing citizens to access services using their own credentials is a key part of Nanaimo’s longstanding policy of providing easy access to the information residents and businesses need to live and do business here.

If you work for a local government and are interested in sharing information and/or code, please get in touch with Nanaimo’s IT department!

-J

P.S. As always, I am writing from a personal perspective. Opinions here are my own, and are not necessarily shared by my employer.

Google Place Pages Indexable? Not really…

Posted on by

Mashable hints that Google’s new Place Pages are potentially indexable. And indeed, they are. If you click on the Link button on any of the places pages, you can see that Google has given each of them a pseudo-URL:

455 Wallace Street, Nanaimo - Google Maps - Google Chrome

While these URLs are certainly interesting, on closer inspection you can see that they aren’t exactly following Google’s established best practices for publishing dynamic web content. They can be reached by any number of URLs, and Google does not use the Canonical meta tag:

http://maps.google.com/places/ca/nanaimo/wallace-st/455/-city-of-nanaimo-city-hall

http://maps.google.com/places/455/wallace-st/nanaimo/-nanaimo-city-hall

http://maps.google.com/places/nanaimo/-nanaimo-city-hall

Not only does this mess up indexing, but it means that there is no common URL for things like Google SideWiki to latch onto for aggregating the comments made on these pages. For instance, you can see a comment at this URL:

http://maps.google.com/places/ca/nanaimo/-city-hall (or here if you don’t have SideWiki)

But not at this one:

http://maps.google.com/places/ca/nanaimo/wallace-st/-city-of-nanaimo-city-hall

It appears that rather than treating each place or business to its own unique hierarchical identifier on the web, Google is instead just parsing the URL for search terms, turning slashes into commas and dashes into spaces (mostly – special case for business names).

This is unfortunate, because if this hierarchical system was in fact in place, then this series of URLs would be very cool and spatially related:

http://maps.google.com/places/ca/bc/nanaimo/wallace-st/-city-of-nanaimo-city-hall

http://maps.google.com/places/ca/bc/nanaimo/wallace-st

http://maps.google.com/places/ca/bc/nanaimo

http://maps.google.com/places/ca/bc

http://maps.google.com/places/ca

(most of these but the last give you what you’d expect, so I was initially fooled into thinking this was more intelligent than it looked for a while).

While there is lots of potential for Google Place Pages to be cool, as it stands it’s just a slight advancement on using mod_rewrite to turn your URL into parameters. As @ajturner said, they’re using the web, but not part of the web.

-J

Vancouver’s Open Data

Posted on by

Congratulations to the City of Vancouver on the launch of their Open Data Catalogue.

They have launched with what looks like a couple dozen datasets (including orthophotography and other GIS data), with a custom license agreement.

This is a great start, and I understand that the folks at Vancouver are working on pushing out more data sets as rapidly as possible. Make sure to take their survey if there is something in particular you are interested in.

-J

Live from FMEUC, it’s the Tim and Jason show!

Posted on by

OK, so better late than never. At the always-awesome FME User Conference, Tim Taylor and I did a short presentation on Nanaimo’s use of FME Server.

I think we did OK, but I definitely need to spend a bit more time polishing both my presentation and the slides next time.

Check out other great FME UC Videos on Safe’s user conference website. There is a lot of valuable videos, with case studies and technical presentations which will show you how your business processes could be improved by using FME.

As an aside, I count myself fortunate to live within driving distance to two of the best geospatial conferences in the world. In these times of tight budgets, I am incredibly grateful to be able to attend both the FME User Conference and GeoWeb.

-J

FWTools FTW … because GDAL FTW didn’t sound as cool!

Posted on by

I’ve received a bunch of compliments on the performance of the NanaimoMap MapGuide / Fusion application that the City of Nanaimo launched in beta last week.

There is a lot involved in making a web map perform, especially if you are not leveraging tile caching. One part of the story is hardware, and I’m lucky enough to share space on a dual quad-core machine with 4GB RAM and relatively fast disk. Another part is proper generalization of the vector data for display; no point in carrying sub-micron precision on a map that will generally be displayed at 1:500 or smaller. And of course, there’s MapGuide’s inherent speed when properly configured. This leaves out one of the most important parts though: raster data.

Raster data is big, brutish and hard to work with, and optimizing raster access is often one of the most important parts of delivering a successful web map. Users have come to expect “satellite” imagery on their web maps, and complain when it doesn’t perform as well as Google Maps. One of the best ways that I have found of flipping and folding raster data is Frank Warmerdam’s FWTools, which wraps GDAL and some other utilities in a single easy-to-use package.

My starting point consisted of:

  • 79 TIFF + Worldfile images, 10cm resolution, about 1.1GB each
  • 14 TIFF + Worldfile images, 30cm resolution, about 600MB each

So, I was working with about 100GB of images, none of which were optimized for web-based display, and which did not contain the spatial reference information that the FDO Raster Provider (also based on GDAL!) works best with.

The first thing I did was set up a batch process to optimize the individual images. This involved three steps:

1. Obtain a correct .prj file containing the WKT spatial reference information for my images. The easiest place for me to get this was SpatialReference.org, but you might just have one hanging around.

http://spatialreference.org/ref/epsg/26910/

2. Reprocess the image into a Tiled GeoTIFF, with no compression and a relatively large internal block size, and specifying the projection file obtained above. The caret (^) is the DOS line continuation character:

gdal_translate ^
-co "TILED=YES" ^
-co "PROFILE=GEOTIFF" ^
-co "INTERLEAVE=BAND" ^
-co "BLOCKXSIZE=512" ^
-co "BLOCKYSIZE=512" ^
-a_srs utm83-10.prj ^
infile.tif ^
outfile.tif

You can obtain more information on gdal_translate and the GeoTIFF options on the GDAL website. Depending on your source data and intended use, other values could be more appropriate, and you really should experiment.

3. Create internal pyramids in each image so that the entire image does not need to be fetched when zoomed out. This is one of the easiest performance gains you can get if you can afford the extra disk space.

gdaladdo -r gauss output.tif 2 4 8 16 32 64 128

Once this was done, I had a really decent set of fast images to work with, but these would only be appropriate to load at large scales when only one or a very few of the images need to be opened on each map view. For smaller scales, I needed to reduce the size of the images being processed, and also reduce the number of files being accessed on each fetch. I decided to go with a simple two-tier approach: Load the individual images at scales larger than some fixed value, and load a single overview image at scales smaller than that value.

The only problem was that I did not have an appropriate overview image. I wanted something that was relatively small, highly optimized, and which had white fill in its nodata areas. Fortunately GDAL and the awesome folks in the #gdal channel at freenode came to the rescue again, this time with four steps.

1. The first thing I needed to do was build a list of all of the images I wanted to have as part of the overview and feed these into the gdalbuildvrt command to build a single virtual image. You could do this manually, but I have the awesome GnuWin32 utilities installed so used these instead; they’re almost enough to make me not miss the days when I spent most of my time in Unix:

find images/ -name "*.tif" | xargs gdalbuildvrt -resolution highest all_images.vrt

2. Because I wanted a white background on my overviews, I then edited the all_images.vrt, adding a <NoDataValue/> section at the top of each of the three <VRTRasterBand /> sections:

<VRTRasterBand dataType="Byte" band="1">
<NoDataValue>255</NoDataValue>

3. The gdalinfo command gave me the dimensions of the virtual image, each of which I then divided iteratively to give me reasonable overview dimensions which I could feed into gdal_translate.

gdal_translate ^
-outsize 53120 14000 ^
-co "TILED=YES" ^
-co "PROFILE=GEOTIFF" ^
-co "INTERLEAVE=BAND" ^
-co "BLOCKXSIZE=512" ^
-co "BLOCKYSIZE=512" ^
all_images.vrt ^
all_images.tif

When this completed, I deleted the all_images.tif.aux.xml file because I did not want to carry the additional metadata that GDAL maintains in that file.

Careful with sizes here. If you’re using an application that supports it, you can specify the -CO “BIGTIFF=YES” option to generate files larger than 4GB, but you’re likely better off generating an intermediate level of aggregated and resampled tiles instead.

4. The final step was to once again generate internal pyramids to allow for better performance at small scales:

gdaladdo -r gauss all_images.tif 2 4 8 16 32 64 128

Once these two data sets were processed, I simply used MapGuide Maestro to make two raster data connections. For the first data connection, I added all of the individual TIFF images to a composite raster type, and Maestro generated a configuration document which allows MapGuide to know which image to access for a given extent. For the second layer, I just pointed to the overview GeoTiff. I then created layers for these, experimented until I found the scale where the overview image started looking pixelated, and set the layers’ view scale properties accordingly. There are some notes on working with rasters in the Maestro documentation.

More performance could probably be gained by having an intermediate level where the coverage area was aggregated into larger tiles before being combined into one large overview image, but for the initial launch this was deemed to have high enough performance.

On my production server, I’m lucky enough to have a fast, high-spindle-count RAID shelf dedicated to storing these uncompressed TIFFs, and they scream off the disk. My test server is VMWare-based, and disk performance and space are both at a premium. In this case, I still used the TIFF overview map, but at large scales I access a set of tiled MrSID files instead. This seemed like a decent compromise given the constraints, but did seem to thrash the CPU a bit.

GDAL was one of the first open source geospatial applications I tried (not counting GRASS and MOSS) and is constantly coming in handy, whether I’m reprojecting, adding spatial reference information to images, or converting between formats.

Thanks to hobu (Howard Butler), FrankW (Frank Warmerdam) and EvenR (Even Rouault) from the #gdal IRC channel on freenode for helping me work my way to this solution. Amazing support!

-J