1. Theming, with ColorBrewer Suport

Being able to theme maps based on data distribution is a basic mapping function, and this release of Maestro delivers. Adding support for ColorBrewer means that you can be confident that your colour scheme is visually distinct and cartographically appropriate for the message that you are conveying. The Maestro UI automatically constrains the colour choices based on the underlying data categories:

2. Expression Editor

MapGuide and the underlying FDO data providers support a powerful expression language, and previously you were on your own to write valid expressions. Thanks to Jackie Ng, MapGuide Maestro was able to use the same expression editor that FDO Toolbox is using, giving expression-completion, valid value extraction, and more:

Check out Jackie’s posts on the FDO Expression Editor and the follow-up where he talks about the addition of value auto-completion.

3. Resource Validation

The Maestro resource validator walks from application to map to layers to data, warning if it detects any common errors such as broken references or potential performance issues like unmatched projections. This is an invaluable tool for troubleshooting problems in your maps:

4. Improved XML Editor

While Maestro’s GUI is great for most purposes, there are times when you need to access the full power of MapGuide XML configuration syntax, like when you’re editing complex XML-based stylization elements. Maestro makes it easy for you to edit any resource as XML simply by right-clicking on it and choosing “Edit as xml”. Maestro 2.0 comes with many improvements to the XML editor, including validation against the schema, cursor position (important when tracking down errors), and the ability to attach arbitrary files to the current resource, which is critical when making bitmap-based symbols:

5. Profiling

Profiling allows you to easily find the performance bottlenecks on a given map, or quickly determine whether changes to theming or other items are having an impact on performance:

6. Package Management

MapGuide packages are a zipped export of the DBXML repository including XML resources, associated binary files, and a manifest. Typically these are managed on the server, but Maestro allows you to create, edit, and load these files via the GUI. This can be really handy when you’re migrating changes between servers. One of my favourite tricks is to export both test and prod as packages, unzip them, and compare using Beyond Compare (not free, but worth every penny).

7. Custom Resource Templates

Any time you are creating a new data source, layer definition, map definition, etc, you are basically just creating a new XML document. If you find that you are always performing certain steps as part of creating these resources, you can create your own custom resource types with customized versions of these XML documents. For instance, I like to start layer creation using the version 1.3 of the LayerDefinition schema, with a default scale range of 1:1->1:500,000, and none of the feature types being displayed. All I had to do was create a new LayerDefinition with these settings, save it to the MapGuideMaestro\Templates directory with a name like “Nanaimo Layer.xml”, and it shows up when I want to create a new resource:

8. Duplicate Resource

This might not seem like a big deal, but when creating dozens of similar layers it can be a huge timesaver. Simply right-click on any resource and choose “Duplicate” and a copy of that resource is created and ready for you to customize:

9. Colour-Coded Resource Tree

When editing many resources at once, things can get a bit confusing. Crispin Hoult from 1Spatial contributed a feature that colour-codes currently open resources in green, and resources that have unsaved changes in pink. The currently active resource comes up in a darker shade. This feature makes it much easier to keep track of what’s going on in your work area, and is surprisingly useful:

10. General Usability

OK, maybe this doesn’t count as a feature, but a LOT of thought has been put into how various user interactions work, and countless small refinements have been made. A few examples:

• inserting a new layer into a map definition when you have a group selected inserts the layer into that group, and places the layer into the overall draw order right after the bottom-most layer in that group
• you can now right-click on any resource and copy its ID (like Library://Nanaimo/Data/MyFile.FeatureSource) to the clipboard, which can be incredibly useful when writing code to access resources
• you can multi-select many layers for insertion into a MapDefinition at once
• maestro keeps track of references when you rename or move resources, prompting you for whether you want dependent resources updated

All of the little enhancements in this release added together have saved me hours of work (I’ve been using the pre-release versions for a few months).

All-in-all this is a very impressive release, with countless new features and enhancements to existing functionality. Give it a spin, and I’m sure you’ll turn up your own favourites!

Thanks Kenneth, and great work.

-J

There is a lot involved in making a web map perform, especially if you are not leveraging tile caching. One part of the story is hardware, and I’m lucky enough to share space on a dual quad-core machine with 4GB RAM and relatively fast disk. Another part is proper generalization of the vector data for display; no point in carrying sub-micron precision on a map that will generally be displayed at 1:500 or smaller. And of course, there’s MapGuide’s inherent speed when properly configured. This leaves out one of the most important parts though: raster data.

Raster data is big, brutish and hard to work with, and optimizing raster access is often one of the most important parts of delivering a successful web map. Users have come to expect “satellite” imagery on their web maps, and complain when it doesn’t perform as well as Google Maps. One of the best ways that I have found of flipping and folding raster data is Frank Warmerdam’s FWTools, which wraps GDAL and some other utilities in a single easy-to-use package.

My starting point consisted of:

• 79 TIFF + Worldfile images, 10cm resolution, about 1.1GB each
• 14 TIFF + Worldfile images, 30cm resolution, about 600MB each

So, I was working with about 100GB of images, none of which were optimized for web-based display, and which did not contain the spatial reference information that the FDO Raster Provider (also based on GDAL!) works best with.

The first thing I did was set up a batch process to optimize the individual images. This involved three steps:

1. Obtain a correct .prj file containing the WKT spatial reference information for my images. The easiest place for me to get this was SpatialReference.org, but you might just have one hanging around.

http://spatialreference.org/ref/epsg/26910/

2. Reprocess the image into a Tiled GeoTIFF, with no compression and a relatively large internal block size, and specifying the projection file obtained above. The caret (^) is the DOS line continuation character:

gdal_translate ^
-co "TILED=YES" ^
-co "PROFILE=GEOTIFF" ^
-co "INTERLEAVE=BAND" ^
-co "BLOCKXSIZE=512" ^
-co "BLOCKYSIZE=512" ^
-a_srs utm83-10.prj ^
infile.tif ^
outfile.tif


You can obtain more information on gdal_translate and the GeoTIFF options on the GDAL website. Depending on your source data and intended use, other values could be more appropriate, and you really should experiment.

3. Create internal pyramids in each image so that the entire image does not need to be fetched when zoomed out. This is one of the easiest performance gains you can get if you can afford the extra disk space.

gdaladdo -r gauss output.tif 2 4 8 16 32 64 128


Once this was done, I had a really decent set of fast images to work with, but these would only be appropriate to load at large scales when only one or a very few of the images need to be opened on each map view. For smaller scales, I needed to reduce the size of the images being processed, and also reduce the number of files being accessed on each fetch. I decided to go with a simple two-tier approach: Load the individual images at scales larger than some fixed value, and load a single overview image at scales smaller than that value.

The only problem was that I did not have an appropriate overview image. I wanted something that was relatively small, highly optimized, and which had white fill in its nodata areas. Fortunately GDAL and the awesome folks in the #gdal channel at freenode came to the rescue again, this time with four steps.

1. The first thing I needed to do was build a list of all of the images I wanted to have as part of the overview and feed these into the gdalbuildvrt command to build a single virtual image. You could do this manually, but I have the awesome GnuWin32 utilities installed so used these instead; they’re almost enough to make me not miss the days when I spent most of my time in Unix:

find images/ -name "*.tif" | xargs gdalbuildvrt -resolution highest all_images.vrt


2. Because I wanted a white background on my overviews, I then edited the all_images.vrt, adding a <NoDataValue/> section at the top of each of the three <VRTRasterBand /> sections:

<VRTRasterBand dataType="Byte" band="1">
<NoDataValue>255</NoDataValue>


3. The gdalinfo command gave me the dimensions of the virtual image, each of which I then divided iteratively to give me reasonable overview dimensions which I could feed into gdal_translate.

gdal_translate ^
-outsize 53120 14000 ^
-co "TILED=YES" ^
-co "PROFILE=GEOTIFF" ^
-co "INTERLEAVE=BAND" ^
-co "BLOCKXSIZE=512" ^
-co "BLOCKYSIZE=512" ^
all_images.vrt ^
all_images.tif


When this completed, I deleted the all_images.tif.aux.xml file because I did not want to carry the additional metadata that GDAL maintains in that file.

Careful with sizes here. If you’re using an application that supports it, you can specify the -CO “BIGTIFF=YES” option to generate files larger than 4GB, but you’re likely better off generating an intermediate level of aggregated and resampled tiles instead.

4. The final step was to once again generate internal pyramids to allow for better performance at small scales:

gdaladdo -r gauss all_images.tif 2 4 8 16 32 64 128


Once these two data sets were processed, I simply used MapGuide Maestro to make two raster data connections. For the first data connection, I added all of the individual TIFF images to a composite raster type, and Maestro generated a configuration document which allows MapGuide to know which image to access for a given extent. For the second layer, I just pointed to the overview GeoTiff. I then created layers for these, experimented until I found the scale where the overview image started looking pixelated, and set the layers’ view scale properties accordingly. There are some notes on working with rasters in the Maestro documentation.

More performance could probably be gained by having an intermediate level where the coverage area was aggregated into larger tiles before being combined into one large overview image, but for the initial launch this was deemed to have high enough performance.

On my production server, I’m lucky enough to have a fast, high-spindle-count RAID shelf dedicated to storing these uncompressed TIFFs, and they scream off the disk. My test server is VMWare-based, and disk performance and space are both at a premium. In this case, I still used the TIFF overview map, but at large scales I access a set of tiled MrSID files instead. This seemed like a decent compromise given the constraints, but did seem to thrash the CPU a bit.

GDAL was one of the first open source geospatial applications I tried (not counting GRASS and MOSS) and is constantly coming in handy, whether I’m reprojecting, adding spatial reference information to images, or converting between formats.

Thanks to hobu (Howard Butler), FrankW (Frank Warmerdam) and EvenR (Even Rouault) from the #gdal IRC channel on freenode for helping me work my way to this solution. Amazing support!

-J

If you’ve got a few minutes to play with it, please join us here:

NanaimoMap Beta

It’s in beta because of the issues that will likely be shaken out by more widespread use, and because we have not yet built out the layers and search functionality required to match our current MapGuide 6.5 ActiveX-based mapping portal CityMap. This will be completed before the end of the year.

Thanks!

-J

P.S. This application was developed by DM Solutions Group. We’re running Fusion 1.1 with the latest test build (r4114) of MapGuide. We wouldn’t have been able to launch–even in beta–without some last minute fixes by Trevor Wekel of OTX Systems and Haris Kurtagic of SL King. From a personal perspective, these guys are both amazing to work with, moderately priced for the value they offer, and are great resources if you’re stuck with a problem in MapGuide core that you can’t fix on your own. As always, the opinions offered on this blog are my own, not necessarily those of my employer.

one of my projects just went live – http://www.exploreaustralia.net.au/

This is a pretty cool implementation. I believe that the maps on this site (e.g. Victoria) use OpenLayers, hitting an S3-hosted tile set generated by MapGuide Open Source, and show off some of MapGuide’s advanced renedering capabilities.

-J

Jackie Ng has a better description of this than I could write.

Apart from the improved scalability and stability of this release, I’m most excited about the WiX-based open source MSI installer that I got to work on alongside Jackie and Kenneth. Huge learning curve, but not having to count on Autodesk resources to build new installers should allow us to push new releases as needed. It also means that someone who understands the installer can easily create custom deployments for internal use.

Anyway, please give this beta a try. If you notice anything it’s not doing right, please let us know.

-J

In the following dialog, either leave the default of all checked, or choose to only pre-render certain zoom levels of your data, the choice is yours (probably best to start with the smallest scale – in this case 100,000):

Then click "Build tiles now" and sit back and watch the tiles cook:

Of course, this isn’t practical for a production environment where you would want to batch this operation.  That’s where the second option comes in; clicking on "Save as script" generates a Windows batch file similar to the following that you can script to your heart’s desire:

"C:Program FilesOSGeoMapGuide MaestroMgCookerCommandline.exe" batch
   --mapdefinitions="Library://Samples/Sheboygan/MapsTiled/Sheboygan.MapDefinition"
   --basegroups="Base Layer Group"
   --scaleindex=7
   --mapagent="http://localhost:8008/mapguide/mapagent/mapagent.fcgi"

I have no idea if the MgCookerCommandLine.exe works under Mono, but given that the rest of Maestro seems to work fairly well there I would not be surprised.

Great job Kenneth; this is something we have been sorely missing since MapGuide 1.2, and it’s great to see it as part of Maestro!

-J

By linking to open source components such as MapGuide, FDO, POCO, CTemplate, and libkml, the MapGuide REST framework is very simple and powerful.  With absolutely no programming, a site administrator can build out customised HTML and KML representations of their data, and enable access to default GeoJSON and FDO-based XML representations.  Pair these representations with static HTML pages that expose "service metadata", and you have data that is easily crawlable by search engine spiders, and easily understood by anyone wanting to access the data for other applications.  Enough talk; let’s go over some examples from the City of Nanaimo’s live web site.

Data URI:  http://maps.nanaimo.ca/data/ 

This is where all of the City’s RESTful GeoData will be exposed.  Currently, there is a single resource type exposed for Property information, but more is on the way and will be interlinked when it makes sense.  The index page for this directory is a static HTML page pointing to various features of the Property resource type.

Property Resources:  http://maps.nanaimo.ca/data/property/ 

This currently displays a page that allows users to search by one of three attributes by HTML forms, and contains links to OpenSearch (with draft Geo) documents that define some of the other search and representation capabilities.

OpenSearch link:

<link
  rel="search"
  type="application/opensearchdescription+xml"
  title="Properties by Street"
  href="/data/property/search_by_street.xml"
>

As well, there is a link to a Street Index which acts as a form of sitemap, providing web crawlers with an equal-length trip to every property in the City.

Example Property HTML Representation:  http://maps.nanaimo.ca/data/property/123170.html

As much as this looks like a regular web page, it is cram-packed with features that make it distinguishable as a geospatial resource.  Most visibly, it displays an image of the property’s shape in context, and provides a link to the KML representation of that property.  Digging a little bit deeper, the page header contains links to alternate representations:

<link
  rel="alternate"
  type="application/vnd.google-earth.kml+xml"
  title="KML representation of 2143 AARON WAY"
  href="http://maps.nanaimo.ca/data/property/123170.kml" >
<link
  rel="alternate"
  type="application/json"
  title="GeoJSON representation of 2143 AARON WAY"
  href="http://maps.nanaimo.ca/data/property/123170.json" >
<link
  rel="alternate"
  type="text/xml"
  title="XML representation of 2143 AARON WAY"
  href="http://maps.nanaimo.ca/data/property/123170.xml" >

and some simple geotags:

<meta name="ICBM" content="49.18882, -123.9785">
<meta name="geo.region" content="CA-BC">
<meta name="geo.placename" content="Nanaimo">
<meta name="geo.position" content="49.18882;-123.9785"> 

Finally, within the page content itself, you will find both adr and geo microformats.

<div class="adr">
  <span class="extended-address"></span>
  <span class="street-address">2143 AARON WAY</span>,
  <span class="locality">NANAIMO</span>,
  <span class="region">BC</span>
  <span class="country-name" style="display:none;">CANADA</span>
</div>

<div class="geo">
  <span class="latitude">49.18882</span>,
  <span class="longitude">-123.9785</span>
</div>

Example Property KML Representation:  http://maps.nanaimo.ca/data/property/123170.kml

The KML representation uses Atom Author and Link elements to define the publisher of the data, and to store the KML’s URI and that of its alternate HTML representation.  It also uses a combination of ExtendedData and BalloonStyle entities to separate data from presentation, though I have also included a link to the HTML representation in the Description entity for earlier clients.  Finally, it uses external Style and Schema definitions to reduce the size of the file and to maintain consistency across the site.  I’d encourage you to view the source of the KML file to see how these elements work together.

Example JSON Representation:  http://maps.nanaimo.ca/data/property/123170.json

Example XML Representation:  http://maps.nanaimo.ca/data/property/123170.xml

Example PNG Representation:  http://maps.nanaimo.ca/data/property/123170.png

The native GeoJSON and FDO XML representations are provided by the extension.  No customization is possible, which ensures a consistent implementation from site to site.  The PNG representation is also native (not template), but allows you to specify a MapGuide map definition and a layer for selection.

"Endpoints" (because I’m not sure what to call them):

http://maps.nanaimo.ca/data/property/index.kml

This is a static file that allows KML clients to interact with the property information dynamically through View Based Refresh.  Basically, this file tells Google Earth to request the .kmz endpoint below using a BBOX element that will be used to return only features in view, or a warning if the viewport is too large.

http://maps.nanaimo.ca/data/property/.html

http://maps.nanaimo.ca/data/property/.kmz

http://maps.nanaimo.ca/data/property/.json

http://maps.nanaimo.ca/data/property/.xml

These allow users to perform queries on the data.  There are quite a few variations possible, but they basically support standard FDO filters, bbox, count, and start GET parameters.  There are also some funky parameter patterns that allow for easy searching from HTML forms, in the format: filter_[0-9]_(and|or)_(equal|like|likeleft|likeright|lt|gt)_attributename

http://maps.nanaimo.ca/data/property/.schema

This is a special "endpoint" that allows a smart client to determine the data schema for the XML representations of the feature.

Applications

Now, providing read-only representations of these features is pretty cool on its own, and that is all that the City of Nanaimo is prepared to do at this point.  However, the extension can be configured to allow full RESTful data access:  GET, POST, PUT, DELETE.  Combined with a yet-to-be-released FDO REST provider, users of applications that implement FDO (such as AutoCAD Map 3D, Safe Software‘s FME, 1Spatial‘s MapRelate and Radius Studio, and MapGuide) will be able to read, write, update, and delete data from this extension the same way as they would from local data stores.  I was experimenting with this against an internal server using Map 3D, and it was pretty cool to be able to edit data directly from a web site.

Applications that implement GeoJSON, such as Safe Software’s FME can also take advantage of this format to read and (untested) write to the exposed data.

The GeoJSON representation has a considerable upside for light-weight mobile data editing, for instance in an OpenLayers application running inside Google Chrome (or another Gears-enabled browser) and for minimal-packet data access from mobile phone applications.

All of these are really cool applications from a geospatial professional standpoint, but they don’t really address the primary goal of the original requirement:  enabling easier access to spatial information by residents.  That’s OK though, just by making these representations available and well-linked, the search engines will pick them up and do this part of the work for me. 

This site has been active for less than two weeks, and Google is already ranking some of the property reports first in the results for generic searches within British Columbia and specific searches elsewhere.  For instance, when I search from home for "881 Hunter St" (one of the properties Google has indexed) it comes up as number 1.  When Haris searches from Slovenia, that isn’t the case, but if he searches for "881 Hunter St, Nanaimo, BC" then it does.  Most municipalities have high authority for their location name, and this is a great way of taking advantage of that authority to provide a higher level of service to our residents.

Weaknesses

The current implementation has a few issues.  Unfortunately these will require a substantial refactoring to resolve, so we’re looking at another month or so before we have a public-ready build of the code. Some of these problems are as follows:

• HTTP status codes are often not correct.  For instance, a search that returns a single feature should redirect to that representation rather than displaying the single feature under the search representation.
• Fine control over methods is required, such as enabling/disabling specific methods, requiring SSL, requiring certain group access, etc.
• Existing configuration system grew in odd ways as the code was modified, and needs to be de-Frankenstein-ed
• More control over URI patterns needs to be extended to users, for instance to allow the creation of -edit representations that return pre-populated HTML forms.
• Need revision numbering or data checksum to reduce chance of overlapping writes.
• Need parameter/http header overrides for method (to deal with firewalls) and format (to allow selection from HTML forms)
• Other miscellaneous issues.

Updates

Based on feedback, Haris has enabled JSONP for the .json representations. Just add the argument _callback to the URI, with the name of your desired function as the value. e.g.: http://maps.nanaimo.ca/data/property/.json?filter_1_and_equal_street=ROSSTOWN+ROAD&_callback=rosstown

Call for Feedback

If you have a chance to look at the current implementation and see any areas where it could be improved, we’re really interested in making changes required to meet REST best practices.  We’ve been trying to pay attention to current guidance, but without a real implementation it has been hard to truly understand what is needed.

Thanks!

-J

http://vancouver.ca/humanresources/jobs/09-0012O2P.htm

Vancouver is a beautiful place and the people I know in that group are all good folks.

-J

If you’re using MapGuide Open Source 2.0 (2.0.2 recommended) and are not happy with the default blue selection colour, you can easily change it on a per-install basis by modifying the mapviewerajax template file. This is located at:

(INSTALLDIR)\WebServerExtensions\www\viewerfiles\ajaxmappane.templ

You will need to find the function called RequestMapImage, which includes a line that looks like this:


url = webAgent +
"?OPERATION=GETDYNAMICMAPOVERLAYIMAGE&FORMAT=PNG&VERSION=1.0.0&SESSION="
+ sessionId + "&MAPNAME=" + encodeComponent(mapName)
+ "&SEQ=" + Math.random();


You will need to modify this to change the VERSION to 2.0.0, and add the BEHAVIOR and SELECTIONCOLOR parameters:


url = webAgent +
"?OPERATION=GETDYNAMICMAPOVERLAYIMAGE&FORMAT=PNG&VERSION=2.0.0&SESSION="
+ sessionId + "&MAPNAME=" + encodeComponent(mapName)
+ "&SEQ=" + Math.random() + "&BEHAVIOR=7&SELECTIONCOLOR=FF5300FF";


Before:

After:

The BEHAVIOR parameter is a bitmask that controls what is rendered, and is described in RFC 38. The SELECTIONCOLOR parameter is a hex string in RGBA format, or its integer equivalent. Note for geeks: the A (opacity) portion of the string is ignored; the server code masks this out of the passed value, and then adds 200 (C8) as the line opacity and 160 (A0) as the fill opacity.

Note, the mapagent call allows you the flexibility to add more complicated behaviour to your viewer, such as changing the selection colour dynamically based on the map name, but I’ll leave that as an exercise for the reader (mostly because I’m lazy and use Fusion anyway and wouldn’t benefit from the work).

-J

-J