I think we did OK, but I definitely need to spend a bit more time polishing both my presentation and the slides next time.

Check out other great FME UC Videos on Safe’s user conference website. There is a lot of valuable videos, with case studies and technical presentations which will show you how your business processes could be improved by using FME.

As an aside, I count myself fortunate to live within driving distance to two of the best geospatial conferences in the world. In these times of tight budgets, I am incredibly grateful to be able to attend both the FME User Conference and GeoWeb.

-J

Safe Software’s FME Workbench has the best user interface that I have ever used.  Hands down.  Every time I have to use something with similar functionality such as Microsoft’s SSIS, or even the time I wasted a whole weekend trying to get an open source ETL tool to do what I wanted it to do, I find myself wishing I could be working in FME instead.  It was great when I started using it six years ago, and it’s been getting better with every release.

Although Safe has definitely been busy with FME 2009 (get the whole story on fmepedia’s What’s Fine in 2009 page), it almost seems tame compared to some of the major changes in the last couple releases.  However, behind the scenes there was a major UI refresh.  Three of the new UI features are among the things I think are coolest about this latest release.

First, they created a really intuitive “default settings” interface.  As far as I can remember, previous releases didn’t even have default settings for transformers, and dataset defaults were set through a dialog box hidden in the configuration UI.  It was so much trouble to use that I don’t imagine many people were setting default values unless they had something that REALLY annoyed them:

For 2009, rather than hiding these values deep in a configuration dialog somewhere, they let users set the default as they are working with the data stores and with individual transformers.  This is something I’d love to see more applications do.  Here’s an example:

And another of setting transformer defaults:

Couldn’t be easier.  I predict that many users–even ones that didn’t know this functionality was available in previous releases–will be using this.  My only suggestion would be the addition of a “Save As …” option, so we could have two or three sets of default values for each item.  For instance, I want to have the values for both my testing and production database configurations available.  And I want to have my favourite “Smoothing” and “Thinning” profiles for the Generalizer.

The second feature that I really like about the new UI is Quick Connect.  Mark describes this over at the FME Evangelist blog better than I ever could… make sure to watch his cheesy movie for some time-saving tips. 

I’ve known that I’ve had one of the problems that this feature solves ever since I’ve been using FME.  Basically, it was very hard to connect two transformers that were so far apart that they neither fit on the screen nor were clickable when zoomed-out.  I made a feature request for this, but my potential solution was really clunky.  With this new feature, connecting two trans-workspace transformers is quick, easy, and intuitive.

The final feature that I really like is the ability to edit source feature attributes.  Now normally, source feature attributes don’t change, and even if they do FME provides an effective wizard that allows you to refresh the source feature definitions from their original source without manual intervention.  Where this really comes in handy is when you use the optional SELECT Statement parameter to join to a secondary table:

Now, instead of jumping through lots of hoops to have the joined attributes show up in your workspace, you can enable the option that allows you to redefine the source feature’s attributes:

There are many other nice UI changes in this release, and some of the non-UI changes are really great too.  Things that I may actually use include new XML features (XQuery and XSLT transformers) and new raster features such as RasterExpressionEvaluator (not quite DOCELL yet, but getting there), support for BigTIFF, and the ability to visualize rasters in the FME Viewer.

I’m sure that there’s something for everyone in the feature list; make sure to check out FME 2009 when you get a chance.

-J

p.s. I had to change my theme as part of my WP 2.7 upgrade, and while I was at it I hacked WordPress so that XMLRPC would work properly with SSL and tried out Windows Live Writer (with a twitter notify plugin). Seems to have worked OK so far…

Over the past month, developers at Safe Software and the author of the FDO SQLite provider have put some time into ensuring that the SQLite provider will work properly with FME 2009. Reading worked fine out of the box, but writing required a bit of effort. FME needed datastore creation and schema writing added to their generic FDO writer, and the FDO SQLite provider needed to account for the way that FME writes to multiple schemas.

Here’s how you can take advantage of this provider in FME (and in other FDO 3.3 consumers, such as MapGuide Open Source 2.0):

• Download the unofficial binaries for the SQLite provider from my site
• Open this zipfile and copy the SQLiteProvider.dll file into your FDO directory (default c:\Program Files\FME\plugins\fdo\)
• Make a backup of the providers.xml file in that directory, and then edit the original, adding the contents of the sqlite_provider_entry.xml file in an appropriate location.

Once this installed, writing to SQLite from within FME is dead easy…

1. Add new FDO Destination Dataset:

2. Go to Settings and specify OSGeo.SQLite.3.3 as the provider name:

3: Specify the filename you want to write to:

4. Optionally, set a spatial reference system, and click on OK:

That’s it; now you can start adding tables to your SQLite file as you would any other destination dataset in FME!

As far as I know, Safe will not be distributing the SQLite provider directly with FME 2009 (it’s still in beta) primarily because the provider is not officially being released for FDO 3.3, and partially because the provider is still under heavy development. Fear not, though. I am building this provider against the 3.3 branch as often as necessary, and will post binaries as I do.

The relative ease with which this format was supported by FME can be attributed to Safe’s foresight in exposing FDO directly, rather than just using it behind-the-scenes in their SDF3 writer. They also allow FME to act as an FDO provider, which enables users of products that use FDO for their data layer (such as AutoCAD Map 3D) to access the full range of formats that FME supports.

-J

It is also the best place to corner an FME developer. I managed to grab Tom Weir, Safe’s KML guru, and go over some of the changes in KML support with FME 2008. During a presentation on the first day of the conference I had included an “easter egg” where I spoke about how to enable active mouse-overs in KML using FME. To my chagrin, after a couple minutes with Tom I realized that my advice was not exactly best practice, and with FME2008 becomes downright ridiculous.

Here’s the before shot from my slide deck (zipped workspace):

And the after shot once I applied what Tom showed me (zipped workspace):

Obviously, the FME 2008 press release should read: “KML Support in FME: Now with 50% less fat!”

KML FME has been generating multi-geometry for information points for quite some time, so that cuts most of the data wrangling out to generate the info point and merge the features into multi-geometry. And FME 2008 will automatically generate StyleMap elements for you if you follow a couple sneaky tricks.

First, when you create each KMLStyler, set its name to the style ID you want it to receive:

And second, on your geometry set the kml_target_style_normal and kml_target_style_highlight attributes to the IDs that you created in the KMLStylers:

That gets my embarrassment out of the way, but doesn’t even begin to touch on the extent of KML 2.2 support in FME 2008. Another issue that I have written about is extended data or schema support, and I am happy to say that FME deals with this. Attributes are stored in your output KML as extended data by default, and it is easy to generate a BalloonStyle template. Here’s my first take on this support, which does a great job of separating data from presentation (zipped workspace):

And a quick look at the new basic editor which is included in FME and used for modifying BalloonStyle templates:

Which gives us this KML output (source).

There is a going to be a lot more to the KML 2.2 support in FME 2008, including generation of image pyramids for PhotoOverlays, but I’ll leave it to you to explore those on your own.

-J

I am really looking forward to this conference, and am confident that it will measure up to the outstanding event that Dale, Don, and their team put on in 2006. The technical quality and value for users of FME is truly outstanding, and sets this conference apart from many other industry events.

I’m getting into town on Wednesday night and staying at the conference hotel. Drop me a line if you’re going to be there and want to hang out.

-J

FME is usually quick to support new formats (like KML and GeoRSS) but this time Safe has surpassed themselves, getting early JSON and GeoJSON support into their betas before the GeoJSON specification has reached a release version. Tonight I took some time out to play with this new support.

The basis of JSON support in FME is provided by two new read/write formats: JSON and GeoJSON. These new formats are augmented by two new transformers: JSONExploder and JSONExtractor. To get started, I’m going to show you how to extract data from a JSON source I happen to have lying around (it’s publicly available at Yahoo Pipes), which looks like this:

When you first import this data source into FME, it is imported with the top level of attributes broken out. In this case, Pipes returns a top-level object with several attributes (link, description, etc) that you can see in my test workspace below:

Now, this isn’t much use, because my features are hidden inside the “items” attribute. In order to get them out, I first need to explode my single object into multiple features. The new JSONExploder transformer comes to the rescue here:

Now, I have a unique feature for each of my feed items, but I really want some of the nested attributes. In particular, I want the description from the root of the item, and the nested y:location["lat"] and y:location["lon"] attributes. The JSONExtractor makes it easy to pull these out into new attributes:

And once adding a couple more of these, each of my features has some nice attributes attached to it, which I could then turn into points if I wanted:

OK, so that’s kinda cool from a straight ETL standpoint. I can take in JSON, mess with it, and then pump it out into whatever format I want. But the fun stuff is when you start getting into GeoJSON. Fortunately there are a couple early adopters, Christopher Schmidt and Howard Butler, who gave me some feeds to play around with. The first of these comes from Christopher’s super-flexible FeatureServer application (check it out, it’s open source):

The features displayed on this OpenLayers map can be easily downloaded from FeatureServer in GeoJSON format (or KML, or GeoRSS, or whatever). The URL for the GeoJSON representation is:

http://featureserver.org/featureserver.cgi/scribble?format=geojson

Pulling this into FME is as simple as creating a new FME data source, and specifying the URL:

As you can see, you can then treat this data like any other spatial data source:

Now, for a final example… Howard has a GeoJSON resource collection of counties in Iowa, accessible in a pattern something like this:

http://geoservices.hobu.biz/political/json/johnson

Now we could take this feature, in its source projection of UTM Zone 15N NAD83, but Howard’s put together a really nifty (non-commercial use only, unless you want to pay Howard some $$$) JSON-based web processing projection service. Not only that, but he’s also made it smart enough to interpret projections referenced locally, but also from the oh-so-cool (and built as a collaborative effort between Christopher and Howard) SpatialReference.org. So, all you need to do is feed it the URL of your source data, the url of your source CRS (http://spatialreference.org/ref/epsg/26915/) and the url of your destination CRS (http://spatialreference.org/ref/epsg/4326/). Like so:

http://geoservices.hobu.biz/project/?url=%22http://geoservices.hobu.biz/political/json/johnson%22&inref=http://spatialreference.org/ref/epsg/26915/proj4/&outref=http://spatialreference.org/ref/epsg/4326/proj4/

And, as just another link in the dynamic web chain, FME can read this transformed JSON feature:

Now, for desktop FME users, this gives us “Pipes on Steroids”: all the mashup flexibility of Yahoo Pipes, with the huge format support and rich processing model of FME. As cool as this is, I think the real power will be seen whenever Safe integrates this functionality into their Server product. It will allow them to play well on both the “enterprise” traditional GML/WFS/etc level and on the neogeography JSON/GeoRSS/KML mashup level with a single product from a single (or multiple if you want) data source. For organisations that want turn-key interoperability solutions, FME Server is going to rock your world.

-J

P.S. I’m thinking about getting a personalized plate that says GEO JSN :)

In this case, it was particularly painful. I’m really interested in GeoRSS because I see the potential that it has for increasing the level of information my municipality can deliver to its residents. For instance, we could offer insight into current housing starts with a feed of new construction building permits.

For me, the beauty of GeoRSS is that it has value even without the maps. With a single format, I can publish information that our residents can view in their favourite feed reader, while allowing more sophisticated users to benefit from the spatial information.

As you can see from the following screenshots of a GeoRSS file displayed in Google Reader and Google Maps, I’ve finally found a little time to play around with the GeoRSS support in FME. You can click on the second image to see how it works. Keep in mind that the data is static, and not guaranteed to be accurate for any use:

Until now, much of the GeoRSS that I’ve had a chance to look at has been point-based, so it’s nice to have a way of generating some more complex elements in Simple and GML varieties and, now that Google Maps supports GeoRSS, an easy way to visualise it.

So anyway, how did I generate this GeoRSS file? It was pretty easy, and I’ll take you through the steps.

First, I had to set some basic parameters for the feed in FME. This image shows the basic workflow, from create a single feature with no geometry, through to writing the feature out to the “Feed” output:

Inside of the AttributeCreator, I’m setting some basic values for the feed as a whole:

Pretty simple, eh?

Next, I pull in some data from our property records database, and merge it with an Oracle Locator spatial table which holds the City’s parcel base:

And finally, I set the same the entry-specific GeoRSS parameters, this time using attributes of the building permit records instead of static text to fill in the blanks:

And that’s all she wrote. I still have to find time to fine-tune some of the attributes (such as dates and link), get some approvals, and work out the logistics of getting the data out to our external server nightly. It sure is nice that the technical part is so easy though.

If you deal with spatial data on a regular basis and haven’t evaluated FME, you’re doing yourself a disservice. Pick up the latest beta from their site (for GeoRSS support), request an evaluation license, and play around a bit. Everyone’s usage pattern is different, but I was able to justify the cost of my initial license and training with the time it saved over about six months. Safe Software deserves a lot of credit. Apart from the usefulness of FME, their support is incredible. They are the only company that I have ever dealt with where I felt I was getting real value from the software maintenance plan.

And, in case you’re wondering, they didn’t buy me a nice laptop or a 3D mouse to use while writing this. I’m just a satisfied customer :)

-J

There’s something to be said for blogging when something’s fresh in your mind. (Sorry Glenn, not even close to real-time on this one) And no, it’s not because I had too many Crown & Cokes. After a few days away, my family needed more attention than my mistress (though her big CRT was glowing seductively) and time plays some interesting tricks with memory. I’ll do my best to give you the highlights though. If you haven’t read about Day One or checked out the conference home page yet, do that first. One thing that I do remember is giving Matt a workspace that will likely give him nightmares, and make him wander around muttering about crazy users.

The morning featured Don and Dale giving a “Road Ahead” presentation in their casual funny style, complete with some really bad puns. One of the most important items that I got out of this were their plans to deal with the next hardware tidal wave with support for 64 bit operating systems and multi-process capabilities to take advantage of the soon to be ubiquitous multi-core CPUs. I wasn’t entirely clear on how the multi-process support was going to be implemented, but it sounded similar to the way that Apache sets up multiple children to listen and respond to requests. As well, there were plans for additional formats as well as improved support for existing formats. One of the most exciting items here was GeoRSS support. I know that this might not sound like a lot, but imagine using GeoRSS as a web-based data synchronisation tool (insert/update/delete). Yes, I suppose WFS and Filter could be used in the same way, but GeoRSS is new and trendy. :)

Rich Geometry is another important area, as it allows for the preservation of nasty items like a linestring with an embedded arc. This is a godsend for anyone that has to deal with CAD data on a regular basis. I spoke about Python improvements in the previous article, but I just noticed that there’s a recent story related to this on the Safe blog: Vancouver Python Workshop and FME. Raster support has seen an order-of-magnitude improvement in performance as well; a projection task was shown that was reduced from over an hour to less than two minutes in processing time. This is great news. Now I can put in some unreasonable requests for improvements in the raster processing capabilities. First I’ll have to apologize though. I interrupted the presentation and made a rather nasty comment about one of the 3rd party integration products that, while there are strong reasons for implementing it, doesn’t hold a candle to FME Workbench for usability. I wasn’t nearly as tactful as I should have been, and I hope that I didn’t scare away any potential users. Anyway, I’m sure that I missed some of the new features; hopefully the slides from the conference will be out by the middle of next week some time.

After their presentation, Dale and Don had Dmitri come up on stage and do some demos. I’m a bit fuzzy on these, but I am pretty sure that they were on the new third-party transformers. These are sold by FME, but are developed by other organisations. MRFCleaner (developed by MRF Geosystems) allows for some very complex automated quality assurance and cleaning. Many of the functions (overshoots, undershoots, intersections, etc) supported by MRFCleaner could be managed in FME previously, but they take countless transformers and are a real nightmare to maintain. I am seriously considering adding MRFCleaner to my licensing. ArcFitter (I didn’t catch the developer’s name) CurveFitter (developed by Tom Inloes of TCI Corporation) allows you to start recovering from the legacy of GIS data models that only supported points, lines, and polygons by retrofitting your stroked curves with fancy new arcs. As well as being a better representation of certain features, this can give you a reduction in storage space, especially in heavily splined curves.

After this came a set of presentation sessions. I was planning to go to the Best Practices for FME talk, but instead ended up sitting with Tom (one of the developers) chatting about KML 2.1 support. Warning: it’s not safe to sit down with me and talk for any length of time. I might just get around to discussing my thoughts on how open source geospatial will cause some proprietary vendors to reconsider their core competencies, and others to reorganise as their bread-and-butter components become commoditised. I believe that there will continue to be a place for well designed, efficient, and innovative products like FME, but that a lot of the fat in the industry will be trimmed.

Lunch was again good. I don’t know if I’ve said this, but the networking at this conference was incredible. I was lucky enough to speak with folks who were doing things as varied as: using FME to aid in moving away from a VAX/VMS-based system, building financial asset reports for taxation purposes directly from drawings, and using FME’s geometric processing capabilities to build complex cartographic output. There are not enough hours in the day to do all of the cool things that FME makes possible, so it’s nice to get some vicarious exposure to others who are working on them.

After lunch came the lightning talks. These little five-minute show and tell pieces were great fun, accompanied by computer-generated lighting sound effects, and a virtual audience that would clap you down if the thunder wasn’t loud enough. This was a great format to learn a lot in a little while, and I hope that this tradition continues in future conferences. I think that if they can get enough volunteers, they should allocate even more time to these tidbits.

The lightning talks were followed by another set of user sessions. This time, I went to the FME and Cartography session. This was really interesting to me on a personal note, as I am currently banging my head against trying to join some Autocad blocks up to some Oracle Points, and bringing them into MapInfo maintaining colour, size, rotation, and other goodies. Not fun at all, but these presenters seemed to have done some pure magic in this area. Hans (an FME MVP) gave a great presentation on how he used FME to improve the cartographic value of a street map, while SRG (another FME MVP) provided a very tasty recipe for SVG a la banana. I hadn’t looked into SVG much before this, but it sure looks powerful and flexible. During this session, a fully symbolised vector-based topographic map was shown running in Internet Explorer. Cool.

I won’t say too much about the ice cream (yum), the closing comments, or my request for flashing lights and running numbers in Workbench. Suffice to say, I had a great time at the FME UC, learned tonnes of stuff and got to speak with a lot of really smart people. I also felt like a bit of a celebrity, which was really odd. My wife doesn’t want me to go to another FME conference for a while. She says my ego’s already large enough… :)

-J

OK, OK, sorry for the cheezy intro. It’s true though. Let me tell you why.

Starting with the important stuff: Safe Software really know how to do swag and marketing in general. This is hands-down the best bag that I’ve ever got at a conference, though the Research in Motion WES conference came close. Inside the bag was the regular paperwork, but also a great water bottle, an organiser and a T-Shirt (they actually asked my size). Nice going conference team!

OK, that’s not really where the value lies, but it was definitely a nice way to start off the day.

I’ve been active in the FME community for a bit over three years now, and know a lot of folks (staff and users) solely through email. This has been a great opportunity to meet these people and others that I have never seen before, and compare notes on how FME is enriching our lives. The conference is intimate enough (around 100 people I think) that it’s possible to at least say hi to everyone, and there are many staff and expert users here to help out with your problems and discuss any ideas you might have in helping FME evolve to be an even better product. Everyone that is here is extremely accessible, and all seemed to be having fun, from the co-founders through the developers and other staff. It can not be emphasised enough how much value can be obtained solely from the networking available at this kind of conference.

The official kick-off featured a couple sessions about the history of Safe Software and FME, and I personally found these very interesting. Don Murray and Dale Lutz, Safe’s co-founders, showed how a great idea and hard work have grown Safe Software from a two-person operation into the strong organisation it is today, with over 70 employees. This was inspirational if nothing else. The second presentation was by Mark Sondheim and Peter Friesen from the BC Goverment. They recounted first hand the wild west of spatial interchange formats in the 90′s, and how the history of Safe Software was closely linked to the initial implimentation of the SAIF (Spatial Archive Interchange Format) standard. I’m ashamed to admit that I was nowhere near as visionary as Mark and Peter, and wondered at the time why we needed another format that I couldn’t even read from Arc/Info. If you haven’t heard of SAIF, take a minute to look it up. You’ll find that it shares a lot of concepts with GML. If you combine this precursor to GML with pioneering roles in PostGIS and the Java Topology Suite (I had similar unenlightened thoughts about JTS when I saw the RFP…) the effect that these folks have had on our industry and open source geospatial in general is amazing. Now if only they could do something about providing publicly funded base mapping data in BC for free ;)

The afternoon consisted of two workshop sections, with two streams each. In the first section I co-presented a session on the use of FME in my workplace in the same time slot with two other municipal-related sessions. For the next section, I chose to go to the Scripting stream, where there were sessions talking about using TCL within FME, using FME functions in Python, and some of the new Python-related enhancements in the FME betas. All very cool stuff. The slides will be available after the conference at the FME User Conference 2006 site, as is the agenda if you want to see if I’ve missed anything.

After the educational sessions, it was time for some fun and games. Three other FME MVPs (Hans, Peter, and SRG) competed along with me for the coveted FME Idol award. As we were working, Ulf Mansson took the opportunity to showcase some of his latest FME Art, which was very cool, and a bit distracting. Even with the distractions, I am happy to say that after some stiff competition I walked away with some serious hardware… er… glassware :)

To wrap this up, I should talk about some other important features of any conference: food, entertainment, and venue. Here, the conference team came through for us again. The food at the conference center was reasonable, and the dinner and entertainment at Storeum were excellent. The conference is being held at the Morris J. Wosk Centre for Dialogue, a rather unique space that is very condusive to communication. This was also the location of the GeoWeb conference earlier this year. I’m not sure how well it would do with a larger group, but it worked well for us today.

Anyway, let me know if you have any questions, and I’ll do my best to answer them. I am really looking forward to tomorrow.

-J

P.S. If anyone from Autodesk is reading this post: Please, Please, Please work with Safe Software to give use a better solution for reading DWG with Object Data!

Those of you who have used Safe Software‘s FME Workbench know that it is a great tool for performing spatial extract/transform/load (ETL) operations. It’s a unique way of setting up extremely powerful location data transformations without having to write a single line of code (but you can… in TCL and, more recently, Python). Workbench is an environment similar to DTS and ESRI’s Model Builder; FME’s implementation just happens to be more user-friendly and powerful:

Two recent developments in this field have made me into one extremely happy camper.

The first of these developments is the new version of DTS packaged with SQL Server 2005, which contains enough upgrades that it warrants a fancy new name: SQL Server Integration Services (SSIS). With this release, Microsoft has introduced a two-plane design paradigm, with the outer plane controlling process flow, and the inner plane controlling data flow. I can’t cover much here; suffice to say that it’s extremely powerful, easy to use, and that this feature alone makes the upgrade from 2000 worthwhile. The Microsoft SQL Server Developer Center is a good place to start learning more about SSIS if you’re interested.

The second of the recent developments ties into the first. Safe Software has done a soft launch on a beta FME SSIS plugin that encapsulates all of the cool Spatial ETL capabilities of FME inside SQL Server! Now I can set up a central SSIS workflow that manages all of my spatial transformations. I can also (and this is a big plus) make use of SQL Server’s native data transformation capabilities. This means that I can join disparate tabular data mid-transform to various spatial data sources before outputting a final product. I can also set up rollback routines without resorting to complicated scripting.

The Safe engineers have done a great job on this integration. Apart from some very FME-ish property and data source dialogs (which is a good thing™) it works seamlessly within the context of SSIS. I am happy to see that they managed to get the visualiser to work from within SSIS as well.

Here are some example shots from their web page:

Opening an ArcSDE data source

Setting the properties of a LengthCalculator transformer

Running the SSIS package (ohhh pretty colours… would like to see this in Workbench)

Viewing spatial results in FME Viewer

Viewing CSV results in text editor

Incidentally, this is not the only integration project that Safe has done. If you use Informatica they have also set up a beta program for FME integration with PowerCenter. If you’re an ArcGIS user, the Data Interoperability Extension gives you the same kind of integration with Model Builder (along with native data access to dozens of formats not supported directly by ESRI) and the 9.2 edition will come with a refresh to the FME 2006 GB code base.

I am amazed at Safe’s dedication to the leading edge. First there was Google Earth KML support, next came support for the SDF+ data format (an open format used for MapGuide Open Source and by Autodesk in its geospatial applications) and now this. What’s next? GeoRSS support?

-J

P.S. If you haven’t signed up for the First Annual FME User Conference, do it now. Vancouver is a great city, and I’m sure we’re going to have lots of fun learn lots of useful stuff there.