Tuesday, April 07, 2009

ESRI Formats - Back to the Future?

Remember when "coverage" was the holy grail? "Boy, if I could just read/write coverage, I could interoperate with the GIS folks no problem."

And then things seemed to get better. Most data was in "shape files" and there was a published specification which more-or-less matched the data generated by the ESRI tools. And lots of folks were using ArcView, so you could share "avl" files (layer style) and "apr" files (project file) and even write third-party tools to consume them directly (they were just a funny text format after all).

And now they are back-sliding. Layers and projects are "lyr" and "mxd" binary formats generated by ArcGIS and not consumable by third party tools. Data is starting to be moved into "file-based geodatabase" (FGDB) and the only tools that can peer into those are Arc*. (The old geodatabase had a lot of drawbacks, but at least you could open it in a third-party tool.) There are no specifications for the data formats, for the layer formats, for the project formats.

I thought we in the computer profession had this discussion and came to a consensus: vendors that tried to use proprietary file formats as a lock-in were to be avoided if possible and castigated if necessary. Even old Microsoft had to play the "open format" game, at least in mixed company.

It's not just data, though the rise of the closed FGDB format over the last couple years is bad enough. The "metadata", the style files and the project files, are a key piece of "operational data" that can be incorporated into workflows automatically – if you can read and write them. With format lock-in in place, the only way to automate your work flows is through the vendor-approved channel.

Really, I thought it was decided – they don't get to do this to us anymore.


Joshua Campbell said...

Cheers to that!

How did all the read/write access in GDAL/OGR get developed?

Is it as simple as someone paying for the development of the conversion tool, or is the format completely locked?

Jarlath O'Neil-Dunne said...

On the remote sensing side there are a few vendors that provide strong support for ESRI geodatabases: SOCET GXP, ERDAS, and ENVI. I don't know what sort of licensing arrangements they have, but they make it happen.

Paul Ramsey said...

@Jarlath, Personal Geodatabase or File-based Geodatabase? Safe's FME does file-based geodatabase, but only in the "for ESRI" addition, which screams licensed DLL to me.

Jarlath O'Neil-Dunne said...

ERDAS and ENVI both support personal and file geodatabases. SOCET GXP only supports personal geodatabases at this time. ITT (makers of ENVI) are an ESRI business partner. ERDAS and ESRI split months ago yet ERDAS continues to offer farily robust geodatabase support.

Geodatabase support is only part of the equation, the other part is supporting version X.X of the geodatabase. This is a realy headache even if you are operating in ESRI land. I keep legacy copies of personal and file geodatabases kicking around for when I need to share data with folks running pre-9.3 ArcGIS.

James Fee said...

Paul, you are correct, you need ArcObjects to use the ESRI "formats" with FME.

ESRI will have an API for the File Geodatabase at 9.4 (probably beta in August, final release in March 2010). Think .NET probably, maybe Java.

Paul Ramsey said...


C ???!?!?

Herb said...

Vendors will continue to "do it" to us so long as uninformed buyers are willing to pay the bucks. What I find particularly insidious are lock-in vendors pretending to be open source by using the word "Open" in their name and then claiming that it's nit-picking to continue to be purist about "open source" definitions. Hey, open source matters. More and more, buyers are getting tired of getting locked-in.

Bart van den Eijnden said...

We need to go with the flow and go back to reverse-engineering :-)

Bill said...

Well said, Paul. I have been thinking a lot lately about the similarities between the FGDB and the coverage in terms of openness. The FGDB in my mind is the "coverage 2.0".

Ian Turton said...

Once you realize that ESRI are in fact crack dealers, it all makes sense.

For anyone who has never heard me rant on this before think "go on just have a little taste, it's free [while you are a student]" and see if you can spot the difference. (Apart from the costs of course, even the best crack dealer can't get you to pay out thousands of $ each year).

Passenger57 said...

re: ESRI as crack dealers

When you were a poor student I bet you appreciated having free access to all that software.

Would you prefer that they charged students? Or are you suggesting that their software should be free for everyone?

Disclosure: former ESRI(UK) and ESRI Australia employee, now working for state government, with no vested interests....

John Callahan said...

I get more upset at those enterprise licenses, such as with large federal and state governments. I've dealt with several people just during this past year whose company/agency "strongly encouraged" them to use ESRI because of the license package they purchased. And the administrators who are making these purchases don't want their staff to deal (get trained in) with any new sw, especially the IT support staff and security personnel.

ESRI started really pushing those enterprise licenses right around the same time ArcGIS Server came out. Coincidence??? I think not! (ESRI has had some very bright people at the top; I believe they've known exactly what they're doing, I just disagree with it.)

I thought (almost) giving the sw to K-12 schools was a great idea at the time. It's popular and easy to use. It's the universities that needed to be smarter. Nowadays, I push K-12 toward QGIS, MapWindow, or something like that.

@Paul, loved the post. Thanks! I've been thinking the exact same thing for years about the binary mxd/lyr files. I can't stand them. However, I actually thought FGDB was a good idea, within the ESRI world, a step up from PGDB (is Microsoft really a third party to ESRI?), that's with the understanding that FGDB would have a published API.

My disclosure: former ESRI cool-aid drinker, ESRI employee, and current FOSS enthusiast!

About Me

My Photo
Victoria, British Columbia, Canada

Blog Archive


bc (44) it (35) postgis (24) video (15) enterprise IT (11) icm (11) gis (9) sprint (9) foi (8) open source (8) osgeo (8) enterprise (7) cio (6) foippa (6) foss4g (6) management (6) politics (6) spatial it (6) outsourcing (5) mapserver (4) bcesis (3) boundless (3) email (3) opengeo (3) oracle (3) rant (3) COTS (2) architecture (2) cartodb (2) deloitte (2) esri (2) hp (2) idm (2) javascript (2) natural resources (2) ogc (2) open data (2) openstudent (2) oss (2) postgresql (2) technology (2) vendor (2) web (2) 1.4.0 (1) HR (1) access to information (1) accounting (1) agile (1) archive (1) aspen (1) bcpoli (1) benchmark (1) buffer (1) build vs buy (1) business (1) business process (1) c (1) career (1) cathedral (1) client (1) cloud (1) code (1) common sense (1) consulting (1) contracting (1) core review (1) crm (1) crockofshit (1) cunit (1) custom (1) data science (1) data warehouse (1) design (1) development (1) digital (1) environment (1) essentials (1) evil (1) exadata (1) fcuk (1) fgdb (1) fme (1) foocamp (1) foss4g2007 (1) ftp (1) gdal (1) gds (1) geocortex (1) geometry (1) geoserver (1) geotiff (1) google (1) google earth (1) government (1) grass (1) hadoop (1) iaas (1) icio (1) imagery (1) industry (1) innovation (1) integrated case management (1) introversion (1) iso (1) isss (1) isvalid (1) jpeg (1) jts (1) lawyers (1) mapping (1) mcfd (1) media (1) microsoft (1) money (1) mysql (1) new it (1) nosql (1) nrs transformation (1) oipc (1) opengis (1) openlayers (1) paas (1) pgconfsv (1) pirates (1) policy (1) portal (1) proprietary software (1) public accounts (1) qgis (1) r (1) rdbms (1) recursion (1) redistribution (1) regression (1) rfc (1) right to information (1) saas (1) salesforce (1) sardonic (1) scandal (1) seibel (1) sermon (1) server (1) siebel (1) snark (1) spatial (1) standards (1) statistics (1) svr (1) taxi (1) tempest (1) texas (1) tired (1) transit (1) tripledelete (1) twitter (1) uber (1) udig (1) uk (1) uk gds (1) verbal culture (1) victoria (1) waterfall (1) wfs (1) where (1) with recursive (1) wkb (1)