Fake Steve on the Borg and Freetards

Good analysis in a sarcastic candy coating:

Open source has ridden the classic Gartner hype cycle. Three years ago was the "peak of inflated expectations," and VCs would fund anything with "open source" in its name or business plan. Now the cycle has moved on to the "trough of disillusionment." Reality has set in. Nobody is making money. They're in the Slough of Despond.

Bear in mind that FakeSteve is the Voice of the Valley, and measures using the standard Valley metrics: VC funding, license sales, lines of copy in Fortune. Open source has been through two hype cycles now (from cycle one, remember the VA Linux IPO?) and keeps on trucking. There are more things in heaven and earth, FakeSteve, than are dreamt of in the Valley.

Timmy's Telethon #6

And finalement:

6. Third party business applications: It could be argued that an enterprise GIS exist to support business requirements which often call for a third party client solution. Are vendors building COTs apps against these third party solutions?

I think the wording is a little off here, and it should be "are vendors building third party COTS solutions against these open source apps?"

And the answer is "sure!" Not as many as against the ESRI stack, but that is to be expected: ISVs support things where there is demand for support, and the market leader obviously drives a lot more demand. However, the amount of ISV support for open source is greater than zero, and growing. On the PostGIS side, for example:

• Safe Software's FME supports it
• Mapguide supports it
• Ionic Red Spider supports it
• ESRI's Interoperability Extension supports it
• Manifold supports it (!!!)
• CadCorp SIS supports it
• ArcSDE 9.3 supports it
• MapDotNet Server supports it

Most of the web-services side of things are supported via open standards, so Mapserver slots into all kinds of infrastructures and third party tools via the WMS standard, for example. I imagine if the WMS standard did not exist, more software would talk directly to Mapserver via it's own CGI dialect, but WMS made that redundant.

And on the client side, I found this nugget from Bill Dollins' ESRI Fed UC review intriguing:

So the take away was “we’ve got the server to crunch your data and give you good analysis results, display it any way you want.” There was no OpenLayers demo but it was mentioned several times and something that should be able to leverage new APIs.

In the open source world, OpenLayers is already the "default map component" for web apps, it is interesting to see it already banging on the doors to the proprietary world as well.

"Risk"

Great quotation on the OSGeo-Discuss list:

The brutal truth is this: when your key business processes are executed by opaque blocks of bits that you can't even see inside (let alone modify) you have lost control of your business. You need your supplier more than your supplier needs you—and you will pay, and pay, and pay again for that power imbalance. You'll pay in higher prices, you'll pay in lost opportunities, and you'll pay in lock-in that grows worse over time as the supplier (who has refined its game on a lot of previous victims) tightens its hold.

I heard Chris Di Bona of Google say exactly this at a conference in DC last year with respect to Google's own strategic software direction. For anything that is core to their business, they use open source if it is good enough, or build it themselves if it is not. Being beholden to a vendor would be a huge competitive disadvantage to them.

Like Cattle to Slaughter

When I first got a cell-phone, some 10 years ago, I deliberately chose the hip new alternative on the block, FIDO, because their costs were competitive, and they didn't have a "N-year contract" system. You paid your bill until you were done with the service, and then you stopped – sounded fair to me.

However, the Canadian industry has consolidated down to 2 major carriers and a handful of small regional ones, and FIDO was bought up by the Rogers behemoth. They still operate as an independent brand, but clearly things are changing. For one thing, they have added multi-year contracts, and are now pulling maneuvers to try and get people to leave behind their month-to-month plans.

My wife got a call from a marketer – would you like a better plan? we'll give you two months free and blah blah blah blah blah. She was holding a screaming baby at the time, and said "yeah, sure". Never say yes to a salesman in haste. Turns out the new plan was a 3-year contract.

Today, I found out that the phone I have was quietly switched to a 3-year plan when they sent me a new handset last year. Surprise! FIDO had sent new handsets in the past, so I didn't notice, but apparently the cost of my new phone was that I am now in cell-phone shackles until January of 2010.

You're not a customer to the phone company, you're an ambulatory money beast. Mmoooooooo.

Timmy's Telethon #5

It's not all about Timmy, Timmy worries about the little people too!

5. Business model: I think I’m missing the business opportunity behind open source. Perhaps its altruism or a hobby… but that only goes so far. How are people leveraging this to pay the bills? Are services where the money is at?

Do you litter? When you put your garbage in a can, instead of dropping it on the street, is it altruism, or self-interest? After all, you're keeping your own environment clean along with everyone else's. What about the fellow who actually bends over and picks up someone else's empty coffee cup to put into the can?

Open source turns software into a commons, like the environment, but because we are used to thinking of software as property, our minds find it hard to figure out why open source software aggregates effort and gets cleaner, faster, stronger. Unlike picking up litter, which everyone can do, improving open source can only be done by a small percentage of the population who can understand and work on the code, and that makes it even harder for the rest of the population to understand what is going on. WTF are these lunatics up to, moving bits of used paper into cylindrical holding devices?

The people improving open source fall into lots of categories, but here are some broad ones:

• The altruist / tinkerer. The most popular media archetype, the altruist / tinkerer probably accounts for the smallest amount of effort on mature open source technologies, but occasionally will create a new technology from scratch that is so good and compelling that it gathers other types who then keep it alive and move it into wider utility.
• The service provider. The most widely understood business model, running from the one man band (take a bow, Frank Warmerdam) to billion-dollar companies like Red Hat. They will sell you support, or custom development, directly on the software. Because they are tightly associated with the software, this is the easiest model for "vendor minded" folks to mentally grasp.
• The systems integrator. Working with the open source software, but not necessarily on the software, the system integrator is easily drawn into fixing bugs and adding new functionality on projects as the client requires. Systems integrators love open source because it allows them to meet client needs without being stuck behind a vendor's development priorities ("we've got your bug report on file for the next release", "that feature will be available in 18 months").
• The company man. Easily the least appreciated member of the open source pantheon, because he is not paid to work on open source. He is paid to work on fish inventories. Or carbon models. Or inventory management. Or tax collection. But he has a bit of discretionary time, and he uses it to make the tools he works with work better.

Notably missing from this list is "the billionaire" and "the venture capitalist". That is because, while there is money to be made in open source, there is not a lot of money to be made. People who try to put up fences in the open source commons find that all the rest of the players end up routing around them, and the value slowly drains from their little patch of land, leaving them only with the tried and true proprietary model to fall back on.

Attention Neotards: Your GPS Sucks!

Reading through an update from Open Street Map (incredibly, it has taken them mere months to load TIGER into their futuristic system) I came across this little gem regarding aerial photography:

Since the TIGER map data was produced from aerial photography, and was originally intended to assist Census Bureau officials in the field, such problems [misalignment of roads] are bound to occur and are unlikely to have undergone official correction.

I'm not sure what to mock first, the leap to the conclusion that TIGER data problems are a result of aerial photography, or the related conclusion that GPS tracing is somehow superior to aerial photography. They are of course, closely related in the mind of the neotard: "Hmm, TIGER is old; aerial photography is old; TIGER sucks; aerial photography is old and expensive (so it sucks); therefore, the reason TIGER sucks is because aerial photography sucks!"

(Admission, I don't know why TIGER positions are so bad in places. The answer may well be the source, but it cannot be hung off of aerial photography. Much of TIGER is sourced from lower levels of government then stitched in, so it is probably a pastiche of capture methods. I wouldn't be surprised if some of it was digitized off of paper road maps. Or if some of it has not been positionally updated in 25 years.)

Oh, pity the poor paleotards, who don't know any better, wasting good money flying about taking error prone "photographs", instead of doing the smart thing and walking around with a Garmin. (Is that a Garmin in your pocket, or are you just happy to see me?)

I admit, I suffered from "GPS is magic" syndrome for quite a while, but I had the fortune to be exposed to people whose job it is to make base maps, who have to ensure that the lines they place on the map (in the database) are as close to "true" location as possible, given the technology available. That exposure taught me some useful things about source data collection, and one thing it taught me is that GPS traces are extremely hard to quality control.

The GPS has a very hard job to do. It has to read in multiple signals from satellites and calculate its location based on very, very, very small time differences. What happens when the signals intermittently drop because of trees overhead blocking the signal? Or bounce off of a nearby structure? The unit makes errors. Which would be fine, except it's hard to isolate which observations are good, and which ones are bad. Too often, GPS track gathering falls back on heuristics that delete "zingers" (really obviously bad readings, determined visually or with a simple algorithm) and assume every other observation is "good". If you delete zingers and take enough repeated traces, you can slowly get averaged lines that converge towards meter-level accuracy. However, the need for multiple traces radically increases the manpower/cost of gathering decent data, and the accuracy level does max out after a while.

The answer to getting good positional information over a large area is to tie together the strengths of GPS systems and the strength of remote sensing (aerial, satellite) systems.

• Take a picture from above (or better, borrow one, from the USDA, or USGS, that has already been "orthocorrected"). This provides a very good source of relative measurement information: you can determine very precisely the distance and bearing between any points A and B. But it has no absolute positioning: how do I know the longitude/latitude of A and B?
• Find several points at the edges of your photograph that are clearly visible from above and identifiable from the ground. Take your GPS to those points, and take long, long, long collections of readings (a hour is good) at each one. Take those readings home, and post-process them to remove even more error. Average your corrected readings for each point, you now have "ground control points".
• Use those control points to fit your aerial picture to the ground in some nice planar projection (like UTM). Digitize all the rest of your locations directly off the picture.

Take a deep breath. You are now a paleotard, but a happy, contented paleotard with very well located data.

This posting derives from a very interesting discussion on Geowanking from some months back. The question was "how do I map my two acre property at very high accuracy?", and while the initial guess of the poster involved using a variation on GPS tracing, the best final answer was a hybrid solution.

Wheels Within Wheels

Sean says he supported Obama because he is more “electable”. So, what are we to make of this notice in the New York Times?

Senator John McCain’s presidential campaign said Thursday that it stood by a year-old pledge made with Senator Barack Obama that each would accept public financing for the general election if the nominee of the opposing party did the same.

This isn't about campaign financing, since that issue would keep until after the Democratic nominee is decided. This is about pumping up Obama by treating him like the presumptive nominee, putting let more wind in his sails heading into Texas and Ohio, where an Obama win will put a stake through the heart of the Clinton campaign. Could it be the McCain campaign doesn't share Sean's take on who the more electable Democrat is?

Keep your Stinking Rasters Out of My Database

W00t! Yeah! Sing it brother!

http://spatialgalaxy.net/2008/02/15/rasters-in-the-database-why-bother/

Timmy's Telethon #4

Out of left field:

4. Compatibility: This problem reaches across the board but when it comes to open source vs. closed source; from what you’ve seen is it a wash? I must admit that I’m inclined to stick with the devil that I and everyone else knows. Additionally doesn’t the nature of open source introduce opportunities for proprietary stovepipes?

This one I frankly do not understand, but perhaps it is in the nature of the word "compatibility".

In general, open source software is wildly more interoperable and therefore "compatible" with different proprietary stovepipes than the proprietary alternatives. Mapserver, for example, can pull data out of dozens of file formats, as well as SDE, Oracle, geodatabase, and the usual open source suspects like PostGIS. Because open source development priorities are "scratch the itch" and people in real offices need to do real work, one of the first features requested and funded is almost always "connect to my proprietary database X".

(This is not just about Mapserver either: Geoserver, uDig, gvSIG, QGIS, even good old GRASS all have better multi-format connectivity than leading proprietary brands. Note the word "leading"... the non-leading proprietary brands tend to have good connectivity too, but the market leader uses lack of interoperability as a means to protect their lead.)

If "compatibility" is really a synonym for "does it work with ArcMap", then indeed there is a problem, but it is not on the open source side. ESRI ties ArcMap tightly to their own stovepipe for very good (to them) reasons of competitiveness and market protection. ArcMap sells ArcSDE licenses, not vice versa. (How many times have you heard someone say "this SDE stuff is great! if only I had a mapping tool to work with the data...")

I'm open to suggestions as to what part of open source "nature" actually lends itself to proprietary stovepipes. That part I don't get at all.

Timmy's Telethon #3

Neither rain nor sleet nor dark of night will stop me:

3. Sociology / Psychology: There is so much more to pitching a solution to those who control the purse strings than technical logic. Selling open source to the powers that be is… well it’s tough. Are there consultants that can be hired to answer the hard questions and make the decision makers feel comfortable? I would love to see more real world business cases for comprehensive GIS environments that must cater to diverse requirements.

25 years ago, selling anything that wasn't IBM to the powers-that-be was tough. I expect selling open source to remain tough, but get easier and easier over time as people with Internet-centric mindsets percolate up into decision making positions for larger organizations.

Sure, there are consultants, as in any field. Real-world business cases are available, but few and far between. When you ask people to write their own case studies, generally the folks with interesting studies are too busy to do it, so you end up with a bunch of academic stuff. And going out to gather the things is a lot of work. And often some of the most compelling studies are found by accident!

I compile case studies for PostGIS, and every 12 months I ask on the postgis-users mailing list (1500 subscribers) for folks to volunteer for case studies. I'll do the writing and editing, they just need to talk to me on the phone or on email. I don't get many responses!

When I was at FOSS4G 2006 two years ago, it was mentioned in passing to me that "oh yes, IGN is using PostGIS now". So I asked for a contact name, tracked the guy down, and extracted the story from him. It's a great case study! But getting it required a curious combination of persistence and luck. Same with the North Dakota study, which I got via friend-of-a-friend referral.



The request for information about "comprehensive GIS environments" feels like trolling for someone to say "just drop ESRI"! And that would be silly. Open source has lots of solutions that do things that ESRI products do, but there is no comprehensive drop-in replacement solution. Even if there were, the switching costs alone would probably make in uneconomic.

So here is what to say to the powers-that-be. Don't foam at the mouth, don't wear jams and a backwards baseball cap. Recognize that change is slow, and there are sound financial reasons for that:

For an organization just getting started with open source, it provides advantages at the margins: not in reworking your existing systems, but giving you flexible options when building new ones. The existing systems should be left running until they hit a natural end-of-life, either when they become out of date, or so expensive to run/pay maintenance on that the switching cost actually becomes acceptable. Evaluate the cost of change regularly. Sometimes not changing is the more expensive option, and it is important to know what that time arrives.

Whither Canada?

It has been a bit over ten years since I started Refractions Research, and in that time most everything that has happened has been unanticipated. The best laid plans rarely yielded what was expected, though they often brought good surprises.

For a founder, one of the side effects of success in a company is that your role is constantly changing. In a company of two, your job is radically different than in a company of twenty. This is both good, in that it provides lots of variety, and bad, in that it can result in being “promoted” into a role that might not be a good fit.

And so it is with me. I have moved from being someone who works on software and problem solving to someone who pays other people to work on software and problem solving. This has not, as it turns out, been a recipe for job satisfaction in the long run.

Back when I had no wife/house/child/child I was able to balance out my role by just working more – spent all day on meetings and email? no problem, play with technology at night.

But that doesn't work any more. I don't feel like I have the leadership focus I used to, nor do I get to spend enough time on technology and problem solving. Since I am doing neither job well right now, I am going to pick the one I like more, and do that for a bit.

So I have decided to make a big change. I am going to leave the company I started, and return whence I came, to independent consulting. Solo consulting is tricky, because the work is “gappy”, but it has a good rhythm: in the gaps, you learn new things; in the peaks, you apply those things.

I don't know what other opportunities might come along (best laid plans and all) but in the meanwhile some quality time with my computer and some concrete problems sounds great. I am looking forward to getting my hands dirty in the guts of the many open source projects I have thus far only experienced as a user.

Refractions is a big (relatively, 25 people) company now, and has momentum – good institutional clients, great delivery skills, stable business processes. I have ensured over the last six months that I am not in the critical path for any of our projects. The leadership team has been operating independently of me for a number of months, and can keep building the company into the future. The staff are an excellent group of folks, the best collection of geospatial smarts in the Pacific North-west.

I will miss spending regular time with the folks at the office, but will continue to do occasional work with Refractions as an associate, so there will be lots of opportunities to keep in touch, have lunches and beers, and keep the creative juices flowing.



Thanks everyone, it has been an amazing experience, I look forward to working with you again in my next incarnation(s).

Timmy's Telethon #2

Moving on:

2. Risk: Our landscape is changing so fast there is an extreme amount of exposure to dumping resources into a solution that isn’t supported. If any one component of the enterprise stack changes you’re vulnerable and I trust those I’m paying to cover my maintenance than I do an ethereal community. When it comes to supporting my clientele I need tangible support resources. How good are the support resources for open source solutions? Is there comprehensive up to date documentation? Can I call someone in an a oh $*!&% moment?

I think this article at InformationWeek (number three in the Google search for “open source support”) sums it up well. There is not one answer, there are a number of answers, and you need to choose the one(s) that make sense for your needs.

• Product support, from specialist companies with expertise in particular components (Refractions for PostGIS, DM Solutions for Mapserver, TOPP for Geoserver)
• Stack support, from generalist companies putting together mixes of components (Wheregroup)
• Community support, much maligned, but better than the technical support provided with most proprietary packages
• Training, from folks like the companies above, or specialist trainers like Open Technology Group
• Hiring project developers, an often unappreciated source of top notch trouble shooting and knowledge,
• Consultants, who have to know the tools they use

For me, the money quote is:

CIOs would be well-advised not to buck the open-source trend. On the contrary, they should assume responsibility for open-source initiatives and ensure that their companies have the right support structures in place... They'll find the more mature an open-source software project, the more mature support options its users enjoy.

Open source requires you to assume responsibility, which is hard for an organization man, with years of CYA behind him, to do. In exchange for taking responsibility for your own infrastructure, you are rewarded with a software ecosystem where there is more than one source of support.

What do open source organizations do when their support provider isn't up to snuff? They get a new support provider.

What do ESRI customers do when ESRI support isn't up to snuff? They bitch about it on James Fee's blog.

Timmy's Telethon #1

So, addressing Timmy's concerns about open source geospatial:

1. Staffing: The specialized skills necessary to build and maintain an open source app are hard to come by. There is a premium on any specialization, is the talent pool to build and support these open source solutions deep enough to maintain continuity in staff skills?

There is, as with any great argument, a kernel of truth in this item, but it is wrapped in a thick, low-calorie, blanket of misdirection, like a corndog at a state fair.

So, should you be concerned about staffing your open source application? You should, to the extent that:

1. the skills required to understand and maintain it take a long time to learn, and
2. the skills required to understand and maintain it are in short supply.

Note that you have reason for concern only if both conditions occur: the skill must be both difficult to learn and in short supply.

Timmy sees that, compared to proprietary toolsets, people with prior experience with open source tool sets are fewer and farther between, and leaps to the conclusion that there is a skills provision risk.

However, the skills necessary to work with open source geospatial applications are either easy to pick up quickly, or transferable from other domains.

• PostGIS: Already worked with Oracle Spatial or ArcSDE's "new" spatial SQL feature? You already know PostGIS.
• Mapserver: Learn the .map file and you are good to go. No harder than picking up enough AXL to be useful. Budget a couple days of learning time.
• OpenLayers: Already worked with Google Maps? You've got the concepts down pat. You'd better know Javascript, but that's a transferable skill and you'll need that for any non-trivial application.
• Geoserver: Point and click through the interface. Do you known enough to deploy a WAR into production? If you installed ArcIMS, you already do.

The slight disadvantage open source has in providing decent tutorial-level guides for new users is offset by the advantage in access to a very helpful user community and direct access to the development community.

Summary: No matter whether you're building on ESRI or open source, if you are building something complex your staff will have to learn a few new skills. Their prior experience with core concepts like programming and IT will serve them well in both domains, and the learning curve will be no worse either way.

Caveat: It's possible Timmy is talking about people who will only learn a new skill if force fed it through a training course, and even then will only retain 50% of what they are taught, who write point-and-click recipes and stick them up on their cubicle walls, who think that "re-boot the server" is a genuine solution. If those are the people whose "skills" he is worrying about, I can only say "go with God, Timmy, peace be unto you, you have larger problems than proprietary vendor lock-in".

Weblicious

I've spent a fair amount of time over the last month re-doing three web sites:

• the Refractions site;
• the uDig site; and,
• the PostGIS site.

Some of it, like the PostGIS site, has been easy, simply re-skinning the existing content (though I'm anticipating reworking the content there to make it more newbie friendly and harmonized). Other bits, like the uDig site, are brand new, and include some big additions like a gallery of projects using uDig. And the Refractions site includes piles of new content like case studies, that have taken many days to write up. And I'm only about 50% through my list of candidates.

PostGIS users will find this nugget in the Refractions site fun: a potted history of PostGIS.

Timmy's Telethon #0

In the comment thread at James Fee's posting on building an open source application in an "ESRI" shop, "timmy" provides the most complete laundry list of incumbent vendor objections to open source I have seen in some time.

The list is far too comprehensive to do in one post, so I'll do them one at a time.

As a general note, many of the items are not really specific to open source or geo-spatial – they could be used by any incumbent market-leading vendor to attack a smaller competitor.

There is also an apples/oranges thing going on here, since the default GIS vendor (ESRI) is at a different point in the technology adoption cycle than open source. Open source can't strongly appeal (yet) to conservative late adopters, and ESRI is finding it hard (at the moment) to appeal to technically savvy early adopters. (Technology book recommendation: Crossing the Chasm is a must-read for anything thinking about the software market.)

Magick!

I've always like ImageMagick, and frequently it is the first thing I install when I set up a new computer. For OS/X, I have found that MacPorts make the installation pretty painless, although it takes a while to compile all the dependencies.

Like most ImageMagick users, I have rarely scratched the surface of what this toolkit can do — I mostly use it for simple format conversions and image re-scaling. However, I recently had a image processing problem that went beyond the ordinary — I wanted a general purpose tool that would take any input image, scale it to 200 pixels wide, and create nice rounded corners, with transparency where the pixels used to be.

Basically, to go from this:



To this:



Since I plan on doing this a lot, I don't want to fire up a graphics program every time and point-and-click my way to nirvana. Enter ImageMagick, my old friend!

#!/bin/bash

# Usage:
# ./roundclip.sh [inputfile]

# Output width
OUTPUTWIDTH=200
# Corner size
CSIZE=20

IMG=$1
EXT=${IMG##*.}
BASE=`basename $IMG .$EXT`
OUTFILE=PNG8:${BASE}_round.png
TMPFILE1=/tmp/${BASE}-1-$$.png
TMPFILE2=/tmp/${BASE}-2-$$.png
MASK=/tmp/mask-$$.png
TRANSPARENT=/tmp/transparent-$$.png

# Scale the input down to our desired width
convert $IMG -scale ${OUTPUTWIDTH}x $TMPFILE1

# Find out the height and width of the working file
DIM=`identify -format %wx%h $TMPFILE1`
W=`identify -format %w $TMPFILE1`
H=`identify -format %h $TMPFILE1`

# Calculate the lower corner coordinates
X=$(($W - 1))
Y=$(($H - 1))

# Make a clipping mask with rounded corners
convert -size $DIM xc:black \
        -fill white \
        -draw "RoundRectangle 0,0 $X,$Y, $CSIZE,$CSIZE" \
        $MASK

# Make a transparent underlay
convert -size $DIM xc:transparent $TRANSPARENT

# Place the masked input image onto the transparent underlay
composite -compose src-over $TMPFILE1 $TRANSPARENT $MASK $TMPFILE2

# Convert to the output format, and do some color reduction
convert $TMPFILE2 -quality 90% $OUTFILE

# Clean up the temporary files
/bin/rm $TMPFILE1 $TMPFILE2 $MASK $TRANSPARENT

There are probably much more efficient ways to do this, with fewer intermediate steps, but I am not a guru yet.

Using other drawing and blurring techniques, it's also possible to create drop-shadows on the fly too...