Timmy's Telethon #0

In the comment thread at James Fee’s posting on building an open source application in an “ESRI” shop, “timmy” provides the most complete laundry list of incumbent vendor objections to open source I have seen in some time.

The list is far too comprehensive to do in one post, so I’ll do them one at a time.

As a general note, many of the items are not really specific to open source or geo-spatial – they could be used by any incumbent market-leading vendor to attack a smaller competitor.

There is also an apples/oranges thing going on here, since the default GIS vendor (ESRI) is at a different point in the technology adoption cycle than open source. Open source can’t strongly appeal (yet) to conservative late adopters, and ESRI is finding it hard (at the moment) to appeal to technically savvy early adopters. (Technology book recommendation: Crossing the Chasm is a must-read for anything thinking about the software market.)

Magick!

I’ve always like ImageMagick, and frequently it is the first thing I install when I set up a new computer. For OS/X, I have found that MacPorts make the installation pretty painless, although it takes a while to compile all the dependencies.

Like most ImageMagick users, I have rarely scratched the surface of what this toolkit can do – I mostly use it for simple format conversions and image re-scaling. However, I recently had a image processing problem that went beyond the ordinary – I wanted a general purpose tool that would take any input image, scale it to 200 pixels wide, and create nice rounded corners, with transparency where the pixels used to be.

Basically, to go from this:

To this:

Since I plan on doing this a lot, I don’t want to fire up a graphics program every time and point-and-click my way to nirvana. Enter ImageMagick, my old friend!

#!/bin/bash

# Usage:
# ./roundclip.sh [inputfile]

# Output width
OUTPUTWIDTH=200
# Corner size
CSIZE=20

IMG=$1
EXT=${IMG##*.}
BASE=`basename $IMG .$EXT`
OUTFILE=PNG8:${BASE}_round.png
TMPFILE1=/tmp/${BASE}-1-$$.png
TMPFILE2=/tmp/${BASE}-2-$$.png
MASK=/tmp/mask-$$.png
TRANSPARENT=/tmp/transparent-$$.png

# Scale the input down to our desired width
convert $IMG -scale ${OUTPUTWIDTH}x $TMPFILE1

# Find out the height and width of the working file
DIM=`identify -format %wx%h $TMPFILE1`
W=`identify -format %w $TMPFILE1`
H=`identify -format %h $TMPFILE1`

# Calculate the lower corner coordinates
X=$(($W - 1))
Y=$(($H - 1))

# Make a clipping mask with rounded corners
convert -size $DIM xc:black \
    -fill white \
    -draw "RoundRectangle 0,0 $X,$Y, $CSIZE,$CSIZE" \
    $MASK

# Make a transparent underlay
convert -size $DIM xc:transparent $TRANSPARENT

# Place the masked input image onto the transparent underlay
composite -compose src-over $TMPFILE1 $TRANSPARENT $MASK $TMPFILE2

# Convert to the output format, and do some color reduction
convert $TMPFILE2 -quality 90% $OUTFILE

# Clean up the temporary files
/bin/rm $TMPFILE1 $TMPFILE2 $MASK $TRANSPARENT

There are probably much more efficient ways to do this, with fewer intermediate steps, but I am not a guru yet.

Using other drawing and blurring techniques, it’s also possible to create drop-shadows on the fly too…

Transparent PNG for IE6

I’ve been wrapping our web content in some new web designs, and one of the issues I have encountered is supporting transparent PNG on IE6. It can be done, and all it requires is a relatively unobtrusive hack that uses the IE-only “behavior” CSS attribute.

Stangely, though, while our web developer could serve pages that worked with this trick, when I implemented them on our own servers, it didn’t work! It took a while to realize that the problem wasn’t how I was implementing the hack (“check the URLs”, “are you line stripping the files?”, “make sure the files aren’t missing”) but rather from where I was serving the hack. Namely, from an old server running Apache.

IE6 would not execute the hack, which was bundled in an IE-only “behavior” file, with a .htc extension. It would load it, I could see that in the logs, but it never did anything. The problem was that my old Apache wasn’t serving up the .htc file with the mime-type that IE wanted on it.

So, one quick entry in /etc/mime.types and an Apache restart later:

text/x-component htc

And we’re golden.

Chemistry Change

I am a sucker for catastrophe, so I am fortunate to live in this age of (as the Chinese might say) “opportunity”. We are privileged to be witness to the twin cataclysms of peak oil and global warming. It may not have the immediacy and drama of the sack of Rome but, baby, it’s got size.

In the past year, I finally got to see “An Inconvenient Truth”, which really is worth seeing, even if you know all about global warming already, because it is such a clean compendium of all the issues, scientific, political, cultural, that we are navigating on our way into this crisis. It brings everything into a neat package, tied with a bow, a truly great work of documentary film making.

However, the real eye-opener for me in the last year was an under-appreciated article in the New Yorker, “The Darkening Sea” by Elizabeth Kolbert. The gist of the article is that by changing atmospheric chemistry (by injecting CO2 in higher concentrations) we are also changing oceanic chemistry (as the oceans absorb the extra CO2). The eye-opener is just how deep through the ecosystem this change in ocean chemistry reaches. I cannot recommend this article highly enough, it’s one of the best of the last couple years.

Counting Votes

This is orthogonal to geospatial, but c’est la vie. Apparently the United States’ quadrennial meltdown over how to count things is firing up again.

A big part of the problem appears to be the de-centralization of the US institutions responsible for conducting elections (in some cases each county gets to decide how to manage the electoral process) and another part is that the people responsible for running elections are themselves elected partisans (the state “secretaries of state”, for example Katherine Harris and Kenneth Blackwell).

Rather than address these structural issues at the root (have elections run by a single federal organization with leadership acceptable to both parties), the USA keeps trying to fix the problem with new and better machines.

For reference, here is how we do it in the civilized world.

British Columbia provincial elections are managed by Elections BC, an independent agency of the government, the head of which is chosen by an all-party committee of the legislature. Federal elections are managed by Elections Canada, ditto.

Each polling station handles a number of polls, each of which has a few hundred people in it. Each poll has its own vote box. Each ballot is numbered, and comes from a book of ballots, so that the number of ballots that end up in a box can be reconciled to the number of voters in a poll, and to the books which were assigned to each poll. The polling stations are staffed by temporary paid workers, two workers to each box, so the box is never unattended. Parties are allowed to have “scrutineers” in the station, who may observe the process. Parties often keep an independent count of how many people have voted, and who has voted (this information is fed back to the central campaign and used to drive get-out-the-vote efforts, by calling known supporters who have not shown up to vote earlier in the day).

Paper ballot in sealed cardboard box. It's that easy

At the end of the day, each box is audibly counted, under the eyes of the scrutineers and the poll supervisor. It takes about an hour to an hour and a half to count all the ballots in a poll. Because the scrutineers are keeping tallies during the count, boxes will sometimes get re-counted if the tallies don’t match.

The effect of having an open process (everyone gets to watch, right down to the individual votes for each poll counts) both increases confidence in the process and the accuracy of the result, because many eyes are working on the problem at once. The poll supervisor reports the totals to his superior in the district, who in turn reports to the central electoral authorities. And the scrutineers independently report the poll numbers to their party headquarters, where they are totaled up to provide the politicians with an early snapshot of the race. Multiple eyes, multiple paths for the data to flow, physical ballots for post-facto processing.

In the event of a really close election, things slow down a lot, because a panel of judges has to sit down in a room and hand-count the complete set of votes for the whole riding. This takes about a week. It’s the difference between parallel processing (each box counted simultaneously on election day) and sequential processing.

The reason such a primitive system can work better than all the fancy US computers is because the standards for things like ballots, handling ballots, counting ballots, box security, etc, are all set and managed centrally, by an independent agency (not an elections equipment vendor). It is basic logistics. Standard processes mandated and followed consistently make logistically difficult problems (like accurately gathering and counting 2 million individually marked ballots) achievable.

Enough with the machines, USA, get a decent organization and return to the pen and paper!