Paul Ramsey

Twice as Dirty (and now with more Shame!)

2012-01-03T11:41:00.000-08:00

For all those Americans, who think your political culture is uniquely corrupt or tarnished by money or special interests, I give you, direct from Canada, in all its astroturfing glory, "Ethical Oil".

http://www.ethicaloil.org/

My favorite part is the donation area, "Please consider making a $5, $10 or $15 donation..." because of course ethicaloil.org relies "on small donors like you to sustain our grassroots advocacy". They won't take money "from foreign corporations, foundations, governments, or lobbyists", so it's a good thing there's lots of money available from CANADIAN corporations, foundations and lobbyists.

I took the kids to school this morning after reading this and especially this so my tolerance for people who think time-shifted mass murder is ETHICAL is a little lower than usual today.

SVR #2

2011-12-16T10:30:00.000-08:00

Hello, all my dark minions in the consulting industry!!! Have you ever low-bid a contract, knowing that once you got deep into it, the client would be as professionally invested in the success of the project as you, and would carry the can back to management for more funding? Come on, chums, you can be honest with me, didn't we row together at Oxford?

Sadly, our old mates at Oracle got their hands caught in the cookie jar recently. They thought they were building a strategic vendor relationship with Montclair State University. Everyone was friends, all pulling together for success, and then some loser decided to knife them in the back instead of being chums!

"When issues arose during the course of the project, it became clear that MSU's leadership did not adequately understand the technology and the steps necessary to complete the project," [Oracle] stated. "Instead of cooperating with Oracle and resolving issues through discussions and collaboration, MSU's project leadership, motivated by their own agenda and fearful of being blamed for delays, escalated manageable differences into major disputes."

Right ho! Instead of "cooperating" and "resolving through discussions and collaboration" (oh! and an extra $15M!) they created a major dispute. Bollocks! It's this kind of unfriendliness and lack of trust that can turn a super strategic friendship and awesome partnership relationship into a garden variety contractual business arrangement, and who wants one of those?!?

And, to add to the betrayal, I guess someone at MSU used to work in the consulting industry (zounds!)

"This is a textbook example of how to file a legal action against a vendor for failure to deliver," said analyst Ray Wang, CEO of Constellation Research, who reviewed the updated complaint on Wednesday.

MSU made some smart moves to protect itself, such as documenting all conversations and interactions with Oracle, and working out an escalation procedure in the event the project ran into problems. It also was wise to use real-life use cases for the demonstrations, Wang added.

I'm glad I live in a jurisdiction where clients and vendors know how to get along.

SVR

2011-12-16T08:30:00.000-08:00

I find much to love in the BC CIO's "IM/IT Enablers Strategy v1.5 for Citizens @ the Centre: B.C. Government 2.0" (well, perhaps not the name!) but there is one section that chills me to the core: Strategic Procurement.

At the heart of "strategic procurement" is the "strategic vendor relationship" (SVR), wherein "enhancing the government's relationship with key vendors [emphasis mine] will lead to more agility in responding to new needs, or making full use of emerging technologies". What part of working with "key vendors" enhances government's power in the vendor/customer relationship? Where do market forces come into play?

How will we know which vendors are strategic and which ones are a waste of our time? Will we play golf with them? And those non-strategic vendors, what of them? Do they get to play golf too?

Verily, there is only one place this leads, and the name of the beast is "Master Standing Agreement", or more colloquially, the "[HP|IBM|Accenture] Always Gets a Piece Act". The same actors will be arranged in the marketplace, but the small ones will only get to access work via large ones, who will always get a (the most) lucrative piece.

It's nice that the BC IT bureaucracy is coming to grips with its co-dependent relationship with the big consultancies, but unfortunate that the reaction is to formalize co-dependence as desirable in the master strategic plan.

(Note to readers: I've heard talk of a "vendors solutions center" or something like that floating around, anyone have any links or documents they can share?)

This (and everything else, it seems) reminds me my favourite technology joke, from circa 1995: How many Microsoft engineers does it take to change a lightbulb? None, Bill Gates just declares darkness to be the new standard.

F*ck You. Pay Me.

2011-12-15T09:09:00.000-08:00

I really enjoyed Mike Monteiro's talk, "F*ck you. Pay me." and it curiously fits into my current writing jag: finding value in your "cost centers".

The curse of IT is that it is a cost center, not a profit center. This is true even for an internet company like Amazon — the profit is in selling those damn books not in running the web site (OK, not true anymore, with Amazon Web Services, Amazon is actually monetizing IT directly now, but cut me some slack here).

As a cost center, the only way IT can directly contribute to the bottom line is to reduce costs. And if IT is managed in a silo, separate from line business (or worse, outsourced), things can go seriously sideways. Reduced services and responsiveness from IT can reduce effectiveness or hamstring line business areas that do generate profits. In chemistry terms, IT is not the reactant, it's the catalyst, but it's still critical to generating the reaction!

What does this have to do with Mike's profane little talk? One of his main points is that, in running his creative business, his legal advisor has been absolutely critical. As Mike says (pointing to his lawyer) "this guy makes me money". Well, no, taking the narrow view, the lawyer only ever receives money, he never gives it back. But in the global view, looking at all the money the company didn't lose due to bad contracts, or due to broken customer relationships, or due to misunderstandings of obligation, Mike's lawyer has generated far more value than he has billed out. He's a quiet profit center.

If you treat it right, IT can do that too.

More In House

2011-12-14T09:50:00.000-08:00

Hot on the heels of my post about doing work in house, InfoWorld's Deep End column takes a look at build-vs-buy.

Relying solely on support contracts and generic solutions is a good way to self-limit the agility and performance of any business. In short, more gurus equals less hand-wringing and stress all around.

In an era when software is eating the world, information agility is key to competitiveness (or, in government terms, "effective service delivery"), when competitors are investing heavily in brain power, why would any organization dumb itself down?

Do It In House

2011-12-07T09:33:00.001-08:00

This morning, I was struck by this nice write-up about how the Hungarian railway accurately geo-located and inventoried their assets:

The work was done entirely by MAV employees which made it much less expensive than if external contractors would have had to have been used. Overall it is estimated that as a result of using internal resources and a GPS/GLONASS-based approach the project was 16 times more efficient than a traditional survey. And the project generated a lot of pride among MAV employees who carried out the work because it was such a remarkable achievement from a data collection, management and quality perspective.

Here's is an incredibly stupid thing for a consultant to say, but nonetheless: if you can do it in house, why wouldn't you? Even if it's a bit of a stretch, your in house resources:

have a predictable cost structure
already understand your core business
have a feel for the historical reasoning behind business processes

New IT infrastructure is strategic almost by definition. Why would you outsource your most important strategic initiatives? If you're anticipating failure, perhaps it's a good idea. But if you succeed, you've just invested in building intellectual capital in a population of people outside your organization. And you've lowered the engagement of your core staff in the future of the organization.

Familiarity breeds contempt, and it's all too common that management is most contemptuous of the people they are most familiar with: their own staff. Hence the lure of the shiny consultant (love me! I'm shiny!).

2012 Code Sprint

2011-11-25T09:20:00.000-08:00

One of the nerdy highlights of my year is the always the annual North American code sprint (previously held in Toronto (2009), New-York (2010) and Montreal (2011)). It started out as a MapServer, C-Tribe kind of thing, but has also had participants from the OpenLayers and GeoServer community too. Basically, if you have an open source geoproject and a team of more than one, the sprint is an excellent opportunity to get face time and serious progress under your belt.

This year, the sprint is on Bainbridge Island near Seattle and the nature of the venue (all inclusive, room, board, meeting space) means pre-registration is de riguer. So if you're coming, please register now.

Federal NDP Leadership Poll

2011-11-14T14:25:00.000-08:00

The Federal NDP is in a leadership race, which means that candidates who have paid their entrance fee have access to the membership list, some 100,000 Canadians like myself. As a political observer and data fiend, who had access to such a list myself only this spring, I love watching to see how people make use of it: do we get the standard policy screed, the informative candidate-is-visiting message, or something more devious... like the below!So, an email arrives stating:

Dear Member of NDP,

I would really appreciate your participation in a study we’re currently conducting amongst members of the federal NDP.

I recognize that you’re busy, so this survey is very straightforward and can easily be completed online at your convenience, in about 15 minutes. Please complete the survey before Tuesday November 15th.

All information provided by respondents will be kept strictly confidential and used only for legitimate research purposes. Study sponsors will not have access to your name, address or phone number.

To begin the survey, simply click on the link below. If your email does not support hotlinks, copy and paste the link into your browser.

http://...omitted...

If you encounter any problems, please contact me at the e-mail address below.

Thank you in advance for participating in our survey.

Regards,

Agnes Klich
Project Management Team Leader
klich@logitgroup.com

The fact that they have the NDP membership list, and the content of the poll, lead me to believe it is associated with a leadership campaign in some way. But it's been done anonymously. The campaign that has done this both (a) gets the data and (b) pretty much ensures that anyone else trying the same gambit will get a much lower response rate. The poll itself is very long, I wonder how many full responses they get? I also wonder if there will be any blowback for using the list in this way? Based on the content of the poll, which campaign do you think is behind it? If the answer seems obvious, and you think there will be blowback, could the poll in fact be the product of a different campaign? Ain't politics grand?

Update: Just to leave no stone unturned, I asked the researcher who commissioned the study, but the answer is not illuminating:

The Logit Group is a Gold Seal Member Agency of the Marketing Research and Intelligence Associaton (MRIA), Canada's governing body for all market research firms. As such, we conform to all regulations related to privacy and confidentiality. In this instance, the organization that provided us with member lists required that the survey be completed in a confidential, or "blind" method (whereby the survey sponsor was not identified at the outset). In return, the responses of individual members (such as yourself) would not be attributed to you specifically when reported back (survey findings would only be provided in aggregate form).

Which language would you prefer to complete this survey in? Dans quelle langue préférez-vous répondre à ce sondage?

English / Anglais
French / Français

We are interviewing members of the NDP. Are you a member of this party?

How long have you been a member of the NDP?

Less than 1 year
1 to 2 years
2 to 5 years
5 to 10 years
10 to 15 years
15 to 20 years
Over 20 years

And in which province do you currently reside?

Alberta
British Columbia
Manitoba
New Brunswick
Newfoundland & Labrador
Nova Scotia
Ontario
Prince Edward Island
Quebec
Saskatchewan
Yukon
Northwest Territories
Nunavut

Are you...

Male
Female

Were you active for the NDP in the last federal election in May, 2011? (As a candidate, campaign worker, fundraiser, etc.)

As you may know, a leadership campaign has been scheduled for March 2012. Each member of the NDP will be eligible to cast a vote for their choice of leader.

As far as you know today, are you likely to be voting in this leadership election?

Yes
No
Not Sure / Depends

Thinking about the next federal election, do you think the NDP will win more seats, will win fewer seats or will win the same number of seats as it has currently?

Win more seats
Win fewer seats
Win same number of seats
Don't know

How confident are you that with a new leader, the NDP can defeat Stephen Harper in the next federal election and take over government?

Very confident
Somewhat confident
Not very confident
Not at all confident

How much do you think the NDP's chances of defeating Stephen Harper depends on who is selected as the new leader?

Depends a great deal
Depends somewhat
Depends a little
Does not depend at all

When you think about the federal NDP as a whole, where would you describe yourself in your own political thinking relative to the party?

Far to the right
Moderately to the right
In the centre
Moderately to the left
Far to the left

Some people say that they would like to know where each of the leadership candidates stand and want them to spell out in detail the policy direction they want to take the party in.

Others say that they don't want the candidates to present detailed policy because the party members and caucus should have a say in the party's future direction after the leadership campaign.

Which of these two points of view best represents your own thinking?

Want to know details
Party members should decide
Don't know / Not sure

Each political party has an establishment: a group of people who have been important to, and actively involved with, the party for some time either provincially, federally or both. What is your impression of the establishment of the NDP? Is it very favourable? Somewhat favourable? Somewhat unfavourable? Very unfavourable?

Very favourable
Somewhat favourable
Somewhat unfavourable
Very unfavourable

Do you consider yourself to be a member of the NDP establishment?

Yes
No
Don't know

There has been some discussion about whether the NDP should at some point discuss a relationship or merger with the Liberal Party and the federal level in order to unite the centre left of the political spectrum. What would be your opinion of a move to discuss such a relationship or merger at some point in time? Would you be strongly in favour? Somewhat in favour? Somewhat against? Strongly against?

Strongly in favour
Somewhat in favour
Somewhat against
Strongly against

As you may know, one of the leadership candidates is proposing that wealthier Canadians should pay more in taxes. Would this make it more or less likely that you would support a candidate who proposed this, or would it make no difference in your support?

More likely
Less likely
Makes no difference

One of the policy issues being debated these days is the creation of a carbon tax for individuals and corporations who use hydrocarbon fuels and the use of rewards for those who reduce their carbon usage. Would you be strongly in favour, somewhat in favour, somewhat against or strongly against the NDP supporting the introduction of a carbon tax?

Strongly in favour
Somewhat in favour
Somewhat against
Strongly against

Different people have been telling us that they are looking for various things in the next leader.

From your own personal point of view, how important is it that the leader you choose ... (READ EACH OF THE FOLLOWING STATEMENTS) ... ? Is it extremely important, very important, somewhat important, not very important or not important at all? (REPEAT FOR EACH STATEMENT)

Be closest ideologically to yourself
Be able to work with, and appeal to, the members of the party and the electorate in Quebec
Be someone who will be able to put together the best possible campaign team
Be someone who will provide the party with much the same direction as Jack Layton
Be someone who understands the concerns of your province
Be someone who will provide the party with a new direction
Be able to hold the party together
Be a currently elected MP
Be impeccably bilingual in English and French
Have experience in government
Be someone who has been a long term NDP member
Be someone who will be able to beat Stephen Harper in a debate
Be someone who is able to appeal to those who have traditionally voted NDP
Be someone who will have the support of unions
Be supported by the establishment of the party
Be someone who runs a positive leadership campaign and avoids attacking other leadership candidates
Be able to win the next federal election
Be someone with a broad base of support in caucus
Be someone who is able to appeal to those who have traditionally not voted NDP

Now we would like to ask you to think about the candidates who have declared for the NDP leadership.

What is your impression of ... (READ EACH CANDIDATE'S NAME) ... as a leadership candidate? Is it very favourable? Somewhat favourable? Somewhat unfavourable? Or very unfavourable? (REPEAT FOR EACH CANDIDATE)

Thomas Mulcair
Robert Chisholm
Brian Topp
Paul Dewar
Nicki Ashton
Nathan Cullen
Romeo Saganash
Martin Singh
Peggy Nash

Overall, based upon what you know about these individuals or have heard, where would you put each of these candidates on the political spectrum in relation to the Federal NDP? Far to the right? Somewhat to the right? In the centre? Somewhat to the left? Far to the left?

Robert Chisholm
Brian Topp
Nathan Cullen
Thomas Mulcair
Paul Dewar
Peggy Nash
Martin Singh
Nicki Ashton
Romeo Saganash

Thinking about some of the leadership candidates, in your view, which of them ...

Is most likely to win the leadership
Will be most supported by the establishment of the party
Is running the most positive leadership campaign and not attacking other leadership candidates
Will best understands the concerns of your province
Is best able to appeal to those who have traditionally voted NDP
Will be able to put together the best possible campaign team
Will be best able to work with, and appeal to, the members of the party and the electorate in Quebec
Will be best able to beat Stephen Harper in a debate
Will have the broadest base of support in caucus
Will have the support of the unions
Is best able to hold the party together
Is someone who will best provide the party with much the same direction as Jack Layton
Has been the longest term NDP member
Is someone who will best provide the party with a new direction
Is best able to win the next federal election
Is best able to appeal to those who have traditionally not voted NDP
Has most experience in government
Is closest ideologically to yourself

If the leadership vote was being held today, which of the leadership candidates would you be voting for? (Please select one only.)

Nicki Ashton
Robert Chisholm
Nathan Cullen
Paul Dewar
Thomas Mulcair
Peggy Nash
Romeo Saganash
Martin Singh
Brian Topp
Undecided

How likely is it that you will change your mind and support another candidate before election day?

Very Likely
Somewhat likely
Somewhat unlikely
Very unlikely

Who would be your second choice for leader?

Nicki Ashton
Robert Chisholm
Nathan Cullen
Paul Dewar
Thomas Mulcair
Peggy Nash
Romeo Saganash
Martin Singh
Brian Topp
Undecided

Who would be your third choice for leader?

Nicki Ashton
Robert Chisholm
Nathan Cullen
Paul Dewar
Thomas Mulcair
Peggy Nash
Romeo Saganash
Martin Singh
Brian Topp
Undecided

Are there any candidates for leader whom you could never vote for? (Please select all that apply.)

Nicki Ashton
Robert Chisholm
Nathan Cullen
Paul Dewar
Thomas Mulcair
Peggy Nash
Romeo Saganash
Martin Singh
Brian Topp
None - Could vote for any of them

Which of the candidates do you think Jack Layton would have wanted to succeed him as leader?

Nicki Ashton
Robert Chisholm
Nathan Cullen
Paul Dewar
Thomas Mulcair
Peggy Nash
Romeo Saganash
Martin Singh
Brian Topp
Undecided

Finally, I want to ask you about some of the personalities in the NDP.

If the following personality supported a particular leadership candidate, would that make it more or less likely that you would support that leadership candidate, or would it make no difference as to whom you supported.

Joy MacPhail
Adrian Dix
Lorne Nystrom
Daryl Dexter
Libby Davies
Dwain Lingenfelter
Greg Selinger
Ed Broadbent
Roy Romanow

I would like to ask your reactions to the following statements.

Please tell me whether you strongly agree, somewhat agree, somewhat disagree or strongly disagree with each of the following.

The NDP needs to move away from trade unions to win government.
Given how well the NDP did in Quebec in the past election, the next leader of the NDP should be from Quebec.
The next leader of the NDP needs to address inequality as a top priority
The NDP can win government without becoming Liberals by running as responsible, principled New Democrats
I would never vote for a leadership candidate who is not already a Member of Parliament
I would never vote for a leadership candidate who has close ties to the labour movement
The NDP needs to move closer to the political centre in order to be elected to government
It is important that the next NDP leader be a woman
I would never vote for a leadership candidate who was not perfectly bilingual

Some people say that the next NDP leader must continue with a positive approach to politics in order to motivate progressive voters to defeat Conservatives.

Others say that the next NDP leader must take a more confrontational approach to Stephen Harper because he needs to be faced down in order to be defeated in the next election.

Which of these statements is closest to your own point of view?

Take positive approach
Take confrontational approach
Don't know

Some people say that the federal NDP has been basically on the right track since Jack Layton was elect ed leader and it should continue with this proven approach under a new leader.

Others say that the NDP has been on the wrong track and won't win government unless the new leader makes fundamental changes to the workings of the party.

Which of these statements is closest to your own point of view?

On right track - continue
On wrong track - fundamental changes needed
Don't know

And finally, a few questions about you...

Which of the following best describes your current marital status...?

Single, never married
Married / cohabiting
Separated / divorced
Widowed

Do you, or did you, work in the private or public sector?

Private
Public
Neither

Are you currently, or have you ever been, a member of a union?

Are you a full-time student?

How would you describe the community in which you live?

Urban
Suburban
Rural

While I know that most of us would classify ourselves as Canadians first, can you please tell me what your specific ethnic background is?

White / Caucasian
Black / African Canadian / Caribbean Canadian
Hispanic / Latino
Asian
South Asian
Middle Eastern
Aboriginal / First Nation

Which of the following categories best reflects your total annual household income before taxes?

Less than $25,000
$25,000 to $50,000
$51,000 to $75,000
$76,000 to $99,999
$100,000 plus
Prefer not to say

Manager, Strategic Management

2011-11-13T08:43:00.001-08:00

Every week, LinkedIn kindly sends me a list of "jobs I might be interested in", which I have to say is an interesting feature, given the data they have to work with. Like the early days of Google advertising, it's fun to see what the algorithm comes up with as "relevant" to me. And this week I got this awesome, awesome, awesome, awesome entry:

Manager, Strategic Management
The Manager, Strategic Management is accountable for leading the development, maintenance and evaluation of corporate planning, performance management, benchmarking, risk management, and reporting programs. The position reports to the Chief Executive Officer of BC Assessment and works directly with the governing Board of Directors to facilitate strategic planning and risk management sessions. The position exercises considerable latitude and independence to oversee and develop a coordinated and consultative corporate plan, risk and performance management culture across BC Assessment. In this role, the position is expected to manage the corporate planning cycle to achieve a top to bottom linking of mandate and vision of operational business activities including the annual and year-over-year alignment of budgets, resource allocation, performance and risk management programs. This position leads a small team, including senior program analysts and a research officer.

People I trust tell me BC Assessment has so much money, they really do eat $16 muffins for breakfast, but the existence of the "Manager, Strategic Management" is all the proof I need.

FOSS4G 2011 Keynote

2011-09-20T10:20:00.001-07:00

Here's my latest foray into the wild and wacky world of Speaking in Front of Too Many People: a keynote slot at FOSS4G 2011. Fortunately a quick 20 minutes, and well received!

Update: Now available in hi-def with extra me courtesy of Peter Batty:

Architecture of Participation

2011-09-17T22:38:00.001-07:00

I told a number of folks at FOSS4G 2011 that I thought this year's event was the "best FOSS4G ever" (HT, Juan Antonio Samaranch) but that wasn't just tongue in cheek. 2011 was the biggest ever, but only a few attendees more than Barcelona in 2010. Yet somehow I felt more energized, more connected, like I had more conversations, than in 2010.

I think the reason for my impression has a lot to do with venue. Barcelona was in a very large conference center, with the rooms fairly spread out and almost too much room for people to expand into. Further, there were no large social areas near the venue. The result was the attendees dissipating after the end of the day's programming.

In Denver, many of the attendees were in the Sheraton, co-located with the program venue. The Winkoop brewery provided a space sufficient to bring in hundreds of attendees for the welcome social. The Sheraton building itself included two pubs capable of seating hundreds, and hundreds of attendees did in fact sit there. The gravitational effect was of people walking by, seeing their FOSS4G comrades, and joining in the group themselves. It was hard to drink alone at FOSS4G 2011!

We had a similar dynamic in 2007, though honestly, not as good, since our "main meeting pub" was closed one night for a private event, and we never had the thought to simply book it ourselves for our own group (live and learn). Nor were we able to co-locate the event with a conference hotel (using the Empress Hotel which abuts the Victoria convention centre would have been prohibitively expensive.)

I think the lesson for future organizers is to thing very carefully about venue and connectivity and social gravity. Ensure there is a social space where many can fit and can find each other for fortuitous meetings. Try to keep all the components of the event (venue, rooms, social areas) as close together as possible. Give as many opportunities (welcome social, random social, exhibitors social, event gala) for mixing as possible. Avoid the sit-down event (which locks folks into a handful of interlocutors) in favour of the stand-up (which allows free flowing).

Seeing how well Denver did makes me almost want to try again and see if I can do better. Almost.

FOSS4G Time

2011-08-20T17:38:00.000-07:00

A journey of 1000 miles starts with one step, and if you're wondering whether open source is in your future, the first step should be a trip to FOSS4G 2011 in Denver this year.

My favorite local open source geospatial success story, Pierce County, in Washington, started four years ago, when the GIS manager sent a small contingent north to attend FOSS4G 2007 in Victoria. From the knowledge and enthusiasm garnered there they have been able to start to re-make their infrastructure into a flexible mixed environment of open source and proprietary tools.

But it all starts with the basic knowledge. What's out there? Who can build with it? What are the best choices? You'll never have a more concentrated opportunity to answer all your questions about open source geospatial than FOSS4G.

I'll be there all week, teaching a workshop on PostGIS (sorry, sold out) and giving talks on introductory PostGIS and advanced PostGIS trickery.

See you all in Denver!

Open source is not free (as in beer) ...

2011-08-15T10:36:00.000-07:00

Last week, I took the time to ridicule a post at the PBBI blog on open source, which really boiled down to a critique of the very flimsy open source argument "open source is free (as in beer) so it's a budget panacea!". I hope not many open source advocates are retailing that one, but I'm sure one or two of us still are.

In my idealistic younger days I read a lot of Chomsky, and one of the general principles I took away from him is the idea that as citizens we have a duty to critique and criticize our own governments first and foremost. Beating our chests about the behaviour of some despot on the other side of an ocean might be cathartic, but if we're really interested in improving the commonweal it is our job as citizens of democratic societies to make our own governments better first. (Like, why does Canada sell asbestos as a building supply to third world countries when back home we recognize that it's too dangerous for our own buildings?)

Anyhow, in that spirit, it is really my responsibility, not PBBI's, to point out the disingenuousness of "open source is free (as in beer)" as a slogan.

So, first the counterargument.

I think most folks recognize that implementation costs will wash out software acquisition costs for any moderately complex system build. But we still, in our lizard brains, feel like the acquisition cost of proprietary software has to be a systemic problem. Perhaps because it's so painful at a personal level to whip out the credit card and plunk down a few hundred bucks for something like Microsoft Office.

However, when you examine the numbers, it gets very clear very fast that service costs are an order of magnitude larger than software acquisition costs. The BC Public Accounts tell the story for my home province: For the past ten years, ESRI has taken in between $1M and $2M a year pretty consistently selling GIS software. Over those same years, the consulting companies in town I would call "geospatial companies" have in aggregate billed about 10 times that annual figure providing services.

Services rule! The profit margins in services are notoriously lower than for software, and I'm sure ESRI made more pure profit on their $1M per year. But the bulk size of service revenues more than compensates. That's why IBM and HP are growing into services. Because there's so much room to grow in there.

Second, the refinement.

While "open source is free (as in beer)!" can be oversold, it is not actually untrue. Open source does actually have a price-tag of $0, and proprietary has a price-tag of > $0. So how to refine the slogan to not over-emphasize price? I have been using a modification which I hope focusses people more clearly on where the price can be important.

"Open source has a zero dollar capital cost."

So, given that the cost of services out-weighs software for most system building, you might not care about software acquisition cost if you're deploying the finished product once. But if you're deploying a system across 10 or 100 nodes, then the capital cost of deployment might be an extremely important economic variable.

In some cases (*cough* Oracle *cough*) the cost of just one node can be enough to push people over the edge on initial capital cost. But most proprietary software is more moderately priced, and it takes more nodes before the pure price differential is worth feeding into a comparison exercise.

I lay out more non-monetary reasons for bringing open source to the management table in my talk A Manager's Guide to Open Source.

Proprietary software is not the future you think it is...

2011-08-12T14:03:00.000-07:00

OK, so this morning James Fee tweeted this nugget of late 90's "open minded thinking about open source" from a proprietary vendor.

And if you think I can leave that alone... you just don't know me that well!

I'd go through and work on the arguments point-by-point, but there's hardly any argument there. It's almost all innuendo and unsupported assertion. So I just re-wrote it from the opposite point of view largely using search-and-replace. It holds up pretty well, because there's no content, just bluster. My favourite part is the paragraph on PostGIS, which I left unchanged, since it's actually a pretty powerful argument against proprietary software (your pro-proprietary argument is that Oracle is going to kick your ass if you use open source? awesome!)

Enjoy.

Proprietary software is not the future you think it is...

Read those words carefully. Read what they are, and understand what they are not. It could mean “open source zealot dismisses proprietary software”. It doesn’t. What it means is that while many preach the virtues of proprietary software, meaning supported, reliable, and guaranteed uptime with no technical expertise required – I do not subscribe. Closed platforms, be it maps or software, are fantastic and I believe a part of the IT industry as far as we can see into the future. OpenGeo, for example, uses proprietary software in its own products so the likes of the OpenGeo Suite connect easily to an ESRI ArcSDE server. But let me give you a couple of examples where I am coming from.

I saw a presentation a couple of years ago from a older gentleman – about my age – who had a 30 minute slot at a conference. For 25 minutes he exalted the virtues of his product, and how he can take this piece of software, and solve all kinds of business problems with just the touch of a button and you – yes you – can touch that button too. But he was also pushing his wares, and spent the final five minutes telling you how his product would be even more useful if you also bought all this other products. So the end result in that case was: this tool is great, but to be actually useful you have to buy all the other stuff. Come again? You mean its good, but I can only unlock the value by standardizing on your whole platform? But you just spent 25 minutes telling me how great this new stuff is and now I’m still being asked to open my wallet even more? Heckling ensued from the peanut gallery.

Let’s take another scenario. You have decided to invest in an standard vendor solution. To put this all together you decide to employ one or two low-end hacks from a technical training farm. Over the following 12 months these guys struggle (because they aren't very bright) but in the end you have something that works, and only requires a nightly reboot.

But a year or two down the line, your management decides they need the system to support ten times as many users, and provide 99.9% up-time. Maybe you don't have the budget to license that many cores. Now you've got to re-architect and re-write the system to scale and not require reboots anymore. And there's a problem. Your low-end hacks can't figure out how to do anything that's not in the "introduction" section of the manual or can't be done with the GUI. Given the choice between learning how to script a solution and spending 5 hours clicking the same buttons over and over, they choose the latter. Any why not? They are on salary, after all. You can see where this is going.

The point of course, is that proprietary software is not 100% reliable or scalable with zero effort. And proprietary software may not even be the best engineered solution. I am not asking you to ignore proprietary software and only use open source, because that in some ways puts you back to square one. Proprietary software has credible value to both system integrators providing sustainable solutions and individual organizations implementing systems.

Take Oracle as an example. It has been the industry standard for years, and is in many instances a credible alternative to the likes of PostgreSQL. But are you going to pay more or less for that Oracle DBA over the PostgreSQL DBA? Assuming it’s the same, or perhaps a little more for a specialist, then we’re into pure software licensing. I can guarantee Oracle aren’t about to roll over because of pricing. If a company has a site agreement with Oracle are you free to pick up another database? Even if it is free – or free to download? As some people say “spatial is special” we are increasingly seeing that IT departments disagree. Spatial is increasingly not viewed as special and won’t be treated as such.

I’ve already seen firsthand, organizations that have tried and given up on building a sustainable product to sell based entirely on proprietary software, and organizations do a complete U-turn on what they were told were "off-the-shelf" systems because of spiralling costs and concerns over maintenance. It’s not to say you shouldn’t take that path, but it is to say, it’s not easy, and in the long run it might not even be possible. The crux behind all of this is money. If we all had loads it wouldn’t matter so much. We’d buy whatever system we wanted, and lots of consultants to wire the black boxes together. But now more than ever we don’t have money. If an offer seems too good to be true, then it probably is. Proprietary software will continue to evolve and grow and I’ve no doubt more and more individuals and organizations will participate. Some organizations will use proprietary software for the press releases and "president's leadership" awards, others for the free lunches with sales reps. I still believe we are looking for products which give us the best of both worlds. Extensibility where we are comfortable, but somehow reducing the risk and ongoing support to minimal predictable cost.

A couple of weeks ago I said a more informed decision is likely to be a better one. Next time someone tells you proprietary software is cheaper overall and more reliable, have a little think about it. Use what is right, not what someone else says might be cheap.

Paul R

PS – This blog is using blogger – we use it because it’s a great tool for the job, not just because it’s closed and proprietary!

Data vs Reports

2011-07-20T13:53:00.000-07:00

I'm sure I'll have to spend a lot more time trolling the DataBC site before I have discovered all the jewels therein amongst the dross, but one thing I noticed while searching for "birth" was a large number of entries like this entry from Vital Statistics: "Table 25 - Infant Mortality by Gestational Age and Birth Weight, British Columbia, 2005"

Is this data? Kinda. There's numbers. It's an Excel sheet. But let's get real here, this is a report. It's a summary. That's why Vital Statistics has hundreds and hundreds of "data set" entries like this in the catalogue instead of the number they should have: ONE.

There is one database of births and deaths in the province. Strip out the names and addresses, leave the causes, dates, ages, postal codes, and release THAT please, instead of the hundreds of year-by-year (not even longitudinal summaries! arg!) summaries.

Guh, Goh!

2011-06-29T10:27:00.000-07:00

Remember Google Wave? Maybe not, but you might remember the hype. The 1 hour video introduction, the breathlessly handed out friend-of-a-friend invitations. Google Buzz, similarly. Google Health. Orkut. Knol. The list goes on.

The upside of trying lots of things is that you sometimes catch a winner, I guess. Maps and location were winners, but they have a natural synergy with the search / advertising business.

The downside is that Google has burned up all it's cachet. When 80% of the products you launch bomb, people eventually assume everything you launch is going to bomb. Which, in the self-levitating bootstrapping world of network services and network effects means they have already bombed. Google+, we hardly knew thee.

Visualizing Progress

2011-06-25T11:08:00.000-07:00

Data visualization gets (from me) a bad rap, because so often it is used as a way to make fundamentally unsexy things just sexy enough for management to buy, without reference to whether the item in question is, at core, useful or effective.

Caveat aside, this graph from the GeoTools JIRA side made my hair stand on end.

Yikes, do you close more tickets than get opened? There's a visualization that lights a fire under my butt!

Even when I'm in full ticket-hunting-mode I feel like I sometimes am not making forward progress, since Regina gets into bug-finding-mode and starts opening them up on me faster. And the last year of adding features to PostGIS (raster, topology) while altering core structures (serialization, indexes) has created an avalanche of tickets. More than fits on a page. Time to get closing!

To the Cloud!

2011-06-20T12:03:00.000-07:00

David Pogue has a little mini-rant today on the use of "cloud" as a generic term to mean "internet" which I found fun, if only because it reminded me of the odd origins of the term.

As far as I can guess, the term leaked out of white-board shorthand used by folks drawing up high-level architecture diagrams. Who it was who popularized that diagram convention is a mystery, probably it is an elaboration on some even earlier trope. Now, thanks to Microsoft marketing, the term means... nothing? everything? kung fu music? computers-can-magically-solve-anything?

IT as Obstacle

2011-06-17T09:18:00.000-07:00

Getting the re-tweeting rounds today, an article on why business are moving to the cloud. The answer? To avoid their own IT departments.

Their top reasons for going around IT? The need to respond quickly to changes in the market, self-sufficiency of their IT-savvy workforce, and the easy availability of top-quality it services that can be bought without long implementation or testing (cloud and SAAs apps, primarily).

Watch out, the natives are restless.

Update: Check out the comment thread in the slashdot.org posting of this article for the good, bad and ugly of IT reaction to being sidelined. Ranges from "stupid users" to some thoughtful comments on the nature of organizations.

Outsourcing in BC

2011-06-17T07:50:00.000-07:00

Every year, in a fit of morbid curiousity, I cruise the detailed schedule of payments in the BC Public Accounts as a way of keeping tabs on the local IT industry. Who is up? Who is down? Who is gone?

One of the certainties, year after year, is that IBM's take just keeps getting bigger. I thought it was pretty impressive when they were doing $30M / year, but last year they cracked $100M.

Another trend has been the slow squeeze of the market place from one with a mix of companies, ranging from 10-person consultancies to 100-person regional integrators to the big international firms, to a market dominated almost completely by the internationals. The mid-size companies were largely bought up, and the small ones squeezed out. Deal sizes have kept growing, making it hard for smaller companies to compete, while the migration of infrastructure roles to outsourcing companies has given those same companies an built-in edge in competing for systems work. Even long-time regional heavy-weight Sierra Systems is getting beaten down.

There are many big mistakes being made by government as this process goes on, but perhaps the biggest is the long-term sole-source outsourcing agreement. The fantasy that costs can be managed in a single vendor environment, that the contract just needs the right review clauses and penalties, continues to suck in government outsourcers. There's no true competition in a system where a vendor only needs to compete every 5-10 years (and even then, from the position of the incumbent). Expecting market efficiencies to work in your favour in such as situation is folly.

Postscript: In 2009/2010, HP and IBM combined to bill the BC government over $200M, mostly in services. IBM's 2010 annual report indicates 34.7% gross margin for IBM Global Services. The HP annual report shows a similar margin for services. The implication is, of the $200M we are spending with HP and IBM, $70M is going to profit, not to providing service. Put another way, if the government chose to provide those services internally, it could pay all the staff at HP/IBM salary rates and still afford to employ 50% more people to do the work.

Superstar CIOs

2011-06-16T16:04:00.000-07:00

The superstar CIO is necessarily going to be a bit of a flimflam artist. He has to be, because in order to get into the position, he's had to dazzle his non-technical superiors. That means eye-candy, first and foremost, delivered fast. Delivered fast, you say, how can he do that with his sclerotic organization? Easy, he contracts off to the side or builds a private skunk-works.

So, from Vivek Kundra (US Federal CIO) on the one hand we have the flimflam of data.gov, and various high level dashboard trackers (recovery.gov anyone?), tossed up quickly by a tiger team. Not indicative of transformation, but rather the opposite: in order to get anything completed quickly the CIO has to actually route around his existing organization. But there is also some substance.

Some of the substance is pure management, like the review of failing projects. Some is technological silver-bullet-worship (of a sort I can appreciate, being a long-time silver bullet lover) like the move to government use of cloud infrastructure (both external and self-managed). (Because the answer is always another layer of abstraction!)

But one thing comes through clearly watching Mr. Kundra: it's a really big organization and he's just one guy. The culture will out-live him. The men in grey suits will win. In some ways a full-on flimflam artist would be better, because at least then I wouldn't be disappointed when nothing changes.

Postscript: Kundra did manage to change some things but many of his marquee initiatives were separately funded initiatives that Congress is now de-funding for political reasons. And the news today is that he's leaving his post for academe to take a well-earned break. Did he push the culture in the right direction? Probably so!

IT Revolution

2011-06-16T08:00:00.000-07:00

Back when I first started working in the British Columbia government, in 1993, there existed a Crown Corporation called "BC Systems Corporation", BCSC. Through the 1980s, major pieces of government operations (timber tenuring, financial tracking, health insurance) had been computerized using the finest technology of the era (mostly mainframes) and BCSC had grown up as the central IT branch of the government. They ran the big machines, the terminals that accessed them and were taking on the newfangled PCs increasingly showing up on the desks of drones like me (I answered correspondence, there still being a brisk trade in "mail" at the time). The extent to which BCSC was wedded to the technology of the previous era can best be understood by noting that one of their last major accomplishments was writing a web browser, Charlotte, for the 3270 green-screen terminal.

Being stocked full of the finest minds in computerdom, BCSC briskly alienated pretty much all the folks it was supposed to be serving. It's services were too expensive, the old mainframe systems too inflexible, the response times too long. Shortly after I left government to finish my graduate degree, BCSC was blown up and its functions distributed to IT departments in the Ministries themselves.

And so the process began anew. Just as BCSC had stitched together systems from multiple ministries into a centralized empire over a decade, so too did the centralizing imperative begin to work immediately once again. First e-mail. Then major data centre needs. First by choice, then by fiat, gradually the Ministry IT departments were stripped of line responsibility for their systems and left with a residue of business analysts and managers.

A new wrinkle was added this time around though, in that rather than centralizing entirely to another agency, many of the services were centralized and then bundled off to outside companies. So desktop support was re-centralized and taken over by IBM. The result has been service just as poor as the old days (worse, in fact! back in the BCSC days there was a PC tech co-located right on our office floor, a luxury most government workers could hardly imagine now) with prices just as high.

BCSC was blown up when Ministries realized that they could provide the same services for themselves that BCSC did using PCs, and started doing so. Once the mainframe boys weren't necessary any more, they were dispensed with, joyfully. Ministries wanted to Get Things Done, and the main role of BCSC was to say You Can't Do That.

Today we have another round of folks saying You Can't Do That. The desktop outsource vendor says you can't have a new computer, and if you do get one, you can't install any software on it. The server outsource vendor says you can't afford a system to run your code on. Or if you can, then you need to go through a three month process to get the server first. And the CIO's office says you can't run that programming language because it's not government standard.

And here comes the cloud. Just as the PC freed the Ministries from the mainframe boys, the cloud is going to free folks from IT once again. Need a web site? Need a small system? Maybe an App Engine deployment. Need some data processing? Maybe an EC2 server will do the trick. It's all well under credit card spending limits. (My first government supervisor, bless her heart, bought me a copy of MS Access on her department credit card so I could avoid the All Databases Must Be Oracle directive and actually Get Some Work Done On My Computer).

The revolution is coming, it'll start quiet, but it'll get ugly. Hopefully soon.

Postscript: Michael Weisman, in a comment on my last post points to an article in the Washington Post about superstar CIO Vivek Kundra (coincidentally, the topic of another upcoming post of mine). Says Mr. Kundra, "People have better access to information technology at their homes than they do at work, and that’s especially true in the public sector". And another great quote from the article: Asked what he typically hears from workers about government- or corporate-provided technology, Kundra said, “It’s not a question of whether they don’t like it. They despise it.”

Kundra seems to have hit upon a winning formula: stop saying "no" and start saying "yes". The career risk of saying "no" is now higher than the risk of saying "yes". Welcome to the new conservatism. Give the people what they want, or things will get... unpleasant.

Broken Enterprise IT

2011-06-15T15:30:00.000-07:00

For as long as I have been doing technology, I have watched enterprise IT and said to myself "this seems incredibly broken, why do people put up with this?" This includes the period when I was doing consulting and running teams that specifically had to mould their reporting and processes to fit into the requirements of government (enterprise) IT. Actually doing it didn't make it seem any more sane.

At this point, I should explain what seemed wrong. The processes around IT actions in the enterprise environment (and my experience was in government, but my glancing exposure to Fortune 500 environments didn't uncover any great differences there) all seemed geared towards slowing down progress. The many layers of approvals and plans, the disconnection between operations and development, the migration of core technical decisions upwards to non-technical management staff.

Compare the outputs of a start-up company, turning out an aesthetically pleasing piece of software usable by ordinary folks within a timeline of a few months to that of an enterprise system building team that may or may not deliver something ugly and slow in a couple years. What's different? The people are generally of the similar capability and intelligence. But the culture and processes they are embedded in are radically different.

As a 25-year-old smart-ass I was always pretty sure a team of 5 working outside the enterprise IT structure could produce a better system faster than a team of any size working within it. As a 40-year-old smart ass, for some reason I still haven't altered that conception. I do, however, have more sympathy for the folks working within the system. As a 25-year-old I naturally assumed they were just stupid. As a 40-year-old I now see them as trapped. Very smart, capable people, stuck within a system that seems built to frustrate and block them from actually Making Their Computers Do Useful Things for Other People.

The genesis of the enterprise IT homunculus seems semi-explainable. If you put a large collection of detail oriented control freaks (hello, computer people) into a single organization, the need of the (detail oriented, control freak) management to avoid non-linearity in their organization will naturally lead to layer upon layer of process and reporting. Otherwise, how can the CIO sleep at night?

But even if the genesis is explainable, that still doesn't make the result good, or the end state desirable. This is still a system wasting the potential of the greater number of its participants. The non-technical masters of these organizations (CEOs, other upper managers) see clearly that they are broken (they manifestly cannot deliver), but the fixes only seem to make things worse: my organization is broken, so I'll outsource to another organization; my organization is broken, so I'll bring in consultants (who carry the same organizational biases); my organization is broken, so I'll go ISO9000 (embrace the disfunction!).

The British Columbia government seems to be in the middle of the outsourcing "fix" right now. The remains of the central IT department are being reviewed for their incredibly high internal pricing, and meanwhile major server and network infrastructure is being outsourced to HP and shipped out of province. The irony is that, should the central IT department be found to be "too inefficient" the "solution" will be another round of outsourcing to a contractor who will end up being (contractually) largely immune to any review.

I think we are only a few years from a serious IT revolution in the provincial government, and by revolution I mean the peasants are going to take up their pitchforks and torches and storm the keep. More on that tomorrow.

In the meanwhile, I am collecting stories of IT transformations that have actually worked. It seems like there are very few if any stories out there. The super-star CIOs seem to rely on various combinations of smoke and mirrors to burnish their reputations, very few seem to have attacked the root of the IT cultures in their organizations. But I would love to hear some stories about those who did and succeeded (or failed).

Keynote May

2011-05-05T08:58:00.000-07:00

It's a little too late now to promote my keynote presentation at ASPRS 2011, since I already gave it on Tuesday.... but! I can still promote my upcoming keynote at PgCon, which I'm looking forward to immensely (and getting nervous about in equal measure).

The ASPRS talk went well (I thought), delivering a little capsule history of open source and reasons why managers might (should) want to start adopting it (I had to shorten my Duluth version of the talk a bit to fit the ASPRS time slot). The reviews afterwards were positive (Twitter is an attention-hog's wet dream (said the attention-hog)).

The PgCon talk will be a new challenge. I'm usually generalizing my technical knowledge up for a less specialized audience of geo-people. But for PgCon I have to narrow and focus my general geo-knowledge down for a very specialized audience of PostgreSQL database developers. Having a room full of the foremost experts and developers of PostgreSQL to talk to is the most daunting audience I've faced since my graduate oral exams.

However, my reward after the keynote will be two days of attending talks by those same foremost experts, so I'm really looking forward to PgCon! Hope to see you there!

Spatial Partitioning in PostGIS

2011-02-18T16:37:00.000-08:00

I've been meaning for a long time to see what an implementation of spatial partitioning in PostGIS would look like, and a trip next week to the Center for Topographic Information in Sherbrooke had given me the excuse to try a toy implementation.

Imaging map data constrained to a 1km square. Here is an example that partitions that square into a left- and right-side, and then inserts the data appropriately into the right table as it comes in. Features that straddle the two halves get put into a third special-case table for handling overlaps.



-- parent table
create table km (
id integer,
geom geometry
);

-- left side
create table km_left(
check ( _st_contains(st_makeenvelope(0,0,500,1000,-1),geom ) )
) inherits (km);

-- right side
create table km_right(
check ( _st_contains(st_makeenvelope(500,0,1000,1000,-1),geom ) )
) inherits (km);

-- border overlaps
create table km_overlaps(
check ( _st_intersects(st_makeline(st_makepoint(500,0),st_makepoint(500,1000)),geom ) )
) inherits (km);

-- indexes
create index km_left_gix on km_left using gist (geom);
create index km_right_gix on km_right using gist (geom);
create index km_overlaps_gix on km_overlaps using gist (geom);

-- direct insert to appropriate table
CREATE OR REPLACE FUNCTION km_insert_trigger()
RETURNS TRIGGER AS $$
BEGIN
    IF ( _st_contains(st_makeenvelope(0,0,500,1000,-1),NEW.geom) ) THEN
    INSERT INTO km_left VALUES (NEW.*);
    ELSIF ( _st_contains(st_makeenvelope(500,0,1000,1000,-1),NEW.geom) ) THEN
    INSERT INTO km_right VALUES (NEW.*);
    ELSEif ( _st_intersects(st_makeline(st_makepoint(500,0),st_makepoint(500,1000)),NEW.geom) ) THEN
    INSERT INTO km_overlaps VALUES (NEW.*);
    ELSE
    RAISE EXCEPTION 'Geometry out of range.';
    END IF;
    RETURN NULL;
END;
$$
LANGUAGE plpgsql;

CREATE TRIGGER insert_km_trigger
    BEFORE INSERT ON km
    FOR EACH ROW EXECUTE PROCEDURE km_insert_trigger();

-- add some data
insert into km (id, geom) values (1, 'POINT(50 50)');
insert into km (id, geom) values (2, 'POINT(550 50)');
insert into km (id, geom) values (3, 'LINESTRING(250 250, 750 750)');

-- see where it lands
select * from km;
select * from km_right;
select * from km where st_contains('POLYGON((20 20,20 60,60 60,60 20,20 20))',geom);

It's still a toy, I need to put more data in it and see how well the indexes and constraint exclusion mechanisms actually work in this case.

ESRI FGDB API: No Servers need Apply

2011-01-25T14:58:00.000-08:00

When downloading the ESRI file-based geodatabase API you are required to accept a license agreement which includes this lovely clause in the section about acceptabe uses of the API:

"Single Use License." Licensee may permit a single authorized end user to install and use the Software, Data, and Documentation on a single computer for use by that end user on the computer on which the Software is installed. Remote access is not permitted. Licensee may permit the single authorized end user to make a second copy for end user's exclusive use on a portable computer as long as only one (1) copy of the Software, Data, and Documentation is in use at any one (1) time. No other end user may use the Software, Data, or Documentation under the same license at the same time for any other purpose.

Note, only one user may use the software, and remote access is not permitted. That is, you can't run a mapping server on top of this API, because then multiple users would be using it, remotely. Impressive. Point to Redlands. I will console myself in the usual way: "better than nothing."

Update: Martin Daly asked me to take a second look, and someone Twittered asking if the clause was boilerplate. On second examination, it could simply be boilerplate used for other desktop-oriented software that has been used without thought as to how it muddies the uses for non-desktop systems. The full text is here for those with ESRI logins. Never attribute to malice what you can attribute to accident, another good rule to live by (but how empty I would be without my little conspiracy theories!). The precautionary principle would dictate not web-serving with it until/unless it is clarified, but in the meanwhile bulk loaders and translators could certainly be put together.

Update 2: Though the I find the license language somewhat unclear, the intent appears to be to allow any application to sit on top of the API. Crisis averted, situation normal.

Update 3: Just got off the plane and the ESRI folks have left me a nice voice mail explaining that the intent of the clause is with respect to developers downloading it (once, for one person) not for uses of derived products. The moral of the story is that ESRI people are pretty darn nice, and I am not so very nice. Something for me to work on in the new year.

Update 4: Received email from ESRI confirming that the final license will be reviewed to ensure there are no ambiguities and that it reflects their intent that the API be usable by any application in any application category and the derived product freely redistributable and royalty free. So to the extent that the current license has any ambiguity it shouldn't be considered a red flag that the final one will.

BC: Power up your Vote!

2011-01-13T10:35:00.000-08:00

I gave this spiel at the Victoria GeoGeeks meet-up last week, but it's worth giving to the world at large: if you want your vote to count in choosing your representatives, exercise it at every opportunity! For those in the USA, that means participating in primary elections, and for us in British Columbia right now that means participating in the party leadership contests.

When you participate in early elections like primaries or leadership races, your vote gets more powerful.

In the last British Columbia election (2009) 1,651,567 people voted to choose the government. But they didn't get to choose the Premier, they just got to choose the party or local candidate. The next Premier of BC will be the leader of either the BC Liberals or the BC NDP. The combined membership of those races will probably be on the order of 60 thousand. So a vote cast in the leadership contests is about 27 times more determinative of who will be the Premier than a General Election vote.

Who wouldn't want their vote to be 27 times more powerful?!? But here's the rub: only party members get to vote.

For the BC Liberals, the election will occur on February 26, but the cut-off for membership is 5PM on February 4. You can sign-up online here ($10 annual membership).

For the BC New Democrats, the election will occur on April 17, but the cut-off is midnight on January 17, and you can sign-up online here ($10 minimum donation).

If you live in BC, this is your chance to have an outsized influence on who makes the decisions for our future. Get to it! I have!

We interrupt this geeky spring...

2011-01-11T09:49:00.001-08:00

To do something completely different. British Columbia, my fair home, is in the midst of an odd period where both major parties are having leadership races. The race for my party of choice (the NDP) has just gotten started, and I believe that pitching in early for what you believe in is a way to have maximum effect. So I'm spending my spring working as much as possible for John Horgan. John's someone who reflects my belief nobody has all the right answers, and he's someone with the personal skills (that I envy) to both listen for the answers and bring folks together around a common direction. And he's fun to be around, which is a nice bonus.

Unfortunately for PostGIS, time doing campaign work takes from time spent on PostGIS 2.0, so unless another PSC member has an equal and opposite infusion of time to mine, things will slow down a bit for the next couple months. I'll still be going to the Montreal code sprint though: some things are sacrosanct.

Victoria GeoGeeks!

2010-12-30T21:58:00.001-08:00

The Victoria geo-geeks are meeting again next week, on Thursday, January 6, at 6pm at the offices of Latitude Geographics. We'll be hearing from four speakers, on the topics of Google Fusion Tables and Open (Geo) Data in Government.

Google Fusion Tables, Intro and Demo - Josh Livni, Google Outreach Engineer (30m)
Open (Geo) Data in BC - Herb Lainchbury and Greg Lawrence (25m)
Open (Geo) Data Rebuttal - Mark Sondheim (5m)

Hope to see all you Victoria geo-geeks there!

Removing Complexities

2010-11-22T15:14:00.001-08:00

My co-worker Alyssa Wright just asked me an interesting question: Vancouver Island is a pretty crinkley island, how would we simplify it to remove the crinkley bits?

Notice the big fjords and other fine work by Slartibartfast. ST_Simplify will not do what we need, since it removes redundant vertices within a tolerance. So the fjords will have fewer vertices, but they'll still be there.

However, it turns out there is a solution, and I didn't even need a long walk in the snow to figure it out. If we buffer by a large amount, then reverse the buffer by the same amount, we'll get something that has a similar shape to the original, but without the crinkly bits.

Buffer out, and the crinkley bits get melted away. But now the island is too big. So buffer inwards.

And now we have something we can use. We could now even run ST_Simplify on this result to drop some unneeded vertices and make it smaller yet.

Looking closer, you can see how we have melted away all the crinkley bits while still following the original (mostly). Note that the new shape is still a strict superset of the original.

FOI @ OpenGov BC

2010-11-16T16:07:00.000-08:00

This is what I did last Wednesday while recovering from my Japanese jet-lag: panel wrangling at OpenGovWest BC. Probably the easiest moderating session I'll ever get, four FOI superstars, just wind them up and let them go.

Freedom of Information (FOI) panel from opengovwestbc.

Victoria GeoGeeks Meetup

2010-11-15T16:01:00.000-08:00

Update: Twelve people RSVP'ed so far for this month's meet!

If you live in Victoria, BC, and you love geotechnology, I hope you'll join the monthly geogeekery meetup we are starting. The first Thursday of every month, at the Sticky Wicket pub. Our first gathering will be December 2, see you there!

Cool Japanese Things #4

2010-11-05T16:35:00.001-07:00

All you can eat/drink reception. You have two and a half hours.... go!
Fast trains. Because faster is better. Really.
Vending machine restaurants. No wait staff, you purchase your order from a vending machine, which gives you tickets you hand to the cooks. Order up!
Cheap food. If not cheap, at least not expensive, dinner out at Osaka pub with slightly more sake than I could handle and full up on (over-the-moon tasty) snack/tapas plates for about $40.

Cool Japanese Things #3

2010-11-01T07:45:00.000-07:00

Umbrella lockers. In the art museum, there are standard left lockers for storing your bags and coats, but also little vertical lockers for locking up your umbrella while you tour the building! Is this a rainy city? I am thinking, yes.
Vertical gas station. Why waste space with gas pumps at ground level when you can just hang the filling hoses down from the ceiling? Hai!
Blogging from the car. Taichi has a mifi box, so the laptop works even while we are travelling underneath the triple-decker freeway, heading back to the guesthouse. Zoom zoom zoom.

Cool Japanese Things #2

2010-10-31T17:49:00.000-07:00

Talks with lots of demos. When you can't understand the words, nothing talks like demos! I learned about bits of HTML5 for rendering, orientation sensors, location awareness and drag-and-drop!
Rice for breakfast. Wrapped in a belt of seaweed to hold it together, a patty of rice about the size of a hockey puck is not that different from a breakfast roll. Should have gotten two.
Insane double decker freeways. Running through a canyon of buildings, there's something very blade-runnerish about the multi-level freeways. How far away is the ground?

Only one not-cool thing, being away from home on Halowe'en. Family is gathered to have wine and snacks right about now and the kids are bouncing off the walls getting ready to head out. Wish I was there.

Cool Japanese Things #1

2010-10-30T15:49:00.001-07:00

I am going to feel the lack of a camera on this trip. I didn't even bother to bring my phone because it would have been a functionless brick.

Techno toilet seats. The one in the hotel had an explanation printed on it about why it was not heated. Nonetheless it still had three knobs. There is much more to learn about Japanese toilet seats. Looking forward to it.
Defogged mirror. I took a hot shower, got out, and there was a neat square of unfogged mirror area right over the sink! Touch it, yes, it's warmed from behind to keep the class clear of condensation.
Hotel kimono. I don't feel like getting dressed yes, and I don't feel like sitting around writing blog posts in the nude. Enter the hotel kimono! It's a little short for me, but that's OK.
Taxi seats with doilies. Taxi this morning had pretty white lace doilies on the backs of the seats. Also a pristine white seat cover on the back. Curiously despite being lovingly cared for the car itself looked like it dated from 1989.

OpenGovWest BC

2010-10-19T12:58:00.001-07:00

There's an open government conference coming up next month, on November 10 at the University of Victoria: OpenGovWest BC! We are gathering some great local stories, from Vancouver and the Provincial and Federal governments, and a great group of attendees — the kinds of folks looking to change the way government works. If you're in the area and want to join in, I encourage you to register soon, the fee is nominal and the venue will fill up fast.

PostGIS Back-up / Restore

2010-09-30T09:47:00.000-07:00

A very common question on PostGIS in production is "how do I upgrade", which is actually a variant on "how do I backup and restore"?

First, for patch version increases (e.g. X.Y.Z -> X.Y.(Z+1)) in PostgreSQL and PostGIS you do not need to do anything at all other than install the new software. The data can remain in place and everything will Just Work.

For minor version increases in PostgreSQL (e.g. X.Y.Z -> X.(Y+1).Z) you need to dump and restore. For minor version increases in PostGIS, you need to do a "soft upgrade", which means leaving the data in place, but running the upgrade scripts (e.g. postgis_upgrade_14_to_15.sql) after you install the software update.

Finally, for major version increases in PostgreSQL and PostGIS (e.g. X.Y.Z -> (X+1).Y.Z) you need to dump and restore.

Which brings me to the actual point I want to make: you can ensure the greatest ease in doing dump and restore of PostGIS data if you ensure that you store no data in the "public" schema.

The "public" schema is where the PostGIS functions and system tables get installed, so if you dump that schema you get all those definitions in your dump. If those definitions are mixed in amongst your data, loading them into a fresh database gets tricky: are the paths to the libraries the same? are there function name clashes? (The utils/postgis_restore.pl script attempts manfully to strip out PostGIS components from a dump file to allow a clean restore, but it is hard to get 100% performance.)

If, on the other hand, all your data is neatly separated into its own schema, you can neatly backup just that schema and avoid having PostGIS system information mixed in with your data. That means you can easily restore your data into any version of PostGIS and PostgreSQL that you like. So upgrades are easy easy easy.

Remember: Store your data in a schema other than "public".

"But Paul," you say, "I already have a full database dump, does that mean I'm SOL?" No, but you will need a strong stomach. First, set up your new PostgreSQL. Create a blank database, load PostGIS into it. Now, load your backup file into that database. You will see lots of errors. However, these errors will be caused by old PostGIS function and type definitions from your dump file conflicting with the existing type definitions in your database. And since you want the new definitions, not the old ones, that's OK. Your load, despite all the noise and errors, should actually work. Once it's done, you can move your data into a nice separate schema, so that next time you can do a clean, error-free restore.

WPS Smackdown

2010-09-20T11:23:00.000-07:00

I was a little surprised last week when Jody Garnett told me that the most contentious session of FOSS4G turned out to be the "WPS compliance testing" talk. Really? "Really", Jody insisted, "Voices were raised."

Over web processing service (WPS) compliance? Well, we are all knee-jerk nerds, so I guess I shouldn't be surprised. The results were presented by just one WPS project, the 52North one, but tested a broad swathe of them. I think the lesson to be learned is that you should be careful testing other people's software, and even more careful when brandishing the results of those tests.

The WMS benchmarking exercise, where each team runs their own service, seems like a better model, though demonstrably hard to coordinate in its own right.

The ZooWPS team has now published their response, so if you are interested in WPS, here's a chance to dig deep into XML response payloads. Yum.

PostGIS 1.5.2rc1

2010-09-20T09:39:00.000-07:00

The PostGIS web site is temporarily down, but the project team is pushing ahead (belatedly) with the 1.5.2 release series in the meanwhile. The reason we are not waiting is the same reason we are belated: 1.5.2 is required to run with PostgreSQL 9.0, which was freshly released recently (we should have released 1.5.2 a couple months ago, really).

You can get the 1.5.2rc1 source code from the PostGIS issue tracking page and wiki: http://trac.osgeo.org/postgis

Please test and post any issues as tickets, we are aiming to drop a final release pretty quickly. Try to confine yourself to posting deal-breaker tickets only.

ST_Intersects and ST_Buffer: No

2010-08-20T16:28:00.001-07:00

If you find yourself writing a query like this:

... WHERE ST_Intersects(ST_Buffer(g1, r), g2)

Stop. Take a cleansing breath. Do this:

... WHERE ST_DWithin(g1, g2, r)

With the carbon emissions you save doing it the efficient way, you can afford to drive to the ice cream store for a well-deserved reward.

Calling all Technoweenies...

2010-08-16T11:18:00.000-07:00

Did you know that OpenGeo is hiring a software engineer? It's true. Let's build something beautiful together.

Take this Tree and Pack it

2010-08-06T15:51:00.000-07:00

Noticing this a little late, but have a peak at these posts from Chris Hodgson about how the R-Tree fails for variable density GIS data, and his approach to a packing process. Unfortunately, packing is a post-facto process, and it's not clear to me how we would do it with the GiST infrastructure that undergirds the PostGIS database. But it's nice to see a good R-Tree and a reminder of just how ugly they can get under the covers.

FOSS4G Reminder

2010-07-27T09:29:00.001-07:00

OK, this is the last time I'm going to remind you: FOSS4G 2010 is coming up fast. September 6-9 in Barcelona, Spain. The presentation call was massive (360 submissions for 120 slots) as was the workshop call. So register soon and make your hotel reservations while you can! That is all.

On Cable Modems

2010-07-22T08:11:00.000-07:00

I had cable guys in my house turning on basic cable, and they noticed that my cable modem was old. When I say "old", I mean "I think I got it when we moved into the house almost 10 years ago". So they said, "here, have a new one". My maximum download speed quadrupled overnight. If you've got an old cable modem, figure an excuse to get a new one.

Network Walking in PostGIS

2010-07-21T14:41:00.000-07:00

One of the new features in PostgreSQL 8.4 was the “WITH RECURSIVE” clause available for queries. It allows you to define a subquery based on a recursive term — fancy language for a function that calls itself. One of the favorite uses of recursion is walking a network. Geospatial applications use networks all the time: electrical grids, stream systems, and storm sewers are all directed networks (they have unidirectional flow).

Here’s an example of network walking using a simple collection of segments. As is common in many GIS applications, the segment are implicitly connected — their end points are coincident with the start points of other segments.

CREATE TABLE network ( segment geometry, id integer primary key );

INSERT INTO network VALUES ('LINESTRING(1 1, 0 0)', 1);
INSERT INTO network VALUES ('LINESTRING(2 1, 1 1)', 2);
INSERT INTO network VALUES ('LINESTRING(1 2, 1 1)', 3);
INSERT INTO network VALUES ('LINESTRING(3 1, 2 1)', 4);
INSERT INTO network VALUES ('LINESTRING(3 2, 2 1)', 5);
INSERT INTO network VALUES ('LINESTRING(2 3, 1 2)', 6);
INSERT INTO network VALUES ('LINESTRING(1 3, 1 2)', 7);
INSERT INTO network VALUES ('LINESTRING(4 2, 3 2)', 8);
INSERT INTO network VALUES ('LINESTRING(3 4, 2 3)', 9);
INSERT INTO network VALUES ('LINESTRING(2 4, 2 3)', 10);
INSERT INTO network VALUES ('LINESTRING(1 4, 1 3)', 11);
INSERT INTO network VALUES ('LINESTRING(4 3, 4 2)', 12);
INSERT INTO network VALUES ('LINESTRING(4 4, 3 4)', 13);

CREATE INDEX network_gix ON network USING GIST (segment);

Visually, the network looks like this:

To walk our network, use a WITH clause that starts with one segment, then repeatedly adds the next downstream segment to the collection. In our case, the “next downstream segment” is defined as a segment whose start point is close to the end point of the current segment. We’ll walk down from segment 6.

WITH RECURSIVE walk_network(id, segment) AS (
    SELECT id, segment FROM network WHERE id = 6
  UNION ALL
    SELECT n.id, n.segment
    FROM network n, walk_network w
    WHERE ST_DWithin(ST_EndPoint(w.segment),ST_StartPoint(n.segment),0.01)
  )
SELECT id
FROM walk_network

Which returns:

 id
----
  6
  3
  1
(3 rows)

From 6 to 3 to 1, correct! Once we have our walker producing the results we want, we can wrap more PostGIS and PostgreSQL functions around the walker to produce a finished product. Here’s a function that takes in an edge identifier and outputs a linestring based on the downstream path.

CREATE OR REPLACE FUNCTION downstream(integer)
RETURNS geometry
LANGUAGE sql
AS '
WITH RECURSIVE walk_network(id, segment) AS (
    SELECT id, segment FROM network WHERE id = $1
  UNION ALL
    SELECT n.id, n.segment
    FROM network n, walk_network w
    WHERE ST_DWithin(ST_EndPoint(w.segment),ST_StartPoint(n.segment),0.01)
  )
SELECT ST_MakeLine(ST_EndPoint(segment))
FROM walk_network
' IMMUTABLE;

And here’s the function in action, generating the downstream path from segment 12.

=# SELECT ST_AsText(Downstream(12));

            st_astext
---------------------------------
 LINESTRING(4 2,3 2,2 1,1 1,0 0)
(1 row)

Check the generated path against our network picture — looks good!

SXSW Geo in a Nutshell

2010-07-07T11:28:00.000-07:00

From Mapping and Geolocation: Turnkey Approaches You Need to Know [MP3] at 19:55.

I think that, at least for us here [at SXSW] this week, Foursquare and Gowalla have been at the center of the universe for a little while.

And I think that [is because] it is something to do with the fact that it's been a problem we've been trying to solve forever. You know you've been thinking "wouldn't it be great if I knew where my friends are so I could find them and we could hang out more" for years and so now this is finally here.

So not only is it an easy way to do something, and solves a problem that we we've been trying to fix forever, it's the first time that location is kind of useful beyond maps. Everyone understands that you need location in maps, because you have to get from point A to point B … I want to know where I am so I can broadcast it, because it's cool that I go to cool conferences (um, isn't it?), and I want my friends to know where I am, and I want to be able to find where they are.

I'm over the hill and I'm not even 40. Now get the hell off my lawn!

How to Survive Foo Camp (Introvert Edition)

2010-06-29T11:24:00.000-07:00

I just read the 2010 edition of Scott Berkun's regular Foo Camp review. It's very good, and makes me want to amplify a couple points for future, introverted attendees.

First, the social dynamics are much as he describes, although I would certainly not argue as he does that pretense is "eliminated". It is somewhat attenuated. The attending population can be segmented in lots of ways, but here's mine:

Foo Regulars	Foo Newbies
Extroverts	Introverts

Extroverts should have a grand time, lots of people to talk to. Regular attending introverts can lean on existing acquaintances when burnt out. But if you, like me, are an introvert, and you aren't part of the "in group" of repeat attendees, you may find yourself feeling socially isolated and very very tired, in the midst of a happy group of very energetic people. You might begin to resent them.

Foo is what it is, though. It is an opportunity to connect with some people you'd otherwise never meet, so all you can do is arm yourself before-hand with some tools and expectations to avoid having a negative experience.

Social Fatigue

First of all, bad news, the Cocktail Party dynamic is going to be a major mode of interaction. "So, what do you do?" You're going to have to answer that (and ask that) a lot, and people will filter you based on your answer. It's best to have an answer that is interesting to a wide swathe of people outside your field. The population of askers will be intellectually sophisticated (Michael Arrington excluded) so the answers that generally work can be surprising.

For me the general truth "I work for OpenGeo doing spatial database programming" was not effective at all. But a specific instance "I'm working on spherical geometry algorithms for location data" worked pretty well. (Unfortunately I was too dazed to really parse that fact until after I left.) Anyhow, on review, the second explanation has lots of potential follow-ups from "why?" for people who understand "location data" to "what's that?" for people who don't. The first explanation is too mushy.

Back to the bad news. Because a lot of your interactions are going to be draining first encounters, you're going to run out of gas pretty fast. Find a seat by the fire, get a quiet corner, and don't feel guilty about not talking. You're tired. You can't. Go to bed early, it's OK – the alternative is just going to make you feel worse.

Note to Parents: Your early-to-bed-early-to-rise conditioning comes in handy sometimes. You can get up early, get a coffee, and join the other parents for a quiet, introvert friendly, discussion around the fire pit while all the 20-something future masters of the universe are sleeping off their hangovers.

The "Formal" Program

Sessions are easy. There's structure, you can talk or not. For a first time attendee, I think this piece of wisdom on the Foo wiki ("Find people whose work you have no supposed interest in and go to their talks" -- Chris DiBona) is absolutely wrong.

You can bring the most value to the group, and meet the people who are going to be of most direct value to you in the future by going to sessions that are directly interesting to you, that fall in your area of expertise, that you have passion for. If you're going to Foo over and over again like Chris, then you have the luxury of sampling. I (foolishly) took the advice and missed going to database sessions with folks like Brian Aker, who says something worth writing down (if you're a database geek) about every 30 seconds. Bad, bad decision on my part.

Blowhards: There will be blowhards. Statistically speaking, the larger the session, the more likely there will be a blowhard in your session. Scott Berkun's 2009 post mentions them, and I like his idea of the "anonymous gong" to help the blowhard achieve self-awareness.

Bring a Pen and Paper

Laptops and iPads are heavy and take time to start up and get into data entry mode. Paper is light and fast. There will be names and ideas you want to remember. Precisely when the nuggets fall from the sky is unpredictable, but you probably won't be sitting down with your device at the ready.

Celebrities

Own your inferiority complex. Yes, you're inferior. Suck it up. Unfortunately, this is "do as I say, not as I do" advice, because I wasn't very good at it.

I didn't tell Kathy Sierra that her 2009 Business of Software talk is currently conditioning all my thinking about building the OpenGeo Suite materials. I just glanced at her out of the corner of my eye.

I did, however, have a nice time talking with (well, listening to) Mitch Kapor. Feed them some rope, provide them a context to tell you what's on their mind. A general request for advice on a general topic will usually do. I asked Kapor about economic development. I should have asked Sierra about our own product issues.

Invisible Celebrities: I actually talked to Chad Dickerson, but I never connected the dots until afterwards that he wrote the "CTO Corner" column in InfoWorld, back when it was a paper publication. That column was the only reason I kept receiving the magazine, it made me feel sane (I am not alone!) to be advocating open source, which was very much a minority position at the time. I would have liked to have told him that.

Meeting People You Want To Meet

If you look at the list and see a name you absolutely want to meet, track them down ahead of time, get their contact information and then contact them during the conference and arrange a time.

The venue is deceptively intimate, but 250 people is a lot of people. You may not randomly run into the person you are looking for. You probably will not. I was absolutely looking forward to talking with James Dixon about open source business models. I even staked out the main passage ways, during transition times. I did not see the man! (Here's where having one of those "iPhone" things would have been handy.)

Surprising Social Norms: Either there is a Silicon Valley thing going on that I'm not acclimated to, or this behavior is a form of status advertisement for high status individuals, but a number of times I saw people pull out their smart phones and check messages while other people were talking to them. I had not encountered this before, though I suppose it is just a variation on the equally noxious answering-your-cell-phone-in-the-middle-of-a-conversation. (Both should be grounds for an immediate caning.)

Us and Them

People sometimes ask, "What can I do to be invited back?" and your best bet is to make a (positive) impression by engaging and presenting.
— Foo Camp Wiki

In general, it's going to be hard to make a good impression if you have no psychological energy. And it's difficult to plan for a session or an Ignite talk when you have no idea what the audience is going to find interesting.

However, having seen the audience, I can now (post-facto (arg!)) offer a generic recipe for an interesting topic: something that is foundational in your specialty field but mostly unknown in the larger world. In my case a talk about general principles of geography, like Tobler's Law or the ecological fallacy or some topical examples of how to lie with maps would have gone over well at Ignite.

It's worth noting, given the mix of regular repeat attendees versus new attendees, that the criteria are not strictly meritocratic — obviously you can also get invited back by being intrinsically "important" in some way to the O'Reilly organization. The "engaging and interesting" criterion is only applied to some.

In Conclusion

Like many of life's memorable experiences ("there was this time I hiked to the top of a mountain in the rain…"), there is a strong possibility that, as an introvert in a highly social milieu without a group of friends to lean on, you might find Foo Camp difficult and unenjoyable in the moment. You might want to leave, a lot of the time. But if you stick with it, you'll come out with some good memories and look back on it as an important experience. You'll also meet some very friendly people who will make you feel welcome despite yourself. Give it a shot.

Shout-out: Thanks to the people who made me feel not like a stranger at Foo Camp, despite my acting like one. Selena Deckelmann, Cary Davis, Joe Hughes, Matt Cutts, Nat Torkington, Shel Kaphan, Toby Boudreaux, Andrew McLaughlin, Laurel Ruma, Emily Jacobi and many others.

Nerds Redux

2010-06-21T17:50:00.000-07:00

I had a chance to re-present my FOSS4G 2009 keynote talk at the Rendezvous OSGeo a Quebec last week, and thanks to the good work of the FOSSLC team, there's now a pretty clean online video of it.

PostGIS @ FOSS4G 2010

2010-06-19T07:00:00.000-07:00

One of the things that tickled me about the presentations selected for FOSS4G 2010 was the number of talks in the list that specifically mention PostGIS:

Beyond PostGIS - New developments in Open Source Spatial Databases
Introducing PostGIS WKT Raster: Seamless raster/vector operations in a spatial database
Introduction of flood evacuation route search system?using QGIS,PostGIS,GRASS and PgRouting
Moving from Oracle/ArcGIS to PostGresql/PostGIS
PostGIS meets the third dimension
PostGIS WKT Raster. An Open Source alternative to Oracle GeoRaster
Running long and complex processes with PostGIS
The State of PostGIS
Tips for the PostGIS Power User

The last two are mine! And one is about not using PostGIS. But still, some interesting talks on the use and future of my favourite spatial database.

PgSQL on EC2

2010-06-18T10:48:00.000-07:00

The theory behind putting a PostgreSQL (and PostGIS) instance on an Amazon EC2 instance with an Elastic Block Store (EBS) file system underneath is pretty straightforward, even for big databases. But when you want those databases to show the kind of properties we have come to expect from our systems, like durability, throughput, and reliability, things get much harder.

This thread on pgsql-general was very illuminating to me. Among the tidbits:

Let's be clear here, physical I/O is at times *terrible*. :)

There's no way we could run this database on a single EBS volume.

We had to fail over to one of our spares twice in the last 1.5 years. Not fun. Both times were due to instance failure.

Basically the assumptions of AWS architecture (virtual instances will be less reliable than real world computers, but that doesn't matter because getting a new one is really easy) don't map well with the requirements of running a classic production database.

There are probably some engineering solutions around for this (GlusterFS, for example, but the core PgSQL would need some serious work and end up looking a lot more like OracleRAC than the currently single-machine set-up.

Finding Corrupt PostgreSQL Data Files

2010-06-02T06:02:00.000-07:00

While PostgreSQL itself will never create corrupt data files, that doesn't stop other processes or hardware failures for corrupting the files underneath the database, which can cause database crashes. Josh Williams of End Point provides a super rundown of how to track and repair a file corruption.

Who's Your Dealer?

2010-05-17T20:36:00.000-07:00

The mapbutcher thinks it's OK to get hooked on proprietary software:

People are interested in the ‘express’ editions because on the surface of it the marketing works on them, they’re familiar with the brand and are attracted by the ‘free’ carrot dangling from the end of the stick. Open Source software starts from that position – it’s already free so open source projects need another carrot to get us hooked.

To which I can only say "Simon! Look at yourself in the mirror, man! Do you want to end up whoring yourself down on Dalgety Street to pay for your ESRI habit?"

OK, metaphor is tricky and fun, because there's so many ways to approach a metaphor. And for this one, it's easy to get distracted by the "drug" side, but the point of my metaphor is not that addictive drugs put you in a subordinate relationship to the drug (though some do) but that they put you in a subordinate relationship to the dealer.

The reason we can all survive and function in society despite our crippling addiction to oxygen is because oxygen is free and plentiful. The pusherman isn't trying to hook new customers because he believes that drugs are wonderful, he's doing it because he wants their money.

Software is "addictive" to organizations. Once you choose a piece of software and implement it, you're going to be "addicted". It's going to be hard to change. There will be withdrawal symptoms. Given that fact, what kind of software do you want to use? Software that is as free as the air you breath? Or software that is only available on terms dictated by someone else?

Your choice. Your future. Your life.

Free like...

2010-05-17T09:10:00.000-07:00

A favorite bon mot of open source critics is that open source software is "free like a free puppy". Tee hee! Open source advocates should remember to keep the rejoinder handy, that proprietary software is "free like a free hit of crack". Oracle "Express", SQL Server "Express", ESRI educational copies, yes I am looking at you.

NerdBA

2010-05-13T18:36:00.000-07:00

I was blessed (or cursed) to go through my undergraduate years from 1989 to 1993, and at that time, I'm sad to say, there wasn't a lot of nerd chic. I did my degree in Mathematics. The computer science student on my dorm floor ended up gainfully employed programming mainframes at Safeway. It was not, shall we say, a glamourous time for young nerds. On the other hand, the pressure wasn't exactly on.

What do young nerds today make of the examples before them, the Brins and Pages and Zuckerbergs? Is there a sense of performance anxiety? Does the start-up zeitgeist weigh on them?

Reading articles about how Mark Zuckerberg screwed his college employers, I am torn. On the one hand, is the satisfaction that the guy who did the actual work and had the particular, detailed vision of the social network got the reward, not his moneyed waspy employers. On the other hand, he clearly screwed them in a very premeditated fashion. What led to this kind of anti-social behavior? The lure of the Big Score? With Brin and Page in the back of your mind, maybe becoming a 20-something billionaire doesn't seem so far fetched? Maybe it's worth violating some social norms for?

Anyhow, now Zuckerberg is part of the pantheon. And it makes me wonder what the psychology is, today, in the nerdly dorm rooms? Is it still OK to go work for Safeway? Or is everyone secretly hoping they can spend a couple years drinking coke, working in the garage, and making it to the NerdBA?

The End of the End of General Purpose Computing

2010-05-05T09:46:00.001-07:00

One of the memes that got slung around a log during the launch of the iPad and associated apologetics for the closed nature of iPad (and iPhone) software development was that we were at the "end of general purpose computing", that the closed and controlled environment was a good thing, and in fact required in order to provide the seamless, user friendly experience necessary to bring computing, finally, to the unwashed masses.

In this meme, the limitations of the new platform—no multi-tasking, applications as unitary bundles that don't share files, the lack of a (user visible) hierarchical file system—were all features, not drawbacks, they were in fact the core benefits that make computing understandable for grandma.

How quickly the worm turns. iPhone OS 4 now introduces multi-tasking, file sharing between applications, and all the associated user interface scruft that goes along with managing those concepts. Also, folders for holding the many application icons in your now-totally-crowded phone screens. And on and on.

So, now that bath water (the blessed "simple enough for grandma" experience) is heading over the balcony edge, can we have our baby back (open application development)?

You know you've arrived...

2010-04-30T18:57:00.000-07:00

...when people start discussing the best way to replace you.

Seen amongst the abstracts in the FOSS4G Community Program Review: "Beyond PostGIS - New developments in Open Source Spatial Databases" and "JASPA, an alternative to PostGIS".

Is that a bright shining light? I'm floating above my body...

FOSS4G Community Program Review

2010-04-30T06:10:00.000-07:00

It's that time of year again! The Community Program Review for FOSS4G has begun!

If you're going to FOSS4G, or seriously thinking about going, the review is a way to ensure that the program includes topics you are interested in. You scroll through a list of all the abstracts the conference has received, and select the ones that are of interest to you. The conference committee tabulates all the results and uses that data to help build the program and to slot talks into rooms of appropriate size.

The number of abstracts is a good deal larger than the number of slots this year, so taking the time to do the review is a good idea if you're planning to attend this year!

Lazyweb: SDE requirements for PostGIS

2010-04-21T08:41:00.000-07:00

Oh, lazyweb, I beseach thee!

Can you tell me, for each version of ArcSDE, what version(s) of PostGIS is/are supported? I'm at the Washington GIS conference, and folks are using SDE on PostGIS, and some of them are using old versions of PostGIS, so I need to know how far back we have to apply patches in order to fully support users who have deployed SDE on PostGIS.

Update: The lazyweb responds, with ArcSDE 10 requirements, ArcSDE 9.3.x requirements and recap: PostGIS 1.4.0 for ArcSDE 10, PostGIS 1.3.2 for ArcSDE 9.3/9.3.1

Also, from Cort Daniel of Pierce County I hear that minor releases as late as PostGIS 1.3.7 and PostgreSQL 8.3.5 work (unofficially) for ArcSDE 9.3.

Update2: Cort also said that while PostgreSQL 8.3.5 worked with ArcSDE 9.3, 8.3.7 did not (date fields stopped working right), which is really really odd. Not sure if anyone else could confirm that.

Surveyor's Sermon

2010-04-19T13:42:00.000-07:00

Last month I had an opportunity to give a 5-minute "Ignite" talk at Where 2.0 in San Jose. I chose my topic because I fear one of the things missing in the technologist enthusiasm for geolocation, particularly in the population of technically astute but non-geo people, is a respect for how locations are actually derived, and knowledge of the provenance of the data that undergirds our new mapping tools.

FOSS4G 2010 Going to be Big

2010-04-16T16:44:00.000-07:00

The presentation abstract deadline has passed for FOSS4G 2010 and there have been 354 abstracts submitted! I'm not sure the exact number of session slots in the planned program, but for comparison 2007 had 120 slots and subsequent events have been similar -- that's about how many 30 minute talks you can fit into two days in a five-track format. In 2007, the biggest FOSS4G so far, we had around 240 submissions for our 120 slots. So, with three submissions for every slot, 2010 is looking like it is going to be gangbusters! Everyone wants to go to Barcelona, I guess they heard about the escudella i carn d’olla.

On the road to Damascus... GPL to BSD

2010-04-06T21:29:00.000-07:00

In response to a Twitter comment I made about my change in attitude towards the BSD and GPL open source licenses, Martin Daly asks:

Blog post on the Damascene (or otherwise) conversion from GPL-ish to BSD-ish please. In your own time.

The reason I was even mentioning the license issue is because I was watching a panel discussion at Where 2.0, in which Steve Coast, in defending the GPL-ish license used for OpenStreetMap, said (paraphrase) that the rationale was keeping a third party from taking the open data, closing it, and working from there. And that rationale pretty closely mirrors how we were thinking when choosing the GPL license for PostGIS, back in 2001.

I have experienced no spectral presence or flashing lights.

However, over the years, I have spent far too much time talking to various corporate folks about how the PostGIS GPL license wouldn't affect their plans to use PostGIS as a database component in their systems. For everyone who came and asked, I am sure many didn't. But in general the license was an impediment to some organizations engaging with the project. So there was a downside to the GPL. And was there an upside? Did the license protect us from privatized forks?

Well, yes, insofar is as it legally prevented them from happening. But it is worth questioning the implicit assumption that such forks are actually harmful to the project.

And here my experience watching the MapServer community (working with the ~~BSD~~ MIT license) was useful. Because over those same years, I watched the MapServer project chug healthily along, even as some third parties did in fact take the code and work on closed forks from time to time.

The lesson I have taken from my observation is that the strength of open source projects resides not in the licenses or the code, but in the communities arrayed about them. Copying the code of MapServer and doing your own thing with it does not stop MapServer developers from working, and in the long run you'll probably be better off working within the community than outside it, to gain from the efforts of others. Trying to maintain a private parallel branch and patch in the changes from the open development will quickly become more effort than it is worth. At the same time, because the BSD license is so permissive, there are no legal impediments for companies to engage with the community.

Look at any healthy open source project and ask yourself: what is more valuable, the code or the community? You could take all the code away from a healthy project, and it would start right up again from scratch and probably do a better job the second time. The value is in the human relationships and the aggregation of cooperating talent.

The GPL tends to keep corporate actors out of the community (for good reasons or bad, that's a separate discussion, but it does). That slows down the development of the project by reducing size of the development pool. Which, counterintuitively, makes forking or ignoring the open project more attractive. Because a slow moving project is easy to beat. A fast moving project is risky to fork, because it is likely that your (relatively understaffed) fork will be left behind by the open project.

So, back to what originally made me delve into license wankery, the question of OpenStreetMap license: the value of the OSM community and philosophy and infrastructure is way higher than the data at this point. And bringing corporate actors into the OSM community would only increase the relative advantage of the open project over any equivalent closed effort.

But changing licenses is a brutally hard thing to do, and it gets harder the longer a license is in place. PostGIS will never change licenses, for example. There are too many developers who contributed in the past under the GPL, and changing license would require the assent of all of them.

But if I ever start a new project, it will definitely be under the BSD.

FOSS4G 2011 Decision Process

2010-03-23T16:15:00.000-07:00

The 2011 FOSS4G siting process has begun. After the 2010 process, where there were too many good proposals and not enough slots, we decided to do a couple things to lower the effort level required by bidders in aggregate.

First, we wanted to be more definitive about our regional siting plan, which is to rotate between Europe (2010), North America (2011) and elsewhere (with elsewhere being Asia (2012) in the current rotation plan). For 2011 the regional arrow points to North America, since 2010 is in Europe.

Second, we wanted to lower the bar to submitting a proposal. In 2010 we got four full proposals, and could choose only one. So 75% of the bidders did a large amount of preliminary research and spade-work for no purpose. For the 2011 process, we will start with a "letter of intent" phase, a short document outlining expressions of interest. Hopefully having all the interested parties publicly declared will promote locations coming together and forming joint bids.

The OSGeo conference committee will vote on the letters and only the top two will be asked to prepare full bid documents. That would still leave one potentially disappointed party at the end of the process, but is a big improvement over leaving three, as happened in the 2008 and 2010 processes. And hopefully the letters phase will encourage groups to consolidate bids so we won't have to even run the final round.

The winning site will still have to submit a proposal and budget, so the Board and Conference Committee have confidence that the local team has thought things through, but the kind of research necessary to prepare a full proposal is a necessary precursor to putting on a conference, so the effort will be put to good purpose.

I'm looking forward to seeing the letters!

How to get Your bug fixed

2010-03-22T09:47:00.000-07:00

Mike Leahy is providing a textbook demonstration of bug-dogging on the PostGIS users list this week, and anyone interested in learning how to interact with a development community to get something done would do well to study it.

Some key points:

He does as much of the work in diagnosing the problem as possible, including combing through Google for references, cutting down the test data as small as possible, trying to find smaller cases that exercise the issue.

He responds very quickly (you'll have to read the timestamps) to questions and suggestions for gathering more information. Since the problem appears initially to manifest only on his machine, any delay on his part risks disengaging the folks helping him.

He prepares a sample database and query to allow the development team to easily replicate the situation on their machines.

And he also gets lucky. The problem is replicable, and the discussion catches the attention of Greg Stark, who recalls and digs up some changes Tom Lane made to PostgreSQL which in turn leads me to find the one-argument-change that can remove the problem. Very lucky, really, it's unlikely I would have been able to debug it by brute force.

NoNoSQL

2010-03-19T15:15:00.000-07:00

I'm really looking forward to seeing the "NoSQL" buzzword head over the top of the hype-cycle and start heading downwards, since I think it's really doing damage in what should be a fairly straightforward discussion of matching customer use cases to appropriate technology.

Because the term is framed ("No!") in opposition to the almost the entire family of existing data persistence technology, anyone who comes to the discussion fresh assumes there's a replacement process going on, wherein NoSQL stands in opposition to SQL.

That's a shame, because the term is only one letter away from a (slightly) less polarizing buzzword: NotSQL. Even then, though, the core discussion would be lost, because the big difference isn't programmatic API versus 4GL. The big difference is use case matching, in particular the high-volume, high-availability use case which has emerged in the age of consumer web services.

People with public-facing web applications face a potentially unconstrained read/write load (in their happiest dreams) and the techniques necessary to scale a traditional RDBMS to match that load proceed from the straightforward at the low end to the increasingly byzantine at the high end.

The scaling story for traditional RDBMS technology just is not great: start by adding servers and extra technology to hook them together; get increasingly smart people to handle your increasingly complex infrastructure; finally, start hacking at your data model to allow even further partitioning and duplication.

The new breed of databases have a great scaling story: once you get set up, scaling requires plugging in new nodes and turning them on. That's it. No model changes, no extra replication and high availability technology.

There's no free lunch though. In exchange for the high-throughput/high-availability you lose the expressiveness and power of SQL. Henceforth you will write your joins and summaries yourself, at the application level. Henceforth performing an ad hoc query may require you to build and populate a whole new "table" (the terminology is highly variable at this point) in your model. And of course this technology is all pretty fresh meat, so just learning enough to get started can be a bit of a slog – kids aren't exactly coming out of school with a course in this stuff under their belt.

So, it's a whole new world, and if you are planning on serving an application where the numbers (hits, pages, requests, whatever) are heading into the 7-digits, it might be good idea to start with this technology (can you tell I can't stand to use the "NoSQL" term? It is just too awful, there really needs to be a non-pejorative term for this application category).

Of course, there's nothing new under the sun. Getting the best performance for a specialized use case requires (a) modifying your data model and (b) using technology that can leverage your specialized data model. This is exactly what OLAP databases have been doing for a generation to provide data analysis on multi-billion record historical databases (special data model, special technology).

Database guru Michael Stonebraker wrote a nice article about the brave new world of databases, called "One Size Fits All: An Idea Whose Time has Come and Gone" in 2005, and the conclusion is that we are going to see increasing fragmentation of database technology based on use case. "NoSQL" (*shudder*) is just the latest iteration in this process.

Meanwhile, I'll put my oar in for the general purpose database: it's easy to run OLAP queries on a general purpose database, you just can't do it once your table size gets over a billion; it's easy to run a public web site on a general purpose database, you just can't do it once your load gets over a million. On the other hand it's well nigh impossible to run even a small web site on an OLAP database and pretty darn hard to build even a small OLAP system on a NoSQL foundation.

Horses for courses folks, horses for courses.

For more of this kind of geekery, see the several articles linked off of "The Case for the Bit Bucket" at the Oracle Nerd blog.

Where 2.0 Drinking Game

2010-03-10T10:56:00.000-08:00

It's that time of year again, and I'll be sitting in the audience with my flask (I hope you will too!) playing the Where 2.0 drinking game. Here's some of my phrases, what are yours?

... find a Starbucks...
... we're releasing an API ...
... friends list ...

Also, take a big slug if someone talks about working with geospatial data more complex than a lat/lon point!

Send us Jeremy and Keyur!

2010-03-05T08:42:00.000-08:00

ESRI (pretty new web page, by the way) has put their open source position on-line and also produced a short podcast with Victoria Kouyoumjian on the same topic.

http://www.esri.com/news/podcasts/audio/speaker/staff_kouyoumjian.mp3

One thing that struck me in the podcast was when Victoria noted that ESRI has sponsored open source events in the past (most notably FOSS4G 2007 directly, but also 2008 and 2009 to a lesser extent through 50°North). She says,

These events allow us the opportunity to engage in conversations and dialogs with various technologists because we want to gain feedback about the needs of open source developers and users. The objective of course is to channel this information back to development so we can reflect this in future products and business decisions in order to best support our customers.

So far ESRI attendance has been at the managerial level, and while I love those guys (hugs to Satish and Victoria!) some real sparks could fly and serious interoperability improvements be made if we started seeing the developers, the project leads and software designers, at the events. We can do better than "channeling" information back to development, let's immerse development in it!

Update: We promise to send them back. Really.

Nothing, Nada, Zip, Bupkus

2010-03-01T09:26:00.000-08:00

There is nothing new under the sun, and I have been wrestling this week with writing out ISO-standard well-known binary from PostGIS.

The most obvious difference is that the type numbers for encoding the presence of Z- and M-dimensions are not the ones described in the old OGC extension document [OGC members only, cited by Martin Daly in 2004, and extended further for PostGIS by Sandro Santilli that year] for WKB. Instead of setting high-bits to indicate the presence of Z and M, as OGC did, the ISO spec simply adds 1000.

So, the ISO geometry number for a PolygonZ is 3 (Polygon) + 1000 = 1003.

The, old OGC geometry number for a PolygonZ is 3 (Polygon) | 0x80000000 = 2147483651.

OGC seems more complex until you note that the function WKB_HASZ(num) can be written (num & 0x80000000). While the ISO test is (num >= 1000 && num < 2000). Setting flags for binary values (has-z, has-m, has-a-piece-of-pie) is nice.

Anyhow, that change was well-known and expected. What I didn't expect was the amount of ambiguity surrounding the definition of an empty geometry in WKB.

To review, the spatial SQL definition includes the concept of an "empty geometry", which is an empty set of a particular geometry type. The empty geometry has more information than a simple database NULL, which is a typeless emptiness. A 'POLYGON ZM EMPTY' has an implied dimensionality. It makes some sense that ST_Intersection() of two disjoint polygons would return a 'POLYGON EMPTY'.

The ISO SQL/MM well-known text specification has clear directions for writing empty geometries of all types. In fact, I've just written two of them above: the type name plus the 'EMPTY' keyword.

For well-known binary, ISO SQL/MM includes the following useless guidance:

i) Case:
i) If <point binary representation> immediately contains a <wkbpoint binary>, then <point binary representation> is the well-known binary representation for an ST_Point value that is produced by <wkbpoint binary>.
ii) Otherwise, <point binary representation> produces an empty set of type ST_Point

Representing an empty point in WKB is hard because there's nowhere obvious to indicate the lack of ordinates. But the ISO specification makes no attempt to solve the problem, they instead provide explicit guidance that is impossible to implement. Basically, if you are reading a WKB POINT and there are doubles after the TYPE number, you have a POINT(x y). If not, you have a POINT EMPTY. All well and good, but how do you distinguish, in a collection of WKB geometries, between the presence of doubles in the byte stream and the presence of another geometry in the stream? You don't.

The ISO guidance for empty Linestrings is even worse!

q) Case:
i) If <linestring binary representation> immediately contains <num>, then <linestring binary representation> is the well-known binary representation for an ST_LineString value. Let APA be an ST_Point ARRAY value with cardinality of <num> that contains the ST_Point values specified by the immediately contained <wkbpoint binary>s. <linestring binary representation> produces an ST_LineString value as the result of the value expression: NEW ST_LineString(APA).
ii) Otherwise, <linestring binary representation> produces an empty set of type ST_LineString.

As with the POINT case, the WKB reader is supposed to magically distinguish between an element of the current geometry (the <num>) in the byte-stream and an element of the next geometry in the byte-stream. And worse, the "clarifying" comment implicitly adds a whole new kind of empty geometry! What if the <num> is present, but the value is zero!?!

This is where the snake starts eating its tail. The way that implementations of OGC WKB have been encoding EMPTY geometries has been to provide the type number and an element count of zero. Back when PostGIS was first getting WKB support, Dave Blasby wrestled with the fact that the specification did not describe how to encode EMPTY. Mateusz Loskot recently published some information showing the WKB EMPTY implementation that Microsoft used for SQLServer. Their implementation is one of the options Dave described five years ago – there's only so many ways to solve this problem.

If ISO didn't like the use of a zero-valued <num> count as a way of indicating EMPTY, they had another option available, which was to follow the original OGC WKB standard and use bitmask flags on their type numbers. There could have been a bitmask for Z, a bitmask for M, and a bitmask for EMPTY. There could even have been a bitmask for SRID, fixing up a huge drawback in WKB, namely that WKB does not include a slot for the SRID, which is an important element in the geometry model.

Sidenote: As a result of WKB not having SRID support, it's not possible to round-trip a geometry through WKB without losing the SRID value. Try this standard SQL and see what happens:
SELECT ST_SRID( ST_GeomFromWKB( ST_AsBinary( ST_GeomFromText('POINT(0 0)', 4326 ) ) ) )
Then try the bastardized PostGIS EWKB format instead:
SELECT ST_SRID( ST_GeomFromEWKB( ST_AsEWKB( ST_GeomFromText( 'POINT(0 0)', 4326 ) ) ) )

As it stands now, the specification is out of synch with the implementations on the ground, which is bad news for the relevance of the specification. I will be implementing EMPTY using the same semantics as SQLServer, which will make the kinds of EMPTY PostGIS can represent slightly richer, but remain backwards compatible to the old schemes.

NYC Sprint: Day 4

2010-02-23T13:36:00.000-08:00

End day was, as last year, a quiet one. Everyone worked on cleaning up their last bits of work before heading home around mid-afternoon. Unfortunately, most left before I had a chance to ask what they finished up!

(For my part, I finished some minimal regression tests on the WKT emitter, changed the emitter to use a stringbuffer for output, and upgraded the stringbuffer to be smarter about performance. Jeff Adams has been working on making our unit test program a little more flexible for developers (easier to add tests, and able to optionally run one test at a time)).

One more time, thanks to the sponsors, we got a great deal done and MapServer, Geoserver, PostGIS and the rest of the tribe are going to be stronger next year thanks to this event.

Special thanks also to Temim Fruchter of OpenPlans who was invaluable in helping me make arrangements in New York for hotels and food and all the things that made the event enjoyable and comfortable. Thanks to OpenPlans for hosting us in their lovely penthouse event room!

NYC Sprint: Day 3

2010-02-22T14:00:00.000-08:00

Third day, best day. There are four MapServer developers now working hard on implementing the rendering plugin changes. Thomas Bonfort is doing the core work, Steve Lime is re-working the old GD renderer, Dan Morissette is creating support for hatched styles, and Assefa is doing KML output.

In Geoserver land, Andrea Aime added support for variable substitution in SLD, which means that URL parameters can now be passed into SLD styling rules, to create dynamic styling effects. Tim Schaub and Justin Deoliveira also demonstrated an application that warms the cockles of my heart: using their new GeoScript extension they made a web-based application that takes in SQL and spits out maps. So now I can type PostGIS queries into a web page and see the results overlaid on a map. Crunchy!

Howard Butler has contributed some work on the auto-projection support in MapServer and is now working on LibLAS Oracle support. He also tracked down an excellent pastrami sandwich. So I am told.

In PostGIS world, Jeff Adams finished his lat/lon formatter and logged his first commit: an impressive complete collection of unit tests, documentation and a working function (ST_AsLatLonText) that can turn POINT(-120.5 12.25) into 12°15'0"N 120°30'0"W. Oliver continues to fix up the text output functions. And I completed my first cut of the WKT output. Curve support really adds a lot of overhead to these things! There are lots of variants and curves have more and sillier formatting rules than linear features. David Zwarg has continued beavering through tickets in the WKTRaster subsystem.

Thanks again to our sponsors, tonight we are heading out to dinner at a Malaysian restaurant in Chinatown.

NYC Sprint: Day 2

2010-02-21T11:42:00.000-08:00

Today the Geoserver, MapServer and OpenLayers teams got together and really showed off the "making things work together better" theme. Starting from Andrea Aime's new "WMS rotation" feature, Daniel Morissette implemented the same feature with the same semantics in MapServer, while Andreas Hocevar implemented client support for the feature into OpenLayers. During testing, a small bug showed up in the Geoserver implementation, which Andrea fixed.

The final result is shown in this screenshot: layers from Geoserver and MapServer viewed and rotated together within OpenLayers.

The MapServer team is now really digging into a couple major goals: reading projection information automatically from data sources; and, making the rendering subsystem pluggable. Here they are deep in discussion on the rendering design:

Automatic projection reading will make it easier for new users to work with data in mixed projections, because they will no longer have to manually populate PROJECTION blocks for their layers. They will be able to just set PROJECTION to AUTO.

The pluggable rendering upgrade is mostly of interest to programmers, but because it will clean up the plumbing in drawing maps, it will allow new features like KML, PDF and SVG output to be added much more easily. In fact, Assefa is currently working on KML output, using the new rendering design.

Keeping to himself quietly, Alan Boudreault is tying the XML Mapfile more tightly into MapServer. In the last revision, XML Mapfiles could be transformed to .map format with a utility. In the next revision, it will be possible to use them directly from the MapServer CGI program.

Over on the Geoserver side, Andrea Aime arrived from Italy last night, and today is adding the WMS GetStyles operation to Geoserver. Andreas Hocevar is working on tying together printing support in Geoserver and GeoExt. The scripting engine crew continues to beaver away on Javascript/Python/Scala scripting in Geoserver.

In PostGIS land, Olivier Courtin has blasted a pile of changes into PostGIS trunk, working on moving GML/KML/SVG support down into the core geometry library, where they will be easier to reuse. On a similar tack, I've been working on writing a new WKT emitter, that is easier to maintain and supports ISO WKT for extended types and dimensions.

Thanks again to our sponsors! In an hour we will be settling in to watch the Canada/USA hockey game on the big screen in the Open Plans penthouse. Go Canada!

Update: Canada disappoints! I blame Brodeur.

NYC Sprint: Day 1

2010-02-20T11:24:00.000-08:00

On a bright day in NYC, we all convened in the sunny Open Plans event room for the first day. As in last years sprint, the morning was spent in planning and discussions, and the afternoon folks began digging in.

The MapServer team talked about release plans for 6.0, and came up with an ambitious release plan. They recognize that not every item on the plan will make the final cut, but hope that most will find either funded or community effort to bring about. Among the highlights (to me):

Pluggable renderer would allow a much cleaner rendering chain, and new renderers for new formats to be more easily added. filterObj to enhance the power of MapServer querying and support OGC Filter fully (and incidentally leverage the power of databases like PostGIS more fully). Named styles to allow re-use of style objects through a map file, instead of repeating the definitions over and over.

I also talked with Steve Lime and Jim Klassen about a bug in the one-pass rendering code that is making complex WFS queries fail. We think we have a solution and Assefa is doing the final tests.

The PostGIS discussions were about our 2.0 roadmap and what the implications of various changes are. Unfortunately, most of my proposed/desired changes are predicated are a large change to the underlying data serialization, so going forward requires a good deal of bravery – I have to burn down the village in order to save it. Olivier Courtin is working on more tractable new features: a polyhedral surface, suitable for storing 3D buildings and other objects that have grown increasingly common.

David Zwarg and Jeff Adams from Avencia joined the PostGIS group, and are working hard already: David on WKTRaster and Jeff on a geographic coordinates formatting routine. Don't tell Jeff, but if he gets the output formatter working, I'm just going to ask him to try and write an ingester.

Justin Deoliveira and Tim Schaub began working on improving the scripting extensions to Geoserver, Tim working on the server-side JavaScript and Justin working on the Python (and David Winslow working on Scala!). Andreas Hocevar has begun a Google Maps V3 API layer for OpenLayers, which will allow Google layers without API keys in OpenLayers (yay!).

As usual, the team had to be driven into the night bodily at the end of the day – it is hard to pry nerds from their code.

Thanks to our 2010 sprint sponsors, for keeping us well supplied with food, drinks and coffee throughout this busy week!

Postscript: At the end of the day, Olivier and I settled on an order-of-operations to move towards the new database serialization in PostGIS. Step one, remove the current places in the code where the serialization pokes up into function code and get it completely isolated underneath a serialize/deserialize layer.

NYC Sprint: Day 0

2010-02-19T09:00:00.000-08:00

Here we go! 23 programmers are winging their way to the Big Apple to take part in the New York code sprint, combining coding talent talent from MapServer, PostGIS, GDAL, OpenLayers, Geoserver, LibLAS, and, and, and!

I am in the air right now, and am looking forward to meeting up with the sprinters at the Broome Street Bar (363 W Broadway) this evening.

Thanks to our 2010 sprint sponsors, for keeping us well supplied with food, drinks and coffee throughout this busy week!

Hotel California

2010-02-18T09:47:00.000-08:00

I recently bought a set of wireless handsets that include a digital answering machine, so I figured I should be economical and cancel my voicemail. This is the internet age, so I pay my bills and manage my phone account online. Go to my account page, a little hunting, and there's all the options available for voice mail.

Note that the check mark next to voicemail service is grayed out! You can add services via the web interface, but you can't remove them! I'd switch to the other phone company, but there isn't one. All I've got is the cable company, and they recently shut off my internet for a week, which put somewhat of a crimp in my working-from-home (became working-from-cafe).

More Googlesoft

2010-02-09T21:25:00.000-08:00

More great news from the G-Men and G-Girls down in Mountain View – social networking goodness is now available from Google, in the form of Google Buzz.

Since their first attempt at a standalone social network, Orkut, never achieved escape velocity (exceto no Brasil!), Google is trying again. Only this time, they are stealing a page from Microsoft and using an existing dominant franchise to force their new offering into the marketplace. Oh, and incidentally crush the incumbents.

The CNET coverage has this perfect sentence:

Google is attempting to do this [compete with Facebook] by taking Gmail, one of its more popular products, and integrating Buzz directly into the Gmail interface.

I mean, the parallel construction just writes itself!

Microsoft is attempting to do this [compete with Netscape] by taking Windows, one of its more popular products, and integrating Internet Explorer directly into the Windows interface.

But Google's not content to just crush a strong competitor by leveraging their market-leading franchise! Remember, these are the smartest people in technology! Today they also announced they'll be crushing a few puny start-ups. Fourquare and Gowalla, with all of 200,000 subscribers between them (divide that number into their $20M in combined venture money for a laugh) will henceforth be known as "that thing I tried before Google Buzz for mobile came along".

(Incidentally, I'm looking forward to the day I receive my last Foursquare-initiated tweet, that charming idea got old incredibly fast. I don't care if you're at the 7-11!)

Keep on sucking up all the oxygen in the room G-People, yours is the fire that purifies, that consumes us down to the ashes of our soul and monetizes the dust left behind.

Chocolate and Peanut Butter?

2010-02-05T12:11:00.000-08:00

My two favourite things, ArcView 3 and PostGIS, together at last...

Die legacy, die!

2010-02-01T17:14:00.000-08:00

'Tis the season to drive a stake through the heart of old technology apparently. Steve Jobs appears to be taking especial glee in killing off Flash as a video format. And Google is doing their damnedest to run IE6 out of town. It takes a certain amount of self-confidence in yourself to pit your unstoppable force against the immovable object of an entrenched de facto standard, but both Apple and Google seemed positioned to carry it off.

In the case of Google, the sheer number of alternatives, ease of access to those alternatives, and popularity of their products will be enough to pry the final population of IE6 users out of their millennial stupor.

For Jobs, the task requires more balls (right now, his crown jewel iPhone is showing lots of holes in web pages where Flash content should be), but is slightly easier, because he doesn't have convince millions of users to change their software, he only has to convince a handful of web sites to add an HTML5 option. And the HTML5 standard on video embedding is just close enough to "ready" that those sites (YouTube, vimeo, etc) will begin providing HTML5-based alternatives to browsers that can support them (like, the iPad, and Safari, and the iPhone).

I wish godspeed to both Apple and Google in their endeavors, since both Flash video and IE6 can't hit the scrapheap soon enough, in my opinion.

Happy Jobsmas Eve!

2010-01-26T09:13:00.001-08:00

How does your family mark Jobsmas Eve? Do you sing 1,2,3,4 and dance around your AirPort Extreme router, like my family? Or do you have a more relaxed, modern Jobsmas Eve, and just play networked iPod games together?

While, obviously, I am excited about the approach of Jobsmas, I find the hand-wringing and obsession about the physical shape of the Gift to be a bit over the top.

No matter what, it's going to be a glass slab about the size of a hard cover book, OK? Does it really matter if the back is silver or black? What gets my Jobsmas noodles in a knot is ... what is it going to do???

It's all about the software, how is this Gift different from the Gifts that have come before? Will the the new feature be something pointless and pretty like "Cover Flow", or something useful and immediately intuitive like "flick to scroll"? Pointless and pretty would be virtual 3D display via head tracking. Useful and intuitive would be pen-less handwriting.

No matter what, I am looking forward to getting up tomorrow, putting on my black turtleneck and black jeans, and celebrating true spirit of Jobsmas, by oggling the Gift and figuring out how it Changes Everything.

New York Code Sprint 2010

2010-01-04T07:30:00.000-08:00

Update: We're expanding from the "C tribe" to the "curly braces tribe" and bringing in some JavaScripters too. Tim Schaub and Andreas Hocevar of OpenLayers/GeoExt will be joining the crew!

Update 2: If you haven't added your name to the wiki and booked your hotel room, you don't exist! GeoExt, OpenLayers folks don't delay!

Update 3: Avencia has joined as a sponsor (and is sending a couple staff to join in the fun). Along with OpenGeo, who are providing us with lovely space and free internet access, that bring our sponsorship docket to LizardTech, Coordinate Solutions, qPublic.net, Farallon Geographics, Avencia and OpenGeo!

Update 4: Other members of the curly braces tribe will be in attendance, in the persons of Justin Deoliveira, Andrea Aime, and Gabriel Roldan from GeoServer. I'm looking forward to working with them on adding PostGIS 1.5 GEOGRAPHY support in GeoServer.

Whoops, it's 2010! How did that happen?

I hope all members of the open source C tribe have gone beyond thinking about attending the New York Code Sprint (February 20 to 23) and moved into the travel planning stage! Don't forget to book your hotel, the room block closes in three weeks.

The list of attendees includes top contributors from MapServer, GDAL, OGR, and LibLAS. I'm feeling a little lonely at the PostGIS table with just Olivier to keep me company. And where are the folks from QGIS, Mapnik and GRASS? We love you guys! Come on, group hug. There you go.

We have four great sponsors now (LizardTech, Coordinate Solutions, qPublic.net, Farallon Geographics) and are looking for a couple more to round out the event. Thanks to our sponsors so far!

Remember to sign up to the mailing list and add yourself to the wiki page so we know you are coming, and you receive all the event notices, it's impossible to plan for you if we don't know you exist. See you soon in the Big Apple!

Thanks to Joel Schlagel for the logo!

FOSS4G 2010 Savings

2010-01-01T19:17:00.001-08:00

If you know you're going to FOSS4G 2010 in Barcelona this year, you can save 20% by taking advantage of the Sooper Dooper Early Registration rates. Rates end January 15, the early bird gets the savings.

Deep Thought

2009-12-22T11:32:00.000-08:00

If "open" is good at making others loose, and less than free is the model for things outside your core revenue stream, why isn't Microsoft giving away free advertising?

Abstracterrific!

2009-12-16T14:58:00.001-08:00

Dan McKinley takes a look under the covers of a couple Python PostgreSQL abstraction layers:

Database client drivers intended for the same database can do drastically different things. By Python standards, the Postgres driver situation is completely schizo. There are a lot of them available - there are five dedicated Postgres drivers listed on the wiki, as opposed to just one for MySQL. People might choose different drivers for licensing reasons, for religious reasons, randomly (because they never did any analysis like I am about to do), or for completely inscrutable reasons because they are just plain out of their minds. You really would not believe how much blood I have seen spilled over Postgres client drivers.

Read it!

Goo Smart by Half

2009-12-15T22:19:00.000-08:00

I typed "postgis" into Google this evening, and in addition to the exquisitely organized first entry there were two ads, one relevant (EnterpriseDB, the Postgres company) and one seemingly utterly random, for an English language school.

Has the Google algorithm made a mistake? How is this relevant? Well, the language school is in Victoria, BC, birth-place of PostGIS, but it gets better than that.

Here's a picture of their sign, which I walked past every morning for almost five years, since the school shares a building (1207 Douglas Street) with Refractions Research, birth-company of PostGIS.

But wait, there's more! The name of the school is "GEOS Language Academy", and the GEOS spatial library was also created in Victoria, specifically to bring spatial predicate algorithms to PostGIS!

Poor Google algorithm, you never stood a chance.

PgPatch!

2009-12-15T13:06:00.000-08:00

It's not much, but it's not nothing, my first ever patch to PostgreSQL proper is part of yesterday's 8.4.2 release:

Fix incorrect logic for GiST index page splits, when the split depends on a non-first column of the index (Paul Ramsey)

It was just a minor syntax error I found because I was reading through that section of the code very, very, very closely (it's true, I copy the work of smarter people than I) while implementing the indexes for GEOGRAPHY in PostGIS 1.5.

Extra, extra, PostGIS Really Fast!

2009-12-10T15:09:00.000-08:00

Because, frankly, I love nothing more than approbation, I am going to quote this comment on "Much Faster Unions in PostGIS" in full:

This is a truly spectacular piece of work. We have often been asked by clients to buffer and merge point datasets with several million points. We attempted this using ArcWhatever (could barely open the points, let along buffer them) and FME, which ran for a week and then gave an out of memory error. So, I do the whole configure, make, make install thing, 4 times, for postgres, goes, proj4 and postgis. After a lot of swearing and running ldconfig a few million times I eventually get postgis to accept that geos really is installed -- MySQL might have more limited spatial functionality, but it sure is a lot easier to build from source. Anyway, I digress. I run a few random queries using the excellent generate series capability in postgres, and manage to create, buffer and merge 100,000 points in a few seconds. Finally, I try this on a real world dataset, namely all of the postal addresses in Wales, 1.4 million or so. With a 200m buffer, this ran on a reasonably pokey 64-bit linux box in 19 minutes. Truly astonishing. Well done. Much as I love MySQL, this was a bit of St. Paul on the road to Damascus moment.

Full credit to Martin Davis, who implemented this technique in JTS. We just borrowed it for database land.

Code Sprint 2010: New York City

2009-12-04T15:26:00.002-08:00

It's official! Following the success of the 2009 Toronto Code Sprint a second event will be taking place in New York City, this winter.

This code sprint will bring together the developers for the MapServer, PostGIS, GDAL, GRASS, OGR, LibLAS, QGIS, Proj4 and other projects that are part of the "C Tribe" of open source software.

Timing: Saturday, February 20 to Tuesday, February 23, 9am to 4pm.

Location: OpenGeo Event Room, 148 Lafayette St, New York, New York

More Information: Wiki Page, Mailing List

Sponsor this event! Last year, sponsorship dollars paid for venue and dinners for the sprinters, who gave up their weekends and more to invest a collective 736 man hours in making their open source project stronger. Thanks again to our 2009 sponsors: Rich Greenwood, OSGIS.nl, Coordinate Solutions, LizardTech, SJ Geophysics, qPublic.net, GDAL! If you want to sponsor this event, and raise your profile with the 2 dozen people who make your software sing, please contact me.

Update: Thanks to our first 2010 sponsors, Coordinate Solutions, LizardTech, and qPublic.net! We are still looking for sponsors for 2010!

Perfect

2009-12-01T19:56:00.000-08:00

The New York Times political blog has instant commentary on Obama's Afghanistan speech tonight, and the one-sentence teaser in the RSS feed speaks volumes:

In an address aimed at rank-and-file Americans, Mr. Obama did not call for sacrifices.

Is "Good Enough" Good Enough?

2009-11-30T17:16:00.000-08:00

The gift that keeps on giving, for me, is performance work. Users love it, and it tickles my cerebral cortex in a way other work just cannot touch. Some of my first substantive patches to MapServer were performance patches, and I've also taken that tack into PostGIS.

One of the things that surprised me while investigating the MapServer QIX index structure was how little a badly built tree impacted real world performance. The QIX file, when built on point files, tends to build in far more depth than it needs, strictly speaking. The extra depth adds theoretical time to traverse the tree for searches and yet... even when I tested quite huge shape files the performance was fine.

Recently I've been looking again at internally indexing geometries for high performance testing of things like distance and intersections. We already have some of this for PostGIS, using the GEOS "PreparedGeometry" construction, but I'd like a native implementation, and something that doesn't necessarily depend on a cached implementation.

There is already a form of internally indexed testing in PostGIS, in the point-in-polygon case, and while reading it over I was struck by the elegance of the underlying assumption: rather than pre-sort the geometry edges, and then build the tree, the implementation simply scans the point array from start to finish and builds parent nodes from each successive pair of nodes. The result is not a perfect tree, by any stretch of the imagination, but... it's good enough. Because the edges are spatially auto-correlated, the parent nodes end up having good locality.

The beauty of this approach to index building is that since there is no sorting step and no re-balancing of the tree or any of that fancy stuff, you can actually build an "index" in O(n) time. A brute force intersection test is O(nm) time. But if you can build your indexes in O(n) time the cost of doing an intersection starts to get closer to O(n+m) time! (Note, I am not a computer scientist and the O() term for the actual tree-on-tree intersection test is beyond my powers.)

Now, since no sorting is being applied in the building of these "indexes" they could theoretically be terrible indexes. But since we're indexing GIS data, and the edges have this wonderful autocorrelation, they are actually "good enough", and obtainable in very little time.

The same tricks apply to building indexes for intersection and distance in geodetic space, which I predict will be in hot demand once people experience just how computationally expensive operations on the new PostGIS 1.5 geography type are!

You get more than what you pay for!

2009-11-23T19:21:00.001-08:00

Entchev reflexively repeats one of the most foolish canards ever hurled against collaborative knowledge building (open source, open data, open knowledge)

You get what you pay for.

Was there ever a putdown more easily falsifiable?

You pay nothing for PostGIS. Is the value to you nothing? You pay nothing for Wikipedia. Is the value to you nothing? You pay nothing for Open Street Map, is the value to you... nothing? Clearly, you, and everyone, gets more than what they pay for.

Now, there are many arguments to be made and beers to be drunk over whether what you get is enough for all possible purposes of all possible people. Some think that the trend in open knowledge is so strong that even that goal is within reach. Me, not so much.

But clearly, everyone has access to these resources, for free, and what they get in return is more than what they paid.

The First Rule of Fight Club...

2009-11-13T14:06:00.000-08:00

Who knew that ESRI was so hip? From James Fee's blog:

Note: If you post any specific 9.4 Beta information (such as quoting forums posts on the Beta forums), expect ESRI to personally contact you. They appear to be monitoring this blog post. You’ve been warned.

Palanterra X3

2009-11-13T08:54:00.000-08:00

James Fee has a nice rant on mapping interfaces today, but the part the piqued my interest was embedded in the screen shot of the National Map. It said "The National Map Viewer BETA: Powered by Palanterra X3"

Interesting! Who or what is this Palanterra X3 who have managed to get a top-of-the-fold "Powered by" on a major USGS site? Oddly, the Google is unusually terse about this topic. Looks like it's an NGA project, and yet it's "Palanterra (TM)". Intra-governmental entrepreneurship at its finest, a carefully imagined brand name that appears only in meeting agendas and powerpoints.

FOSS4G hearts SoTM

2009-11-11T17:07:00.001-08:00

A quiet bit of byplay is going on in the world of extreme geogeekery, as the State of the Map conference, the annual gathering of the OpenStreetMap community, decides where to host their 2010 event. The twist for this year is that an intrepid member of their spanish community has suggested hosting the event in Barcelona, to align with the FOSS4G 2010 event (either right before or right after). The OSM community decides on their site in early December – I'll add my non-binding "ooooooooh, I hopehopehopehopehope they choose Barcelona!"

State of PostGIS

2009-11-05T07:00:00.000-08:00

I gave a 30 minute talk about the past present and future of PostGIS at FOSS4G two weeks ago, and thanks to the efforts of the folks at FOSSLC it's now online as a video.

Geography and MapServer

2009-11-04T15:25:00.000-08:00

Can you use the new PostGIS GEOGRAPHY type with MapServer? Yes! Just make sure your LAYER declares a geographic projection (e.g. "init=epsg:4326", or "proj=lonlat") so the correct coordinates are passed in. For simple DATA definitions(e.g. DATA "thegeog from thetable"), that's all you have to do. I haven't tested out more complex DATA statements yet, but I am pretty sure they should work fine.

PostGIS gets Spherical (Directors Cut)

2009-11-03T13:46:00.000-08:00

Update: Installation instructions have changed slightly since this post was written.

For a business oriented discussion of the new GEOGRAPHY type, see my post on the OpenGeo blog.

So you want to try the new GEOGRAPHY type and see what it can do? Alright then!

If you are running Windows, please follow the directions on the Windows experimental binaries download page on the PostGIS site. Note that these builds might not be the absolute latest versions available.

If you are running Linux, fetch the latest code from SVN (svn checkout http://svn.osgeo.org/postgis/trunk postgis-svn) and then follow the install instructions.

After installing, you should find the usual postgis.sql file which contains the old geometry and now the new geography features too. Install PostGIS and spatial reference information as usual:

createdb mydb
psql -d mydb -f postgis.sql
psql -d mydb -f spatial_ref_sys.sql

I find it useful for testing to load a shape file in the usual way, then add a geography column to it.

shp2pgsql -s 26910 -g geom taxlots.shp taxlots | psql mydb
psql mydb

Once you have a table to play with, it's time to add a geography column, and build indexes. You have to transform the geometry into EPSG:4326 before casting to geography. When you build the index on the geography column, it builds a special index over the sphere which respects the poles and dateline.

alter table taxlots add column geog geography;
update taxlots set geog = geography(st_transform(tgeom,4326));
create index roads_geom_idx on taxlots using gist (geom);
create index roads_geog_idx on taxlots using gist (geog);

How does this magic index work? The "trick" is to be as lazy as possible. I didn't want to write a whole new indexing scheme, and I already had a serviceable R-Tree index. What I needed to do was convert the lat/lon coordinates to a domain where the problems of the singularities at the poles and dateline would go away. The answer is to convert from spherical coordinates (lat/lon) relative to Greenwich into geocentric coordinates (x/y/z) relative to the center of earth. It's easy then to build a 3D R-Tree on the geocentric bounds of the features. Calculating the bounds in 3D is tricky, because the curvature of the features over the spherical surface must be respected, but once that is done, the index performs admirably.

Now you can compare properties calculated on the plane and on the spheroid.

select st_area(geom) as geomarea, st_area(geog) as geogarea from taxlots limit 10;

Note that the units returned by the geography functions are metric. Square meters for area and linear meters for distances and lengths.

How about a quick spatial self-join, to test the indexes?

\timing
-- geography test
select sum(st_area(geog)) from taxlots a, taxlots b where st_dwithin(a.geog, b.geog, 100.0) and b.gid = 1;
-- geometry test
select sum(st_area(geom)) from taxlots a, taxlots b where st_dwithin(a.geom, b.geom, 100.0) and b.gid = 1;

In testing, I have been finding the geography index slightly faster than the geometry index, which is perhaps because I was able to write the geography index binding from scratch, trying to apply the lessons I have taken from reviewing the existing geometry index. In PostGIS 1.5 (coming Christmastime or sooner) the geography and geometry types will coexist, but use different disk serializations and index bindings. In PostGIS 2.0 (fall 2010) the geometry will also be swapped over to a new serialization and index binding and should become as fast (faster) than the geography index.

Finally, try out the transparent integration of the ST_Buffer() function from geometry with the geography type.

select st_area(st_buffer(geog, 2.0)) from taxlots limit 1;

By carefully transforming geometries out to appropriate planar spatial reference systems, you can use the functions built for geometry to provide operations on geographies. The definition for ST_Buffer(geography, double) looks like this:

CREATE OR REPLACE FUNCTION ST_Buffer(geography, float8)
        RETURNS geography
        AS 'SELECT geography(ST_Transform(ST_Buffer(ST_Transform(geometry($1), _ST_BestSRID($1)), $2), 4326))'
        LANGUAGE 'SQL' IMMUTABLE STRICT;

Note that ST_Transform() appears twice, once to transform from geographics to a planar system and again to transform back after the buffer operation is complete. Also note the _ST_BestSRID() function, which analyzes the geography and provides a best guess planar system suitable for carrying out the operation. Right now the system picks an appropriate UTM zone, or a polar stereographic, or falls back to a mercator if there is no good choice.

Acknowledgements: Of course, top billing goes to the funder, who has chosen to remain anonymous. Also, it would have been impossible for me to build the ST_Area() or ST_Distance() functions on the sphere and spheroid without the contributions of David Skea (if you check the PostGIS source code, you'll find he has contributed mathematical magic in the past also). And finally, Regina Obe who has been testing and documenting my work as it is committed, resulting in very effective on-the-spot debugging and fixing of issues as they occur. Thanks everyone!

Googalytics

2009-10-29T09:25:00.000-07:00

If you haven't read Paul Bissett's take on the Goog's entrance into the data collection and LBS businesses, you must, he's nailing it. Start with "GOOG = BORG, and I Mean That in the Nicest Way", and then do "Google Will Open Source National Parcel Map".

Climate @ FOSS4G 2009

2009-10-25T15:37:00.001-07:00

I found the keynote by climate scientist Andy Pitman at FOSS4G very interesting, on a couple levels.

Firstly on the climate level, he started off with a graph of some of the predictions from 1990, when it was becoming clear that climate change was something that policy makers needed to Do Something About. At that time, the scientists put together a range of scenarios, places we would be based on different policy responses, ranging from radical changes in energy production and efficiency to doing SFA. And amazingly, we have managed to do worse than the worst-case scenario envisioned in 1990. With the polar ice on the way out, and glacial (hah hah) progress in getting policy change China and the USA (the only two that matter, if they lead the world will follow), it was very sobering to see just how poorly we are doing at averting the very predictable end of our civilization.

Secondly on the open source level, he was very clear that open source technology (C, Java, etc) and open source programmers (knowing nothing about climate science) probably couldn't be directly useful to the modelling effort. But he did think that the modellers should aspire to a more open source ethic in their science. Most of the models are closed source, and there is little of the open source ethic of communications going on. My main concern about an open source climate model is that the politicization of the whole discussion would destroy any efforts at community building. One of the things projects have to occasionally deal with is "poisonous people", and an open climate model would have to deal with orders of magnitude more of those. It's draining and distracting from the real business at hand.

I hope the climate modellers can figure out how to square the circle of more openness without poisonous people, but in my mind modelling is icing on the cake at this point. We know we need to de-carbonize, it's obvious that changing the chemical balance of our environment, whether locally with things like PCBs or DDT or globally with CO2 is a dangerous crapshoot. At this point, we need the force of will to do it.

(BTW, I think we should be taxing carbon at the source, not taxing emissions (too many tailpipes) but extraction. It's not carbon per se that is the problem, it's extra carbon, the kind we dig and pump out of the ground, that is baking us. Fewer points of taxation and control, easier to enforce and manage, let's tax oil and coal and natural gas at the point of production (or, if exporting jurisdictions refuse to do so, at our borders.)

FOSS4G 2009 Keynote

2009-10-22T20:33:00.001-07:00

Lots of people here have been asking if my keynote is going to be posted online. So, here it is (31MB). Be forewarned, it's a big file. There is also a handheld video available on YouTube. There was a professional video shot too, and hopefully that will go online at the conference site in the next month or so.

Slashdot Explains Sexism in Open Source

2009-10-12T13:55:00.000-07:00

In the spirit of beer commercials and sports highlight shows, Slashdot's editors blithely follow a post about sexism in open source with, just 45 minutes later, a post on a Marge Simpson spread in Playboy.

Let no one say Slashdot does not know its audience. At least, the audience they have, not the one they're missing out on.