Tuesday, February 04, 2014

Introspection Double-Shot

Davy Stevenson has a great post (everyone should write more, more often) on a small Twitter storm she precipitated and that I participated in. Like all sound-and-fury-signifying-nothing it was mostly about misunderstandings, so I'd like to add my own information about mental state to help clarify where I come from as a maintainer.

First, PostGIS is full of shortcomings. Have a look at the (never shrinking) ticket backlog. Sure, a lot of those are feature ideas and stuff in "future", but there's also lots of bugs. Fortunately, most of those bugs affect almost nobody, and are easily avoided (so people report them, then avoid them).

When we first come up an a "major" release (2.0 to 2.1 for example) I expect lots of bugs to shake out in the early days, as people try the new release in ways that are not anticipated by our regression tests. (It's worth noting that the ever-growing collection of regression tests provides a huge safety net under our work, allowing us to add features and speed without breaking things... for cases we test.)

My expectation is that the relative severity of bugs reported decreases as the time from initial release increases. Basically, people will always be finding bugs, but they will be for narrower and narrower user cases, data situations that come up extremely infrequently.

The bug Davy's team ran across broke my usual rules. It took quite a while for users to find and report, and yet was broad enough to affect moderately common use cases. However, is was also something the vast majority of PostGIS users would not run across:

  • If you were using the Geography type, and
  • If you were storing polygons in your Geography column, and
  • If you queried your column with another Polygon, and
  • If the query polygon was fully contained in one of the column polygons, then
  • The distance reported between the polygons would be non-zero (when it should be zero!)

It happens! It happened to Davy's team, and it happened to other folks (the ones who originally filed #2556) — I was actually working on the bug on the plane a couple weeks before. It was a tricky one to both find and to diagnose, because it was related to caching behaviour: you could not reproduce it using a query that returned a single record, it had to return more than one record.

If I was prickly about the report from Davy:

And pricklier still about the less nuanced report of her colleague Jerry:

That prickliness arose because, on the basis of a very particular and narrow (but real!) use case, they were tarring the whole release, which had been out and functioning perfectly well for thousands and thousands of users for months.

Also, I was feeling guilty for not addressing it earlier. PostGIS has gotten a lot bigger than me, and I don't even try to address raster or topology bugs, but in the vector space I take pride in knocking down real issues quickly. But this issue had dragged on for a couple months without resolution, despite the diligent sleuthing of Regina Obe, and a perfect reproduction case from "gekorob", the original reporter.

That's where I'm coming from.

I can also empathize with Jerry and others who ran across this issue. It's slippery, it would eat up a non-trivial amount of time isolating. Having had the time eaten, a normal emotional response would be "goddamn it, PostGIS, you've screwed me, I won't let you screw others!" Also, having eaten many of your personal hours, the bug would appear big not narrow, worthy of a broadcast condemnation, not a modest warning.

Anyways, that's the tempest and teapot. I'm going to finish my morning by putting this case into the regression suite, so it'll never recur again. That's the best part of fixing a bug in PostGIS, locking the door behind you so it can never come out again.

Friday, November 29, 2013

Government HR Throws Up Its Hands

Has it come to this? Has government decided that it is completely incapable of recruiting and training its own technical staff?

Elections BC is not a big shop, and frankly, its technical problems are not that challenging (with the interesting exception of the quadrennial Failure Is Not An Option election event). And yet, they are now building a list of 5 bidders (to be winnowed down to one in the end) to outsource all their systems work to. Maintenance, new builds, the lot of it.

Does government no longer have an HR department? This is a very very very expensive version of Kelly Services except instead of placing secretaries we're now placing IT staff, and at markups that would old Kelly blush.

Place your bets now: who will win? HP? CGI? Accenture? Deloitte? The important thing is that the shareholders get a good return. Which reminds me, time to rebalance the portfolio...

Tuesday, November 26, 2013

Massey Bridge: $125K per house subsidy to suburban developers

Last week I was driving home from the NDP convention with a semi-retired traffic engineer in the backseat (and he was not backseat driving) when we passed under the Fraser River via the Massey Tunney.

"This tunnel provides 3 lanes in the direction of peak flow during rush hour", he said, "and if they build the 8-lane Massey Bridge that will add one lane, which is 2000 cars per hour, which over a 3 hour rush hour is 6000 new commuters. Assuming half the new households they put in here have a commuter the new bridge will support the development of 12,000 new houses."

Now, 12,000 new houses in south Delta/Ladner/Tsawassen will certainly eat up a lot of nice farm land, and that's a tragedy in its own right. But given that number of households, I immediately divided it into the $3B probable cost of a bridge-plus-Richmond-highway-expansion, and was blown away.

Three billion dollars divided by twelve thousand homes is $250,000 per home. Be generous if you like, and assume a much lower number of Fraser crossing commuters in the new developments. It's still hard to get the number lower than $100,000 per new home.

So, a review of the facts:

  • Traffic in the current tunnel is actually falling.
  • So new development is the only reason to build an 8-lane monster bridge over the river.
  • The extra lanes on the bridge will increase capacity by 2,000 cars per hour, which can be extrapolated to 12,000 to 24,000 new homes, depending on your assumptions.
  • The last major bridge/highway project cost 3 billion dollars, and everyone seems to think building a new bridge and widening the freeway north of it will also cost "about" 3 billion dollars.
  • 3 billion dollars divided by 24,000 homes is $125,000 per home.

Which leads me to the inescapable conclusion that BC taxpayers are going to be subsidizing "affordable" new housing south of the Fraser to the tune of about $125K per new home.

Can someone stop the ride now? I'm feeling a bit queezy and I want to get off.

Wednesday, November 20, 2013

Definitive healthcare.gov (and techmanagement.gov in general)

In the comments of my last post I was asked to provide a take on the healthcare.gov fiasco. Since I cannot improve a whit on Clay Shirky's perfect post today, I will settle for just quoting my favorite paragraph.

The vision of “technology” as something you can buy according to a plan, then have delivered as if it were coming off a truck, flatters and relieves managers who have no idea and no interest in how this stuff works, but it’s also a breeding ground for disaster. The mismatch between technical competence and executive authority is at least as bad in government now as it was in media companies in the 1990s, but with much more at stake.

 

Monday, November 18, 2013

BC Electoral Redistribution and Population Dispersion

2013/01/09 Update: I have converted this blog post into a document and submitted it as a comment on the white paper.



I've been asked by my technical readers to stop writing so much about politics, but I cannot help myself, and this week I have the perfect opportunity to apply my technical skills to a local political topic.

History and Background

Like Britain and the USA (and very few other jurisdictions anymore), British Columbia has a first-past-the-post representative electoral system. The province is divided up into "electoral districts" (also known as "ridings" in the British tradition) and each district returns a single member to the Legislature. In each district, the candidate with the most votes wins the district, even if they only obtain a plurality. In the Legislature, the party with most seats forms government.

In such a system, the geographical layout of the districts has a great deal of importance, because it is possible for a party to win a majority of votes in the province, but a minority of seats in the legislature, if the votes are concentrated in particular seats. This actually happened in British Columbia in 1996. It also happened in the USA in the 2000 Presidential election, since their presidential Electoral College effectively acts like a weighted version of a first-past-the-post Legislature.

In a representative democracy, it's important that everyone's vote have the same weight, which means ensuring that the each district has approximately the same number of people in it. As relative populations grow in some regions and shrink in others, districts can become unbalanced, and districts need to be redrawn. In the USA, this "redistricting" process is often driven by partisan considerations, and can lead to districts like this (try out this awesome gerrymandered districts puzzle game):

Fortunately, since 1989 British Columbia's districts have been drawn every 10 years by non-partisan "Electoral Boundaries Commissions", and the primary consideration has been creating districts that are as equal in population as possible while allowing for effective representation.

Effective Representation

BC is a big place and a place of extremes. The smallest district in BC is in downtown Vancouver (Vancouver-West End), with an area of under 500 hectares: it takes less than 30 minutes to walk across it and 48,500 people live there.

The largest district is in the north-west of the province (Stikine), with an area of almost 20,000,000 hectares: about the size of Ireland, Switzerland, Denmark and the Netherlands, combined. Just over 20,000 people live in it. When you're dealing with areas this sparsely populated "effectiveness of representation" begins to have some concrete meaning.

On the other hand, the principle of "one person, one vote" is the corner-stone of democracy, and the goal of an electoral boundary re-distribution is to try to achieve it, as far as possible. There is a tension inherent in the process.

2008 Commission Catastrophe

Population growth in BC over the last generation has been concentrated in the south: mostly in Vancouver and its suburbs, with some in Vancouver Island and Kelowna. Commissions have dealt with this growth through a combination of increasing the number of seats in the Legislature (to suppress the growth of the average seat size), and slowly increasing the size of the rural districts (to keep defensibly close to the average).

In 2008, this process reached a tipping point, as the Commission recommended two new seats, and the transfer of three seats from rural areas to urban areas. Rural BC exploded in anger, and the government of the day rushed in legislation directing the Commission to add more seats than recommended and to avoid removing seats from certain rural areas.

At this point, though the process remained non-partisan (both parties in the Legislature supported the new plan), it had become thoroughly politicized (the carefully considered deliberations of the Commission had been hastily overturned by politicians for public relations purposes).

Formalization of Politicization

No doubt remembering the tumult of the 2008 experience, the current government of BC has released a proposal for the rules governing the next Electoral Boundary commission. The proposal aims to avoid a messy politicization of the process at the end, by politicizing quietly it in advance:

  • The Commission may not recommend adding any further seats to the Legislature
  • The Commission may not remove seats from three protected regions: North, Columbia-Kootenay, and Cariboo-Thompson

The protected regions look like this.

Note that the Okanagan region is isolated from the rest of the "unprotected" areas of the province, making it impossible to juggle population into or out of the region. That means the Okanagan can either gain or lose a whole seat, but never lose a "half" a seat by having population juggled in or out via boundary changes.

Anyone with a passing familiarity with BC electoral geography will recognize that this proposal entrenches an already large and growing deviation from the principle of one-person-one-vote, but I want to calculate just how large, and also to measure the "fairness" of this particular proposal.

Population

The electoral district boundaries of BC are readily available as GIS files online, but do not have population information attached (and would be out of date if they did, since they pre-date the most recent census).

Similarly, the StatsCan boundary files can be downloaded, and the attribute file giving the census 2011 population in each block is also available. There are about 500-800 blocks in each electoral district, making for a very fine-grained profile of where people are concentrated in each district.

I loaded the GIS files into a PostGIS spatial database for analysis. Once the electoral districts (ed) and dissemination blocks (db) were loaded, calculating the electoral district population in PostGIS was a simple spatial join query:

SELECT ed.edabbr, ed.edname, sum(db.popn) as popn
FROM ed, db
WHERE ST_Intersects(ed.geom, db.geom) 
AND ST_Contains(ed.geom, ST_Centroid(db.geom))

The results of this calculation and others in this article can be seen in the spreadsheet I've placed online.

A quick summary of the population results shows that, among other things:

  • The current distribution is extremely lopsided, with the most heavily populated riding (Surrey-Cloverdale, 73042) having well over 3 times the population of the least populated (Stikine, 20238)
  • The current provincial average population is 51765 per riding
  • The average population in the 17 "protected" ridings is 35609, 31% less that the provincial average
  • The average population in the 68 "unprotected" ridings is 55804, 8% higher than the provincial average
  • A vote in the protected regions will be over 1.5 times more "powerful" than one in the unprotected regions
  • Of the 85 ridings, 26 are below average and 59 are above average, indicating that the problem of underpopulation is concentrated in a minority of ridings

There is no doubt that the government proposal will enshrine the regional imbalance in representation, and further worsen it as continued migration into the south pushes the balance even further out of line.

"Fair" Imbalances

Legal challenges to imbalanced representation have resulted in court decisions that indicate that it is constitutional within limits, and with reasonable justification. The limits generally accepted by the courts are +/- 25% of the provincial average, and the starting point of this proposal already exceeds that on average--some individual ridings (like Stikine) will be much worse. Political commentators in BC are already musing about whether ridings built under this scheme would survive a court challenge.

Of more analytical interest is whether the scheme of selecting "protected regions" is a good one for choosing which ridings should receive preferential treatment.

"Representing" a riding involves being available to your constituents, meeting with other orders of government in your riding (cities, school boards), and attending local events. Representation is very much tied up with being where the people are.

  • If the people are all in one place, near together, then representing them is easy.
  • If the people are spread out, in many different localities, then representing them is hard.

Can we measure the "spreadoutness" of people? Yes, we can!

Calculating Dispersion

Each riding contains several hundred census dissemination blocks, each of which has a population associated with it. Imagine measuring the distance between each block, and all the other blocks in the riding, and weighting that distance by the population at each end.

For Vancouver-Fairview, the picture looks like this.

The blocks are fairly regular, the population is all very close together, and the dispersion is not very high.

For Skeena, the picture looks like this.

The population is concentrated in two centers (Terrace and Kitimat) reasonably far apart, giving a much higher dispersion than the urban ridings.

In mathematical terms, the formula for "dispersion" looks like this.

In the database, after creating a table of census blocks that are coded by riding, the calculation looks like this.

WITH interim AS (
 SELECT 
  a.ed,
  max(a.edname) AS edname,
  sum(b.popn * a.popn * st_distance_sphere(a.pt, b.pt)) AS score,
  sum(b.popn * a.popn) AS p2
 FROM popn a, popn b
 WHERE a.ed = b.ed
 GROUP BY a.ed
)
SELECT ed, edname, score/p2/1000 AS score 
FROM interim;

Taking the ratio of the distance scaled populations against the unscaled populations allows populations that are far apart to dominate ones that are close together. Scaling the final result down by 1000 just makes the numbers more readable.

As before, the results of this calculation and others in this article can be seen in the spreadsheet.

Is Regional Protection "Fair"

Using the measure of dispersion allows us to evaluate the government proposal on its merits: does protecting the North, Kootenays, Cariboo and Thompson protect those ridings that are most difficult to represent?

In short, no.

The regional scheme protects some difficult ridings (Stikine) but leaves others (North Island) unprotected. It also protects ridings that are not particularly dispersed at all (Kamloops-South Thompson), while leaving more dispersed ridings (Powell River-Sunshine Coast, Boundary-Similkameen) unprotected.

Among the larger ridings, Skeena is notable because even though it is the 10th largest riding by area, and 10th sparsest by population density, it's only the 17th most dispersed. There are many smaller ridings with more dispersion (Powell River-Sunshine Coast, Nelson-Creston, Boundary-Similkameen). This is because most of the people in Skeena live in Terrace and Kitimat, making it much easier to represent than, say, North Island. Despite that, Skeena's population is 43% below the average, while North Island's is 5% above.

Kamloops-South Thompson is the least dispersed (score 15.2) protected riding, and it's worth comparing it to the similar, yet unprotected, Nanaimo-North Cowichan (score 16.2).

Kamloops-South Thompson (protected) consists of a hunk of Kamloops, and a string of smaller communities laid out to the east for 50KM along Highway 1.

Nanaimo-North Cowichan (unprotected) consists of a hunk of Nanaimo, and a string of smaller communities laid out to the south for 45KM along Highway 1 (and some settled islands).

What is it about Kamloops-South Thompson that recommends it for protected status along with truly dispersed difficult ridings like Stikine? Nothing that I can determine.

Let the Commission Work

The intent of the government's proposal to amend the Redistribution Act is clearly to avoid the firestorm of protest that accompanied the 2008 Commission report, and it's good they are thinking ahead.

They need to think even further ahead: the consequences of having the boundaries enacted, then reversed in court, will be far more disruptive than allowing the Commission to proceed with the necessary work of redistributing BC's districts to more fairly reflect our actual population distribution.

The end result of an unconstrained Commission will be fair boundaries that still reflect the representation needs of dispersed ridings by giving them lower populations within the limits already acknowledged by the courts: +/- 25% with a handful of exceptions (I'm looking at you, Stikine).

I encourage you to explore the data on dispersion, and how it relates to the regional "protection" scheme, in the spreadsheet.


Wednesday, October 23, 2013

UK Government Digital Service

I wrote last year that the UK GDS seemed like a promising approach to revitalizing government IT.

Of late, the GDS Director, Mike Bracken has been doing to rounds of US "new wave IT" organizations, the Code for America group and yesterday the Presidential Innovation Fellows. At code for America, he focussed on the need for difference makers to, above all, "deliver".

After speaking to the Fellows, Bracken sat down with NPR, and was naturally asked about the healthcare.gov debacle.

The real problem is systemic. You actually can't build technology like this. Technologies aren't things that are binary. You don't procure them. They're living systems and you have to have people who look after them and develop them iteratively and change and grow with them and you need those skills in the heart of government.

GDS has over 300 people, skilled folks with a mandate to deliver, building the digital heart of the UK government. And they have been sufficiently successful that CGI UK is complaining about it.

The question for decision makers is: what's more risky, standing up a shop with the chops and mandate to deliver, and risking that they might screw up a few times; or continuing on doing what we're doing now?

Update:

Also, read this wonderful, wonderful, wonderful piece from Merici Vinton on what it takes to build successful IT projects in government. From TFA:

  1. Never build a website that's too big to fail; instead, start small.
  2. Let's do open source when possible (preferably always).
  3. Let's have in house strategy, design, and tech.

Sunday, October 20, 2013

Platforms for Open Information/Government

David Eaves has posted a great talk which covers in generality the point I was circling around in specificity in my post this summer on government email systems: the "access to information" community, both within and outside government, is still partying like it's 1993.

I've got an open FOI request right now for a bunch of data, and the information access officer is scratching her chin and wondering "hm, how are we going to deliver that to you". The problem being their process is optimized around delivering documents, in PDF, via e-mail. (An improvement over the original system--documents, on paper, via the post--but still basically the original process from 1993.)

Eaves tell his audience of information access folks that the big win for them, over the next generation, is in systems. In advocating for and getting systems in government that can include information access metadata from top to bottom, so that it is possible for vast swathes of documents and data to be "open by default".

Have a listen, it's 20 minutes very well spent.

Thursday, October 10, 2013

Being an Open Source Citizen

I was invited to deliver a keynote at FOSS4G 2013 in Nottingham last month, and in a twist it turned out to be a closing slot rather than an opening one. So rather than a call to action, I did something more moralistic, a "sermon" of sorts for everyone going home after a great conference. And then the recording turned out to be no good. So I did it again a couple weeks later at the Boundless company meeting in Denver.

Monday, August 05, 2013

BC Government Email: Defective by Design

Update: Bob Mackin has unearthed a wonderful exchange with an Ontario politico getting grilled on how she managed her email (she deleted thousands a emails pertaining to public policy decision). My favourite quote from the politico: "When I delete emails, I do not have the ability to go back. Perhaps it's a lesson learned that the government can take back, in terms of maybe it shouldn't be political staff who search their emails; maybe it should be an individual in the civil service who has access to inboxes and sent-mail folders and deleted archives -- whatever it is -- to conduct the search."


When the BC Freedom of Information Commissioner Elizabeth Denham released her report on the Liberal "ethnic outreach scandal" this week, the usual scene ensued: journalists grabbed copies, riffed through the pages, found little politically damning, pronounced the report a wet squib, and promptly moved on to other matters (hey, it's the middle of summer).

I think they missed a very important (non-political) story: the Province's management of email clearly sucks.

How does it suck? Let me count the ways.

It sucks Historically

It seems like government e-mail has sucked for as long as government e-mail has existed (although I retain a certain fond nostalgia for the VAX/VMS green screen e-mail of my early days in the Ministry of Forests).

One of my favourite e-mail-based scandals dates back to 2009, when (as part of the "BC Rail scandal") a judge ordered the government to cough up MLA e-mails from the period 2001-2005. As it turned out, the archival e-mail was long gone. E-mail boxes were regularly purged, and even the disaster recovery back-up tapes had been over-written.

Retaining archival e-mails is hardly a novel idea. Long before the BC Rail trail ever got started, the 2001 Microsoft anti-trust trial featured e-mails penned by Bill Gates in the mid-nineties half a decade before. (Microsoft is, incidentally, the company that makes the Exchange e-mail software used by the BC government.)

If secretive corporations can archive their mail, and yet cough it up under subpoena, why can't the BC government? One hand of government considers e-mail a document of record, and another considers it so much transient tissue paper, to be discarded on a regular basis. This bi-polar view of e-mail remains to this day.

It sucks Functionally

A central criticism levelled against the BC government during the ethnic outreach scandal was that government staffers knowingly used private email to skirt FOI requests (a technique once used by one-time Governor Sarah Palin), and the Denham report confirms this:
In response to a question from government investigators as to why Mr. Bonney, his former Communications Director, was routinely using personal email for his correspondence, Minister Yap responded that it was to avoid freedom of information legislation.
However, the report notes that staff were also using private emails just because their government service sucked:
Ms. Lo stated that she sent emails between her government and personal email accounts because she needed to ensure she would have access to them while traveling with the Minister or when otherwise working away from the office.

Ms. Lo stated that she used her government smartphone to check her government email, but it was common for her smartphone battery to run out when she was away from the office.

Mr. Bonney stated that he sent emails between his government and personal email accounts so that he could easily access this information regardless of his location.
And one of only five recommendations in the Denham report is this one:
Government should provide its employees with sufficient technological resources to ensure that they do not have a reason to use personal email accounts in the performance of their government duties.
The government staff I know agree with Denham: it would be nice if government e-mail didn't suck.

It doesn't have to suck

When I started up my cleverelephant.ca infrastructure in 2008, I swore to never run my own e-mail server again, so I signed up for Google Apps. At the time, Google was pushing "enterprise email archive" service called "Postini", which you could add to your account if you were that kind of enterprise. I wasn't, but I wondered what "Postini" was.

All "Postini" does is sit in your institutional email delivery pipe. It take copies of all mails sent and received, and provides an administrative search-and-retrieval interface making it easy for authorized users to pull up any email to or from any person from any time. Why would any company want this capability? Because they are legislatively required to have it, by the US Sarbanes-Oxley law that applies to public companies.

So in 2008 a robust enterprise email archive capability existed, available for a few dollars per account. It seemed very relevant to the BC Rail e-mail scandal that occurred a few months later. After the scandal had run its course, I suspected that enterprise email archiving would be a top government IT priority.

Was it? (Hah!) I recently ran a little test.

But it still does suck

Dave Nikolejsin
Since I have an abiding interest in the genesis and continued thrashings about of the BC Integrated Case Management system, I decided to see what kind of e-mail record the Chief Information Officer, Dave Nikolejsin, generated during the critical period in which the prime contractor was selected, 2009.

So I FOI'ed his e-mail for 2009. All of it.

If BC had an enterprise e-mail archive, they would simply run a query on the archive ("After:2009-01-01 AND Before:2010-01-01 AND (To:Dave.Nikolejsin OR From:Dave.Nikolejsin)") and send me back the result 2 minutes later.

Surprise! They don't have an enterprise e-mail archive. Instead, I got this email.
I am contacting you in regards your above noted FOI request, which you submitted on June 10, 2013 for:

“All e-mail From: or To: Dave.Nikolejsin@gov.bc.ca. Date range is January 1, 2009 to December 31, 2009”.

It is possible that emails sent to/from Mr. Nikolejsin have since been appropriately filed, as these communications may relate to particular issues or topics of note. I anticipate that a significant search would be required to determine this, which could result in a fee.

Is there a particular subject/topic of interest, or perhaps a particular sender/receiver you wish to focus on? I believe that by concentrating your request in this manner, we can decrease the likelihood of a fee being generated. Please let me know if you are open to discuss, or have any questions.
The gist of the message is that e-mail is archived by being printed and filed (old skool!), therefore searching all the files for individual e-mails would be onerous (fair enough). I decided to ask again for a electronic archival search:
I was hoping this would be a simple computer query against a corporate mail archive, not a manual process. Is that not the case?
No, it's not the case.
Mr. Nikolejsin may have email responsive to your specified time line in his Outlook mailbox, which would indeed be searchable electronically. If you would like to focus your request to his mailbox alone, (i.e. excluding records that may have been filed elsewhere), we can certainly accommodate, but I wanted to ensure you were aware of the potential search required to satisfy your request. Please let me know how you would like to proceed.
So instead of a single searchable archive with a single uniform retention policy, the government retains whatever happens to be in the individual Exchange mailboxes of their 20K employees, with retention individually managed by those employees.

Given the option, I decided I was still curious: what is in the BC CIO's Outlook mailbox for the year 2009? This is a busy, important, technically plugged in guy, what does he do with the valuable personal and institutional memory embodied in his e-mail archive?

Answer: He deletes it.

There are four e-mails in the CIOs mail history for 2009. Two of them are bits of login info sent to him by tech support. One is a policy document on Wifi security he sent out to ministry CIOs. One is a notification of a reorganization he sent out. That's it.

By way of comparison, my mail box for 2009 contains at least 2000 e-mails just FROM me. How many are in your box? (Gmail search string: "after:2009-01-01 before:2010-01-01 from:your@email.com")

It sucks at the Highest Levels

This week's FOI report is not not the first that touches on staff use of e-mail. The Commissioner also did an inquiry into the strange case of the "Ken Boessenkool scandal" (it really is amazing this government got re-elected) in which the interview and dismissal of the Premier's Chief of Staff somehow generated no written records at all. The Premier's office explained the conundrum to the Commissioner this way:
The general practice within the Office of the Premier is to communicate verbally in person. Email communications usually consist of requests to make telephone calls or meet in person. Generally, staff members in the Office of the Premier do not make substantive communication relating to business matters via email. Most of the emails are transitory in nature and are deleted once a permanent record, such as a calendar entry, is created.
That's right, the highest executive office in the Province runs entirely "verbally in person". No substantive matters in e-mail. In 2013.

E-mail is an incredibly valuable tool for communication, and I use e-mail tools like Gmail daily to organize and search my voluminous historical archive of communications. Access to my saved e-mail history makes me a smarter and more capable professional. Threaded histories of e-mail conversations and decisions, technical disputes and resolutions, even files ("where's that thing I sent to Joe...") are easily accessed via my e-mail history.

In contrast, our high-level government staff (even down to the CIO?) are deliberately excising this information trail on a daily and hourly basis, and even though doing it makes them less effective and capable, they feel that they have to do it because every email in their archive is treated by the FOI act like a final product, like a "document".

(Un)intended Consequences

One of the unintentionally funny passages in the latest Denham report is this one:
Government should have been taking steps to ensure that it was preserving correspondence relating to the Outreach Plan on its own servers both in the name of good governance as well as for the responsible preservation of documents that could be the subject of future access to information requests
In a world in which staff purge their email regularly to avoid FOI requests, keeping emails "in the government" is actually counter-productive to future investigations. Staff will just delete their e-mails at the end of the day as part of their daily email purge, and no records will exist at all.

At this point, government email for policy and political work is institutionally broken. It can barely be used as e-mail (as most of us understand it) anymore.

I have a completely un-founded suspicion that the parlous state of e-mail archive management in BC is somehow related to the FOI consequences of having a properly implemented archive. What would the political reaction be to an IT project proposal for a high quality persistent e-mail archive? Would there be enthusiasm? Would any IT manager even be so foolhardy as to propose a better archive system to their political masters?

The BC government e-mail system has become defective by design, because fixing the technological underpinnings of government e-mail would only magnify how institutionally broken the system is.

E-mail is only the thin edge of the wedge.  The cost of creating a "record" (audio, video, text) using modern technology is now so low as to be basically zero. Does that mean that everything government does should now be on the record? Or do we now need to be clearer about what "records" are?

The future is not far away...

What about when government has an IP phone system? Why aren't all phone conversations a public record? Every government phone call can be easily recorded, voice scanned into text and made searchable, using today's technology. Check out Google Voice.

What about when government staff are all wearing Google Glass (or something similar), personal or otherwise? We can already FOI hand-written notes from yellow legal pads as "official records" of otherwise private meetings. Government business transacted on private e-mail is still subject to FOI. So why not also have FOI access to first-person recordings of all meetings, shot by the principals using their augmented reality glasses?

The FOI act needs to be re-thought in the light of modern record keeping technology. 

Right now the BC government is managing its FOI exposure by just not properly recording their activities. As a result, incompetence in record keeping is currently a virtue, not a vice. This is not a healthy situation. It would be preferable if government could start properly recording their history, within the context of a revised FOI act that respects the difference between interim and final documentation and decisions.

Friday, July 26, 2013

BCeSIS: A Modest Proposal

No, not "eat all the children" although that would be one way to solve the BCeSIS problem.

Rather, embrace the free market and the spirit of competition to reduce the risk of project failure.

The trouble with spending the entire student information system budget on a single system and vendor is that it is an unhedged bet. If it pans out, you win; if it doesn't you lose 100% of your investment.

In reviewing the state of BCeSIS and recommending future options, the Gartner report had this to say about pursuing an open source option:
There is little evidence of a common, well-developed strategy to develop an Open Source solution in the K-12 software domain and it is anticipated that the development life cycle from defining requirements through deployment would be more than 5 years once a strategy was completed, thus introducing a high degree of risk to the BC education sector. Although some attempts have been initiated to develop K-12 Open Source SIS solutions, there does not appear to be any formal, structured funding mechanism and approach to designing and developing a solution which would meet the future needs of the BC education sector, nor to scale to the extent required to provide a single, province-wide student database.
Leaving aside the circularity of recommending against funding open source development because open source development lacks funding, the seeds of a great idea are actually lying in this paragraph.

  • The OpenStudent project actually got started two years ago, and seem to be more or less on Gartner's five-year estimated schedule (my opinion of Gartner notches upwards fractionally), with three years to go on feature-complete delivery.
  • The OpenStudent project could benefit from incremental funding from the Ministry, a "formal, structured funding mechanism", if you will.

Herewith, the modest proposal:

Rather than committing 100% to the Aspen/Fujitsu approach, commit 95% to it, and commit the other 5% to OpenStudent.  If OpenStudent fails, only a relatively small investment will be lost. If Aspen/Fujitsu fails, the Ministry has a potential fall-back plan in place and ready to go with no additional capital required. In either event, the existence of active competitors will encourage each project to provide the best possible user experience and provide a baseline of comparison for improvement ("oh, OpenStudent is faster? we can beat that", "oh, Aspen has this feature? we can add that")

What do you say, Ministry of Education. You've got the brass balls to bet $100M on a single product and vendor. Do you have the balls to bet a couple million on the possibility your $100M bet might not pay off?

About Me

My Photo
Victoria, British Columbia, Canada

Followers

Blog Archive

Labels