Monday, August 05, 2013

BC Government Email: Defective by Design

Update: Bob Mackin has unearthed a wonderful exchange with an Ontario politico getting grilled on how she managed her email (she deleted thousands a emails pertaining to public policy decision). My favourite quote from the politico: "When I delete emails, I do not have the ability to go back. Perhaps it's a lesson learned that the government can take back, in terms of maybe it shouldn't be political staff who search their emails; maybe it should be an individual in the civil service who has access to inboxes and sent-mail folders and deleted archives -- whatever it is -- to conduct the search."


When the BC Freedom of Information Commissioner Elizabeth Denham released her report on the Liberal "ethnic outreach scandal" this week, the usual scene ensued: journalists grabbed copies, riffed through the pages, found little politically damning, pronounced the report a wet squib, and promptly moved on to other matters (hey, it's the middle of summer).

I think they missed a very important (non-political) story: the Province's management of email clearly sucks.

How does it suck? Let me count the ways.

It sucks Historically

It seems like government e-mail has sucked for as long as government e-mail has existed (although I retain a certain fond nostalgia for the VAX/VMS green screen e-mail of my early days in the Ministry of Forests).

One of my favourite e-mail-based scandals dates back to 2009, when (as part of the "BC Rail scandal") a judge ordered the government to cough up MLA e-mails from the period 2001-2005. As it turned out, the archival e-mail was long gone. E-mail boxes were regularly purged, and even the disaster recovery back-up tapes had been over-written.

Retaining archival e-mails is hardly a novel idea. Long before the BC Rail trail ever got started, the 2001 Microsoft anti-trust trial featured e-mails penned by Bill Gates in the mid-nineties half a decade before. (Microsoft is, incidentally, the company that makes the Exchange e-mail software used by the BC government.)

If secretive corporations can archive their mail, and yet cough it up under subpoena, why can't the BC government? One hand of government considers e-mail a document of record, and another considers it so much transient tissue paper, to be discarded on a regular basis. This bi-polar view of e-mail remains to this day.

It sucks Functionally

A central criticism levelled against the BC government during the ethnic outreach scandal was that government staffers knowingly used private email to skirt FOI requests (a technique once used by one-time Governor Sarah Palin), and the Denham report confirms this:
In response to a question from government investigators as to why Mr. Bonney, his former Communications Director, was routinely using personal email for his correspondence, Minister Yap responded that it was to avoid freedom of information legislation.
However, the report notes that staff were also using private emails just because their government service sucked:
Ms. Lo stated that she sent emails between her government and personal email accounts because she needed to ensure she would have access to them while traveling with the Minister or when otherwise working away from the office.

Ms. Lo stated that she used her government smartphone to check her government email, but it was common for her smartphone battery to run out when she was away from the office.

Mr. Bonney stated that he sent emails between his government and personal email accounts so that he could easily access this information regardless of his location.
And one of only five recommendations in the Denham report is this one:
Government should provide its employees with sufficient technological resources to ensure that they do not have a reason to use personal email accounts in the performance of their government duties.
The government staff I know agree with Denham: it would be nice if government e-mail didn't suck.

It doesn't have to suck

When I started up my cleverelephant.ca infrastructure in 2008, I swore to never run my own e-mail server again, so I signed up for Google Apps. At the time, Google was pushing "enterprise email archive" service called "Postini", which you could add to your account if you were that kind of enterprise. I wasn't, but I wondered what "Postini" was.

All "Postini" does is sit in your institutional email delivery pipe. It take copies of all mails sent and received, and provides an administrative search-and-retrieval interface making it easy for authorized users to pull up any email to or from any person from any time. Why would any company want this capability? Because they are legislatively required to have it, by the US Sarbanes-Oxley law that applies to public companies.

So in 2008 a robust enterprise email archive capability existed, available for a few dollars per account. It seemed very relevant to the BC Rail e-mail scandal that occurred a few months later. After the scandal had run its course, I suspected that enterprise email archiving would be a top government IT priority.

Was it? (Hah!) I recently ran a little test.

But it still does suck

Dave Nikolejsin
Since I have an abiding interest in the genesis and continued thrashings about of the BC Integrated Case Management system, I decided to see what kind of e-mail record the Chief Information Officer, Dave Nikolejsin, generated during the critical period in which the prime contractor was selected, 2009.

So I FOI'ed his e-mail for 2009. All of it.

If BC had an enterprise e-mail archive, they would simply run a query on the archive ("After:2009-01-01 AND Before:2010-01-01 AND (To:Dave.Nikolejsin OR From:Dave.Nikolejsin)") and send me back the result 2 minutes later.

Surprise! They don't have an enterprise e-mail archive. Instead, I got this email.
I am contacting you in regards your above noted FOI request, which you submitted on June 10, 2013 for:

“All e-mail From: or To: Dave.Nikolejsin@gov.bc.ca. Date range is January 1, 2009 to December 31, 2009”.

It is possible that emails sent to/from Mr. Nikolejsin have since been appropriately filed, as these communications may relate to particular issues or topics of note. I anticipate that a significant search would be required to determine this, which could result in a fee.

Is there a particular subject/topic of interest, or perhaps a particular sender/receiver you wish to focus on? I believe that by concentrating your request in this manner, we can decrease the likelihood of a fee being generated. Please let me know if you are open to discuss, or have any questions.
The gist of the message is that e-mail is archived by being printed and filed (old skool!), therefore searching all the files for individual e-mails would be onerous (fair enough). I decided to ask again for a electronic archival search:
I was hoping this would be a simple computer query against a corporate mail archive, not a manual process. Is that not the case?
No, it's not the case.
Mr. Nikolejsin may have email responsive to your specified time line in his Outlook mailbox, which would indeed be searchable electronically. If you would like to focus your request to his mailbox alone, (i.e. excluding records that may have been filed elsewhere), we can certainly accommodate, but I wanted to ensure you were aware of the potential search required to satisfy your request. Please let me know how you would like to proceed.
So instead of a single searchable archive with a single uniform retention policy, the government retains whatever happens to be in the individual Exchange mailboxes of their 20K employees, with retention individually managed by those employees.

Given the option, I decided I was still curious: what is in the BC CIO's Outlook mailbox for the year 2009? This is a busy, important, technically plugged in guy, what does he do with the valuable personal and institutional memory embodied in his e-mail archive?

Answer: He deletes it.

There are four e-mails in the CIOs mail history for 2009. Two of them are bits of login info sent to him by tech support. One is a policy document on Wifi security he sent out to ministry CIOs. One is a notification of a reorganization he sent out. That's it.

By way of comparison, my mail box for 2009 contains at least 2000 e-mails just FROM me. How many are in your box? (Gmail search string: "after:2009-01-01 before:2010-01-01 from:your@email.com")

It sucks at the Highest Levels

This week's FOI report is not not the first that touches on staff use of e-mail. The Commissioner also did an inquiry into the strange case of the "Ken Boessenkool scandal" (it really is amazing this government got re-elected) in which the interview and dismissal of the Premier's Chief of Staff somehow generated no written records at all. The Premier's office explained the conundrum to the Commissioner this way:
The general practice within the Office of the Premier is to communicate verbally in person. Email communications usually consist of requests to make telephone calls or meet in person. Generally, staff members in the Office of the Premier do not make substantive communication relating to business matters via email. Most of the emails are transitory in nature and are deleted once a permanent record, such as a calendar entry, is created.
That's right, the highest executive office in the Province runs entirely "verbally in person". No substantive matters in e-mail. In 2013.

E-mail is an incredibly valuable tool for communication, and I use e-mail tools like Gmail daily to organize and search my voluminous historical archive of communications. Access to my saved e-mail history makes me a smarter and more capable professional. Threaded histories of e-mail conversations and decisions, technical disputes and resolutions, even files ("where's that thing I sent to Joe...") are easily accessed via my e-mail history.

In contrast, our high-level government staff (even down to the CIO?) are deliberately excising this information trail on a daily and hourly basis, and even though doing it makes them less effective and capable, they feel that they have to do it because every email in their archive is treated by the FOI act like a final product, like a "document".

(Un)intended Consequences

One of the unintentionally funny passages in the latest Denham report is this one:
Government should have been taking steps to ensure that it was preserving correspondence relating to the Outreach Plan on its own servers both in the name of good governance as well as for the responsible preservation of documents that could be the subject of future access to information requests
In a world in which staff purge their email regularly to avoid FOI requests, keeping emails "in the government" is actually counter-productive to future investigations. Staff will just delete their e-mails at the end of the day as part of their daily email purge, and no records will exist at all.

At this point, government email for policy and political work is institutionally broken. It can barely be used as e-mail (as most of us understand it) anymore.

I have a completely un-founded suspicion that the parlous state of e-mail archive management in BC is somehow related to the FOI consequences of having a properly implemented archive. What would the political reaction be to an IT project proposal for a high quality persistent e-mail archive? Would there be enthusiasm? Would any IT manager even be so foolhardy as to propose a better archive system to their political masters?

The BC government e-mail system has become defective by design, because fixing the technological underpinnings of government e-mail would only magnify how institutionally broken the system is.

E-mail is only the thin edge of the wedge.  The cost of creating a "record" (audio, video, text) using modern technology is now so low as to be basically zero. Does that mean that everything government does should now be on the record? Or do we now need to be clearer about what "records" are?

The future is not far away...

What about when government has an IP phone system? Why aren't all phone conversations a public record? Every government phone call can be easily recorded, voice scanned into text and made searchable, using today's technology. Check out Google Voice.

What about when government staff are all wearing Google Glass (or something similar), personal or otherwise? We can already FOI hand-written notes from yellow legal pads as "official records" of otherwise private meetings. Government business transacted on private e-mail is still subject to FOI. So why not also have FOI access to first-person recordings of all meetings, shot by the principals using their augmented reality glasses?

The FOI act needs to be re-thought in the light of modern record keeping technology. 

Right now the BC government is managing its FOI exposure by just not properly recording their activities. As a result, incompetence in record keeping is currently a virtue, not a vice. This is not a healthy situation. It would be preferable if government could start properly recording their history, within the context of a revised FOI act that respects the difference between interim and final documentation and decisions.

8 comments:

gingersnap said...

Canada ha become a, cesspool of corruption especially since, Harper's so called majority.

Ex BC Liberal Premier Gordon Campbell is, Harper's favorite henchmen. Gordon Campbell also twice lied and cheated, to win his elections too.

Gordon Campbell was the worst, most corrupt and shameful Premier, in the recorded history of BC.

It's a joke to think, Campbell wouldn't have e-mails destroyed. Campbell point blank refused to say, what the debt from the Olympics was. The papers pulled from the FOI, they were whited out.

Gordon Campbell's election lies were? The BCR wasn't for sale. The HST wasn't on Campbell's radar either.

Campbell used every dirty tactic in the book, to stay in office. Campbell was working with Herr Harper to, totally dismantle BC. We are fully aware of Harper's dirty tactics as well. Campbell thieved and sold everything out of BC, he got his dirty hands on.

Rob Skelly said...

Are you considering a career in journalism, Paul? Because this is exactly the sort of thing I wish one could find in the "papers of record": Commentary by specialists on the functions of government relevant to their specialties.

RossK said...

What Mr. Skelly said.

.

Ian Turton said...

If it helps Paul I have 59 emails from or about you in 2009?

Paul Ramsey said...

@Rob, no career changes for me! Better to make filthy lucre in software development and journalize as a hobby. You're right that the MSM concentrates pretty boringly on the conventional wisdom and politicized topics of the day. I used to point to local US papers (Seattle PI, Milwaukee Gazette) as a counter-example, where non-partisan policy reporting was more common, but since they are all going out of business, that doesn't work so well anymore.

I hope that other hobbyists will pick up some of the pieces of policy reporting in their areas of expertise. Since publishing platforms are free now, the barriers to doing so are a lot lower. And I already read things like Sightline and Gordon Price for urban policy issues, and Bob Mackin and Northern Insights for alternative investigative reporting.

The local newspaper may be dying, but a good feed reader, and a collection of decent local bloggers can go some of the way to replacing it.

Paul Ramsey said...

@RossK, thanks! And regards your post, you're right. I forgot to mention the Dobell Maneuver, which is the sine qua non in FOI avoidance: create no records at all. And actually, sending emails down the memory hold is against the rules, while the Dobell Maneuver is not, which makes it preferable. A real e-mail archival system would make e-mail deletion impossible, which would force a wider swath of government to adopt the pure Dobell Maneuver. Perhaps we can foster a 100% oral culture and bring back the old skills of eidetic memory and long form story telling? The Cabinet Sagas, anyone?

Paul Ramsey said...

@gingershap, I think any view of politics that has to hang on the venality of the participants is fundamentally missing the interesting material. I prefer to think in terms of the systems and how they work regardless of the personalities involved. The email/FOI problem would exist and the dynamic of secrecy would be just as strong for an NDP government, that's what makes it an interesting problem to solve.

Nicklas Avén said...

Nowadays you or your organisation don't have to bother about e-mail archives any more.

From what I understand you can just ask NSA and they can serve you with all your conversations. The best thing is that you can keep track of both your emails and phone calls all together.

I don't know who to contact. Mr Snowden I think is fired since he had a disagreement with his employer about how the service should be advertised. But as mr Snowden claims: "Just give us an email address and we can find all you have said and thought since you came online", Works for you and works for the president.

Give it a try, it is probably the same discs that stats spinning no matter if you ask NSA or googles archive directly.

About Me

My Photo
Victoria, British Columbia, Canada

Followers

Blog Archive

Labels

bc (32) it (26) postgis (17) icm (11) enterprise IT (9) sprint (9) open source (8) osgeo (8) video (8) management (6) cio (5) enterprise (5) foippa (5) gis (5) spatial it (5) foi (4) mapserver (4) outsourcing (4) bcesis (3) foss4g (3) oracle (3) politics (3) COTS (2) architecture (2) boundless (2) esri (2) idm (2) natural resources (2) ogc (2) open data (2) opengeo (2) openstudent (2) postgresql (2) rant (2) technology (2) vendor (2) web (2) 1.4.0 (1) HR (1) access to information (1) accounting (1) agile (1) aspen (1) benchmark (1) buffer (1) build vs buy (1) business (1) business process (1) cathedral (1) cloud (1) code (1) common sense (1) consulting (1) contracting (1) core review (1) crm (1) custom (1) data warehouse (1) deloitte (1) design (1) digital (1) email (1) essentials (1) evil (1) exadata (1) fcuk (1) fgdb (1) fme (1) foocamp (1) foss4g2007 (1) ftp (1) gds (1) geocortex (1) geometry (1) geoserver (1) google (1) google earth (1) government (1) grass (1) hp (1) iaas (1) icio (1) industry (1) innovation (1) integrated case management (1) introversion (1) iso (1) isss (1) isvalid (1) javascript (1) jts (1) lawyers (1) mapping (1) mcfd (1) microsoft (1) mysql (1) new it (1) nosql (1) opengis (1) openlayers (1) oss (1) paas (1) pirates (1) policy (1) portal (1) proprietary software (1) qgis (1) rdbms (1) recursion (1) regression (1) rfc (1) right to information (1) saas (1) salesforce (1) sardonic (1) seibel (1) sermon (1) siebel (1) snark (1) spatial (1) standards (1) svr (1) tempest (1) texas (1) tired (1) transit (1) twitter (1) udig (1) uk (1) uk gds (1) verbal culture (1) victoria (1) waterfall (1) wfs (1) where (1) with recursive (1) wkb (1)