If I hear the words "triple delete" one more time...

… I’m going to tear my ears off. Also “transitory email”. Just bam, going to rip them right off.

Note for those not following the British Columbia political news: While we have known for many years that high-level government staff routinely delete their work email, a smoking gun came to light in the spring. A former staffer told how his superior personally deleted emails that were subject to an FOI request and then memorably said “It’s done. Now you don’t have to worry anymore.” (A line which really should only be delivered over a fresh mound of dirt with a shovel in hand.) The BC FOI Commissioner investigated his allegation and reported back that, yep, it really did happen and that the government basically does it all the time.

The Microsoft Outlook tricks and the contortions of policy around what is “transitory” or not, are all beside the point, since:

  1. there is no reason electronic document destruction should be allowed, in any circumstance, ever, because
  2. electronic message archival and retrieval is a solved problem.

The BC Freedom of Information Act, with its careful parsing of “transitory” versus real e-mails, was written in the early 1990s, when there was a tangible, physical cost to retaining duplicative and short-lived records – they took up space, and cost money to store.

Oh, yes, digital documents cost money to store, but please note, my old CD collection (already a very information dense media) takes up a 2-cube box in my garage, but barely dents the storage capacity of an $10 memory stick in MP3 form. My book collection (6 shelves) hardly even registers in digital form. You use more data streaming an episode of Breaking Bad. Things have changed since 1995. And since 2005.

So why are we still having this conversation, and why does the government have such lax rules around message retention? And let me be clear, the government rules are very, very, lax.

In the USA, public companies are under the Sarbanes-Oxley rules and have extremely strict requirements for document retention, with punishments to match:

“Whoever knowingly alters, destroys, mutilates, conceals, covers up, falsifies, or makes a false entry in any record, document, or tangible object with the intent to impede, obstruct, or influence the investigation or proper administration of any matter within the jurisdiction of any department or agency of the United States or any case filed under title 11, or in relation to or contemplation of any such matter or case, shall be fined under this title, imprisoned not more than 20 years, or both.”

Similarly, in Canada investment companies must keep complete archives of all messages, in all kinds of media:

Pursuant to National Instrument 31-103 … firms must retain records of their business activities, financial affairs, client transactions and communication. … The type of device used to transmit the communication or whether it is a firm issued or personal device is irrelevant. Dealer Members must therefore design systems and programs with compliant record retention and retrieval functionalities for those methods of communication permitted at the firm. For instance, the content posted on social media websites, such as Twitter, Facebook, blogs, chat rooms and all material transmitted through emails, are subject to the above-noted legislative and regulatory requirements.
— IIROC Guidelines for the review, supervision and retention of advertisements, sales literature and correspondence, Section II

Wow! That sounds really hard! I wonder how US public companies and Canadian investment dealers can do this, while the government can’t even upgrade their email servers without losing 8 months worth of archival data:

As it turned out, the entire migration process would take eight months. When the process extended beyond June 2014, MTICS forgot to instruct HPAS to do backups on a monthly basis. This meant that every government mailbox that migrated onto the new system went without a monthly backup until all mailboxes were migrated. Any daily backup that existed was expunged after 31 days. At its peak, some 48,000 government mailboxes were without monthly email backups.
— OIPC Investigation Report F15-03, Page 32

Corporations and investment banks can do this because high volume enterprise email archiving has been a solved problem for well over a decade. So there are lots of options, proprietary, open source, and even British Columbian!

Yep, one of the top companies in the electronic message archiving space, Global Relay, is actually headquartered in Vancouver! Guys! Wake up! Put a salesperson on the float-plane to Victoria on Monday!

Right now, British Columbia doesn’t have an enterprise email archive. It has an email server farm, with infrequent backup files, retained for only 18 months and requiring substantial effort to restore and search. Some of the advantages of an archive are:

  1. The archive is separate from the users, they do not individually determine the retention schedule using their [DELETE] key, retention is applied enterprise-wide on the archive.
  2. Archive searches are not done by users, they are done by the people who need access to the archive. In the case of corporate archives, that’s usually the legal team. In the case of the government it would be the legal team and the FOI officers.
  3. Archive searches can address the whole collection of email in one search. Current government FOI email searches are done computer-by-computer, by line staff who probably have better things to do.
  4. The archive is separate from the operational mail delivery and mail box servers, so upgrades on the operation equipment do not affect the archive.

So, for the next little while, the Commissioner’s narrow technical recommendations are fine (even though they make me want to tear my ears off):

But the real long-term technical solution to treating email as a document of record is… start treating it as a document of record! Archive it, permanently, in a searchable form, and don’t let the end users set the retention policy. It’s not rocket science, it’s just computers.

Keynote at FOSS4G 2015

On my usual bi-annual schedule, I gave a keynote talk at FOSS4G this year in Seoul, about the parallel pressures on open source that the move to cloud computing is providing. On the one hand, the cloud runs on open source. On the other hand, below the API layer the cloud is pretty much the opposite of open: it’s as much a black box as the old Win32 API. And the growth of cloud is paralleled by the shrinkage of infrastructure maintainers in other venues; the kinds of folks who currently use and produce OSS. It’s a big change coming down the highway.

Krugman FTW

“Sometimes I have the impression that many people in the media consider it uncouth to acknowledge, even to themselves, the fraudulence of much political posturing. The done thing, it seems, is to pretend that we’re having real debates about national security or economics even when it’s both obvious and easy to show that nothing of the kind is actually taking place.”
Paul Krugman

Big Data and Data Science Piss Me Off

Get off my lawn!

I don’t talk about this much, but I actually trained in statistics, not in computer science, and I’ve been getting slowly but progressively weirded out by the whole “big data” / “data science” thing. Because so much of it is bogus, or boys-with-toys or something.

Basically, my objections to the big data thing are the usual: probably your data is not big. It really isn’t, and there are some great blog posts all about that.

So that’s point number one: most people blabbing on about big data can fit their problem onto a big vertical machine and analyze it to their heart’s content in R or something.

Point number two is less frequently touched upon: sure, you have 2 trillion records, but why do you need to look at all of them? The whole point of an education in statistics is to learn how to reason about a population using a random sample. So why are all these alleged “data scientists” firing up massive compute clusters to summarize every single record in their collections?

I’m guessing it’s the usual reason: because they can. And because the current meme is that they should. They should stand up a 100 node cluster on AWS and bloody well count all 2 trillion of them. Because: CPUs.

But honestly, if you want to know the age distribution of people buying red socks, draw a sample of a couple hundred thousand records, and find out to within a fraction of a percentage point 19-times-out-of-20. After all, you’re a freaking “data scientist”, right?

BC IT Outsourcing 2014/15

If what goes up must come down, nobody told BC’s IT outsourcers, because they continue to gobble up a larger chunk of the government pie every year.

The BC Public Accounts came out today, and I’m happy to say that the People Who Are Smarter Than You Are managed to book another record year of billings: a $468,549,154 spend, up 8% over last year.

It’s not a victory unless you beat someone else, so good news:

  • Overall government revenue, up 5.4%
  • Overall government spending, up 2.4%
  • Health spending, up 2.8%
  • Education spending, up 0%
  • IT services spending up 8%!!!!

Don’t be sad, kids and sick people, IT services folks are Adding Value and Finding Synergies in ways that you just can’t. In the long run, workshopping the new Management Strategy Realignment Plan is just a better investment than fixing your gimpy hip, or hiring a teaching assistant to help Angry Jimmy focus on his work.

HP Advanced Solutions continues to dominate the category, adding $20M in billings this year alone (How many teachers could that hire? At least 200. Or even more teaching assistants.) In fact, two thirds of the billing growth this year was just HP.

There’s also a new kid in the enterprise software vendor list to keep an eye on: Salesforce.com (SFDC) showed up with a wee $463,053 in billings this year. I expect that to increase mightily in coming years. However, the big money in SFDC work will not be earned by SFDC (even after locking up the entire BC government enterprise back-office, Oracle bills less than $10M a year in software maintenance), but by the consultants providing SFDC “implementation services” (Deloitte, CGI, HP). Watch for a SFDC goldrush as the government starts replacing expensive Oracle systems with… expensive SFDC systems in the cloud.

The best part about hiring big enterprise IT companies like HP, Oracle, Maximus, and CGI to create lots of important Technology Process (and occasionally a bit of Product) for us isn’t the soothingly glacial pace of progress or the fantastic billing rates. It’s knowing that at least 20% of every public dollar spent goes straight to the bottom line of those companies, ensuring that shareholders and institutional investors survive through another year without undue financial hardship.

Until next year, keep on spending, British Columbia!