BC IT Outsourcing 2015/16

It’s BC Public Accounts time again (calm down!), and in this year’s analysis of IT outsourcing we have a surprise result: total spending on IT contractors by the BC central government actually fell in the 2015/16 budget year.

The fall-off is almost entirely the result of a collapse of billings to the central government by IBM, though Deloitte had a small fall-off also.

Other IT contractors continued to bill heartily, including massive category leader HP Advanced Systems (HPAS) who billed a very respectable $163,499,787. For comparison, that’s more than was spent on the Ministries of Energy & Mines, Agriculture, Small Business, and International Trade, combined.

Since 2010, IBM’s billings to central government have fallen in four of the last six years, which leads me to a question: what’s the matter with Big Blue?

After further research, the answer is this: nothing at all. Big Blue is doing just fine.

It turns out that the fall off in IBM billing to central government has been more than offset by a massive increase in IBM billing to the Provincial Health Authorities. The Ministry of Health budget is huge, but much of it is spent by the Health Authorities, and the provincial public accounts only record transfers to the Authorities, they don’t keep track of who the Authorities spend with.

Fortunately, the Authorities also have to publish annual financial statements, and I have now input all that data into my summary sheets, through to 2015 (Health Authorities do not publish their detailed payments data until the fall, so I am one year behind until then). The results are, frankly, staggering…

When Health Authority spending is taken into acount, IBM revenue from the BC government has not fallen at all. It has instead been on an almost unbroken tear upwards, taking total government IT outsourcing spending to just under $700 million dollars in 2014/15.

A reasonable chunk of that billing over the last couple years has been on the PHSA/Vancouver Coastal Health electronic medical records project, a $1B+ firehose of cash that IBM has been slurping on heavily, though to little practical effect.

Since IBM was kicked off the project in mid-2015, we can expect their take to fall in the 2015/16 data, but much of the slack should be taken up by medical software vendor Cerner who have been given the prime contractor role on a no-bid, sole-source basis (I’m sure that will work out fine).

While entering the Health Authority data, I had the opportunity to learn a little about the ecosystem of local vendors who support the health sector. Unsurprisingly, there is little overlap between those companies and the ones I already know of who support central government in Victoria: smaller companies tend to be more specialized.

When you lump every local vendor together, central government and health authority, and plot them up, the result is… underwhelming.

While there has been a trending upwards of local IT contracting over time, it is dwarfed 10:1 by the dollars spent on the large international consultancies. HPAS alone takes in over three times what every local IT services firm in the province bills.

The moral of the story, I think, is: if you have money, you will spend money.

  • As part of my research, I reviewed the school district financials and found very minimal spending on IT consulting: not enough to warrant the effort to enter all the data. School districts are just too poor to waste money on stupid IT consulting.
  • On the flip side, I reviewed the ICBC 2015 list of vendors and found, in just one year, spending of:

    • $5,051,887 on Accenture
    • $23,944,563 on IBM
    • $3,177,241 on HP Advanced Solutions
    • $7,962,866 on Deloitte, and
    • $5,962,057 on Quartech Systems.

    On administrative operating costs of about $350M, that’s $46M of IT spend, over 18%. If you’ve got money, you’ll find a way to spend it. However, since ICBC is not a direct arm of government, I didn’t include their crazy spend in my totals.

Next year is hard to predict: IBM should have a fall-off; offset partially by Cerner, as they pause the EMR project before ramping back up again. HPAS should continue incremental growth. A wild card is the Natural Resources Permitting Project, which hasn’t truly hit its spending stride yet. If it gets going, I expect Deloitte will increase their billing substantially in 2016/17.

Until next year, happy billing BC!

Politics Amid Plenty

“Political leadership is a subtle art in times of plenty. When there are no great crises, there is no public demand for heroic acts. Politics becomes a parlor game, ignored by all but the most devoted citizens, a game practiced most assiduously by those interests–the business associations, trade unions, and single-issue groups–who have a direct stake in the outcome. The game turns on tactics and gestures, on the ability to placate factions, rather than to inspire the masses.”
- Joe Kein, The Natural

“Innovative” Government IT

Jennifer Pahlka from Code for America has published a pair of essays on IT leadership that anyone thinking about “organizational transformation” should read. They are very high level and a bit lengthy, but are full of good ideas:

“Innovative” Government IT

In talking about innovation in the second essay, Pahlka has this wonderful section, about the difference between adopting modern development practices and actually innovating.

The problem is that if you want “government technology as good as what we have at home,” you’re going to have to do things like move to the cloud and test prototypes with actual users.

That’s not innovation. That’s just how tech works today.

The practices and tools that result in good digital services vary from organization to organization, to be sure, but there is a lot in common that the private sector, and increasingly the public sector, pretty much agree on as standard.

When we frame these practices as somehow cutting-edge, risky, or non-standard, we do the mission a great disservice.

Sing it! Adopting open source, agile development and cloud technology are not “innovative” any more, they’re just table stakes, the minimum possible ante upon which to build a responsive technology organization.

The other big take home for me is in the first essay, decomposing the functional roles that are traditionally mushed into a single “CIO” position and pointing out how unlikely they are to match the capabilities of any one person:

  1. Digital services: the services residents use to engage and do business with the City. This can also include APIs and open data programs, though this is often the domain of the other CIO (the innovation officer.)
  2. Back office software: Day-to-day core services like email, human resources management, budgeting, fiscal and accounting that all departments rely on.
  3. Mission IT: The business applications that run the internal processes of departments and agencies. These are often custom, but can now make use of underlying commodity technology.
  4. Infrastructure: Network and connectivity, hosting and device management.

I tend to break the IT roles into just two pieces, but I think Pahlka makes a strong case for all four. My two pieces are:

  • Infrastructural. Email, network, desktop, phones, backup, payroll, finance. All skills that are replicated across every organization in existance, where little domain knowledge is required. With appropriate contracting safe-guards, you can outsource all of this stuff.
  • Strategic. Business systems unique to your organization and all the facets thereof. Back-end, front-end, user experience, API, etc. The IT tools are common, but the domain knowledge of the data and business processes are unique to your organization. You should own both the technology and the people creating it, for maximum flexibility.

Either way, the idea that the folks who are best at handling one category are also good at handing the other is dangerous.

You no more want your ultra-cautious infra manager (“let’s map out a 4 month plan for that…”) running development than you want a cowboy lead developer making decisions (“deploy!”) that might affect network uptime.

Anyways, go read! Time well spent.

Primitive

“Far out in the uncharted backwaters of the unfashionable end of the western spiral arm of the Galaxy lies a small unregarded yellow sun. Orbiting this at a distance of roughly ninety-two million miles is an utterly insignificant little blue green planet whose ape-descended life forms are so amazingly primitive that they still think digital watches are a pretty neat idea.”
- Douglas Adams, HHGTTG

Drowning in Passwords

I like the internet, I use a lot of sites, I don’t fear online shopping or discussion or communication or banking. As a result, I have a pretty healthy footprint of accounts, and I have been finding recently that my brain can no longer keep up.

Drowning in Passwords

So I’ve started using a password manager, LastPass, which works just fine.

Once I fully committed to it, the number of passwords managed started a slow and seemingly relentless climb, as over time I returned to all the many sites at which I had been forced to register accounts in the past. As a result, I’ve learned useful things:

  • The number of sites I have accounts on is far larger than I thought. I have 40 entries in LastPass already, and I imagine I’m only about 2/3 of the way through adding all the sites I use more than once a year. Of those, easily a dozen are what you might call “important”: banks, brokerages, email, OAuth sources (Twitter, Facebook, Google), etc. Far more sites than I kept separate passwords for!
  • The centrality and potential vulnerability of the email account is hard to overstate. For 90% of the accounts I have added, the first step was “reset the password”, since I had forgotten it. (In fact, that was my old access mechanism, since I accessed many sites so rarely.) And “reset” uses access to email as a source of proxy authentication. So, 0wn my email address, 0wn me, entirely. If you haven’t enabled two-factor authentication on your email account yet, you need to, because of this.

Usually, improving security involves things getting more inconvenient but in the case of using a password manager, it has actually been a net improvement. No more time spent trying to remember which of the passwords in my limited brain key-chain I had used for a site. No more reset-password-and-wait for the many sites at which I had no idea what the password was.

LastPass has been very good, and I have only one note/caveat. The password manager works by installing a plug-in into your browser, so if you use multiple browsers (I use both Chrome and Safari regularly) you’ll end up with multiple plugins. The preferences on those plugins are managed separately. Same thing if you use multiple computers (I do, desktop and laptop). Each plugin on each computer has separate preferences.

This is important because the LastPass preferences are, in my opinion, a little loose. Once you provide the master password, by default, the password vault remains unlocked and available until you actually shut down your browser. I can go days without shutting down my browser. So, I changed that preference to a time-out of 15 minutes instead. But I had to change it on every browser and on every computer I owned, which was not intuitive, since the plugin is good about sharing other information to other installs transparently (add a password on one browser, it’s available on all).

So far I’ve been using the free version of LastPass, but as I move into the mobile world I will probably have to buck up for the paid version for support on mobile devices. Given how much it has simplified my life, I won’t begrudge them the dollars.

Bonus paragraph: Isn’t saving all my passwords on a cloud service really dumb? Not so much. The passwords are only ever decrypted locally by the password manager plugin. The cloud just stores a big encrypted lump of passwords, and I have a lot of faith in AES256. However, my security now has one big central point of failure: the master password. But since I only have to remember one master password, I have been able to make it nice and long, so any remote technical attack on my security is unlikely. That just leaves all the other kinds of attack (social engineering, keylogging, device theft, human factors, etc, etc).