BC ICM Footnote

Early this year, the ICM project released a system assessment report from Queensland Consulting which was seized on by the BCGEU and others as showing ICM to be "deeply flawed and serious question remain regarding the software’s suitability for child protection work".

I found two things curious about the Queensland report: first, it was willing to talk about the failing of COTS methodology in a way that is rare in the stodgy conservative world of enterprise IT; and second, a large number of the process recommendations were reiterations of a previous report, a "Readiness Assessment" by Jo Surich, Victoria tech eminence grise and former BC CIO.  If the Surich report was so good that Queensland was quoting from it rather than writing their own material, I wanted to read it, so I filed an FOI.

You can read the full Surich report now, on the BC open info site.

I found two items of particular interest in Surich's report.

First, the date. Surich reported out in April of 2011. Over a year later, the Queensland consultants were re-iterating many of his recommendations. Surich was not listened to. Since a second set of consultants saw the same problems Surich did, letting his recommendations gather dust was a tactical error, to say the least.

Second, the big picture plan. Surich notes that, in addition to replacing all the tracking and reporting systems used in child protect, the Ministry was simultaneously changing the practice model (the "business process", in the usual IT terminology) that social workers use for their cases.

Simultaneously changing the business process and technology is a time honoured and widely replicated failure mode in enterprise IT development, because it makes so much sense.  If you're changing the business process, you'll need to change the systems to match the business process.  So, why not replace the systems at the same time as you change the business process?

Lots of reasons!

• Your mutating business process will constantly change the system requirements underneath you, resulting in lots of back-tracking and re-coding. 
• You won't know if your business process is bad until you deliver your system, which will then require further system changes as you again alter the business process.
• You double down on changes your staff need to ingest, in both tooling and methodology, and triple down as you add in fixes to business process after deployment.
• It's been done before, and it's led to some epic, epic failures.

As I learn more about the background to ICM, I have to ask myself if I would have done any differently. Particularly given the timelines and promises that backstop the huge capital commitment that gave birth to ICM, I find myself saying "no", I'd have made the same (similar) mistakes. I'd have walked down a very similar path. Probably not using COTS, but still trying to do business process and technology at once, trying to deliver a complete replacement system instead of evolving existing ones. Taking a set of rational, correct, defensible decisions leading down a dead-end path to failure.

The Microsoft Era

In many ways, the "Microsoft Era" has been over for quite some time. The most important developments in technology, the ones that change the way we work as IT practitioners, have been coming from other organizations for at least five years, maybe a decade (first open source, then the cloud companies).

But Microsoft has held on, and even garnered some of the aura of the scrappy underdog as it tries to compete in the new "network is the computer" (so close, Sun: right slogan, wrong decade) world. The reason Microsoft has been able to hang in this fight for so long is the continuing "price of doing business" revenue it has been able to extract from its operating system and office automation franchises.

Why, despite its continuing technological failures to deliver useful new functionality into its offerings, is Microsoft still hanging in there, still receiving billions of dollars a year from operating system and office software?

Because it's good enough, and because there is no alternative.

More realistically, because we BELIEVE it is good enough, and there is no alternative. As long as we believe that, we won't spend any time evaluating alternatives to see if they too are good enough.

It is the self-reinforcing belief that Microsoft produces and supports good enough software, and has the business continuity to CONTINUE to produce and support good enough software, that allows conservative IT managers to get to sleep at night, safe in the knowledge that they have backed a winner.

But what if they are backing a loser? Or, more to the point, what if they begin to BELIEVE they are backing a loser?

They are going to start looking for alternatives. And suddenly Microsoft will BE a loser. And the feedback loop will intensify.

All this to emphasize the importance that even ZDNet, yes, ZDNet is starting to lose faith in the market dominance of Microsoft.

I think Microsoft could continue to dominate the important, but no longer growing, desktop market for years, even decades to come. However, I don't think they will.

The analysts have already tracked the decline of Windows relative to tablet and phone operating systems. The CIOs are working on "bring your own device" policies, which will liberate countless desktops from Microsoft monoculture. The trends are not good, and as the trends are publicized more and more, they will only get worse.

Bye bye Microsoft, I wish I could say I'll miss you.

Boston Code Sprint: MapServer

There is a good crowd of MapServer developers here in Boston, too!

Jeff McKenna, Jonas Lund Nielsen, and Sarah Ward have been working hard on the documentation tree. Since the last release "MapServer" hasn't been just MapServer anymore: it also includes MapCache and TinyOWS. The documentation is being reorganized to make the parts of the system clearer, and to make the web site look less like a documentation page.

MapServer eminance grise Daniel Morissette spent the first part of the sprint getting familiar with the new GitHub workflow for working with MapServer. "Overcoming his GitHub fear" in his words.

Thomas Bonfort, the force behind MapServer's migration to git last year, is making another big infrastructural change this year, migrating the build system from autotools to CMake.  This should make multi-platform build support a little more transparent.

Steve Lime, the father of MapServer has been writing up designs for improved expression handling, pushing logical expressions all the way down into the database drivers. This will make rendering more efficient, particularly for things like WFS requests, which often include expressions that could be more efficiently evaluated by databases.  Steve is also working on a design to support UTFGrid as an output format.

Alan Boudreault has been very busy: adding support for the "DDS" format to GDAL, which means that MapServer can now serve textures directly to NASA WorldWind; completing support for on-the-fly generation of contour lines from DEM files; and completing a design for on-the-fly line smoothing.

Stephan Meissl worked on resolving some outstanding bugs with dateline wrapping, and has been tracking down issues related to getting MapServer "officially" certified as an OGC compliant server for WFS, WMS and WCS.

Note that much of the work done by the MapServer crew at the "code sprint" has actually been design work, not code. I think this is really the best way to make use of face-to-face time in projects. Online, design discussions can take weeks and consume 100s of e-mails. Face-to-face, ideas can be hashed out much more effectively, getting to agreed designs in just a few hours.

Boston Code Sprint: PostGIS

Rather than try and do a day-by-day take on the sprint, this year I'll try to outline project-by-project what folks are hoping to accomplish and who is attending.

This year we have the largest contingent of PostGIS folks ever, and work is being done on old-school PostGIS as well as the raster and geocoding portions that I usually look in with a bit of fear and trembling.

Azavea has sent four staff to the sprint. David Zwarg has worked on PostGIS raster at previous sprints and is continuing the tradition this year, focussing on building raster overviews inside the database.  Justin Walgran is updating GDAL to do EPSG code determination for arbitrary WKT strings. Adam Hinz is returning 1.5 semantics to some OGC geometry functions, and exploring SP-GiST implementation for PostGIS. And Kenny Shepard is adding geometry validity checking and policy enforcement to the shape loader utilities. Update: And, Rob Emanuele, who worked on QGIS support for PostGIS raster and the GDAL QGIS plug-in. Thank him, QGIS users!

Regina Obe has been working with Steven Woodbridge on an upgrade to the TIGER GeoCoder, integrating a C module for address normalization and a flatter scheme for storing address information. Early tests have shown this new approach to be much faster than the old PL/PgSQL code, which should make all the bulk geocoders out there very happy.

Stephen Mather is working on polygon skeletonization, to convert lakes and roads to linear equivalent features.

Oslandia has sent two representatives. Olivier Courtin and Hugo Mercier are working on SFCGAL, a C library wrapper around the C++ CGAL computational geometry library. SFCGAL will bridge PostGIS to use the CGAL 3D geometry algorithms, as well as algorithms like line voronoi generation, which will be useful for Stephen's skeleton project.

The two heavyweights of the PostGIS raster world, Pierre Racine and Bborie Park are here. Bborie is improving the performance of expression-based map algebra functions.  Always pushing the leading edge, Pierre Racine is coordinating the raster work, and collecting up new functional routines in PL/PgSQL into a library of utilities.

I'm spending time fixing my bugs in the 2.1 milestone, completing distance calculation for curved geometry types, and on the new pointcloud extension for LIDAR storage in PostgreSQL.

digital.cabinetoffice.gov.uk

Let me preface this by saying that "wow, you guys over there are just uniquely kicking ass" is an evaluation that "you guys" almost always dispute.

I know this from personal experience, since most of the world seems to think that Canada is uniquely kicking ass in the field of open source geospatial, a myth founded on a few hundred thousand dollars in Federal funding that a couple of companies bent themselves into pretzels to obtain and use for open source development work. Often, the exterior impression is simplistic, or only takes in one data point. (If the plural of "anecdote" is not data, neither is the singular.)

All that before I go on and say "the UK seems to be really ahead of the curve on a lot of things that are important in government and IT". In talking about them explicitly, at the least, and perhaps in doing them too.

• I was introduced to the concepts of the open data movement via the UK "Power of Information" report, back in 2008.
• I was interested a couple years ago to hear the UK government was pursuing an explicit procurement policy to put 25% of government contracts into the hands of "small and medium sized enterprises" (SMEs), though apparently the policy has had mixed results.
• I just now learned about the Martha Lane Fox report, "DirectGov 2010 and Beyond: Revolution not evolution" and digital.cabinetoffice.gov.uk.

First, get this: Martha Lane Fox is the "UK Digital Champion", appointed by the government, and the report is just one of the things she has produced. Imagine, a "digital champion"!

Second, check out the core recommendation: rip responsibility for web and digital interaction with citizens out of line ministries, into a dedicated digital office.

Well, they did it, or seem to have done, and the result is digital.cabinetoffice.gov.uk.  It's worth checking out the site, if only for the pictures of youthful nerd-hood, the independent job postings, the attempt to promote a more innovative and active digital culture than one typically finds within government.

The intent is clearly to create a new centre of mass of government IT, with a more innovative and web-oriented culture, and from there push out into the line ministries over time, rather than trying to alter the whole IT culture in one go. In 20 years, will the IT leadership of the UK be made up of digital.cabinetoffice.gov.uk alumni?

Update: Check out their design principles.

Priorities

Apropos of nothing, here's every RFP document issued for the ICM project in BC (and there were 10 different RPF/ITQ/RFI/NOI postings over the years) rendered as a word cloud. Notice the prominence of "child" and "family". When you get down to brass tacks, enterprise IT doesn't much care about what you do, but it very intensely cares about how you do it.
 

Once in a Lifetime: IT in Question Period

Today in question period, the opposition NDP in British Columbia devoted five of their six questions to information technology issues! It's pretty rare that the snooze-fest that is IT rises to the level of political notice, so I don't expect a repeat display again in my lifetime.

Some of the questions:

A. Dix: My question is to the Minister of Citizens' Services. An independent report on integrated case management across ministry computer systems, meant to improve services to vulnerable British Columbians, states that the system may never be fixed. A system that cost over $200 million may have to be thrown out because of mismanagement. The report says that in the bidding phases for this technology, the ministries in question were in chaos, going through so-called transformation with a revolving door of ministers. The children's representative has said: "This report speaks to incompetent stewardship." With such damning comments, and a lack of faith that the system can be fixed, can the minister explain what went wrong and when the government will take responsibility for this mess? 

R. Austin: It's not just in the child welfare system that this mismanagement has occurred but also in education. We know that this government spent almost $100 million on their electronic data system. What we don't know is how much was also spent by school boards on training, implementation and licensing costs. Communities and taxpayers should not have to pay the costs of Liberal mismanagement on these colossal computer projects. Can the Minister of Education tell this House how much in total was spent by districts and the government on this failed computer program?

The systems in question, ICM and BCeSIS, cost the government respectively about $200M and $100M. That is, far beyond the outer edge of where we can intellectually grasp the amount of money involved, or where the effort goes.

Here's one way of understanding what a $200M project budget means, look at what the budget means in terms of people.

Assume a $200K fully loaded staff cost (salary, overhead, benefits, everything). That's a big number, but let's assume experienced professionals being paid commensurately. That means $200M equates to 1000 person-years of effort.  Or, over a 5-year project span, a 200 person staff. What could you deliver, in 5 years, with 200 highly trained professionals?

Here's another way of looking at what $200M spent on IT means. Over the four year period from 2003 to 2007, Facebook grew from Mark Zuckerberg's dorm project to a company with 50 million active users. In that time, during which Facebook had no revenue, investors put just under $40M into the company. So it cost $40M to build the 2007-era Facebook.

So, $200M to build a failed case management system that (poorly) serves a working population of a few thousand civil servants, against $40M to build an intricate social network that serves a population of 50 million.

There's no way to get around it: enterprise IT is hopelessly broken, and much of the money we spend on systems development is wasted. We need a new way of doing business.

Modular Systems Development

Turns out not only am I not insane, I'm not on the bleeding edge or even remotely innovative!

When I say that enterprise IT projects should be broken into small parts, done by independent small teams, and manage risk by managing size, I'm actually parroting the United States Office of Management and Budget, the boringest, stodgiest organization on the planet!

Modular system development mitigates risk, says Daniel Werfel, federal controller for the OMB.

Under modular development, discrete project deliverables get pushed out more often, meaning that reviews and evaluations happen sooner, allowing for the possibility for quick corrective actions that can mitigate financial exposure.

And...

Projects of a more traditional variety--in which about 80 percent of the budget goes to customizing a commercial solution--cannot quickly adapt to changing laws or technology and often have costs rise by hundreds of millions of dollars.

Check out this awesome graphic from the guidance document:

Read the full guidance document if you dare: they didn't get to be the boringest, stodgiest organization on the planet by bringing their "B" game.

COTS uber alles?

I continue to follow the ongoing $180M-and-counting IT debacle of "integrated case management" in British Columbia with interest, perhaps because it's a local catastrophe and not some far-flung disaster.

The latest tidbit is an independent consultant's review of ICM from the perspective of the Ministry of Children and Families (MCF). I hadn't previously appreciated the extent to which the hydra-headed nature of ICM as a project of both MCF and the Ministry of Social Development (MSD) was contributing to disfunction.

In politic terms, the consultants point out that MCF was the weaker partner in the partnership, and therefore got steamrolled during the design phase. MSD had a stronger team, was better prepared, and got what they wanted. MCF was the red-headed step-child.

Another issue the consultants noted was the way the "COTS" requirement and "no customization" made a bad situation worse. No matter what the clients wanted, there was a arbitrary external rule always in play:

The ICM solution [should] not be based on a customized system, but as much as possible rely on an “out of the box” product (with necessary configuration to meet the respective needs of MCFD and MSD).

If you knew the magic incantations, you could sometimes get an exemption, but the MCF team didn't know how to play the game:

It became clear in our interview process that while the project principles did support exceptions to the “no customization”’ rule where there was a solid business rationale, current MCFD leadership did not recognize they could make this appeal, and so remained fully constrained by the design and implementation of a single instance untailored to MCFD needs.

And so the final product rolled out was a design disaster for child protection:

The decision to implement a single instance of the Siebel product required that all users (regardless of ministry or program) share a common set of forms and data attributes, even though they touch on completely different topics and clients, and speak very different practice “languages”. This has also led to many forms being developed with redundant fields and or labels that are meaningless to the majority of users.

In fact, the "phase 2" rollout was such a disaster that the whole COTS "no customization" rule seems be to up for review! Tucked into the consultant report is this little gem of a status update:

The ICM/Deloitte Project Team has also been working on the design and development of an external service provider portal which will utilize a custom user interface (i.e. leveraging the information within Siebel but presented through a completely separate web application). While the service provider portal does not have any direct relationship to MCFD’s Child Protection Services, it does demonstrate that the project team can design and develop a solution that builds on the Siebel platform with a customized user interface and significantly improved usability. This approach will be of value as we consider the revised child protection solution and we see it as a very positive step.

Plan B is now officially under way! The generic Siebel interfaces are being ditched in favour of (gasp!) custom designed user interfaces fit to purpose. Will the Siebel interfaces all be eventually phased out? That's my prediction.

The only question left is how long MSD and MFC will have to pay Siebel licensing for a system that has been delivered over-budget and with insufficient functionality largely to cater to the predictable limitations of the Siebel software and the "no customization" COTS philosophy. A long, long time, I fear.

Is building an enterprise systems a capital expense?

First, terminology: operating versus capital expenses. Bah! Accounting! However, it's important stuff. Wikipedia provides a good example of the difference:

For example, the purchase of a photocopier involves capital expenditures, and the annual paper, toner, power and maintenance costs represents operating expenses.

The key thing to remember is that a capital expense is supposed to convert cash into an asset.

Second, application: most enterprise information system builds these days are funded as capital expenses. Money is spent, and at the end of the day the organization places an entry on the balance sheet saying "System Z is worth X million dollars".

Third, contention: this common IT accounting practice is bullshit.

The reason it is bullshit is that the asset has to be placed on the books with a value, some dollar figure that represents what it is worth. This isn't just some made up number, it's important. This number is a component of the total asset value of the organization, and if you are adding up the value of the organization, it will be added in along with the cash, the real estate, the fixtures, all the other things that we know do have value.

Does the enterprise information system have value? How much? Where does that number come from?

Is the information system value a re-sale value? No. Unlike the photocopier from the Wikipedia example, the enterprise information system has no value outside the organization that built it: it's sui generis, custom-built for the purposes of one organization.

Is the information system value a replacement cost? No. Governments build things like bridges and highways that don't necessarily have a re-sale value (who is going to buy them?) but still have a provable value in terms of their replacement cost. If it takes $100M to build a bridge, it's a fair bet that it'll take $100M to build a second one just like it. Is this true of enterprise information systems? If I build a billing system and it costs me $1M, will it cost you the same thing to build a second one? If you have access to my source code and specifications, you could probably build an even nicer one for 1/10 the cost or less, since you wouldn't have to spend any time at all discovering the requirements or doing data cleansing. There's no expense in materials and relatively little of the labor value ends up embodied in the final product--the system value is not the replacement cost.

Is the information system value durable? No. Long-lived public infrastructure may not be re-sellable, but it often has a useful life span reckoned in multiple decades. Given that, it's fair to say that the cash involved in building it has not been spent, but has been converted into a fixed asset. Do enterprise information systems have that kind of durability? Do I even have to ask?

Does the information system value reflect a one-time acquisition cost? No. The $3B (gulp) Port Mann bridge newly opened on BC's Lower Mainland will have an annual maintenance cost of perhaps $30M a year, %1 of the capital cost. The asset is built, and we get to keep using it almost for free (except for the tolls to cover the loan, ha ha!). Is this true of enterprise information systems? In my experience, information systems budget 10-20% of build cost for expected annual maintenance. So, it's very different again.

Is there any case at all to be made that enterprise information systems can be treated as assets, and hence budgeted as capital expenses? Yes. But it requires that the asset value be assessed very conservatively (the whole build budget is not indicative of final value), and that the value depreciate very quickly (the system has a relatively short lifespan, years not decades).

But rapid asset depreciation is just as hard on a balance sheet as operating spending is! Build a $100M system and depreciate it at 10% a year, all you've done is concentrate $100M of IT spending into a very short period of time and spread out the depreciation over a decade.

So, skill testing question for all you IT practitioners out there. Who will get the better results:

1. the manager who spends $100M in one year on a system build and depreciates his asset over the ensuing decade? or,
2. the manager who spends $10M a year over a decade in incremental system enhancements and improvements?

Note that both approaches have exactly the same effect on the organizational balance sheet. Take your time, don't rush your answers.

Accounting for enterprise information systems as capital expenses is a mistake. It's dubious from an accounting perspective, because the "asset" on the books isn't re-sellable, doesn't hold its value, and doesn't cost nearly its book value to replace. And it's dubious from a practical perspective because it forces system development and maintenance into an incredibly risky and inefficient spending pattern.

Don't do it if you can help it.