PostgreSQL "Compatible" Aurora

PostgreSQL "Compatible" Aurora

While I know full well that Amazon’s marketing department doesn’t need my help, I cannot resist flagging this new development from the elves in Santa’s AWS workshop:

Today we are launching a preview of Amazon Aurora PostgreSQL-Compatible Edition. It offers … high durability, high availability, and the ability to quickly create and deploy read replicas. Here are some of the things you will love about it:

Performance – Aurora delivers up to 2x the performance of PostgreSQL running in traditional environments.

Compatibility – Aurora is fully compatible with the open source version of PostgreSQL (version 9.6.1). On the stored procedure side, we are planning to support Perl, pgSQL, Tcl, and JavaScript (via the V8 JavaScript engine). We are also planning to support all of the PostgreSQL features and extensions that are supported in Amazon RDS for PostgreSQL.

Cloud Native – Aurora takes full advantage of the fact that it is running within AWS.

The language Amazon uses around Aurora is really wierd – they talk about “MySQL compatibility” and “PostgreSQL compatibility”. At an extreme, one might interpret that to mean that Aurora is a net-new database providing wire- and function-level compatibility to the target databases. However, in the PostgreSQL case, the fact that they are additionally supporting PostGIS, the server-side languages, really the whole database environment, hints strongly that most of the code is actually PostgreSQL code.

There is not a lot of reference material about what’s going on behind the scenes, but this talk from re:Invent shows that most of the action is in the storage layer. For MySQL, since storage back-ends are pluggable, it’s possible that AWS has added their own back-end. Alternately, they may be running a hacked up version of the InnoDB engine.

For PostgreSQL, with only one storage back-end, it’s pretty much a foregone conclusion that AWS have taken a fork and added some secret sauce to it. However, the fact that they are tracking the community version almost exactly (they currently offer 9.6.1) indicates that maybe their fork isn’t particularly invasive.

I’d want to wait a while before trusting a production system of record to Aurora PgSQL, but the idea of the cloud native PostgreSQL, with full PostGIS support, excites me to no end. RDS is already very very convenient, so RDS-with-better-performance and integration is just icing on the cake for me.

I, for one, welcome our new cloud database overlords.

NRPP, It's Alive!

Hey, good news zombie lovers, the project I’ve declared dead (or, at least, doomed) is not only still shambling around, it’s going to get the official political glad-handing treatment tomorrow:

NRPP, It's Alive!

We are excited to share an important Natural Resource Permitting Project (NRPP) milestone. Tomorrow—December 7—NRPP’s first service on the NRS Online Services website will be launched in Williams Lake at FrontCounter BC with Minister of State for Rural Economic Development Donna Barnett.

I always feel sad for the poor politician tasked with the “new website” announcement, because honestly, is there any announcement that feels like more of an empty gesture towards real action? “Yes, I understand you wanted $50M for addiction treatment, but… how about this new web site?”

Anyways, while she’s getting her demonstration of the new NRS Online Services website, here’s some questions Minister Barnett might like to ask:

  • Can I drive? Let me use the computer. Why don’t you ever let the Minister use the computer? I’m the Minister, dammit.
  • Why is it so slow? Surely for this kind of money it should be fast.

  • What are all those buttons across the top? What does the pencil mean? I’ve never seen those icons before. What do you mean “have I ever used ArcView?”, what’s ArcView?

  • You really expect me to click all those links to find out what impacts my project? There’s got to 20 of them!

  • Does it work on my phone? Ah, kind of, I see. No, I don’t know what a “bootstrap” is.

  • Honestly, why is it so slow? Am I clicking it wrong?

  • Does this work align with government priorities? Maybe you need to do some more transformation on it.

Congratulations, NRPP on your first step towards transforming the sector, have a great demo!

Addendum: In the press release, Minister Steve Thomson is quoted as “saying” (poor Ministers always sound so stodgy when they “speak” in press releases):

Through the Natural Resource Permitting Project, the Province is making a significant investment in helping communities balance economic development with protecting our natural resources.

My notes:

  • The province is surely making a “siginificant investment” in Deloitte and CGI. Whether that investment ends up “helping communities” is still very much a question in flux.
  • The purpose of NRPP is not “balance” between economic development and protected natural resources. The business case makes clear, the purpose of NRPP is to increase natural resource extraction rates, generating a permanent lift in royalties to offset the (significant) costs of NRPP. Sadly, even on those terms it’s likely to fail.

Such a short sentence, yet still so much misdirection.

NRPP, Still Doomed

I have found it extremely difficult to extract information about NRPP from the government. FOIs have come with very large fee assessments, or documents have been completely redacted – you’d think I was out to get them or something.

NRPP, Still Doomed

New flash: I am not out to get them. In fact, they are the best-run IT mega-project I’ve seen so far in the BC government. But that doesn’t change the fact that, like the dinosaurs before them, they are doomed, dooooomed.

Fortunately, the FOI process can be indiscriminate, information can leak out despite the best efforts of the project team, and last month there was great tidbit about NRPP.

Hiding inside an FOI request to the Environmental Assessment Office (EAO) were a number of interesting documents.

Who’s On First?

In an email on June 24, 2016, Wilf Bangert, the Assistant Deputy Minister (ADM) for NRPP, invited the steering committee of ADMs to a meeting to:

define the meaning of ‘sector transformation’ by documenting a model that will enable its implementation.

Or put another way, after three years and $50M, the “Natural Resources Sector Transformation Secretariat” (NRSTS) still isn’t clear on what “transformation” means, and is looking for guidance.

It’s not too hard to figure out what’s going on.

Lacking any operational mandate themselves, NRSTS has been going around looking for partners with operational permitting processes: “come work with us, we’ll re-structure your business work-flow and make it ‘better’ with cool software we haven’t built yet”.

To me, this sounds good (cool software! better!); to NRSTS, this sounds good (re-structure!); to the permitters it sounds like “we’re going to waste a bunch of your (scarce) time in workshops making you talk about abstractions instead of doing your job, then we’re going to upend your office in the service of ‘transformation’, while we experiment with software that is as yet unwritten”.

You know what happened to the guy who had the first human heart transplant? In IT terms, the procedure was a success – he didn’t die on the table. He died 18 days later of pneumonia, with his heart still pumping away.

Back to High School

After asking for advice on ‘transformation’ from the ADMs, Bangert then tells them he needs a “subject matter expert” (at a “decision making level”!) from each of them to attend “several” workshops in the summer and a week-long workshop in the fall. So, presumably a manager or director, whose time is sufficently low value that it can be donated to NRSTS for days at a time.

Model UN

But that’s not even the best part. The best part is the structure of that five-day fall workshop. NRPP is going to be running (has already run?) a “Model UN” process with all these managers and directors.

My favourite bullet points! From the “How Will it Work” section:

  • An unlimited number of delegates are allowed per Ministry. Attendance and pre-work completion is mandatory before and for the duration of the workshop.

Because more is better, and mandatory homework makes fast friends! From “Who Should Attend”

  • Folks who are highly motivated to make the Natural Resources Sector “processes” work better.
  • Extroverted communicators and people connectors.
  • Introverted thought leaders and thinkers.
  • Creative problem solvers.

Great combination! Anyone not invited?

Still, so far we’re just talking about a standard “consultant-facilitated workshop time vortex”, of a sort we’ve all participated in and/or inflicted on others. The bit that is really transcendent is the “engagement model UN process”:

  • There is a general assembly component
  • Only voting delegates attend
  • Voting on resolutions prepared by committees
  • Decision making body for the process
  • Mandatory that voting delegate attends
  • Fixed time for debate and voting
  • A chairperson oversees
  • Process repeats [emphasis added] until all aspects of the work flow have been reviewed, resolution prepared, and voted on.

How could this possibly go wrong?!?

The Smell of Desperation

Once again, inputs:

Outputs:

  • “What exactly do you mean by ‘transformation’, really?”
  • “Lend us your SME’s for a week, so we can figure out a generic process to stuff your business into.”
  • Also, some unimpressive deliverables.

Desperation

If a core early problem with NRSTS was that nobody wanted to be the first organization to be subjected to their tender mercies, imagine how they are perceived now, as they come up on the end of their Phase 1 funding and still haven’t even figured out what “transformation” means?

Would you trust your staff time and business process to an organization that looks likely to be blown up in the next 24 months? If so, why?

Addendum: Commenters, please weigh in on whether the recent departure of the Executive Director, Technology to work with major project consultant CGI is (a) a sign of good things to come (CGI positioning to win follow-on work in Phase Two) or (b) a sign of imminent disaster (man-in-the-know getting out while the getting is good).

NRPP is Dead

Long live NRPP!

The Natural Resource Permitting Project (NRPP) is now mired down, having failed to deliver on its ambitious promises to transform the sector with “generic frameworks that will support the ‘One Project, One Process’ model”.

But, as my ‘ole grand-pappy used to say to me: “When the going gets tough, the tough redefine success so they can still declare victory.”

NRPP is Dead

Accordingly, success re-definition is under way at NRPP. Success will no longer be a generational transformation in how government manages natural resources; success will now be submitting formerly paper forms using web forms.

But wait, I said NRPP is “mired down”, how can I tell? By measuring the outputs against the inputs.

Lots of Money Going In

NRPP has been ongoing in various forms and names since before 2013, and for at least the last two years has been carrying a staff/consultant complement that I’d estimate costs about $17M per year. I’ve heard estimates of expenditures to date of over $50M, and that is consistent with my back-of-the-envelope calculations.

So, $50M or more in. What’s come out? (Worth remembering, successful $1B start-up companies have been built for less.)

Not So Much Coming Out

In March of 2016, the Executive Director of NRPP gave a progress update to the Deputy Ministers Committee on Transformation and Technology (DMCTT). Good news: “year 2 of the initiative has been delivered on time, on scope and on budget”.

  • Clients can now access NRS online services for guidance, information and map-based data to support applications for authorizations
  • 290 data layers are now accessible through NRS Online Services
  • Hunters will be able to register online for the Limited Entry Hunt in mid-April 2016
  • Legislation will be introduced in Spring 2016 to move selected Fish and Wildlife authorizations to a criteria based notification model

All of these assertions are superficially true, but even from my perch far outside the warm light of the inner circles of government, it’s laughably easy to find substantial caveats and concerns about all four of them.

I really wonder what the is purpose of reporting to high-level “oversight” committees like DMCTT, if the committees just accept the reports and do not bother to do any independent verification and research.
If you have only the information and spin from the project in front of you, no matter how piercing and direct your analysis is, you’re never going to really be able to ask the tough questions, because the key information will be hidden or obfuscated.
This is why so much “oversight” seems to devolve into reductive discussions of schedule and budget, the only metrics that all participants are guaranteed to understand and that all projects are required to provide.
Feel free to deliver a product that fails to meet your user needs – the big boss will never notice. But slip your schedule by 2 weeks, and the fiery wrath of God will descend upon you. Project management and communication is optimized accordingly.

NRPP “Achievements”

I want to look closely at each of the pieces of good news about “year 2”.

Clients can now access NRS online services for guidance, information and map-based data to support applications for authorizations

Let’s do this interactively: go find the NRS online services web site. I’ll wait. Use the Google, start from the Ministry web site, however you like, go to it.

Back? How did it go?

I’ll wager you didn’t find the actual Natural Resource Sector Online Services portal, which though online seems to be linked to from nowhere, outside or inside the BC government.

This puts the claim that “clients can now access NRS online services” a little in doubt. Sure, they “can” access the services, but since the services are basically hidden, do they access the online services?

This new portal is one of the products of the $50M spent so far. It has an “OK” design, a bit wasteful of screen real estate and bandwidth, but clean and not too “last century”.

The portal also has a bunch of content and links to existing processes, which would be more impressive if they were not duplicative of content and links already assembled and put on the web (in the last decade) by Front Counter BC.

The $50M folks at NRPP appear to have mostly taken the content from Front Counter BC and re-skinned it using their modern web design, but provided vanishingly little value beyond that.

Re-packaging existing in-house knowledge and claiming it for your own is an old consultant trick from way, way back. Mark Twain once joked that “an expert is anyone who comes from more than 60 miles away”, and little seems to have changed since his time.

290 data layers are now accessible through NRS Online Services

Indeed they are, at least quite a few layers, I didn’t bother to count. However, like the portal, the mapping application is a recapitulation of functionality that government has been providing for a decade. Way back in 2002, the “Ministry of Sustainable Resource Management” was tasked to “deliver a corporate land and resource information data warehouse”: that is, a collection of all the land information in BC, and a web view of those layers. The warehouse and web maps have been around in various forms ever since.

In many technical respects the NRS map is superior to the old ImapBC (it’s more modular and reusable) but for the purposes of this post note that (a) like the portal it’s carefully hidden from public view and (b) it’s still not a net-new gain of functionality on a project that’s $50,000,000 in.

Hunters will be able to register online for the Limited Entry Hunt in mid-April 2016

Again, this is true, but yet again there’s less there than meets the eye. NRS was going to transform resource tenuring: one account for all users; new modern and modular technology; change the way the land base is managed.

None of that has happened here.

Hunter Number

I took the app for a test drive (not so far as applying for a license, though maybe I should have) and what stuck out for me is:

  • The business process is basically “paper form on the web”. You still need a special “Hunter Number” to apply – the business process clearly hasn’t been transformed at all, nor integrated into a “one process” framework.
  • Technologically, if you peel back the web code and look underneath, the whole thing is being managed by a system called “POSSE”.

Why is it significant that POSSE is being used to manage this web form? Because POSSE is the system used by, wait for it… Front Counter BC! The same folks NRS cadged their portal content from.

So the “new” Limited Entry Hunt app has the same smell as the portal itself. Finding themselves unable to meet their stated goals of business transformation and new technology, NRPP is now building Potemkin deliverables using old business process and old technology.

Of course, having met one deliverable by giving up on “transformation” and just stuffing existing business process into web forms, what are the odds that NRPP will go on to do the same for the whole portfolio and then declare “victory”? Very high, very high indeed.

Legislation will be introduced in Spring 2016 to move selected Fish and Wildlife authorizations to a criteria based notification model

This was the only promise not tied to technology deliveries, and sadly it looks like it perished at the hands of a government too tired out to pass substantive legislation. I searched the Hansard for the spring 2016 session and did not find any evidence that the legislation was introduced.

Recap

On one side of the ledger:

On the other side of the ledger:

  • A “portal” nobody can find, full of content other people assembled.
  • A map nobody can find, full of content that has been accessible for a decade.
  • An app built on old technology using the same old business process.
  • Legislation that was not introduced.

Here are the things we cannot blame this on:

  • Stupid people
  • Bad intentions
  • Political shenanigans
  • Graft or corruption

Here are the things we can blame this on:

  • Excessive size and ambition of the project
  • Elevation of process over product

NRPP was/is a mistake. It’ll deliver something, in the end, but that something won’t be worth 10% of the money that is spent to achieve it. Hopefully NRPP is the last of the “transformation” projects to come out of government, and future business process improvement/integration efforts can evolve incrementally over time, at 10% of the cost and 10% of the risk.

Reporters in Politics

I frequently write things that people in the government IT community probably find hurtful. I do it deliberately, because I think the issues should be aired.

As I semi-jokingly tell my family, with reference to my former career as a government IT consultant: “I want to make sure I never work in this town again.”

At this point, I’m pretty sure that I have succeeded.

Reporters in Politics

Bill Tieleman has a very even-handed write-up on the troubling rush of journalists into the BC Liberal Party’s tent over the past few years.

In fact, Tieleman’s take is so even-handed it displays the same fault that he’s bending over backwards to not describe: he’s avoiding saying hurtful things in order to retain future relationships.

But the question isn’t whether Darling or Johal were biased toward the BC Liberals. It’s how much their years of hard-earned fairness will help their new party sell some dubious claims — even about issues the two journalists may have ripped them on in the past.

Tieleman is a communications expert with a professional interest in remaining a respected member of the club of communications experts, and he very carefully avoids impugning the motivations of any of the subjects of this article, while larding out praise for his subjects’ “hard-earned fairness”.

Similarly, journalists have a professional interest in maintaining good relationships with their fellow professionals, the communications and PR staff (and their bosses) in the organzations they report on.

Why would they not?

The exit ramps of journalism are few and far between: corporate PR, and government communications. For a very few well-known press celebrities, like Jas Johal or Steve Darling, perhaps a direct leap to politics (with a winning party, if you still want a pay-cheque, n’est pas).

In these tenuous times, it is perhaps too much to ask, but I’d like my journalism to come from folks who aren’t so chummy with the people they report on. And I don’t just mean “I’d like a job later” chummy. I also mean “we’re all just blokes doing a job together” chummy. The kind of chummy that leads to events like the Press Gallery dinners in Ottawa and the Correspondents Association Dinner in DC, where the press and their subjects mingle and share comedy stylings and even the most egregious policy decisions can be played for a joke (remember George W. Bush looking for WMDs in a “hilarious” bit from 2004).

I don’t think many journalists write the stories that will never let them work in this town again.

In fairness, not many of them can afford to. It’s probably not a coincidence that some of the harshest takes on BC government policy and politics come from journalists and professionals who are safely retired.

Precarious employment has a way of disciplining folks, no matter if they are blue collar mill workers or white collar journalists: you’ve got to go along to get along.

It used to be that free speech only belonged to those who owned a printing press. Now it belongs to those beyond the reach of the marketplace: are you secure enough not to care what anyone thinks of you? Write away. Otherwise: you know the party line, stick close to it.