The Geography of Manufacturing

It’s a shame I’m not a geographer, because there are probably some interesting sociological nuggets hiding in the way manufacturing has dispersed across the USA over the past 40 years. This weakly reasoned article in Salon includes the point that auto manufacturers are locating their US plants in the South. Why so?

Union avoidance probably has something to do with it. But the price efficiency of a dependent labour force has to balanced against the inefficiency of being separated from a concentrated nexus of talent. Jane Jacobs, in the “Economy of Cities”, talks about the self-reinforcing nature of the skills networks that develop in cities, and you don’t have to look far to see them in action. Silicon Valley, Hollywood, the Research Triangle, etc, etc.

Oddly, the USA has some world class geographic nexuses of intellectual talent, while the manufacturing base has dispersed. Intellectual work is highly portable (I’m writing in a coffee shop, linked into a client’s VPN, and compiling a library on their machine) and virtual. Manufacturing work is heavy, with physical objects to move around. Why aren’t all the auto parts manufacturers, secondary assembly plants and final auto assembly still located in Michigan?

Clearly the cost of moving things around has fallen so much in recent years that factory sites can be chosen on other bases (prevailing wages, tax holidays) instead. I would also guess that the automation of parts of the assembly line has lowered the required skill specialization of line workers, so that having access to a pool of qualified talent in a factory hub has become less important too.

I wonder if increases in the cost of transport will cause some manufacturing to geographically re-integrate, so Boeing (for example) will stop shipping wings around the world, and their parts suppliers will find it more economical to set up satellite production closer to final assembly in Seattle.

Maybe if I beg, FortiusOne will dig up some cool maps on this topic. Please? Pretty please?

Valgrinding PostGIS

UPDATE 2021/09/07: So, notwithstanding the below, running a PostgreSQL backend through valgrind will only very rarely tell you anything useful. The only reason I found it useful, back in 2008, is that memory errors I was tracking down were generated by GEOS, which allocates using the system memory allocator.

PostGIS/PostgreSQL on the other hand, allocates using a “heirarchical memory manager” inside PgSQL, which is why the code is full of allocations using palloc() and not malloc(). The PgSQL memory manager allocates big “memory context” blocks, and then when cleaning up after a query or other big piece of work, drops the whole context at once. This results in all sorts of good things for PgSQL: memory locality, reduction in system calls to things like free(), faster allocations of small objects.

However, since only the allocation of the context and the freeing of the context use system memory calls, valgrind has no visibility into the small allocations your application code might be making. It only sees these big allocations / frees, and will never see the specific allocations / frees in your application code.

Clever developers might be thinking: “I know, I’ll just over-ride palloc() and pfree() in the PgSQL code, so that all allocations go via the system, and then valgrind will have to pick them up”. Except, because of the memory context system, lots of code doesn’t bother to pfree() all allocations, since the system will just free the whole context at the end.

As a result, if you try to push all allocations to malloc() and free() and then run valgrind, valgrind will say that PgSQL leaks like a sieve, and any mistakes you might want to find in your application code will be drowned out in the avalanche of un-freed memory valgrind is now finding.

So, you want to be a super-hero? How about tracking down memory leaks and otherwise using valgrind to make PostGIS a cleaner and better system? However, getting PostGIS into valgrind can be a bit tricky.

First of all, what is valgrind? It’s a tool for finding memory leaks and other memory issues in C/C++ code. It only runs under Linux, so you do need to have sufficiently portable code to run it there. Many memory checking tools rely on “static code analysis”, basically looks at what your code says it does and seeing if you have made any mistakes.

These kinds of tools have to be very clever, since they not only need to understand the language, they have to understand the structure of your code. Valgrind takes the opposite tack – rather than inspect your code for what it says it does, it runs your code inside an emulator, and sees what it actually does. Running inside valgrind, every memory allocation and deallocation can be tracked and associated with a particular code block, making valgrind a very effective memory debugger.

In order to get the most useful reports, you have to compile your code with minimal optimization flags, and with debugging turned on. To grind both GEOS and PostGIS simultaneously, compile GEOS and PostGIS with the correct flags:

# Make GEOS:
CXXFLAGS="-g -O0" ./configure
make clean
make install

# Make PostGIS:
CFLAGS="-g -O0" ./configure --with-pgconfig=/usr/local/pgsql/bin/pg_config
make clean
make install

Once you have your debug-enabled code in place, you are ready to run valgrind. Here things get interesting! Usually, PostgreSQL is run in multi-user mode, with each back-end process spawned automatically as connections come in. But, in order to use valgrind, we have to run our process inside the valgrind emulator. How to do this?

Fortunately, PostgreSQL supports a “single user mode”. Shut down your database instance (pg_ctl -D /path/to/database stop) first. Then invoke a postgres backend in single-user mode:

echo "select * from a, b where st_intersects(a.the_geom, b.the_geom)" | \
  valgrind \
    --leak-check=yes \
    --log-file=valgrindlog \
    /usr/local/pgsql/bin/postgres --single -D /usr/local/pgdata postgis

So, here we are echoing the SQL statement we want tested, and piping it into a valgrind-wrapped instance of single-user PostgreSQL. Everything will run much slower than usual, and valgrind will output a report to the valgrindlog file detailing where memory blocks are orphaned by the code.

Google SLA Quest

I host my email using Google Apps, and I even go so far as to pay the $50/year for “Premier” service to have no ads on my account. Having self-hosted infrastructure for 10 years at Refractions (I was President, CEO, and sysadmin for too many years), it has been pretty pleasant to have zero maintenance issues keeping my email online, my spam well filtered, my web pages up.

However, he who lives by the cloud, dies by the cloud.

Yesterday, my web mail access was unavailable for about one hour, exceeding the 99.9% availability per month guaranteed in the Google Apps SLA. For those doing the math, 99.9% availability means that 45 minutes per month of downtime is considered acceptable.

So, I am going to try and get my account credit of three free days of service (retail value, 3 * $50 / 365 = $0.41) from Google. (Does anyone else think that’s a pitiful SLA credit? If they go below 95% uptime (37 hours of downtime) they will credit you a whole 15 days of service – $2.05.)

Step one, who the heck do I contact? Unfortunately, the SLA is completely silent on the matter. Presumably someone in “support”, so I go hunting through the support pages. There we find a reference to 24x7 support for Premier customers (my $50 in action!). And at the bottom of that page, the answer:

You can submit a priority email request as follows:

  • Log in to your Google Apps control panel.
  • Click on ‘Domain Settings,’ then on ‘Account Information.’
  • Click ‘Contact us online.’

So, I submit my request for account credit, and wait. I get an auto-response right away saying they have created a ticket. So the clock is running on Google support, started at 4:30pm, Monday August 11. How long to a response? Updates will follow.

Update (Aug 12, 11:40pm): Received a human response from the Google collective. No answer on the SLA credit yet.

Hello Paul,

Thanks for your message.

I’ve reviewed your report and have forwarded the information you provided to the appropriate team for additional investigation. I appreciate your patience and apologize for any inconvenience this may have caused.


The Google Apps Team

Update (Aug 15, 5:46pm): More signs of life at the GooglePlex:

Hello Paul,

Thanks for your response.

Our billing team is currently reviewing this case. We will let you know when we’ve processed your service credit request.

You can continue to contact the support team, who can advocate on behalf of your domain in situations such as these.

Thank you for your continued support and patience.


The Google Apps Team

Update (Aug 27): The big G has decided that the August down-times warrant a big Mea Culpa. Clearly committed to keeping faith in the cloud high, they are extending a larger credit that strictly required to all us Premiere customers (not just whiners like me):

Given the production incidents that occurred in August, we’ll be extending the full SLA credit to all Google Apps Premier customers for the month of August, which represents a 15-day extension of your service. SLA credits will be applied to the new service term for accounts with a renewal order pending. This credit will be applied to your account automatically so there’s no action needed on your part.

Top marks to the big G, way to not be Evil!

Optimize, Optimize, Optimize!

I have been trying to squeeze every last erg of performance into a Mapserver installation for a client. Their workload is very raster-based, pulling from JPEG-compressed, internally-tiled TIFF files. We found one installation mistake, which was good for a big performance boost (do not fill your directories with too many files, or the operating system will spend all its time doing file searching). But once that was identified, only the raster algorithms themselves were left to be optimized.

Firing up the Shark, my new favorite tool, I found that the cycles were being spent about 60% in the JPEG decompression and 30% in the Mapserver average re-sampling. Decompressing the files did make things faster, but at a 10x file size penalty – unacceptable. Removing the average re-sampling did make things faster, but with very visible aesthetic penalty – unacceptable.

Using two tools from Intel, I have been able to cut about 40% off of the render time, without changing any code at all!

First, Intel publishes the “Integrated Performance Primitives “, basically a big catch-all library of multi-media and mathematics routines, built to take maximum advantage of the extended instruction sets in modern CPUs. Things like SSE, SSE2, SSE3, SSE4 and so on.

Ordinarily, this would only be of use if I had a couple weeks to examine the code and fit the primitives into the existing code base. Fortunately for me, one of the example programs Intel ships with IPP is “ijg”, a port of the Independent JPEG Group reference library that uses IPP acceleration wherever possible. So I could simply compile the “ijg” library and slip it into place instead of the standard “libjpeg”.

Second, Intel also publishes their own C/C++ compiler, “icc”, which uses a deep understanding of the x86 architecture and the extensions referenced above to produced optimized code. I used “icc” to compile both Mapserver and GDAL, and saw performance improve noticeably.

So, amazingly, between simply re-compiling the software with “icc” and using the “ijg” replacement library to speed up the JPEG decompression, I’ve managed to extract about 40% in performance, without touching the code of Mapserver, GDAL, or LibJPEG at all.

Strong Like Ox!

I’ve put my Mapserver slides online from my recent gig at GeoWeb. We’ll be in town until Friday! Try the veal!