22 Sep 2008
I wrote previously about how using the Intel “Integrated Performance Primitives” (IPP) and the Intel compiler (ICC) helped to improve performance in a Mapserver raster rendering system by about 40%.
I was feeling pretty proud, until the client pointed out that the outputs, while fast were, um, sort of wrong:
What was going on here?!
Well, the problem looked to be in the JPEG writing, and Mapserver writes its JPEG via libgd by default, so I looked at the JPEG code in libgd for clues. There was little to find. LibJPEG is provides a very simple interface to writing out images, and the code in GD (gd_jpeg.c) was identical to the sample image compression code in the Intel IJG code base (cjpeg.c).
I only had two clues to go on: first, the artefacts would go away, if we created the JPEGs in a “progressive” (interlaced) mode; second, the Intel cjpeg utility could create correct JPEGs in non-interlaced mode.
After taking a brain-break of a couple days, I found the problem while grepping the code and watching TV. Setting up a JPEG compressor requires that you provide the compressor with a buffer-fill function of your own design, so you can provide input from all kinds of sources. The implementation of the buffer-fill routine in the Intel example code had a switch in it, that checked if the mode was progressive or not!
In the case of non-progressive JPEG, the Intel function would allow the compressor to only partially read the buffer if it wanted. Then it would move the un-read bytes to the front of the buffer, and fill in the rest with new bytes. The standard JPEG library expects the compressor to always read the full buffer.
Patching GD to use this partial buffer logic, and JPEG compression worked, I was getting what looked like good ouputs! So, I put the patch in place and told the client and, um, there were still problems:
Gah! The artefacts were smaller, but they were still there. However, having my previous solution in hand, it was easy to find this bug. The artefacts were in a nice blocky patterns, just like the internal TIFF tile structure, which pointed to the JPEG decompressor having the same buffer behavior as the compressor.
So this time I opened up the GDAL library (that Mapserver uses to read the TIFFs) and found the JPEG reading code in the TIFF driver. Sure enough, the Intel decompressor only partially read the buffer, while the standard decompressor was expected to read the whole thing every time.
A small patch to the GDAL TIFF driver, and another patch to the GDAL JPEG driver (for completeness) and now we finally had IPP-accellerated JPEG input and output without artefacts.
Moral of the story: Although the Intel IJG is API-compatible with the standard libjpeg, the slightly different expected behavior of the empty_output_buffer and fill_input_buffer functions means that most existing code will require slight modifications before it can use the Intel libraries – they cannot just be dropped into place as a binary-only update.
06 Sep 2008
I haven’t seen anyone explicitly say this yet, so I will do so:
Google Chrome is not a web browser, it just happens to be shaped like one. Chrome is Google’s entry into the web application platform wars, currently being waged by Adobe (FLEX) and Microsoft (Silverlight). The horse Google is backing is the (oft maligned) one currently in the lead, DOM/Javascript, Chrome is an attempt to ensure that DOM/Javascript doesn’t lose that lead for reasons of speed or stability.
Chrome + Gears can deliver stand-alone and web-enabled apps. Next out of the gate from Google, I expect some debugging / IDE tools to support DOM/Javascript development in Chrome, ala Firebug+.
04 Sep 2008
I “just got back” (actually, last week) from a week at the Alders Beach Resort, a lovely place about 200km north up Vancouver Island. The Alders isn’t so much a “resort” in the conventional sense (well, in the conventional sense of 50 years ago), but rather a collection of mildly run-down cabins, full of families and kids, located on a tidal beach, tucked into the woods, comfortably far from pretty much anything stressful.
I had lots of time on my hands, and one can only watch children play with sand for so long, before the mood strikes, so what I did with my holiday was try to figure out how to build sand sculptures.
The Alders has been around for 50 years, and there were people at the resort last week who had been going to 15 years straight, whose kids grew up there. It is going to be a family holiday staple for us, for many years to come.
22 Aug 2008
From Andrew Turner, the Urban Spoon spatial-temporal map of restaurant seeking. Among the insights you can glean from this data presentation:
- People look for restaurants in cities full of people, not in the wilderness.
- People don’t look for restaurants when they are asleep.
Perhaps a zoomed in view of San Francisco would be more interesting.
21 Aug 2008
It’s a shame I’m not a geographer, because there are probably some interesting sociological nuggets hiding in the way manufacturing has dispersed across the USA over the past 40 years. This weakly reasoned article in Salon includes the point that auto manufacturers are locating their US plants in the South. Why so?
Union avoidance probably has something to do with it. But the price efficiency of a dependent labour force has to balanced against the inefficiency of being separated from a concentrated nexus of talent. Jane Jacobs, in the “Economy of Cities”, talks about the self-reinforcing nature of the skills networks that develop in cities, and you don’t have to look far to see them in action. Silicon Valley, Hollywood, the Research Triangle, etc, etc.
Oddly, the USA has some world class geographic nexuses of intellectual talent, while the manufacturing base has dispersed. Intellectual work is highly portable (I’m writing in a coffee shop, linked into a client’s VPN, and compiling a library on their machine) and virtual. Manufacturing work is heavy, with physical objects to move around. Why aren’t all the auto parts manufacturers, secondary assembly plants and final auto assembly still located in Michigan?
Clearly the cost of moving things around has fallen so much in recent years that factory sites can be chosen on other bases (prevailing wages, tax holidays) instead. I would also guess that the automation of parts of the assembly line has lowered the required skill specialization of line workers, so that having access to a pool of qualified talent in a factory hub has become less important too.
I wonder if increases in the cost of transport will cause some manufacturing to geographically re-integrate, so Boeing (for example) will stop shipping wings around the world, and their parts suppliers will find it more economical to set up satellite production closer to final assembly in Seattle.
Maybe if I beg, FortiusOne will dig up some cool maps on this topic. Please? Pretty please?
