Friday, May 18, 2007

Mass Shape File Load into PostGIS

I needed some test data to do some performance investigations, and had to load 235 shape files, all of identical schema. Here's what I did.

First, get the table schema into the database, by loading a small file, and then deleting the data. We delete the data so we can loop through all the files later without worrying about duplicating the data from the initial file.

shp2pgsql -s 3005 -i -D lwssvict.shp lwss | psql mydatabase
psql -c "delete from lwss" mydatabase

Then use the shell to loop through all the shape files and append them into the table.

foreach f (*.shp)
foreach? shp2pgsql -s 3005 -i -D $f -a lwss | psql mydatabase
end

Note the "-a" switch to tell shp2pgsql we are in append mode, rather than the default create mode. Add a spatial index, and we're done.

psql -c "create index lwss_gix on lwss using gist (the_geom)" mydatabase

Seven hundred thousand line segments, ready to play!

psql -c "select count(*) from lwss" mydatabase
count
--------
755373
(1 row)

4 comments:

Dylan said...

Hi Paul,

Nice example. I used a somewhat similar approach to loading in a couple hundred NCSS digitial soil survey shapefiles into our online soil survey. I posted some of the notes here.

cheers,

JasonBirch said...

Hmm. 1:50k watershed atlas stream segments?

Did you load the whole province, or just a region or two?

;)

-J

Paul Ramsey said...

Whole province, Jason. Go big or go home. Took all of five minutes. And the performance test turned out great: PostgreSQL supports bitmap merging of index results for versions 8.1+, and that allows spatial and attribute indexes to be cleanly merged for fast AND conditions even when the individual indexes are not particularly selective on their own.

Clinton said...

Hi I have tried your script, see below and it keeps on giving errors: ERROR: syntax error at or near "01060000....
invalid command \N.
If I try the old shp2pgsql method, it inserts certain data, but the rest gives me the following error:

psql:cad_ec.sql:568: ERROR: current transaction is aborted, commands ignored until end of transaction block.

Please could you help.

About Me

My Photo
Paul Ramsey
Victoria, British Columbia, Canada
View my complete profile

Blog Archive