Working with Shapefiles

This is an attempt to summarise the options available to the php coder, when working with Geospatial data. Bear in mind that what follows is the thinking-out-loud of a GIS newcomer, and thus a work in progress.

 

Light Approaches

 

Given that the Unix way of doing things is to try to use a sequence of light tools that do only one thing each, thats a good place to start. Oddly if you research this you will find startlingly few php librarys to work with geographic vector data. Heres what i found in 2010.

 

1. GEOCLASS

This is an abandoned project circa 2003, capable of extracting gis data from a database, importing e00 files and rendering to png. However e00 is an obsolete proprietary format, for which no open source tool can export to. It also suffers from an almost complete lack of documentation. This all limits it use, however it is stand alone class with no prerequisites.

[http://www.multimediamotz.de/GeoClass/]

[http://www.multimediamotz.de/GeoClass/manual/]

 

2. IMAGE-GIS

This is a PEAR class with similar capabilities and weaknesses to GeoClass, (ie:e00).The source code has some minimal documentation, and requires additional pear libs Cache_Lite Image_Color & XML_SVG. Abandoned 2004.

[http://pear.php.net/package/Image_GIS/]

 

3. JJFMAPPER

This is a standalone class capable of importing shapefiles, and rendering to png. It is well documented and relatively simple to setup. Doesnt support some modern shapefile variants. Unsure about projection support. Abandoned 2006.

[http://jjfmapper.sourceforge.net/]

 

4. SHAPEFILE

This class is a shapefile parser. However it requires dbase support to be enabled for PHP. Undated.

[http://www.phpclasses.org/browse/file/6786.html]

 

5. OPENSVGMAPSERVER

This package can fetch GIS data from a mysql database, and render a map in svg. They also provide a tool to export from arc explorer to mysql. Abandoned 2002.

[http://www.freegis.org/database/viewobj?obj=241]

 

So theres a theme emerging here, no one is trying to maintain GIS related code for PHP.

 

Perl / Command line Solutions

 

Looking into perl and general command line linux tools, we can see a slightly better situation. The linux programs GDAL/OGR/PROJ are actively developed, robust, fast and capable. They do a surprising amount of GIS heavy lifting for little learning curve. There are also a number of perl cpan libs that deal with geographic vector information.

 

First get the debian packages, we are talking debian lenny here.

aptitude install gdal-bin libgdal-perl proj librsvg2-2 libimage-librsvg-perl zip unzip

 

Add some cpan libs (always a bit tricky with debian, which has its own perl libs dependency system)

perl -MCPAN -e shell

install Geo::Point

install Geo::Proj4

install Geo::ShapeFile

install SVG

 

Then you can convert shapefiles to a range of other formats easily, eg:

//make a kml, and kmz

ogr2ogr -f KML myshape.kml myshape.shp -dsco

    NameField=RegionName

zip myshape.kmz myshape.kml

 

//make a gpx , but note that gpx cant handle polygons

ogr2ogr -f GPX   myshape.gpx myshape.shp -dsco

      GPX_USE_EXTENSIONS=YES

 

//make a mapinfo mif/mid file, a nice human parsable format

ogr2ogr -f `MapInfo File` myshape.mif myshape.shp -dsco

      FORMAT=MIF

 

1. SHPTOSVG

I found this recent perl script that uses those cpan libs to render a shapefile into svg. Requires:Geo::Point, Geo::ShapeFile ,SVG , & Geo::Proj4. Tested as working 2010, not sure about its memory mgmt.   You can use it in PHP via exec.   Has some projection support, but at least one of those cpan libs requires compilation (compilers and production server boxes dont mix).

[http://wiki.github.com/kbh3rd/shptosvg/]

 

eg: usage example

perl shptosvg.pl -x 600 -y 600

    myshape.shp,style=`stroke:#000000;stroke-width:1`

    > myshape.svg

 

2. SVGTOPNG

Looking around for something to do the final step to get a .png, theres   rsvg. Note that rsvg requires a non trivial set of gnome libs, and hence we are well and truly out of light country.

rsvg myshape.svg myshape.png

 

3. GDAL_RASTERIZE

Gdal has a fledgling utilty to make bitmaps of vector data. However lennys version (1.5) doenst support creating the image, so you have to create a blank image first. Also it doesnt support polylines. Problems. If you are willing to upset your package manager and upgrade to the latest version (1.8 at present) then both problems are solved, and you have a nice one package solution. Squeeze isnt that far away i guess. Edit: Squeeze will only ship with 1.6.

[http://www.gdal.org/gdal_rasterize.html]

 

  1. gdal1.5
  2. gdal_rasterize -b 1 -b 2 -b 3 -burn 255 -burn 0 -burn 0   -l test test.shp test.tif

    1. gdal1.8
    2. gdal_rasterize -b 1 -b 2 -b 3 -burn 255 -burn 0 -burn 0   -ts 800 800 -l test test.shp test.tif

 

 

POSTGRES

One thing i forgot to mention is that the postgresql database has quite advanced GIS handling capabilities in the form of its postgis addon. Mysql has only limited spatial awareness by comparison. These things become an issue if you actually want to import the vector data into a database, so that you can extract subsets, do calcs like path length, area, find paths near certain points etc.

[http://postgis.refractions.net/]

 

PROJECTIONS

It is probably also a good time to mention projections. Its important that you have at least a basic understanding of the way that ellipsoid surfaces like the earth are represented as flat maps. I found these useful:

  • [http://www.manifold.net/doc/projections_tutorial.htm]
  • [http://hosting.soonet.ca/eliris/remotesensing/bl130lec2.html]
  • [http://www.uwgb.edu/dutchs/fieldmethods/utmsystem.htm] (UTM)
  • [http://www.ollivier.co.nz/projection/faq.shtm] (NZ)
  • [http://www.utas.edu.au/spatial/locations/] (Tasmania)

 

The main thing is to be really clear about the projections and datums used in all your datasets both vector and raster. If you overlay a set of vector data over a raster base layer then both need to be in the same projection and datum.

  • Counter-intuitively latlong in one datum is different to latlong in another datum!
  • Data originating from most gps gear is latlong wgs84. But check.
  • Shapefiles from various sources may be in any projection and datum.
  • Vector data is trivial to reproject. Raster data can also be reprojected but its very slow and will distort the image.
  • Aerial imagery raster base layers come in variety of proj/datum.
  • Google maps uses mercator, not UTM, they are quite different things.
  • Orthophotos are most often in UTM or some local form of transverse mercator.
  • NZ is in the process of migrating from NZMG to NZTM. The former while complex was considerably more accurate, but the latter is better due to its global compatibility and simplicity. Time to move on. Australia has made the same transition recently as well.
  • Elsewhere, understand UTM regions, each one has its own origin/false origin.

 

When in doubt, for a good local scale map (up to a few hundred kms),   try to get raster layers in the prevailing local transverse mercator projection. Then convert your vectors to the same projection and datum. Then overlay them.

 

We could be content to call this good, but theres room for improvement. To avoid momst of those prerequisites we could do this:

1. using OGR convert our shapefile to mif format, in NZTM.

2. use GD to render to png.

  • Theres a tutorial that explains the basics of extracting from mif format and rendering to png using gd. [http://www.web-max.ca/PHP/]
  • Some extensions to be above at [http://www.easywms.com/easywms/?q=en/taxonomy/term/49].
  • See my attempt WIP [www.zoneblue.org/files/lib-shapeconvertor.zip]

 

Heavier approaches

 

So from what we have found, it seems we need to get out the big guns. GIS is a pretty involved field and my guess is that PHP programmers have simply left it to the experts. So i asked the experts, and they pointed me in the direction of the big two.

 

Theres also some pretty full lists of open source gis tools here:

[http://blends.alioth.debian.org/gis/tasks/workstation]

[http://download.osgeo.org/livedvd/4.0-rc7/osgeolive-docs/overview/overview.html]

[http://opensourcegis.org/]

[http://maptools.org/]

[http://freegis.org/database/?cat=55]

[http://www.misovic.net/2007/05/09/php_geoclasses/]

 

 

1. MAPSERVER

This is a well regarded package, that has been around since the 90s. It can pretty much be used to create fully featured gis web apps, complete with zoom, pan etc ui tools, ouptut to png and svg, tiling, and more. The thing to know is that there are two ways to run it, as a dedicated cgi ie completely seperate from php, or as mapscript which is like an API hook into the mapserver binarys from php. The latter is naturally going to be of much more interest to php coders.

[http://mapserver.org/]

[http://mapserver.org/utilities/shp2img.html]

 

//php api

aptitude install php5-mapscript

//cgi

aptitude install cgi-mapserver mapserver-bin

 

2. MAPNIK

Mapnik is the new kid on the block, its used for open street map among other things. It does much of what mapserver does, but mapnik has anti-aliasing, hence does nice raster renders. The API is pretty much all python, which is good if you work with python i suppose. The map tiles are efficiently rendered using an apache module called mod_tile. However for our present purposes (non live one-off png renders) theres a useful sounding command line utility called Nik2Img.

[http://mapnik.org/]

[http://code.google.com/p/mapnik-utils/wiki/Nik2Img]

[http://trac.mapnik.org/wiki/DebianInstallation]

[http://trac.mapnik.org/wiki/XMLGettingStarted]

[http://komunitasweb.com/2010/08/create-map-from-scratch-using-mapnik/]

 

For lenny you can follow the howto above to get the latest version or you can install an old version simply with :

aptitude install python-mapnik

 

These approaches are obviously going to be faster in cpu terms than a script based solution, especially if you want live renders. They are also going to be maintained, supported via user communitys, accurate and data robust etc. So even though they are overkill for our purpose they seem to be pretty much the way to go.

 


MAPNIK EVALUATION

To follow.

 

MAPSERVER EVALUATION

To follow.

Admin login