Build your own OpenStreetMap Server

Build your own OpenStreetMap server. Build your own what?

OpenStreetMap is the editable World map of everything. It is the Wikipedia of maps. It is to other on-line maps as Wikipedia is to Britannica. And it is awesome in every possible way.

OpenStreetMap is a massive project that started as Steve Coast's, frame-breaking idea in 2004. If I make a map of my neighborhood and give it away, and you make a map of your neighborhood and give it away, then we both have better maps. As of March 2009 there are over 100,000 contributors making maps of their neighborhoods and contributing them to this World wide effort.

OpenStreetMap makes the data and the software available to you with Free Software and free data licenses so that you can use, learn from, teach with, improve upon and share with others what you gain from OpenStreetMap. And you can build your own local copy of OpenStreetMap for your business, school, community group or personal interests.

The project operates on a massive scale as there is an incredible amount of data, there is more data every day, and there are more people using the data every day. OpenStreetMap has to run on several servers, including a handful of API servers and separate database, development, web and tile servers. This article does not cover the creation of a complete OSM datacentre.

It does cover creating a single server from a typical PC circa 2006. These instructions build what OpenStreetMap calls a tile server. That is, a computer that uses the OSM data set to create map images that are suitable for a web site. Not every OpenStreetMap function is supported, but you will be able to create a local map, keep it up to date and customize it for your own purposes.

The key compromise that is required to be able to build this system on a non-super-computer is that we'll only build the map for a small part of the world.

Why would I build my own?

Why indeed? OpenStreetMap.org is already freely available on the internet. Why not just use that? You can and you should. Eventually you may come up with an idea. You might want to make the map work a little differently for you. You might want a map for a special purpose.

Perhaps for cycling, http://www.opencyclemap.org/ OpenCycleMap is a wonderful example of what you can do with the tools and data of OpenStreetMap catalyzed by an idea. Created by Andy Allen and Dave Stubbs, OpenCycleMap uses OpenStreetMap data, then displays it in a way that is useful to cyclists with the emphasis placed on cycle trails, bike shops and bike parking. They've also added elevation contours and hill-shading as cyclists care about hills. Sometimes they are looking for a good challenging climb, and other times they just want to get home with the groceries. The brilliant work of the OpenCycleMap team was recognized with a Commendation from the British Cartographic Society as well as the prestigious Lolcat of Awesomness from the OpenStreetMap community at large.

Or maybe you need to have access to you map even when your internet provider is down. Or when the power is off. Or both. It won't take much for you to see the benefit of having your own piece of OpenStreetMap infrastructure. All you need to start is an idea and the thirst for knowledge.

There are a lot of moving parts to OpenStreetMap. I hope that these steps will make it easier for you to get your first map working. This article is intended to get from bare iron to a working local OpenStreetMap tile server. This one will collect OpenStreetMap data and allow you to render that data into images suitable for use on a web site. Future articles will cover how to use your new server and several customizations that you'll want to try.

Let's get started, shall we?

Install Ubuntu server

I used Hardy Heron 8.04 LTS for this tutorial. Choose the LAMP server and the ssh server.

This Ubuntu Hardy LAMP installation tutorial is a great start for somebody who hasn't installed one before.

Browse to it and get "It Works!" from apache to know that you've got it right.

codebrainz revised this article for Ubuntu 9.04 Jaunty. It can be found at http://codebrainz.ca/index.php/2009/09/27/installing-mapnik-and-friends-on-ubuntu-904/

Add some general tools

sudo apt-get install subversion unzip

Get the Data

Start to download your planet file or extract. It's very big. Plan to use another console or screen to continue while downloading. Download from one of the mirrors like this one, for a faster download.
wget http://ftp.heanet.ie/mirrors/openstreetmap.org/planet-latest.osm.bz2
Get back to the download tomorrow. We've got things to do until then.

Install the Database

A Word About username

As a simplifying assumption, this article will have you install all of the tools to work with the same user. This user is the same user for the Ubuntu login.

Set up the database to hold your copy of the OSM data. We'll use postgresql with PostGIS extensions to make the geographic stuff easier.

First we install the dependencies.
sudo apt-get install postgresql-8.3-postgis postgresql-contrib-8.3

It should respond like this

Configuring postgresql.conf to use port 5432...
 * Starting PostgreSQL 8.3 database server
   ...done.
Setting up postgresql-8.3-postgis (1.3.3-1) ...

edit /etc/postgresql/8.3/main/postgresql.conf in four places. These changes help with the large quantities of data that we are using.
shared_buffers = 128MB # 16384 for 8.1 and earlier
checkpoint_segments = 20
maintenance_work_mem = 256MB # 256000 for 8.1 and earlier
autovacuum = off

edit /etc/sysctl.conf
kernel.shmmax=268435456

Russ Nelson and balrog-k1n remind us that the above only takes effect after a reboot. Making this work immediately requires the following.
sudo sysctl kernel.shmmax=268435456

Disable the two pg_maintenance tasks which would otherwise cause vacuums to interrupt our database loads. We don't want loading the database to get interrupted.
crontab /etc/cron.d/postgresql-common
Add a # before each line to disable it.
Reenable them if you're doing anything else with postgres, or performance will slowly degrade.

Restart postgres to enable the changes
sudo /etc/init.d/postgresql-8.3 restart

It should restart as above.

Configuring postgresql.conf to use port 5432...
 * Starting PostgreSQL 8.3 database server
   ...done.
Setting up postgresql-8.3-postgis (1.3.3-1) ...

create the database

Substitute your username for username in two places below. This should be the username that will render maps with mapnik.
sudo -u postgres -i
createuser username # answer yes for superuser
createdb -E UTF8 -O username gis
createlang plpgsql gis
exit

PostGIS

Set up PostGIS on the postresql database.
psql -d gis -f /usr/share/postgresql-8.3-postgis/lwpostgis.sql
This should respond with many lines ending with

...
CREATE FUNCTION
COMMIT

Substitute your username for username in two places in the next line. This should be the username that will render maps with mapnik.
echo "ALTER TABLE geometry_columns OWNER TO username; ALTER TABLE spatial_ref_sys OWNER TO username;" | psql -d gis
# Should reply with

ALTER TABLE
ALTER TABLE

Enable intarray
psql gis < /usr/share/postgresql/8.3/contrib/_int.sql

Replies with many lines ending with

CREATE FUNCTION
CREATE OPERATOR CLASS

Mapnik from source

Use Mapnik for rendering pretty maps. We'll use both the mapnik.org project and some OpenStreetMap.org mapnik-tools and we'll call them both Mapnik at various times depending on context. Sorry about that. This section deals with the Mapnik.org project.

Install some dependencies.

sudo apt-get install libltdl3-dev libpng12-dev libtiff4-dev libicu-dev
sudo apt-get install libboost-regex-dev libboost-iostreams-dev
sudo apt-get install libboost-filesystem-dev libboost-thread-dev
sudo apt-get install libboost-python1.34.1 libboost-python-dev
sudo apt-get install libfreetype6-dev libcairo2-dev libcairomm-1.0-dev
sudo apt-get install libboost-program-options-dev python-cairo-dev
sudo apt-get install libboost-serialization-dev
sudo apt-get install imagemagick

Get some more dependencies
sudo apt-get install build-essential libxml2-dev libgeos-dev libpq-dev libbz2-dev proj

proj above is called libproj-devel on OpenSuse - reported by kW on #osm

To avoid confusion, we'll put the Mapnik.org project software in ~/src/mapnik, also known as /home/username/src/mapnik

Get the source from svn

mkdir src
cd src
svn co http://svn.mapnik.org/trunk/ mapnik
cd mapnik

Configure mapnik

Note, Updated 14 Aug 2009

Are you having trouble with ImportError: No module named mapnik after upgrading to Jaunty? Here is a fix for ImportError: No module named mapnik

python scons/scons.py

Please note that the optional boost system is not included in Boost 1.34 so the Boost System warning will persist.

Install mapnik
sudo python scons/scons.py install

Let python know about mapnik
sudo ldconfig

Install osm2pgsql

Get the latest osm2pgsql from svn.
svn co http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/
Should reply with several lines and end with

Checked out revision 14116.

Your revision number will vary.

Build osm2pgsql
cd osm2pgsql
make

This should return without warnings or errors.

Set the Spatial Reference Identifier (SRID) on the new database.
psql -f 900913.sql -d gis
Should reply with

INSERT 0 1

OpenStreetMap Mapnik-tools

This is the other Mapnik. That is to say that these are the tools that OpenStreetMap uses to use Mapnik effectively. These will be kept in ~/mapnik also known as /home/username/mapnik.

Get the latest mapnik from svn.
cd ~
svn co http://svn.openstreetmap.org/applications/rendering/mapnik/

Let's make some oceans

Stephen Wright once said, It's a small World but I wouldn't want to paint it. I feel the same about the outlines of the continents. I wouldn't want to have to walk along each of the edges of the Earth with a GPS. OpenStreetMap uses three different sources for this coastline information. Each is freely licensed for use with OpenStreetMap. Each is stored in a different compressed file format. So we get to use a couple of different tools.

The coastline information needs to be in the mapnik directory by default. You can move it elsewhere but you'll have to fix some other settings.
cd mapnik

World boundaries
wget http://tile.openstreetmap.org/world_boundaries-spherical.tgz
tar zxvf world_boundaries-spherical.tgz
That created the world_boundaries directory that we'll use for the rest of this coastline data.

Coastlines from OpenStreetMap
wget http://hypercube.telascience.org/~kleptog/processed_p.zip

Alternate locations of processed_p, for when hypercube is slow/down
http://www.las.ic.unicamp.br/pub/gentoo/distfiles/processed_p-20090514.z...
http://beta.letuffe.org/ressources/
http://tile.openstreetmap.org/processed_p.tar.bz2

unzip processed_p.zip
mv coastlines/* world_boundaries/
rmdir coastlines
That created a new coastlines directory that we won't use. We've moved the data to the world_boundaries directory.

Shorelines for low-zoom layers
cd world_boundaries
wget http://tile.openstreetmap.org/shoreline_300.tar.bz2
tar xvjf shoreline_300.tar.bz
This one didn't create a directory at all so we unpacked it directly into world_boundaries

Testing and Your First Map

Part two of this article is expected in late March. And here it is, Make Your First Map with your OpenStreetMap tile server.

This article is one of a series that involve customizing OpenStreetMap. Please use comments below to request additional OpenStreetMap topics or to offer feedback.

References / Credits

All of the helpful folks on #osm on irc.oftc.net especially jburgess and springmeyer
All of the helpful folks on the OSM mailing list http://lists.openstreetmap.org/
http://wiki.openstreetmap.org/wiki/Mapnik
http://mapnik.org/
http://www.kelvinism.com/tech-blog/installing-mapnik-ubuntu-710/
http://www.ubuntugeek.com/ubuntu-804-hardy-heron-lamp-server-setup.html
Thanks dbaker and acant for copy editing help.
Thanks nelson, codebrainz, Ldp, balrog-k1n and kW for more updates.