Build your own OpenStreetMap Server - Ubuntu 10.04 Lucid Lynx

Build your own OpenStreetMap server. Build your own what?

OpenStreetMap is the editable World map of everything. It is the Wikipedia of maps. It is to other on-line maps as Wikipedia is to Britannica. And it is awesome in every possible way.

This article updates an earlier version, archived here. This current version was written in March 2010.

The current version of this article is based on Ubuntu 10.04 Lucid Lynx. Lucid will be released on 29 April 2010 and be in beta until then. Operating systems in beta are not suitable for production use.

OpenStreetMap is a massive project that started as Steve Coast's, frame-breaking idea in 2004. If I make a map of my neighborhood and give it away, and you make a map of your neighborhood and give it away, then we both have better maps. As of March 2009 there are over 100,000 contributors making maps of their neighborhoods and contributing them to this World wide effort.

OpenStreetMap makes the data and the software available to you with Free Software and free data licenses so that you can use, learn from, teach with, improve upon and share with others what you gain from OpenStreetMap. And you can build your own local copy of OpenStreetMap for your business, school, community group or personal interests.

The project operates on a massive scale as there is an incredible amount of data, there is more data every day, and there are more people using the data every day. OpenStreetMap has to run on several servers, including a handful of API servers and separate database, development, web and tile servers. This article does not cover the creation of a complete OSM datacentre.

It does cover creating a single server from a typical PC circa 2006. These instructions build what OpenStreetMap calls a tile server. That is, a computer that uses the OSM data set to create map images that are suitable for a web site. Not every OpenStreetMap function is supported, but you will be able to create a local map, keep it up to date and customize it for your own purposes.

Why would I build my own?

Why indeed? OpenStreetMap.org is already freely available on the internet. Why not just use that? You can and you should. Eventually you may come up with an idea. You might want to make the map work a little differently for you. You might want a map for a special purpose.

Perhaps for cycling, http://www.opencyclemap.org/ OpenCycleMap is a wonderful example of what you can do with the tools and data of OpenStreetMap catalyzed by an idea. Created by Andy Allan and Dave Stubbs, OpenCycleMap uses OpenStreetMap data, then displays it in a way that is useful to cyclists with the emphasis placed on cycle trails, bike shops and bike parking. They've also added elevation contours and hill-shading as cyclists care about hills. Sometimes they are looking for a good challenging climb, and other times they just want to get home with the groceries. The brilliant work of the OpenCycleMap team was recognized with a Commendation from the British Cartographic Society as well as the prestigious Lolcat of Awesomness from the OpenStreetMap community at large.

Or maybe you need to have access to your map even when your internet provider is down. Or when the power is off. Or both. It won't take much for you to see the benefit of having your own piece of OpenStreetMap infrastructure. All you need to start is an idea and the thirst for knowledge.

There are a lot of moving parts to OpenStreetMap. I hope that these steps will make it easier for you to get your first map working. This article is intended to get from bare iron to a working local OpenStreetMap tile server. This one will collect OpenStreetMap data and allow you to render that data into images suitable for use on a web site. Future articles will cover how to use your new server and several customizations that you'll want to try.

Let's get started, shall we?

Prerequisites

Start with an installation of Ubuntu Lucid Lynx Server for your hardware architecture. Add the LAMP server and SSH server during installation. Once installed, this can be run as a headless server with neither monitor nor keyboard. So ssh to the new box, and let's get started.

Update operating system

Get Ubuntu updates. Yes, you just installed Ubuntu, but it has been getting security and bug fixes. Use this to bring your system up to data as of right now.
sudo apt-get update
sudo apt-get upgrade

Get some system tools

We'll need subversion to get the latest updates from OpenStreetMap and other places. Munin makes pretty pictures of activity on the server. I like screen. htop is neat-o.

sudo apt-get install subversion autoconf screen munin-node munin htop

Organize the file system a bit

cd ~
mkdir src bin planet

Get the latest planet - OpenStreetMap data file

A new planet file is published approximately each week. The mirror and archives of the planet files are here. In March 2010 the planet file was about 8.2GB in length. If you are not interested in the entire planet you can choose to download an extract file instead.

Facing an 8.2GB or larger download, this is an excellent time to consider using screen if you haven't used it before. Screen allows you to operate several terminal windows through one ssh connection. And more. Have a look this screen tutorial if you haven't used it before. Or wait for your download to complete, or use another ssh session to continue.

cd planet
wget http://planet.openstreetmap.org/planet-latest.osm.bz2

Prepare the postGIS database

Use the PostGIS extensions to postgresql for all sorts of geographical goodness. Install the postGIS and prerequisites.
sudo apt-get install postgresql-8.4-postgis postgresql-contrib-8.4
sudo apt-get install postgresql-server-dev-8.4
sudo apt-get install build-essential libxml2-dev
sudo apt-get install libgeos-dev libpq-dev libbz2-dev proj

Install osm2pgsql from the repository

The latest version of osm2pgsql has the most goodies, so we'll use that rather than a package.

cd ~/bin
svn co http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/
cd osm2pgsql
./autogen.sh
./configure
make


Configure the PostGIS database

edit /etc/postgresql/8.4/main/postgresql.conf in four places. These changes help with the large quantities of data that we are using.
shared_buffers = 128MB # 16384 for 8.1 and earlier
checkpoint_segments = 20
maintenance_work_mem = 256MB # 256000 for 8.1 and earlier
autovacuum = off

edit /etc/sysctl.conf
kernel.shmmax=268435456

Russ Nelson and balrog-k1n remind us that the above only takes effect after a reboot. Making this work immediately requires the following.
sudo sysctl kernel.shmmax=268435456

Restart postgres to enable the changes
sudo /etc/init.d/postgresql-8.4 restart

It should restart as above.

 * Restarting PostgreSQL 8.4 database server
   ...done.

Create a database called gis. Some of our future tools presume that you will use this database name. Substitute your username for username in two places below. This should be the username that will render maps with mapnik.
sudo -u postgres -i
createuser username # answer yes for superuser
createdb -E UTF8 -O username gis
createlang plpgsql gis
exit

Set up PostGIS on the postresql database.
psql -f /usr/share/postgresql/8.4/contrib/postgis.sql -d gis
This should respond with many lines ending with

...
CREATE FUNCTION
COMMIT

Substitute your username for username in two places in the next line. This should be the username that will render maps with mapnik.
echo "ALTER TABLE geometry_columns OWNER TO username; ALTER TABLE spatial_ref_sys OWNER TO username;" | psql -d gis
# Should reply with

ALTER TABLE
ALTER TABLE

Enable intarray
psql -f /usr/share/postgresql/8.4/contrib/_int.sql -d gis

Replies with many lines ending with

...
CREATE FUNCTION
CREATE OPERATOR CLASS

Set the Spatial Reference Identifier (SRID) on the new database.
psql -f ~/bin/osm2pgsql/900913.sql -d gis
Should reply with

INSERT 0 1

Load planet into the database with osm2pgsql

Your planet file will have a different date. Use an extract file if your interest is limited to a smaller portion of the planet. This operation will take 30 hours or longer. It is very I/O intensive and you can speed it up with very fast disks.
cd ~/bin/osm2pgsql
./osm2pgsql -S default.style --slim -d gis -C 2048 ~/planet/planet-100217.osm.bz2

Loading the planet file will take some time. How much time it will take depends primarily on the speed of your hard drive system and on the configuration of your system and your available memory. For more details on tuning your Mapnik stack for better performance, see the SotM 2010 session and follow up by Frederik Ramm of GEOFABRIK.

In the interim, let's have a look at your planet import. The first part of the osm2pgsql output looks scary, but is normal.

Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
NOTICE:  table "planet_osm_point" does not exist, skipping
NOTICE:  table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE:  table "planet_osm_line" does not exist, skipping
NOTICE:  table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE:  table "planet_osm_polygon" does not exist, skipping
NOTICE:  table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE:  table "planet_osm_roads" does not exist, skipping
NOTICE:  table "planet_osm_roads_tmp" does not exist, skipping
Mid: pgsql, scale=100, cache=4096MB, maxblocks=524289*8192
Setting up table: planet_osm_nodes
NOTICE:  table "planet_osm_nodes" does not exist, skipping
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_nodes_pkey" for table "planet_osm_nodes"
Setting up table: planet_osm_ways
NOTICE:  table "planet_osm_ways" does not exist, skipping
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_ways_pkey" for table "planet_osm_ways"
Setting up table: planet_osm_rels
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_rels_pkey" for table "planet_osm_rels"

Don't be concerned by the NOTICE: entries above. All normal.

Next, osm2pgsql will start reading the compressed planet file.

Reading in file: /home/nerd/planet/planet-100217.osm.bz2

As osm2pgsql reads the planet file it will give progress reports. The line below will refresh every few seconds and update the numbers in brackets. This part of the import takes a long time. Depending on your server, it will take between hours and days.

Processing: Node(10140k) Way(0k) Relation(0k)

As the import proceeds, the Node number will update a couple of times per second until complete, then the Way number will update not quite so quickly, roughly every second or two. Finally the Relation number will update but at a slower rate, roughly once per minute. As long as you can see these numbers advancing the import process is still operating normally for your server. Do not interrupt the import process unless you have decided to start over again from the beginning.

Processing: Node(593072k) Way(45376k) Relation(87k)
Exception caught processing way id=110802
Exception caught processing way id=110803
Processing: Node(593072k) Way(45376k) Relation(474k)

The exceptions shown above are due to minor errors in the planet file. The planet import is still proceeding normally.

The next stage of the osm2pgsql planet import process also will take between hours and days, depending on your hardware. It begins like this.

Node stats: total(593072533), max(696096737)
Way stats: total(45376969), max(55410575)
Relation stats: total(484528), max(555276)

Going over pending ways
processing way (752k)

The processing way number should update approximately each second.

Going over pending relations

node cache: stored: 515463899(86.91%), storage efficiency: 96.01%, hit rate: 85.97%
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Sorting data and creating indexes for planet_osm_line
Sorting data and creating indexes for planet_osm_roads
Sorting data and creating indexes for planet_osm_polygon
Committing transaction for planet_osm_point
Sorting data and creating indexes for planet_osm_point
Stopping table: planet_osm_nodes
Stopping table: planet_osm_ways
Stopping table: planet_osm_rels
Building index on table: planet_osm_rels
Stopped table: planet_osm_nodes
Building index on table: planet_osm_ways
Stopped table: planet_osm_rels
Completed planet_osm_point
Completed planet_osm_roads
Completed planet_osm_polygon
Completed planet_osm_line
Stopped table: planet_osm_ways

This should mean that you import is complete and successful.

Install Mapnik library

The Mapnik library is the first of two items sometimes called Mapnik. The other item is a collection of tools that OpenStreetMap uses to invoke Mapnik.

The official and up-to-date Mapnik Installation Instructions are here.

You might find that this procedure works as well.

Get some dependencies for building the Mapnik library.
sudo apt-get install libltdl3-dev libpng12-dev libtiff4-dev libicu-dev
sudo apt-get install libboost-python1.40-dev python-cairo-dev python-nose
sudo apt-get install libboost1.40-dev libboost-filesystem1.40-dev
sudo apt-get install libboost-iostreams1.40-dev libboost-regex1.40-dev libboost-thread1.40-dev
sudo apt-get install libboost-program-options1.40-dev libboost-python1.40-dev
sudo apt-get install libfreetype6-dev libcairo2-dev libcairomm-1.0-dev
sudo apt-get install libgeotiff-dev libtiff4 libtiff4-dev libtiffxx0c2
sudo apt-get install libsigc++-dev libsigc++0c2 libsigx-2.0-2 libsigx-2.0-dev
sudo apt-get install libgdal1-dev python-gdal
sudo apt-get install imagemagick

Build Mapnik library from source.

cd ~/src
svn co http://svn.mapnik.org/tags/release-0.7.1/ mapnik
cd mapnik
python scons/scons.py configure INPUT_PLUGINS=all OPTIMIZATION=3 SYSTEM_FONTS=/usr/share/fonts/truetype/
python scons/scons.py
sudo python scons/scons.py install
sudo ldconfig

Confirm that Mapnik library is installed.
python
>>> import mapnik
>>>
If python replies with the second chevron prompt >>> and without errors, then Mapnik library was found by Python. Congratulations.

Install Mapnik tools

The Mapnik tools are the second item sometimes called mapnik. This is a collection of tools from OpenStreetMap for making effective use of the Mapnik library.

cd ~/bin
svn co http://svn.openstreetmap.org/applications/rendering/mapnik


Install prepared world boundary data

Mapnik uses prepared files to generate coastlines and ocean for small scale maps. This is faster than reading the entire database to render zoom levels from zero to nine.

This section now includes the additional shape files that were added to OpenStreetMap default styles in mid-2010. Beware of the long, strange looking links with the repeated http. They are unlikely to copy / paste directly. Use copy link location or equivalent.

cd ~/bin/mapnik
mkdir world_boundaries
wget http://tile.openstreetmap.org/world_boundaries-spherical.tgz
tar xvzf world_boundaries-spherical.tgz
wget http://tile.openstreetmap.org/processed_p.tar.bz2
tar xvjf processed_p.tar.bz2 -C world_boundaries
wget http://tile.openstreetmap.org/shoreline_300.tar.bz2
tar xjf shoreline_300.tar.bz2 -C world_boundaries
wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/...
unzip 10m-populated-places.zip -d world_boundaries
wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/...
unzip 110m-admin-0-boundary-lines.zip -d world_boundaries


Render your first map

The database is loaded and the tools are installed. Let's test everything together. Remember to replace username with your username.

cd ~/bin/mapnik
./generate_xml.py --dbname gis --user username --accept-none
./generate_image.py


View image.png to confirm that you have rendered a map of England. Congratulations.

References and credits

Thank you to the developers who wrote all of these Free Software tools.
Thanks to:
All of the helpful folks on #osm on irc.oftc.net especially jburgess, Ldp and springmeyer
All of the helpful folks on the OSM mailing list http://lists.openstreetmap.org/
http://wiki.openstreetmap.org/wiki/Mapnik
http://mapnik.org/
Thanks dbaker and acant for copy editing help on the previous version of this article.
Thanks nelson, codebrainz, Ldp, balrog-k1n and kW for more updates on the previous version of this article.

Lynx photo CCBYND Tambako the Jaguar on Flickr.