Build your own OpenStreetMap server. Build your own what?
OpenStreetMap is the editable World map of everything. It is the Wikipedia of maps. It is to other on-line maps as Wikipedia is to Britannica. And it is awesome in every possible way.
This article updates an earlier version, archived here. This current version was written in March 2010.
The current version of this article is based on Ubuntu 10.04 Lucid Lynx. Lucid will be released on 29 April 2010 and be in beta until then. Operating systems in beta are not suitable for production use.
OpenStreetMap is a massive project that started as Steve Coast's, frame-breaking idea in 2004.
If I make a map of my neighborhood and give it away, and you make a map of your neighborhood and give it away, then we both have better maps. As of March 2009 there are over 100,000 contributors making maps of
their neighborhoods and contributing them to this World wide effort.
OpenStreetMap makes the data and the software available to you with Free Software and free data licenses so that you can use, learn from, teach with, improve upon and share with others what you gain from OpenStreetMap. And you can build your own local copy of OpenStreetMap for your business, school, community group or personal interests.
The project operates on a massive scale as there is an incredible amount of data, there is more data every day, and there are more people using the data every day. OpenStreetMap has to run on several servers, including a handful of API servers and separate database, development, web and tile servers. This article does not cover the creation of a complete OSM datacentre.
It does cover creating a single
server from a typical PC circa 2006. These instructions build what OpenStreetMap calls a
tile server. That is, a computer that uses the OSM data set to create map images that are suitable for a web site. Not every OpenStreetMap function is supported, but you will be able to create a local map, keep it up to date and customize it for your own purposes.
Why would I build my own?
Why indeed? OpenStreetMap.org is already freely available on the internet. Why not just use that? You can and you should. Eventually you may come up with an idea. You might want to make the map work a little differently for you. You might want a map for a special purpose.
Perhaps for cycling, http://www.opencyclemap.org/ OpenCycleMap is a wonderful example of what you can do with the tools and data of OpenStreetMap catalyzed by an idea. Created by Andy Allan and Dave Stubbs, OpenCycleMap uses OpenStreetMap data, then displays it in a way that is useful to cyclists with the emphasis placed on cycle trails, bike shops and bike parking. They've also added elevation contours and hill-shading as cyclists care about hills. Sometimes they are looking for a good challenging climb, and other times they just want to get home with the groceries. The brilliant work of the OpenCycleMap team was recognized with a Commendation from the British Cartographic Society as well as the prestigious Lolcat of Awesomness from the OpenStreetMap community at large.
Or maybe you need to have access to your map even when your internet provider is down. Or when the power is off. Or both. It won't take much for you to see the benefit of having your own piece of OpenStreetMap infrastructure. All you need to start is an idea and the thirst for knowledge.
There are a lot of moving parts to OpenStreetMap. I hope that these steps will make it easier for you to get your first map working. This article is intended to get from bare iron to a working local OpenStreetMap tile server. This one will collect OpenStreetMap data and allow you to render that data into images suitable for use on a web site. Future articles will cover how to use your new server and several customizations that you'll want to try.
Let's get started, shall we?
Minimum hardware requirements for your tile server are modest by modern computer standards. You may find that you can run your tile server from a ~2005 desktop computer with more than 1GB of memory. But that is not recommended. The operations on large files and large databases suggest that you provide as much RAM as you can. Some OpenStreetMap developers use machines with 32GB or more of RAM. Many import and update operations will be disk-bound, so get the fastest disks possible for production use, and be aware that testing on typical disks will be substantially slower.
- Bare minimum hardware requirements
- RAM: 1 GB
- Disk: 250 GB
- Recommended minimum hardware requirements
- RAM: 4 GB
- Disk: 1 TB
- potential production hardware requirements
- RAM: 64 GB
- Disk: Many TB of 15,000 RPM RAID
Start with an installation of Ubuntu (10.04) Lucid Lynx Server for your hardware architecture. Add the LAMP server and SSH server during installation. Once installed, this can be run as a headless server with neither monitor nor keyboard. So ssh to the new box, and let's get started.
Update operating system
Get Ubuntu updates. Yes, you just installed Ubuntu, but it has been getting security and bug fixes. Use this to bring your system up to data as of right now.
sudo apt-get update
sudo apt-get upgrade
Get some system tools
We'll need subversion to get the latest updates from OpenStreetMap and other places. Munin makes pretty pictures of activity on the server. I like screen. htop is neat-o.
sudo apt-get install subversion autoconf screen munin-node munin htop
Organize the file system a bit
mkdir src bin planet
Get the latest
planet - OpenStreetMap data file
planet file is published approximately each week. The mirror and archives of the
planet files are here. In March 2010 the planet file was about 8.2GB in length. If you are not interested in the entire planet you can choose to download an extract file instead.
Facing an 8.2GB or larger download, this is an excellent time to consider using screen if you haven't used it before. Screen allows you to operate several terminal windows through one ssh connection. And more. Have a look this screen tutorial [Link removed.] if you haven't used it before. Or wait for your download to complete, or use another ssh session to continue.
Prepare the postGIS database
Use the PostGIS extensions to postgresql for all sorts of geographical goodness. Install the postGIS and prerequisites.
sudo apt-get install postgresql-8.4-postgis postgresql-contrib-8.4
sudo apt-get install postgresql-server-dev-8.4
sudo apt-get install build-essential libxml2-dev libtool
sudo apt-get install libgeos-dev libpq-dev libbz2-dev proj
Install osm2pgsql from the repository
The latest version of osm2pgsql has the most goodies, so we'll use that rather than a package.
svn co http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/
Configure the PostGIS database
edit /etc/postgresql/8.4/main/postgresql.conf in four places. These changes help with the large quantities of data that we are using.
shared_buffers = 128MB # 16384 for 8.1 and earlier
checkpoint_segments = 20
maintenance_work_mem = 256MB # 256000 for 8.1 and earlier
autovacuum = off
Edit kernel parameter shmmax to increase maximum size of shared memory.
sudo sysctl -w kernel.shmmax=268435456
sudo sysctl -p /etc/sysctl.conf
Restart postgres to enable the changes
sudo /etc/init.d/postgresql-8.4 restart
It should restart as above.
* Restarting PostgreSQL 8.4 database server ...done.
Create a database called
gis. Some of our future tools presume that you will use this database name. Substitute your username for
username in two places below. This should be the username that will render maps with mapnik.
sudo -u postgres -i
createuser username # answer yes for superuser
createdb -E UTF8 -O username gis
createlang plpgsql gis
Set up PostGIS on the postresql database.
psql -f /usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql -d gis
This should respond with many lines ending with
... CREATE FUNCTION COMMIT ... DROP FUNCTION
Note: recent change requiring a new path to postgis.sql, is now reflected above.
Substitute your username for
username in two places in the next line. This should be the username that will render maps with mapnik.
echo "ALTER TABLE geometry_columns OWNER TO username; ALTER TABLE spatial_ref_sys OWNER TO username;" | psql -d gis
# Should reply with
ALTER TABLE ALTER TABLE
Set the Spatial Reference Identifier (SRID) on the new database.
psql -f ~/bin/osm2pgsql/900913.sql -d gis
Should reply with
INSERT 0 1
Load planet into the database with osm2pgsql
Your planet file will have a different date. Use an extract file if your interest is limited to a smaller portion of the planet. This operation will take 30 hours or longer. It is very I/O intensive and you can speed it up with very fast disks.
The osm2pgsql command that follows has several parameters. Because this process takes a long time to complete, it is worthwhile to take a moment to consider before starting.
- style file name.
- Use slim mode. Do this. Trust me. Slim mode is required if you can't hold the complete node data in RAM during the import. Even if you have enough RAM and can hold all the node data, you probably want to use slim mode. Slim mode is required if you plan to later apply updates to your data base rather than reloading from scratch.
- data base name.
- RAM cache size in MB. Make this as large as you can.
- Location of your planet file.
./osm2pgsql -S default.style --slim -d gis -C 2048 --number-processes=1 --cache-strategy=dense ~/planet/planet-100217.osm.bz2
[note from 26 October 2011] - osm2pgsql development is currently aiming for performance improvement. The new parameters above, for
--number-processes=1 --cache-strategy=dense should be safe.
Loading the planet file will take some time. How much time it will take depends primarily on the speed of your hard drive system and on the configuration of your system and your available memory. For more details on tuning your Mapnik stack for better performance, see the SotM 2010 session and follow up by Frederik Ramm of GEOFABRIK.
In the interim, let's have a look at your planet import. The first part of the osm2pgsql output looks scary, but is normal.
Using projection SRS 900913 (Spherical Mercator) Setting up table: planet_osm_point NOTICE: table "planet_osm_point" does not exist, skipping NOTICE: table "planet_osm_point_tmp" does not exist, skipping Setting up table: planet_osm_line NOTICE: table "planet_osm_line" does not exist, skipping NOTICE: table "planet_osm_line_tmp" does not exist, skipping Setting up table: planet_osm_polygon NOTICE: table "planet_osm_polygon" does not exist, skipping NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping Setting up table: planet_osm_roads NOTICE: table "planet_osm_roads" does not exist, skipping NOTICE: table "planet_osm_roads_tmp" does not exist, skipping Mid: pgsql, scale=100, cache=4096MB, maxblocks=524289*8192 Setting up table: planet_osm_nodes NOTICE: table "planet_osm_nodes" does not exist, skipping NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_nodes_pkey" for table "planet_osm_nodes" Setting up table: planet_osm_ways NOTICE: table "planet_osm_ways" does not exist, skipping NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_ways_pkey" for table "planet_osm_ways" Setting up table: planet_osm_rels NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_rels_pkey" for table "planet_osm_rels"
Don't be concerned by the
NOTICE: entries above. All normal.
Next, osm2pgsql will start reading the compressed planet file.
Reading in file: /home/nerd/planet/planet-100217.osm.bz2
As osm2pgsql reads the planet file it will give progress reports. The line below will refresh every few seconds and update the numbers in brackets. This part of the import takes a long time. Depending on your server, it will take between hours and days.
Processing: Node(10140k) Way(0k) Relation(0k)
As the import proceeds, the
Node number will update a couple of times per second until complete, then the
Way number will update not quite so quickly, roughly every second or two. Finally the
Relation number will update but at a slower rate, roughly once per minute. As long as you can see these numbers advancing the import process is still operating normally for your server. Do not interrupt the import process unless you have decided to start over again from the beginning.
Processing: Node(593072k) Way(45376k) Relation(87k) Exception caught processing way id=110802 Exception caught processing way id=110803 Processing: Node(593072k) Way(45376k) Relation(474k)
The exceptions shown above are due to minor errors in the planet file. The planet import is still proceeding normally.
The next stage of the osm2pgsql planet import process also will take between hours and days, depending on your hardware. It begins like this.
Node stats: total(593072533), max(696096737) Way stats: total(45376969), max(55410575) Relation stats: total(484528), max(555276) Going over pending ways processing way (752k)
processing way number should update approximately each second.
Going over pending relations node cache: stored: 515463899(86.91%), storage efficiency: 96.01%, hit rate: 85.97% Committing transaction for planet_osm_roads Committing transaction for planet_osm_line Committing transaction for planet_osm_polygon Sorting data and creating indexes for planet_osm_line Sorting data and creating indexes for planet_osm_roads Sorting data and creating indexes for planet_osm_polygon Committing transaction for planet_osm_point Sorting data and creating indexes for planet_osm_point Stopping table: planet_osm_nodes Stopping table: planet_osm_ways Stopping table: planet_osm_rels Building index on table: planet_osm_rels Stopped table: planet_osm_nodes Building index on table: planet_osm_ways Stopped table: planet_osm_rels Completed planet_osm_point Completed planet_osm_roads Completed planet_osm_polygon Completed planet_osm_line Stopped table: planet_osm_ways
This should mean that you import is complete and successful.
Install Mapnik library
The Mapnik library is the first of two items sometimes called
Mapnik. The other item is a collection of tools that OpenStreetMap uses to invoke Mapnik.
The official and up-to-date Mapnik Installation Instructions are here.
You might find that this procedure works as well.
Get some dependencies for building the Mapnik library.
sudo apt-get install libltdl3-dev libpng12-dev libtiff4-dev libicu-dev
sudo apt-get install libboost-python1.40-dev python-cairo-dev python-nose
sudo apt-get install libboost1.40-dev libboost-filesystem1.40-dev
sudo apt-get install libboost-iostreams1.40-dev libboost-regex1.40-dev libboost-thread1.40-dev
sudo apt-get install libboost-program-options1.40-dev libboost-python1.40-dev
sudo apt-get install libfreetype6-dev libcairo2-dev libcairomm-1.0-dev
sudo apt-get install libgeotiff-dev libtiff4 libtiff4-dev libtiffxx0c2
sudo apt-get install libsigc++-dev libsigc++0c2 libsigx-2.0-2 libsigx-2.0-dev
sudo apt-get install libgdal1-dev python-gdal
sudo apt-get install imagemagick ttf-dejavu
Build Mapnik library from source.
svn co http://svn.mapnik.org/tags/release-0.7.1/ mapnik
python scons/scons.py configure INPUT_PLUGINS=all OPTIMIZATION=3 SYSTEM_FONTS=/usr/share/fonts/truetype/
sudo python scons/scons.py install
Confirm that Mapnik library is installed.
>>> import mapnik
If python replies with the second chevron prompt
>>> and without errors, then Mapnik library was found by Python. Congratulations.
Install Mapnik tools
The Mapnik tools are the second item sometimes called
mapnik. This is a collection of tools from OpenStreetMap for making effective use of the Mapnik library.
svn co http://svn.openstreetmap.org/applications/rendering/mapnik
Install prepared world boundary data
Mapnik uses prepared files to generate coastlines and ocean for small scale maps. This is faster than reading the entire database to render zoom levels from zero to nine.
This section now includes the additional shape files that were added to OpenStreetMap default styles in mid-2010. Beware of the long, strange looking links with the repeated http. They are unlikely to copy / paste directly. Use copy link location or equivalent.
tar xvzf world_boundaries-spherical.tgz
tar xvjf processed_p.tar.bz2 -C world_boundaries
tar xjf shoreline_300.tar.bz2 -C world_boundaries
unzip 10m-populated-places.zip -d world_boundaries
unzip 110m-admin-0-boundary-lines.zip -d world_boundaries
Render your first map
The database is loaded and the tools are installed. Let's test everything together. Remember to replace
username with your username.
./generate_xml.py --dbname gis --user username --accept-none
View image.png to confirm that you have rendered a map of England. Congratulations.
References and credits
This tutorial is one of a series of tutorials for OpenStreetMap beginners. Find more tutorials here.
Thank you to the developers who wrote all of these Free Software tools.
All of the helpful folks on #osm on irc.oftc.net especially jburgess, Ldp and springmeyer
All of the helpful folks on the OSM mailing list http://lists.openstreetmap.org/
Thanks dbaker and acant for copy editing help on the previous version of this article.
Thanks nelson, codebrainz, Ldp, balrog-k1n and kW for more updates on the previous version of this article.
Thanks wnoronha for additional patches.
This archived article was originally published Fri, 03/26/2010 - 19:13.