Build your own OpenStreetMap Server - Ubuntu 10.04 Lucid Lynx

SUNY Buffalo campus as seen on OpenStreetMap.org in March of 2009
Build your own OpenStreetMap server. Build your own what?

OpenStreetMap is the editable World map of everything. It is the Wikipedia of maps. It is to other on-line maps as Wikipedia is to Britannica. And it is awesome in every possible way.


This article updates an earlier version, archived here. This current version was written in March 2010.

The current version of this article is based on Ubuntu 10.04 Lucid Lynx. Lucid will be released on 29 April 2010 and be in beta until then. Operating systems in beta are not suitable for production use.

OpenStreetMap is a massive project that started as Steve Coast's, frame-breaking idea in 2004. If I make a map of my neighborhood and give it away, and you make a map of your neighborhood and give it away, then we both have better maps. As of March 2009 there are over 100,000 contributors making maps of their neighborhoods and contributing them to this World wide effort.

OpenStreetMap makes the data and the software available to you with Free Software and free data licenses so that you can use, learn from, teach with, improve upon and share with others what you gain from OpenStreetMap. And you can build your own local copy of OpenStreetMap for your business, school, community group or personal interests.

The project operates on a massive scale as there is an incredible amount of data, there is more data every day, and there are more people using the data every day. OpenStreetMap has to run on several servers, including a handful of API servers and separate database, development, web and tile servers. This article does not cover the creation of a complete OSM datacentre.

It does cover creating a single server from a typical PC circa 2006. These instructions build what OpenStreetMap calls a tile server. That is, a computer that uses the OSM data set to create map images that are suitable for a web site. Not every OpenStreetMap function is supported, but you will be able to create a local map, keep it up to date and customize it for your own purposes.

Why would I build my own?

Why indeed? OpenStreetMap.org is already freely available on the internet. Why not just use that? You can and you should. Eventually you may come up with an idea. You might want to make the map work a little differently for you. You might want a map for a special purpose.

British Cartographic Society web logoPerhaps for cycling, http://www.opencyclemap.org/ OpenCycleMap is a wonderful example of what you can do with the tools and data of OpenStreetMap catalyzed by an idea. Created by Andy Allan and Dave Stubbs, OpenCycleMap uses OpenStreetMap data, then displays it in a way that is useful to cyclists with the emphasis placed on cycle trails, bike shops and bike parking. They've also added elevation contours and hill-shading as cyclists care about hills. Sometimes they are looking for a good challenging climb, and other times they just want to get home with the groceries. The brilliant work of the OpenCycleMap team was recognized with a Commendation from the British Cartographic Society as well as the prestigious Lolcat of Awesomness from the OpenStreetMap community at large.

Or maybe you need to have access to your map even when your internet provider is down. Or when the power is off. Or both. It won't take much for you to see the benefit of having your own piece of OpenStreetMap infrastructure. All you need to start is an idea and the thirst for knowledge.

There are a lot of moving parts to OpenStreetMap. I hope that these steps will make it easier for you to get your first map working. This article is intended to get from bare iron to a working local OpenStreetMap tile server. This one will collect OpenStreetMap data and allow you to render that data into images suitable for use on a web site. Future articles will cover how to use your new server and several customizations that you'll want to try.

Let's get started, shall we?

Prerequisites

Start with an installation of Ubuntu Lucid Lynx Server for your hardware architecture. Add the LAMP server and SSH server during installation. Once installed, this can be run as a headless server with neither monitor nor keyboard. So ssh to the new box, and let's get started.

Update operating system

Get Ubuntu updates. Yes, you just installed Ubuntu, but it has been getting security and bug fixes. Use this to bring your system up to data as of right now.
sudo apt-get update
sudo apt-get upgrade

Get some system tools

We'll need subversion to get the latest updates from OpenStreetMap and other places. Munin makes pretty pictures of activity on the server. I like screen. htop is neat-o.

sudo apt-get install subversion autoconf screen munin-node munin htop

Organize the file system a bit

cd ~
mkdir src bin planet

Get the latest planet - OpenStreetMap data file

A new planet file is published approximately each week. The mirror and archives of the planet files are here. In March 2010 the planet file was about 8.2GB in length. If you are not interested in the entire planet you can choose to download an extract file instead.

Facing an 8.2GB or larger download, this is an excellent time to consider using screen if you haven't used it before. Screen allows you to operate several terminal windows through one ssh connection. And more. Have a look this screen tutorial if you haven't used it before. Or wait for your download to complete, or use another ssh session to continue.

cd planet
wget http://planet.openstreetmap.org/planet-latest.osm.bz2

Prepare the postGIS database

Use the PostGIS extensions to postgresql for all sorts of geographical goodness. Install the postGIS and prerequisites.
sudo apt-get install postgresql-8.4-postgis postgresql-contrib-8.4
sudo apt-get install postgresql-server-dev-8.4
sudo apt-get install build-essential libxml2-dev
sudo apt-get install libgeos-dev libpq-dev libbz2-dev proj

Install osm2pgsql from the repository

The latest version of osm2pgsql has the most goodies, so we'll use that rather than a package.

cd ~/bin
svn co http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/
cd osm2pgsql
./autogen.sh
./configure
make

Configure the PostGIS database

edit /etc/postgresql/8.4/main/postgresql.conf in four places. These changes help with the large quantities of data that we are using.
shared_buffers = 128MB # 16384 for 8.1 and earlier
checkpoint_segments = 20
maintenance_work_mem = 256MB # 256000 for 8.1 and earlier
autovacuum = off

edit /etc/sysctl.conf
kernel.shmmax=268435456

Russ Nelson and balrog-k1n remind us that the above only takes effect after a reboot. Making this work immediately requires the following.
sudo sysctl kernel.shmmax=268435456

Restart postgres to enable the changes
sudo /etc/init.d/postgresql-8.4 restart

It should restart as above.

 * Restarting PostgreSQL 8.4 database server
   ...done.

Create a database called gis. Some of our future tools presume that you will use this database name. Substitute your username for username in two places below. This should be the username that will render maps with mapnik.
sudo -u postgres -i
createuser username # answer yes for superuser
createdb -E UTF8 -O username gis
createlang plpgsql gis
exit

Set up PostGIS on the postresql database.
psql -f /usr/share/postgresql/8.4/contrib/postgis.sql -d gis
This should respond with many lines ending with

...
CREATE FUNCTION
COMMIT

Substitute your username for username in two places in the next line. This should be the username that will render maps with mapnik.
echo "ALTER TABLE geometry_columns OWNER TO username; ALTER TABLE spatial_ref_sys OWNER TO username;" | psql -d gis
# Should reply with

ALTER TABLE
ALTER TABLE

Enable intarray
psql -f /usr/share/postgresql/8.4/contrib/_int.sql -d gis

Replies with many lines ending with

...
CREATE FUNCTION
CREATE OPERATOR CLASS

Set the Spatial Reference Identifier (SRID) on the new database.
psql -f ~/bin/osm2pgsql/900913.sql -d gis
Should reply with

INSERT 0 1

Load planet into the database with osm2pgsql

Your planet file will have a different date. Use an extract file if your interest is limited to a smaller portion of the planet. This operation will take 30 hours or longer. It is very I/O intensive and you can speed it up with very fast disks.
cd ~/bin/osm2pgsql
./osm2pgsql -S default.style --slim -d gis -C 2048 ~/planet/planet-100217.osm.bz2

Loading the planet file will take some time. How much time it will take depends primarily on the speed of your hard drive system and on the configuration of your system and your available memory. For more details on tuning your Mapnik stack for better performance, see the SotM 2010 session and follow up by Frederik Ramm of GEOFABRIK.

In the interim, let's have a look at your planet import. The first part of the osm2pgsql output looks scary, but is normal.

Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
NOTICE:  table "planet_osm_point" does not exist, skipping
NOTICE:  table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE:  table "planet_osm_line" does not exist, skipping
NOTICE:  table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE:  table "planet_osm_polygon" does not exist, skipping
NOTICE:  table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE:  table "planet_osm_roads" does not exist, skipping
NOTICE:  table "planet_osm_roads_tmp" does not exist, skipping
Mid: pgsql, scale=100, cache=4096MB, maxblocks=524289*8192
Setting up table: planet_osm_nodes
NOTICE:  table "planet_osm_nodes" does not exist, skipping
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_nodes_pkey" for table "planet_osm_nodes"
Setting up table: planet_osm_ways
NOTICE:  table "planet_osm_ways" does not exist, skipping
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_ways_pkey" for table "planet_osm_ways"
Setting up table: planet_osm_rels
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_rels_pkey" for table "planet_osm_rels"

Don't be concerned by the NOTICE: entries above. All normal.

Next, osm2pgsql will start reading the compressed planet file.

Reading in file: /home/nerd/planet/planet-100217.osm.bz2

As osm2pgsql reads the planet file it will give progress reports. The line below will refresh every few seconds and update the numbers in brackets. This part of the import takes a long time. Depending on your server, it will take between hours and days.

Processing: Node(10140k) Way(0k) Relation(0k)

As the import proceeds, the Node number will update a couple of times per second until complete, then the Way number will update not quite so quickly, roughly every second or two. Finally the Relation number will update but at a slower rate, roughly once per minute. As long as you can see these numbers advancing the import process is still operating normally for your server. Do not interrupt the import process unless you have decided to start over again from the beginning.

Processing: Node(593072k) Way(45376k) Relation(87k)
Exception caught processing way id=110802
Exception caught processing way id=110803
Processing: Node(593072k) Way(45376k) Relation(474k)

The exceptions shown above are due to minor errors in the planet file. The planet import is still proceeding normally.

The next stage of the osm2pgsql planet import process also will take between hours and days, depending on your hardware. It begins like this.

Node stats: total(593072533), max(696096737)
Way stats: total(45376969), max(55410575)
Relation stats: total(484528), max(555276)

Going over pending ways
processing way (752k)

The processing way number should update approximately each second.

Going over pending relations

node cache: stored: 515463899(86.91%), storage efficiency: 96.01%, hit rate: 85.97%
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Sorting data and creating indexes for planet_osm_line
Sorting data and creating indexes for planet_osm_roads
Sorting data and creating indexes for planet_osm_polygon
Committing transaction for planet_osm_point
Sorting data and creating indexes for planet_osm_point
Stopping table: planet_osm_nodes
Stopping table: planet_osm_ways
Stopping table: planet_osm_rels
Building index on table: planet_osm_rels
Stopped table: planet_osm_nodes
Building index on table: planet_osm_ways
Stopped table: planet_osm_rels
Completed planet_osm_point
Completed planet_osm_roads
Completed planet_osm_polygon
Completed planet_osm_line
Stopped table: planet_osm_ways

This should mean that you import is complete and successful.

Install Mapnik library

The Mapnik library is the first of two items sometimes called Mapnik. The other item is a collection of tools that OpenStreetMap uses to invoke Mapnik.

The official and up-to-date Mapnik Installation Instructions are here.

You might find that this procedure works as well.

Get some dependencies for building the Mapnik library.
sudo apt-get install libltdl3-dev libpng12-dev libtiff4-dev libicu-dev
sudo apt-get install libboost-python1.40-dev python-cairo-dev python-nose
sudo apt-get install libboost1.40-dev libboost-filesystem1.40-dev
sudo apt-get install libboost-iostreams1.40-dev libboost-regex1.40-dev libboost-thread1.40-dev
sudo apt-get install libboost-program-options1.40-dev libboost-python1.40-dev
sudo apt-get install libfreetype6-dev libcairo2-dev libcairomm-1.0-dev
sudo apt-get install libgeotiff-dev libtiff4 libtiff4-dev libtiffxx0c2
sudo apt-get install libsigc++-dev libsigc++0c2 libsigx-2.0-2 libsigx-2.0-dev
sudo apt-get install libgdal1-dev python-gdal
sudo apt-get install imagemagick

Build Mapnik library from source.

cd ~/src
svn co http://svn.mapnik.org/tags/release-0.7.1/ mapnik
cd mapnik
python scons/scons.py configure INPUT_PLUGINS=all OPTIMIZATION=3 SYSTEM_FONTS=/usr/share/fonts/truetype/
python scons/scons.py
sudo python scons/scons.py install
sudo ldconfig

Confirm that Mapnik library is installed.
python
>>> import mapnik
>>>

If python replies with the second chevron prompt >>> and without errors, then Mapnik library was found by Python. Congratulations.

Install Mapnik tools

The Mapnik tools are the second item sometimes called mapnik. This is a collection of tools from OpenStreetMap for making effective use of the Mapnik library.

cd ~/bin
svn co http://svn.openstreetmap.org/applications/rendering/mapnik

Install prepared world boundary data

Mapnik uses prepared files to generate coastlines and ocean for small scale maps. This is faster than reading the entire database to render zoom levels from zero to nine.

This section now includes the additional shape files that were added to OpenStreetMap default styles in mid-2010. Beware of the long, strange looking links with the repeated http. They are unlikely to copy / paste directly. Use copy link location or equivalent.

cd ~/bin/mapnik
mkdir world_boundaries
wget http://tile.openstreetmap.org/world_boundaries-spherical.tgz
tar xvzf world_boundaries-spherical.tgz
wget http://tile.openstreetmap.org/processed_p.tar.bz2
tar xvjf processed_p.tar.bz2 -C world_boundaries
wget http://tile.openstreetmap.org/shoreline_300.tar.bz2
tar xjf shoreline_300.tar.bz2 -C world_boundaries
wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/...
unzip 10m-populated-places.zip -d world_boundaries
wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/...
unzip 110m-admin-0-boundary-lines.zip -d world_boundaries

Render your first map

The database is loaded and the tools are installed. Let's test everything together. Remember to replace username with your username.

cd ~/bin/mapnik
./generate_xml.py --dbname gis --user username --accept-none
./generate_image.py

View image.png to confirm that you have rendered a map of England. Congratulations.

References and credits

Thank you to the developers who wrote all of these Free Software tools.
Thanks to:
All of the helpful folks on #osm on irc.oftc.net especially jburgess, Ldp and springmeyer
All of the helpful folks on the OSM mailing list http://lists.openstreetmap.org/
http://wiki.openstreetmap.org/wiki/Mapnik
http://mapnik.org/
Thanks dbaker and acant for copy editing help on the previous version of this article.
Thanks nelson, codebrainz, Ldp, balrog-k1n and kW for more updates on the previous version of this article.

Lynx photo CCBYND Tambako the Jaguar on Flickr.

help needed

sudo apt-get install postgresql-8.4-postgis postgresql-contrib-8.4, the script always fail at this portion

Hi marco, What was the error

Hi marco,

What was the error message? Which distro are you using? Which repositories have you enabled?

I do have the same problem,

I do have the same problem, it seems that the package does not exist. My error message is:
"E: Konnte Paket postgresql-8.4-postgis nicht finden"
Which means "could not find this package"

Same questions to you,

Same questions to you, Bikeman2000,

Which distro are you using?
Type this,
cat /etc/issue.net
It should reply,

Ubuntu lucid (development branch)

Which repositories have you enabled?
Type this,
grep '^deb .* universe$' /etc/apt/sources.list
It should reply,

deb http://us.archive.ubuntu.com/ubuntu/ lucid universe
deb http://us.archive.ubuntu.com/ubuntu/ lucid-updates universe
deb http://security.ubuntu.com/ubuntu lucid-security universe

How about you?

postgresql-8.4

hi rw,
thanks for replying, I was kind of mess up, I was on ubuntu 9.10 and trying to following the instruction made for ubuntu 10, so I guess it was the reason it was not working, now I got the ubuntu 10 installed and following the procedure to run again, I will let you know how it comes out.

later

Marco

Hi marco, Even the previous

Hi marco,

Even the previous article was not for 9.10, but for 8.04, so you may find some differences.

Good luck,

I am using Ubuntu 9.10 and

I am using Ubuntu 9.10 and am surprised that this specific Postgresql 8.4 package is for 10.04 only. Currently I am trying to install postgresql with postgis.
I found postgresql 8.4 with postgis but the file "postgis.sql" ist missing.

Anybody knows a working command to install postgresql with postgis in Ubuntu 9.10 (I do not care if it is 8.3 or 8.4 or a developer version)?

Hi Bikeman2000, I don't have

Hi Bikeman2000,

I don't have a tutorial specific to building an OpenStreetMap tile server on Karmic 9.10, but I did find this link for you. It looks like Tom Wardill has a solution for you

    It suggests that what you are seeing in Karmic is:

  • postgresql 8.4 is default
  • postgis was not built for postgresql 8.4 in Karmic
  • an automated installation won't work without help

http://blog.isotoma.com/2010/03/installing-postgis-on-ubuntu-karmic-koal...

If it works for you on Karmic, you should thank Tom.

Thanks this helped a lot.

Thanks this helped a lot. The psql-command worked but i don't have the file "_int.sql".

Forget the last comment, I

Forget the last comment, I reinstalled the postgis-packages from this tutorial and after that I had all the .sql files.

Currently I am importing the .osm file for my city into the database

Wonderful! Congratulations.

Wonderful! Congratulations.

This tutorial is very

This tutorial is very precise and comfortable since the reader only has to copy/paste the commands into the terminal.

But in two places the reader does not get the results because even a superuser does not have the rights to modify the files "/etc/postgresql/8.3/main/postgresql.conf" and "/etc/sysctl.conf"

First I started gedit and opened the files. Saving my edits was denied due to my userrights. After that I started gedit as a superuser and was surprised that I still did not have the rights to save the file.
Then I opened the folder with Nautilus and opened the file properties. I saw that I am not the owner of the file and I was not able to change the ownershipt there.
So I had to search the web for a solution and finally I found the "chown" command to change it.

Could you add the following lines to "edit /etc/postgresql/8.3/main/postgresql.conf in four places...":
sudo chown root /etc/postgresql/8.3/main/postgresql.conf
sudo gedit /etc/postgresql/8.3/main/postgresql.conf

and give the hint that the user has to close gedit or else he cannot continue in the terminal?

The same to "edit /etc/sysctl.conf":
sudo chown /etc/sysctl.conf
sudo gedit /etc/sysctl.conf

I'm not able to reproduce

I'm not able to reproduce your bug regarding gedit with either 9.04, 9.10 or 10.04 Beta1. I do see a couple of similar reports after a search, but those appeared to a) relate to a version in 8.04, or b) be fixed with sudo

Check the following

- make sure your operating system is up to date. sudo apt-get upgrade; sudo apt-get update
- try another editor. Perhaps nano? sudo nano /etc/postgresql/8.3/main/postgresql.conf

Good luck

Finally it seems I made

Finally it seems I made it.

Only at "Render your first map" I got an error message:
PostGIS: SRID warning, using srid=-1
Traceback (most recent call last):
File "./generate_image.py", line 37, in
mapnik.load_map(m,mapfile)
RuntimeError: PSQL error:
FEHLER: Relation »planet_osm_polygon« existiert nicht
LINE 3: from planet_osm_polygon
^
Full sql was: 'select * from
(select way,aeroway,amenity,landuse,leisure,man_made,military,"natural",power,shop,tourism,name
from planet_osm_polygon
where landuse is not null
or leisure is not null
or shop is not null
or aeroway in ('apron','aerodrome')
or amenity in ('parking','university','college','school','hospital','kindergarten','grave_yard')
or military in ('barracks','danger_area')
or "natural" in ('field','beach','heath','mud','wood')
or power in ('station','sub_station')
or tourism in ('attraction','camp_site','caravan_site','picnic_site','zoo')
order by z_order,way_area desc
) as leisure
limit 0'
(encountered during parsing of layer 'leisure')

I did not import the whole planet.osm and used an extract for my region. Did this cause the error?

hmmmm. "PostGIS: SRID

hmmmm.

"PostGIS: SRID warning, using srid=-1"

Make me wonder if you have missed this step?

psql -f ~/bin/osm2pgsql/900913.sql -d gis

"FEHLER: Relation »planet_osm_polygon« existiert nicht"

Makes me wonder about your extract import. Did it seem to import without error? How long did the import take?

You did not load data

I make same error. Forget to load data. Try this:

./osm2pgsql -S default.style --slim -d gis -C 2048 ~/planet/YOUR_OSM_FILE.osm.bz2

Feedback

Hello !

Thanks for this very nice tutorial !
I did not achieve to generate properly a postgre/postgis database from a regional OSM extract last saturday, on 9.10.
This was mainly because I couldn't setup Postgis in my database
The day after, I found your blog :-)

I figured out that my initial choice of Postgis version (1.5.1) was wrong.
I reinstalled postgreSQL & Postgis packages per your tutorial and I got better results !

I have few comments though :
Same issue as for some comments above, I am new to linux / Ubuntu, and editing the conf files as root user was not that easy for me.
I finally used the command "sudo nano /etc/postgresql/8.3/main/postgresql.conf" and that allowed to save the modifications.

Use of osm2pgsql returned an error stating that postgreSQL was maybe not listening on port 5432.
I guess this is because access rights setting for localhost was not set to "trust", but "md5" instead.
(in /var/lib/pgsql/data/pg_hba.conf)

This is now fixed, and I can generate my database and perform SQL queries with pgadmin that return coherent results.

Thanks again !

Finally I did it under

Finally I did it under Ubuntu 9.10. I created the image of England and made a copy of the script which generated an image of my city. I will post my solution later.

THANK YOU VERY MUCH FOR THIS TUTORIAL

Yay! \o/

Yay! \o/

osm2pgsql uses autoconf now.

On my first try I only had 500 MB of ram so I built a new virtual server, this time with 8GB ram and also I used Ubuntu 10.04. I was following these new instructions when I came to building osm2pgsql. There was no configure file. Someone on the IRC said to use the autogen.sh, but that calls autoconf, so I had to install that.

sudo apt-get install autoconf
./autogen.sh
./configure
make

So far it's been great and I'm importing the OSM data now.

Thanks!

Thank you. Updated for

Thank you. Updated for autoconf in the article.

libboost woes

While the DB was importing I switched to another 'screen' (thanks for showing me that one) and started working on Mapnik prerequisites.
I cut and pasted from the libboost section and got:

E: Couldn't find package libboost1.41-dev

and more.

I checked the repositories as suggested above, and they were as shown.

IRC members suggested just using 1.40 so I'm working on it.

This would be a great item for a BitNami stack!

Thanks for all your help :)

Hmm, my installation showed

Hmm, my installation showed that libboost1.41-dev was up to date. When I remove boost and re-installed, I see your error. I've reverted this tutorial to boost 1.40 until I learn what is up with 1.41.

Thanks!

Great guide and right to the point. Miles better than anything else I've found so far. Thank you!

Thanks!

Just finished the first render of the UK - seems to work very well. Thanks again! Just a quick correction to the above - the generate_image.py produces an image.png, not image.py.

Thank you for the kind

Thank you for the kind words, and for the correction.

Installation with Ubuntu 8.04 and Postgre 8.3

Hi, everythign works fine until psql -f /usr/share/postgresql/8.3/contrib/postgis.sql -d gis

The file postgis.sql is just missing...

Everyting before went jsut as written in this tutorial. Does anyone have a clue what I could have been missing?

Thank you.
MW

Hi MW, Thanks for stopping

Hi MW,

Thanks for stopping by and having a look at this article.

You find that the previous version of this article is for installing an OpenStreetMap tile server on Ubuntu 8.04, the current version is for Ubuntu Lucid lynx, 10.04.

The postgis file name has changed since 8.04. From the archived article, try this

Set up PostGIS on the [Ubuntu 8.04] postresql database.
psql -f /usr/share/postgresql-8.3-postgis/lwpostgis.sql -d gis

And that should move you along to the next step. Consider following the archived article. Lucid will be in beta for another couple of days as of this writing.

Thank you for answering so

Thank you for answering so quickly.

In fact I have found this other page: http://www.ioexception.de/2010/02/16/openstreetmap-rendering-mit-mapnik/

Which guided me to this other sql file.

Unfortunately I saw too late that I have to stop the maintenance script for postgresql as it kills the server process...

I am now trying again to import the osm into the db... wish me luck ;-)

PS: I have found this page for mod_tile
http://trac.openstreetmap.org/browser/applications/utils/mod_tile/readme...

Isn't there an easier way of having osm on the webserver?

Thx
MW

Still many errors

Hi,

unfortunately there are quite a few errors occuring while trying to insert the data into the database with this command:
./osm2pgsql -S default.style --slim -d gis -C 2048 ~/planet/planet-100217.osm.bz2

2010-04-18 01:58:48 CEST HINWEIS: Tabelle »planet_osm_point« existiert nicht, wird übersprungen
2010-04-18 01:58:48 CEST HINWEIS: Tabelle »planet_osm_point_tmp« existiert nicht, wird übersprungen
2010-04-18 01:58:49 CEST HINWEIS: Tabelle »planet_osm_line« existiert nicht, wird übersprungen
2010-04-18 01:58:49 CEST HINWEIS: Tabelle »planet_osm_line_tmp« existiert nicht, wird übersprungen
2010-04-18 01:58:49 CEST HINWEIS: Tabelle »planet_osm_polygon« existiert nicht, wird übersprungen
2010-04-18 01:58:49 CEST HINWEIS: Tabelle »planet_osm_polygon_tmp« existiert nicht, wird übersprungen
2010-04-18 01:58:49 CEST HINWEIS: Tabelle »planet_osm_roads« existiert nicht, wird übersprungen
2010-04-18 01:58:49 CEST HINWEIS: Tabelle »planet_osm_roads_tmp« existiert nicht, wird übersprungen
2010-04-18 01:58:49 CEST HINWEIS: Tabelle »planet_osm_nodes« existiert nicht, wird übersprungen
2010-04-18 01:58:49 CEST HINWEIS: CREATE TABLE / PRIMARY KEY erstellt implizit einen Index »planet_osm_nodes_pkey« für Tabelle »planet_osm_nodes«
2010-04-18 01:58:49 CEST HINWEIS: Tabelle »planet_osm_ways« existiert nicht, wird übersprungen
2010-04-18 01:58:49 CEST HINWEIS: CREATE TABLE / PRIMARY KEY erstellt implizit einen Index »planet_osm_ways_pkey« für Tabelle »planet_osm_ways«
2010-04-18 01:58:49 CEST HINWEIS: Tabelle »planet_osm_rels« existiert nicht, wird übersprungen
2010-04-18 01:58:49 CEST HINWEIS: CREATE TABLE / PRIMARY KEY erstellt implizit einen Index »planet_osm_rels_pkey« für Tabelle »planet_osm_rels«

In the postgre logs there are many error messages saying that some columns are missing. Then after hours of importing I get error message and the import stops.

2010-04-18 03:22:26 CEST LOG: Background-Writer-Prozess (PID 31586) wurde von Signal 9 beendet: Killed
2010-04-18 03:22:29 CEST LOG: aktive Serverprozesse werden abgebrochen
2010-04-18 03:22:48 CEST WARNUNG: breche Verbindung ab wegen Absturz eines anderen Serverprozesses
2010-04-18 03:22:52 CEST DETAIL: Der Postmaster hat diesen Serverprozess angewiesen, die aktuelle Transaktion zurückzurollen und die Sitzung zu beenden, weil ein anderer Serverprozess abnormal beendet wurde und möglicherweise das Shared Memory verfälscht hat.
2010-04-18 03:22:53 CEST TIPP: In einem Moment sollten Sie wieder mit der Datenbank verbinden und Ihren Befehl wiederholen können.
2010-04-18 03:22:53 CEST ZUSAMMENHANG: COPY planet_osm_roads, Zeile 1
2010-04-18 03:22:53 CEST WARNUNG: breche Verbindung ab wegen Absturz eines anderen Serverprozesses
2010-04-18 03:22:53 CEST DETAIL: Der Postmaster hat diesen Serverprozess angewiesen, die aktuelle Transaktion zurückzurollen und die Sitzung zu beenden, weil ein anderer Serverprozess abnormal beendet wurde und möglicherweise das Shared Memory verfälscht hat.
2010-04-18 03:22:53 CEST TIPP: In einem Moment sollten Sie wieder mit der Datenbank verbinden und Ihren Befehl wiederholen können.
2010-04-18 03:22:53 CEST ZUSAMMENHANG: COPY planet_osm_rels, Zeile 1
2010-04-18 03:22:45 CEST WARNUNG: breche Verbindung ab wegen Absturz eines anderen Serverprozesses
2010-04-18 03:22:53 CEST DETAIL: Der Postmaster hat diesen Serverprozess angewiesen, die aktuelle Transaktion zurückzurollen und die Sitzung zu beenden, weil ein anderer Serverprozess abnormal beendet wurde und möglicherweise das Shared Memory verfälscht hat.
2010-04-18 03:22:54 CEST TIPP: In einem Moment sollten Sie wieder mit der Datenbank verbinden und Ihren Befehl wiederholen können.
2010-04-18 03:22:54 CEST ZUSAMMENHANG: COPY planet_osm_line, Zeile 1
2010-04-18 03:22:53 CEST WARNUNG: breche Verbindung ab wegen Absturz eines anderen Serverprozesses
2010-04-18 03:22:54 CEST DETAIL: Der Postmaster hat diesen Serverprozess angewiesen, die aktuelle Transaktion zurückzurollen und die Sitzung zu beenden, weil ein anderer Serverprozess abnormal beendet wurde und möglicherweise das Shared Memory verfälscht hat.
2010-04-18 03:22:54 CEST TIPP: In einem Moment sollten Sie wieder mit der Datenbank verbinden und Ihren Befehl wiederholen können.
2010-04-18 03:22:54 CEST ZUSAMMENHANG: COPY planet_osm_point, Zeile 1001077
2010-04-18 03:22:54 CEST WARNUNG: breche Verbindung ab wegen Absturz eines anderen Serverprozesses
2010-04-18 03:22:54 CEST DETAIL: Der Postmaster hat diesen Serverprozess angewiesen, die aktuelle Transaktion zurückzurollen und die Sitzung zu beenden, weil ein anderer Serverprozess abnormal beendet wurde und möglicherweise das Shared Memory verfälscht hat.
2010-04-18 03:22:54 CEST TIPP: In einem Moment sollten Sie wieder mit der Datenbank verbinden und Ihren Befehl wiederholen können.
2010-04-18 03:22:54 CEST ZUSAMMENHANG: COPY planet_osm_nodes, Zeile 207502220: »239224827 209015140 -739239709 \N«
2010-04-18 03:22:46 CEST WARNUNG: breche Verbindung ab wegen Absturz eines anderen Serverprozesses
2010-04-18 03:22:52 CEST DETAIL: Der Postmaster hat diesen Serverprozess angewiesen, die aktuelle Transaktion zurückzurollen und die Sitzung zu beenden, weil ein anderer Serverprozess abnormal beendet wurde und möglicherweise das Shared Memory verfälscht hat.
2010-04-18 03:22:55 CEST TIPP: In einem Moment sollten Sie wieder mit der Datenbank verbinden und Ihren Befehl wiederholen können.
2010-04-18 03:22:55 CEST ZUSAMMENHANG: COPY planet_osm_ways, Zeile 1
2010-04-18 03:23:41 CEST WARNUNG: breche Verbindung ab wegen Absturz eines anderen Serverprozesses
2010-04-18 03:24:02 CEST DETAIL: Der Postmaster hat diesen Serverprozess angewiesen, die aktuelle Transaktion zurückzurollen und die Sitzung zu beenden, weil ein anderer Serverprozess abnormal beendet wurde und möglicherweise das Shared Memory verfälscht hat.
2010-04-18 03:24:03 CEST TIPP: In einem Moment sollten Sie wieder mit der Datenbank verbinden und Ihren Befehl wiederholen können.
2010-04-18 03:24:04 CEST ZUSAMMENHANG: COPY planet_osm_polygon, Zeile 1
2010-04-18 03:24:05 CEST LOG: could not send data to client: Broken pipe
2010-04-18 03:24:05 CEST CONTEXT: COPY planet_osm_polygon, line 1
2010-04-18 03:24:05 CEST STATEMENT: COPY planet_osm_polygon (osm_id,"access","addr:flats","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","bridge","boundary","building","construction","cutting","disused","embankment","foot","highway","historic","horse","junction","landuse","layer","learning","leisure","lock","man_made","military","motorcar","name","natural","oneway","operator","power","power_source","place","railway","ref","religion","residence","route","service","shop","sport","tourism","tracktype","tunnel","waterway","width","wood","z_order","way_area",way) FROM STDIN
2010-04-18 03:24:06 CEST LOG: alle Serverprozesse beendet; initialisiere neu
2010-04-18 03:24:07 CEST LOG: Datenbanksystem wurde unterbrochen; letzte bekannte Aktion am 2010-04-18 03:20:05 CEST

I know it is full of german, but I guess that many terms in english explain what the issue is...

I will start all over again, just not sure what could have gine wrong.

The installation of Mapnik was successfull though.

Thx for helping.

PS: unfortunately it is not possible to reply or comment on the Ubuntu 8.04 install howto...

MW

All worked well!

Thanks for this - I got it all up and running very easily. The only part I got stuck was when I wanted to generate all the tiles, when I ran ./generates_tiles.py I kept getting the error that /home/alex/svn.openstreetmap.org/applications/rendering/mapnik/osm-local.xml couldn't be found. But I got this fixed by editing the mapfile =... line in generate_tiles.py to point to /home/alex/osm/bin/mapnik/osm.xml instead and then all worked fine. Is there a better way of setting 'MAPNIK_MAP_FILE' variable? (I'm very new to python).

Where am I to create dirs ?

Thanks for this tutorial I will use.
Just a little question... I will put mapnik on my computer. And I don't know where I must create the folders with the command :
mkdir src bin planet
Thank you for your reply that, I'm shure, will come soon.

I've been putting all of

I've been putting all of this in the /home/username directory. I'll add a hint in the article.

mod_tile

Great blog post, so much easier to follow than the wiki page and I got everything up and running without a hitch,

Are you going to do another post on getting mod_tile running? :)

xml2-config missing

Under Karmic Koala, I had the error message
"Exiting... the following required dependencies were not found:
- xml2-config (xml2-config program | try setting XML2_CONFIG SCons option)"
when building mapnik.
I solved it by installing libxml2-dev :
sudo aptitude install libxml2-dev

Best regards

openstreetmap server for Red hat enterpise linux or Suze

Hello Friends,
Could anybody please tell me whether it is possible to configure openstreetmap tile server on Red Hat Enterprise Linux or Suze O.S?
Any help would be greatly appreciated.

Thanks a lots!
Balram

Hello Balram, Sure, it

Hello Balram,

Sure, it should be possible. I did a partial installation on RHCE 5.4 but went back to Debian / Ubuntu. Too many prerequisite packages were only available via external repositories. Not sure about SUSE. YMMV.

FYI.

you can combine lines like:

bunzip2 shoreline_300.tar.bz2
tar xvf shoreline_300.tar

into: tar xvfj shoreline_300.tar.bz2

no reason to run 2 different commands.

Problem

HI,

I find your blog very usefull since I'm trying to accomplish the exact same thing. However I do get an error.

I want to import the data with the latlong option, since I need to see the coordinates, but when I add the -l I get the following error:

Using projection SRS: 4326(LatLong)
table planet_osm_point does not exist, skipping
table planet_osm_point_tmp does not exist, skipping
SELECT AddGeometryColumn('planet_osm_point', 'way', 4326, 'POINT', 2 );
failed: ERROR: AddGeometryColumns() - invalid SRID
CONTEXT: SQL statement "SELECT AddGeometryColumn('','', $1 , $2 , $3 ,
$4 , $5
)"
PL/pgSQL function "addgeometrycolumn" line 5 at SQL statement

When i don't use the -l everything works fine. Any idea's on how to fix that?

DejaVu fonts

Thanks a lot for the tutorial

I had to install the ttf-dejavu package for rendering the image
Only ttf-dejavu-core was installed

And how do we implement

And how do we implement search on the osm db?

Hi anonymous, thanks for

Hi anonymous, thanks for coming by. Weren't you here before? Maybe it was somebody else with the same name. :-)

How do you search on the OSM database? That depends on what you are doing. As a user, I often just use the OpenStreetMap search box, on the left side of the page at http://openstreetmap.org/

On my local tile server I usually write a postgres or PostGIS query.

I might use XAPI.

For .osm files I use grep or sometimes sed if I'm feeling a bit saucy.

If I had a bunch of clients wanting location search that 'just works', I suppose I would set up a local instance of the amazing Nominatim, by Twain.

But really it depends on too many things to just say, "Do this specific thing." Tell me more?

Load planet into database

On "Load planet into the database with osm2pgsql" I got an error message along the lines of
Connection to database failed... Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"

Diagnosis, run ls -la /var/run/postgresql and you see a line like .s.PGSQL.5433. Eeek, did you notice it complains about 5432 but has 5433?

Solution, specify the port by changing the command to be ./osm2pgsql -S default.style --slim -d gis -C 2048 -P 5433 ~/planet/planet-100217.osm.bz2.

These instructions are great Richard, thanks.

I'm glad you got that sorted

I'm glad you got that sorted out, Gregory! Any idea why you were running postgres on port 5433? Did you have two postgres servers running or something?

I also run into this

I also run into this socket-domain problem and you provided me with the right solution, so thanks! However, I also had to add -H localhost to the code.

And also thanks for the writer of this article, very helpful!

GEOS via SVN results in OSM2PGSQL-Make-error

First let me thank you for this profound description, which is straight forward to a working result.

I am new to the OSM subject and doing my first steps so far.

For several reasons I would like to use both the GEOS library and the OSM2PGSQL library via SVN. However when I start MAKE for OSM2PGSQL it results into one error (obviously raised by the GEOS library). It might be a missing INCLUDE-header or something different, which is easy to fix by somebody who is familiar with compiling from the source - but difficult for a newbie.

Is there any chance to use the GEOS-SVN library rather than the (older) package? Did anybody already try?

Kind appreciation,

Kartograph

Oops, my last comment got

Oops, my last comment got stuck (unless it's in moderation?).

I had problems with "Load planet into the database with osm2pgsql" and so changed the command to specify the port.
./osm2pgsql -S default.style --slim -d gis -C 2048 -P 5433 ~/planet/planet-100217.osm.bz2

generate_image.py will also fail. I'm working out if I need to edit that or if there is a way I can allow the server to accept port 5432.

Port problems in postgres?

Port problems in postgres? Interesting. Gregory reported a similar problem above. I wonder what has changed?

What do you get from this?
grep -R "postgres" /etc/services

I have:

postgresql	5432/tcp	postgres	# PostgreSQL Database
postgresql	5432/udp	postgres

And from postgresql.conf?
grep "port" /etc/postgresql/8.4/main/postgresql.conf

I have:

port = 5432				# (change requires restart)

How do Updates work? I do

How do Updates work? I do have an openstreetmap installation of Europe and would like to upgrade now to world. Is it enough to just import the planet with osm2pgsql?

Yes. If you use osm2pgsql

Yes. If you use osm2pgsql to import the planet file you'll drop the old Europe database tables and start from zero with the planet file.

Ideas?

Thanks for the very well done tutorial. I am looking to implement an api to querry tiles from my local server. I'd also like to implement a similar functionallity to the OSM server where a user can querry my tile server via an http request. Do you have any suggestions? Thanks much!

XAPI

How can you attach XAPI to this local server?

SRID - psql -f ~/bin/osm2pgsql/900913.sql -d gis

Sir
I have run all steps successfully except the following one,which in turns has stopped me in installing osm server,

Set the Spatial Reference Identifier (SRID) on the new database.
#psql -f ~/bin/osm2pgsql/900913.sql -d gis

This command is giving the following problem:
/var/lib/postgresql/bin/osm2pgsql/900913.sql: No such file or directory

what is the reason of it..only this is not working..
Help me as soon as possile...

Thank's in advance.

Hi Parveen, Welcome. Use

Hi Parveen,

Welcome.

Use locate 900913.sql to find your location for the SRID file. Then use your location in the psql command.

mod_tile and update/ expiry?

Hi Richard,

this is a really neat tutorial and I often like to refer people to it to explain how to set up a tile server as it is the best documentation I have seen so far!

Now it would be really cool, if you could extend the tutorial to also include the setting up of mod_tile and renderd (potentially in a separate article). The iceing on top of the cake would of cause be if it even included an explanation of keeping the db up to date and expire updated tiles.... :-)

Thanks again

P.S. it would also be good if you could rename the article to "Build your own openstreetmap _tile_ server" to distinguish it from the rails_port server.

Hi apmon, and thanks! I

Hi apmon, and thanks!

I like the idea of adding an article on mod_tile, renderd and tirex. Plus minutely mapnik and expiry. One of these days! Anybody with a supply of extra 'Round 'Tuits would be nice. ;-)

Your suggestion on renaming for clarity is a good one. I'll put that renaming off until the next revision. I fear breaking incoming links.

generate_image.py creates bad png file

We setup OSM accrding to this article... when we run generate_image.py the resulting png wont open. Any ideas what might be wrong?

Thanks

Hi Kent, What error message

Hi Kent,

What error message do you see? Is any .png generated, or just a 0-length file?

update and thanks

*update needed*
It seems that now two more files are needed:
http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/...
http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/...
Done that...

... it works! Thanks for this guide :-)

Ciao

RuntimeError: 110m_admin_0_boundary_lines_land does not exist

RuntimeError: d:/SVN/mapnik/world_boundaries/110m_admin_0_boundary_lines_land does not exist (encoun
tered during parsing of layer 'necountries')

Cause: two required shapefiles are missing.

Workaround: edit layer-shapefiles.xml.inc and remove the necountries layer

OR

Download the shapes :-) ( thnx to VSK http://forum.openstreetmap.org/viewtopic.php?pid=85606 )

10m-populated-places.zip

110m-admin-0-boundary-lines.zip

I have updated the article

I have updated the article to include the additional shape files.

Let's leave your work around here, just in case.

The urls there got

The urls there got obfuscated some way. would be nice if you can fix this, so it's possible to use c&p

Error on last command

Thank you first for this superbly written guide.
I was able to follow it right up to the very last step, then error hits:

mn@IOSOSM:~/bin/mapnik$ ./generate_image.py
Traceback (most recent call last):
File "./generate_image.py", line 37, in
mapnik.load_map(m,mapfile)
RuntimeError: /home/mn/bin/mapnik/world_boundaries/110m_admin_0_boundary_lines_land does not exist (encountered during parsing of layer 'necountries')

Maybe you could give me a hint on what might have gone wrong.

Thank you.

Thank you, streitberg.

Thank you, streitberg.

Check the "Install prepared world boundary data" section again. I have updated it to include the additional shape files.

thanx

Thanx a lot for the excellent manual.
I was able to generate my own cyclemap for Belgium.
Thanx to Holland! http://git.openstreet.nl/ (hartelijk dank dat jullie dit ter beschikking stellen!)

I incorporated their overlay into osm.xml.

I managed to get tile-server up and running on localhost.
Copied it with Tangogps for off-line use :)

Thanks for the tutorial

Thanks for these steps! I believe this is the first Linux tutorial that I followed which actually works! Too bad I'm using a virtual machine with slow disks, so it's been importing the data for the last two days now ;-)

Anyone remember how many ways and relations it has to import? It's at "Node(704211k) Way(18843k) Relation(0k)" at the moment...

Don't find osm2pgsql

if i have the instruction ./osm2pgsql -S default.style --slim -d gis -C 2048 ~/planet/planet-100217.osm.bz2 it doesn't work, say's me "it doesn't exist osm2pgsql, why happen this, i follow all step's right.

if you can help me im very thnxs.

Hi jcervantes and thank you

Hi jcervantes and thank you for writing.

The ./osm2pgsql says to use the osm2pgsql command and to find the osm2pgsql command in the Present Working Directory. So this will only work if you are in the ~/bin/osm2pgsql directory.

Alternatively, you could specify the full path to osm2pgsql, and the default.style, like this

~/bin/osm2pgsql -S ~/bin/osm2pgsql/default.style --slim -d gis -C 2048 ~/planet/planet-100217.osm.bz2

Como acceder desde mi navegador al servicio montado de OSM?

Saludos...
Segui el tutorial para montar el servidor OSM, ya me genera el mapa de ejemplo: imagen.png, pero ahora quiero probarlo para que me aparezca el mapa desde el navegador... Como hago esto?

How to access my server OSM from the browser web?

Hi!
I followed the tutorial for mounting the OSM server, I generated the map example: imagen.png, but now I try to bring up the map from the browser ... As I do this?

Hi. Thanks you for writing.

Hi. Thanks you for writing. This tutorial does not address putting OSM on the web. With some requests and enough time, I'll write a tutorial for that.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <q> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <img> <h1> <h2> <h3> <h4> <pre> <sup> <sub> <blockquote>
  • Lines and paragraphs break automatically.

More information about formatting options