Difference between revisions of "Nominatim for Traffic Accident Database"

Revision as of 12:38, 5 March 2015

>> return Cedeus_IDE

Objective

Develop a script for automated geo-referencing of Chiles transit accident database.

Tools

• Nominatim
• MS-Access / LibreOffice : need to load the MS Access DB from Carabineros de Chile in OpenOffice/LibreOffice: http://askubuntu.com/questions/187389/is-it-possible-to-open-an-access-2010-database-file-without-using-wine-or-virtua/519571#519571
• iPython + Notebook

Installing Nominatim

• Nominatim installation manual: https://wiki.openstreetmap.org/wiki/Nominatim/Installation

Setup of Nominatim VM

1. copying basicubunutu1404.vdi file and renaming to nominatim.vdi
2. creating the nominatim VM on CedeusDB (ip.18):
   • VBoxManage createvm --name nominatim --ostype Ubuntu_64 --register
   • VBoxManage modifyvm nominatim --memory 4096
   • VBoxManage modifyvm nominatim --cpus 4
   • VBoxManage modifyvm nominatim --nic1 nat
   • VBoxManage storagectl nominatim --name "SATA Controller" --add sata --controller IntelAhci
   • assign the (old) disk image a new uuid

     VBoxManage internalcommands sethduuid nominatim.vdi

   • attach the (old) disk image:

     VBoxManage storageattach "nominatim" --storagectl "SATA Controller" --port 0 --device 0 --type hdd --medium nominatim.vdi

   • VBoxManage storagectl nominatim --name "IDE Controller" --add ide --controller PIIX4
   • set the nat rules (ports):

     VBoxManage modifyvm nominatim --natpf1 "ssh,tcp,,20022,,22"

     VBoxManage modifyvm nominatim --natpf1 "apache,tcp,,20080,,80"

     VBoxManage modifyvm nominatim --natpf1 "pgsql,tcp,,20432,,5432"

3. optional - set VRDE port:
   • VBoxManage modifyvm nominatim --vrdeport 7761 (Note: 3389 is the default port anyway)
4. check all settings with VBoxManage showvminfo nominatim
5. start the VM

   VBoxHeadless -s nominatim --vrde on &

   the VM will listen on port 7761 - connect with GTKvncViewer to check

   • check if ssh connection works
6. change the computers name in /etc/hosts and /etc/hostname
7. restarting Apache gives the error message: "apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 ..."

   => solve this by adding ServerName nominatim as the last line in /etc/apache2/apache2.conf file

8. shut the VM down to resize the HDD if necessary
   • check the HDD size: VBoxManage showhdinfo nominatim.vdi returns:

UUID:           e148d758-00ac-4b60-a067-d5c97316069d
Parent UUID:    base</br>
State:          created
Type:           normal (base)
Location:       /home/ssteinig/nominatim.vdi
Storage format: VDI
Format variant: dynamic default
Capacity:       100000 MBytes
Size on disk:   6224 MBytes
In use by VMs:  nominatim (UUID: be795392-3786-4f3b-9714-a9445f91855d)

• so I resized the disk to 200GB as, according to Nominatim infos the disk size for a full planet is 700GB

  VBoxManage modifyhd nominatim.vdi --resize 200000

Setup of Nominatim itself

Install needed libraries

• install PHP 5
• install Postgres
• install PEAR
• install GCC, GIT, protbuf, wget, etc.
• install Osmosis

=> see https://wiki.openstreetmap.org/wiki/Nominatim/Installation#Software

sudo apt-get install build-essential libxml2-dev libgeos-dev libpq-dev libbz2-dev libtool automake libproj-dev
sudo apt-get install gcc proj-bin libgeos-c1 git osmosis libgeos++-dev
sudo apt-get install php5 php-pear php5-pgsql php5-json php-db
instead of: >> sudo apt-get install postgresql postgis postgresql-contrib postgresql-9.1-postgis postgresql-server-dev-9.1
  use: >> sudo apt-get install postgresql postgis postgresql-contrib postgresql-9.3-postgis-2.1 postgresql-server-dev-9.3
sudo apt-get install libprotobuf-c0-dev protobuf-c-compiler

and perhaps

sudo apt-get install git autoconf-archive

Edit PostreSQL settings

perhaps edit Postgres settings (before data import) sudo nano /etc/postgresql/9.3/main/postgresql.conf

=> see https://wiki.openstreetmap.org/wiki/Nominatim/Installation#Software

My machine has only 4GB, so I changed:

• shared_buffers = 128MB => 512MB
• work_mem = 50MB (previously uncommented, and set to 1MB)
• maintenance_work_mem = 2GB (previously commented, and set to 16MB) //=> This value should be reduced again afterwards to avoid swapping when autovacuum runs.
• effective_cache_size = 2GB (previously commented, and set to 128MB)
• synchronous_commit = off (previously commented, and set to on)
• checkpoint_segments = 100 (previously commented, and set to 3)
• checkpoint_timeout = 10min (previously commented, and set to 5min)
• checkpoint_completion_target = 0.9 (previously commented, and set to 0.5)

and for initial import only:

• fsync = off (previously commented, and set to 'on')
• full_page_writes = off (previously commented, and set to 'on')

=> switch both values back to 'on' after import, to avoid database corruption.

Note, I did a restart of the postgresql service after to have the settings applied.

First Nominatim Installation

Get the source code

I used the last stable release 2.3.1:

wget http://www.nominatim.org/release/Nominatim-2.3.1.tar.bz2
tar xvf Nominatim-2.3.1.tar.bz2

and rename the folder

mv Nominatim-2.3.1 Nominatim

Compiling

now compile the software

cd Nominatim
./configure
make

Note, the warning about missing lua libraries can be ignored. Nominatim does not make use of osm2pgsql's lua extension.

Configuration

create the configuration file local.php in the sub folder settings with the following content:

<?php
  // Paths
  @define('CONST_Postgresql_Version', '9.3');
  @define('CONST_Postgis_Version', '2.1');
  // Website settings
  //@define('CONST_Website_BaseURL', 'http://mysite/nominatim/');
  @define('CONST_Website_BaseURL', 'http://146.155.17.18:20080/nominatim/');

for further params see the file settings/settings.php.

Download (optional) data

Wikipedia rankings (downloading took me 20+ mins @PUC)

Wikipedia can be used as an optional auxiliary data source to help indicate the importance of osm features. Nominatim will work without this information but it will improve the quality of the results if this is installed. This data is available as a binary download.

wget --output-document=data/wikipedia_article.sql.bin http://www.nominatim.org/data/wikipedia_article.sql.bin
wget --output-document=data/wikipedia_redirect.sql.bin http://www.nominatim.org/data/wikipedia_redirect.sql.bin

(=> requires to be in the Nominatim folder). wikipedia_article.sql.bin had a size of 1.2GB and redirect 240MB. Combined the 2 files are around 1.5GB and add around 30GB to the install size of nominatim. They also increase the install time by an hour or so.

Creating postgres accounts

Creating the importer account

The import needs to be done with a postgres superuser with the same name as the account doing the import.

sudo -u postgres createuser -s <your user name>

You should ensure that this user can log in to the database without requiring a password (e.g. using ident authentication). This is the default on most distributions. See trust authentication for more information. (sstein: it looks like it is not mean ident but trust identification... see below)

Some more PostgreSQL Access configs

Note: To have access from outside, I also changed in /etc/postgresql/9.3/main/postgresql.conf':

listen_adresses = '*'

And in /etc/postgresql/9.3/main/pg_hba.conf I did set

local   all             all                                     peer

to

local   all             all                                     ident

as well as:

host    all             all             127.0.0.1/32            md5

to

host    all             all       0.0.0.0       0.0.0.0         md5

(perhaps I also need to modify /etc/postgresql/9.3/main/pg_ident.conf at some point, if I want to map a system user name to a certain db user name)

Do a sudo service postgresql restart

Then I logged in and changed my ssteinig user password this way for the standard db postgres:

sudo -u ssteinig psql postgres
alter user ssteinig with password 'myNewPassWord';

Now I tested if I can connect to the DB:

• connection from command line to DB postgres: psql -U ssteinig -d postgres -h localhost

=> This worked, but it asked me for a password. I thought this should not happen after setting local to ident?

However, when I changed from ident/md5 to trust, I could login without password:

# "local" is for Unix domain socket connections only
local   all             all                                      trust
# IPv4 local connections:
host    all             all        0.0.0.0     0.0.0.0           trust
# IPv6 local connections:
host    all             all             ::1/128                  trust

got this from here: http://suite.opengeo.org/4.1/dataadmin/pgGettingStarted/firstconnect.html

• connect from outside (cedeusgis1) with pgAdmin (ssteinig, db: postgres, 20432)

  => This worked as well.

Create website user

Create the website user www-data:as a PostgreSQL database role

createuser -SDR www-data

For the installation process, you must have this user. If you want to run the website under another user, see comment in section Set up the website. You must not run the import as user www-data or root.

Nominatim module reading permissions

Some Nominatim Postgres functions are implemented in the nominatim.so C module that was compiled in one of the earlier steps. In order for these functions to be successfully created, PostgreSQL server process has to be able to read the module file. Make sure that directory and file permissions allow the file to be read. For example, if you downloaded and compiled Nominatim in your home directory, you will need to issue the following commands:

chmod +x ~/Nominatim
chmod +x ~/Nominatim/module

Note, the instructions say chmod +x ~/src/Nominatim ...but this must be for the installation from git source?

Import and index OSM data

First download a Planet File or a planet extract, for example from Geofabrik. Using a file in PBF format is recommended. I downloaded the Luxemburg Dataset for testing and then Chile. I used a cache of 1.2GB

Now start the import:

./utils/setup.php --osm-file /home/ssteinig/luxembourg-2015_2mar.osm.pbf --all --osm2pgsql-cache 1200 2>&1 | tee setup.log

=> This will create a new DB "nominatim" (with PostGIS extension enabled automatically). Importing the wikipedia articles was most time consuming. Together > 1h.

ToDo : reverse data import settings for PostgreSQL (see above)