Introduction

The Overpass API serves various various Overpass downstream tools for the Jura Mountains mapping. Summarised here is the installation and operation of Overpass API and the various downstream tools.

It should be noted that the use case for this Overpass API installation is not to continuously make available new OpenStreetMap diff updates. Instead, the aim is to simplified installation and set up to service downstream tools over a limited time and spatial extent for a specific project such as a  construction project for infrastructure.


Overpass API

The Overpass API is currently in version 0.7.55. Installation on Ububtu 18.04 LTS following the Overpass installation guide, the OSM wikiBreki , and ZeLonewolf is straightforward.

With an apache2 webserver already installed,  install a C++ compiler and the expat library with the terminal commands:

  • sudo apt-get install g++ make
  • sudo apt-get install expat libexpat1-dev zlib1g-dev

In a directory /home/user/op for user "user", get and decompress Overpass API:

  • wget http://dev.overpass-api.de/releases/osm-3s_v0.7.55.tar.gz
  • tar -zxvf ../Downloads/osm-3s_v0.7.55.tar.gz

For simplicity, change the directory name osm-3s_v0.7.55 to osm3, move to  osm3 and allow root permission:

  • cd osm3
  • ./configure --prefix="`pwd`" 

and compile with:

  • make 

and set permissions:

  • chmod -R 755 /home/user/op

To configure apache2, edit with say nano:

  • sudo nano /etc/apache2/apache2.conf

by adding:

<Directory /home/ertert/op/osm3/html>
Options FollowSymLinks
AllowOverride All
Order deny,allow
Allow from all
</Directory>

Open a port (say 8091):

  • sudo nano /etc/apache2/ports.conf

by adding:

  • Listen 8091

Create and edit a file called "overpass.conf" :

  • sudo nano /etc/apache2/sites-available/overpass.conf

with:

....................................

<VirtualHost *:8091>
   ServerAdmin webmaster@localhost
   DocumentRoot /home/user/op/osm/html
   ScriptAlias /api/ /home/usert/op/osm3/cgi-bin/
   <Directory "/home/user/op/osm3">
      AllowOverride None
      Options Indexes FollowSymLinks
      Require all granted
   </Directory>
   <Directory "/home/user/op/osm3/cgi-bin/">
      AllowOverride None
      Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
      Require all granted
   </Directory>
   ErrorLog ${APACHE_LOG_DIR}/error.log
   CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

....................................

Install cgi and the filter module:

  • sudo a2enmod cgi
  • sudo a2enmod ext_filter

Then load overpass and restart apache2:

  • sudo a2ensite overpass.conf
  • sudo service apache2 reload

The OverpassAPI database is populated using .bz2 files. bzip2  can be installed with:

  • sudo apt update
  • sudo apt-get install bzip2

and an OSM history (full metadata) file is compressed. The Jura Mountains  was started with the file switzerland-internal.osh.pbf file from Geofabrik that was clipped with osmium-tool and saved as an osm file:

  • osmium extract -b 5.95,6.55,46.30,46.80 --with-history --set-bounds /home/user/files/switzerland-internal.osh.pbf -o /home/user/files/switzerland-internal.osm
  • bzip2 /home/user/files/switzerland-internal.osm

To populate the Overpass API database, keeping the history, in /home/user/op/osm3 run:

  • nohup /home/user/op/osm3/bin/init_osm3s.sh /home/user/files/switzerland-internal.osm.bz2 /home/user/op/osm3/db /home/user/op/osm3 --meta & tail -f nohup.out

To use Rules with Overpass API, copy the Rules directory which is in the github Overpass src directory into the database directory /home/user/op/osm3/db

To run overpass API with history (namely with what is called "attic support"), run:

  • nohup /home/user/op/osm3/bin/dispatcher --osm-base --attic --db-dir=/home/user/op/osm3/db &

"attic" refers to prior versions of data in an OSM map.  In order to query changes with time of an object using diff and adiff (augumented diff) commands attic support is needed (see OSM wiki and Overpass-Turbo query example at http://overpass-turbo.eu/s/dMf). For the use case consideredhere, it is useful to be able to monitor the evolution of a project with time.

The Overpass API dispatcher is started if the lock file osm3s_v0.7.55_osm_base appears in the db directory and/or in /dev/shm.

A simple test of the web API is to run:

  • wget --output-document=test.xml http://local:8091/api/interpreter?data=%3Cprint%20mode=%22body%22/%3E

which gives:

....................................

--2020-12-05 23:43:51--  http://192.168.1.118:8091/api/interpreter?data=%3Cprint%20mode=%22body%22/%3E

Connecting to local:8091... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/osm3s+xml]
Saving to: ‘test.xml’
test.xml   [ <=>  ]  249  --.-KB/s  in 0s  2020-12-05 23:43:51 (30.3 MB/s) - ‘test.xml’ saved [249]

....................................

where test.xml says that OSM data is made available under licence.

Another simple test is to run:

  • /home/usert/op/osm3/bin/osm3s_query --db-dir=/home/user/op/osm3/db

and as indicated by the prompt, to copy-paste the following into the prompt:

  • node["amenity"="restaurant"]; out skel;

followed by typing Control D. This should return a list of nodes (of restaurants).

Useful commands are:

  • /home/user/op/osm3/bin/dispatcher --status

and

  • /home/user/op/osm3/bin/dispatcher --terminate

which also removes the lock file or files.

A useful additional functionality is the management of areas offered by the Areas extension of Overpass API. It constitutes a new data type area besides the OSM data types node, way and relation that is computed by the Overpass API server. Areas allow a search to be carried out within an OSM way or relation.

Areas are made available with:

  • nohup /home/user/op/osm3/bin/dispatcher --areas --db-dir=/home/user/op/osm3/db &

It may be nececessary to set permissions:

  • sudo chmod 666 /home/user/op/osm3/db/osm3s_v0.7.55_areas

Rules are run from /home/user/op/osm3 using:

  • nohup  /home/user/op/osm3/bin/rules_loop.sh ../osm3/db &



Map changes

Broadly speaking, two approaches are used to develop downstream tools for monitoring OSM map changes:

  • external API;
  • osm-qa-tile + tile-reduce analysis stack.

Focusing here on the use of external API's, there are several widely used approaches:

  • openstreetmap.org/api
  • overpass 
  • ohsome.

Although Overpass API downstream tools are available, classification is difficult because many of the tools combine more that one approach, more than one use case and for some applications both frontend (client-side) and backend (server-side) components. Tools include:

For the use case being considered here, namely construction activities such as project planning, management, certification and procurement,  one assumes that Overpass, OpenStreetMap and ohsome API's will have been set up (see the list of tools). An additional fairly complex installation of a frontend-backend system such as OSMCha to monitor map changes is probably overkill. Instead, we focus on relatively simple, pure Javascript browser-based applications that have map-style user interfaces and can be added as a directory to a web server directory (apache2 in our case).

Accordingly, Latest-Changes is made available. It uses both the OpenStreetMap and Overpass API's and links to a slightly simplified local version of Achavi and to a local OpenStreetMap site to provide further details of edits. 

Being able to subdivide a project area into small areas that can monitored easily and quickly in much the same way as the ohsomeHeX history explore would be useful. Achavi allows a small area to be selected and this partly meets the need. Better would be OSM Edits Per Tile for which only the frontend code is available. Extending WhoDidIt may be a possibility.


ohsome API

To somewhat complete this overview, something should be said about the ohsome API.  The ohsome API uses oshdb databases to offer powerful analytical capabilities for OSM data aggregation, data extraction and user statistics. It is described elsewhere how oshdb databases are created along with an ohsome API. Together they provide what the developers refer to as an ohsome OSM History Analytics Platform (see list of applications).

The platform's main frontend applications are ohsomeHeX for history analysis and the ohsome Dashboard, but packaged source code is not available.

5 December 2020


Email: peter@peterboswell.com
Phone: +41792989666
Skype: live:petergboswell
WhatsApp: PeterBoswellcom
Messenger: petergboswell
Telegram: PeterBoswellBot
LinkedIn
Facebook
Twitter
OneName