Introduction

The application of OpenStreetMap (OSM) technologies at a relatively local scale (see Jura mapping) for use with various aspects of construction projects needs to be able to track the evolution of tags and attributes that are assigned to spatio-temporal map objects. The OSHDB data store combined with the ohsome API  possibly offers the most flexible framework for monitoring this evolution. ohsome2label which uses ohsome and OSHDB  to identify "labelled" regions ( primarily for training samples for satellite imagery machine learning)  illustrates the types of local-scale applications that are supported by OSHDB and the ohsome API. 

While the implementation of these technologies is fairly straightforward, there are some aspects that benefit from a presentation of a simple workflow.  

At the global level, new OSM data are added continuously, while existing data are modified, and stale or incorrect data are removed such that the history of each object that ever existed in OSM lives on in the main OSM database. At a local level, a project using OSM technologies has local databases to store both locally generated data and OSM data. Being able to monitor, retrieve and analyse this data as clearly necessary.

A "latest full history planet file" comprising OSM's complete editing history is available, along with regional  and city-level extracts from various sources. So-called diff files that give the complete set of changes to OSM between two points in time are also available.

The use case we are dealing with is where a project has databases for rendering raster and vector map tiles as well as databases for an OSM "Rails Port" to manage the local editing of a project map. The OSM database will be synchronised with the rendering databases and with other backend datasources such as the databases used by oshdb API and by the Overpass API to serve many different types of applications

For our use case, xml files downloaded from the OpenStreetMap Rails port website database (called "openstreetmap" in descriptions of standard installations) are used to update the various backend databases.

OSHDB ETL


The core component of the OSHDB framework is the OSHDB ETL (extract - transform - load) tool that generates a local database of OSM data, together with its history, from an "openstreetmap" database.

The workflow given below is for Ubuntu 18.04LTS, for a user "ertert". 

It is best to build the OSHDB ETL from source using Maven (see instructions). Although the focus is the ETL tool, the complete OSHDB must be cloned:

  • git clone https.//github.com/GIScience/oshdb.git

The Maven pom file in /home/ertert/oshdb needs adjusting according to the instructions. Moreover, in the HeiGIT snapsots repository declaration, old versions needed "libs-snapshot-local" to be replaced by "libs-releases-local". This change is not needed in the latest OSHDB version (version 0.5.10, although we have problems compiling this new version).

Before using ETL, as per the instructions, in /home/ertert/oshdb:

  • mvn clean build

It is wise to test the ETL using a reliable OSM history file. History files for say European countries are available from Geofabrik, where one must login using an OSM account.  Download an .osh.pbf file to /home/ertert/oshdb/oshdb-tool/etl and to extract data, run as terminal commands in /home/ertert/oshdb/oshdb-tool/etl:

  • mvn clean compile
  • mvn exec:java -Dexec.mainClass="org.heigit.bigspatialdata.oshdb.tool.importer.extract.Extract" -Dexec.args="--pbf /home/ertert/oshdb/oshdb-tool/etl/switzerland-internal.osh.pbf -tmpDir ./tmpFiles --timevalidity_from 2017-01-01" -Dexec.cleanupDaemonThreads=false

Terminal output for extraction is:

  • extracting key table ...sorting tags by frequency .. done!
  • sorting roles by frequency ... done!
  • extract done in 2.157  s

Before the transformation step the file /home/ertert/oshdb/oshdb-tool/etl/extract_meta needs to be edited. One is not sure what is possible, so in general it is probably best to change the line: 

  • extract .region={"bbox":[]}

to:

  • extract.region= {"type": "Polygon","coordinates": [[[-10.0, -10.0], [10.0, -10.0],[10.0, 10.0],[-10.0, -10.0]]]}

where the coordinates of this GeoJSON geometry object correspond to the maps bounds.

The map bounds are given by the bounding box given by the Osmium extended information command. With Osmium installed, the terminal command that reports the coordinates  is:

  • osmium fileinfo -e /home/ertert/oshdb/oshdb-tool/etl/switzerland-internal.osh.pbf

For the transform step, one generally needs to increase the Java heap size before running the step: 

  • export _JAVA_OPTIONS=-Xmx4096m
  • mvn exec:java -Dexec.mainClass="org.heigit.bigspatialdata.oshdb.tool.importer.transform.Transform" -Dexec.args="--pbf /home/ertert/oshdb/oshdb-tool/etl/switzerland-internal.osh.pbf  -tmpDir ./tmpFiles"

Terminal output is:

  • Transform:
  • available memory; 5447 mb
  • Found Way at 8520455
  • Found Relation at 13695401
  • maxMemory for transformation: 5444 mb
  • start transforming nodes ... COMPLETE
  • transformer save ToDisk ./transform_node_==-== ... done! 34788392 bytes

(repeated for ways and relations)

  • transform done in 8.307 s

For loading (into an H2 database) H2 is needed. Installation of H2 at say /home/ertert/oshdb/h2 with server.properties at /home/ertert.h2.server.properties is straightforward. The terminal command to load is:

  • mvn exec:java -Dexec.mainClass="org.heigit.bigspatialdata.oshdb.tool.importer.load.handle.OSHDB2H2Handler" -Dexec.args="-tmpDir ./tmpFiles --out /home/ertert/oshdb/oshdb-tool/etl/switzerland-internal --attribution '© OpenStreetMap contributors' --attribution-url 'https://www.openstreetmap.org/copyright'"

Note that the H2 database name is not given an extension (the extension .mv.db is added during execution).

Terminal output during execution speaks of "load tags ... loading roles ... loading to grid" followed by a succession of insert statements of the type "insert 15:4123456 done!" and finishing with "loading done in 9.387 s".

History files


The files needed for the present use case are those generated from the OSM Rails port Postgresql database (called "openstreetmap") with the Rails port directory at "/home/ertert/osm-site". For a localised construction project, the map area is generally comparatively small. For the rendering and Overpass databases, after edits have been made, the usual practice is therefore to output the entire database rather than changes. Using Osmosis, the terminal command is:

  • osmosis --read-apidb host="localhost" database="openstreetmap" user="ertert" password="ertert"  validateSchemaVersion="no" --write-xml file="/home/ertert/osm-site/full_database.xml"

For OSHDB, preserving history data reliably seems to require:

  • osmosis --read-apidb-change host="localhost" database="openstreetmap" user="ertert" password="ertert" intervalBegin="2017-01-01_00:00:00" validateSchemaVersion="no" --write-xml-change file="/home/ertert/osm-site/oshdb_database.xml"

This limitation of history data to relatively recent data is not an issue for the use case under consideration.

It is also found that conversion to the .osh.pbf history format required by OSHDB should be carried out using Osmium:

  • osmium  cat /home/ertert/osm-site/oshdb_database.xml -o /home/ertert/osm-site/oshdb_database.osh.pbf
oshdb_database.osh.pbf can then be processed using the OSHDB ETL workflow described above.
ohsome API


The ohsome API set-up instructions make it clear that Java 11 or higher is needed. For use with Maven, it is only necessary to clone/download the src directory and the pom.xml file to say /home/ertert/ohsome  

The terminal command at /home/ertert/ohsome:

  • mvn -DskipTests=true clean package

builds the project, which is run using the command:

  • java -jar /home/ertert/ohsome/target/ohsome-api-1.3.0-SNAPSHOT.jar --database.db=/home/ertert/osm-site/oshdb_database.mv.db

ohsome API is then available at http://localhost:8080, with swagger documentation at http://localhost:8080/swagger-ui.html

On the swagger page,  the selection of say "Count - Compute the count of point/linear/polygonal element OSM elements"  followed by clicking "POST /elements/count/groupBy/tag" and then "Try it" on the next page gives a page showing parameters. The default is for buildings for each year since 2017. 

Clicking "Execute" gives the the Request URL string, which when submitted via a browser address bar gives the reults. If the bounding box parameters ("bboxes") correspond to an area that is outside the map area a error may be thrown or the values are zero. It is probably wise to have a map area for the OSHDB database that is a bit larger than the bboxes setting for ohsome API. 

ohsome2label


ohsome2label can be set up in a virtual environment, say in /home/ertert/ohsome2label, using the terminal command:

  •  git clone https://github.com/GIScience/ohsome2label.git

in /home/ertert. Then cd to ohsome2label and run:

  • python3 -m venv .
  • source bin/activate

Then install ohsome2label with:

  • pip install ohsome2label

An example config.yaml file is provided. In the osm section, the url can be changed to point to the local ohsome API at http://localhost:8080/elements/geometry and the bboxes entry to the map area.

For the image section, a bing virtualearthnet api_token can be found on the web.

The terminal commands to create  geojson (vector) files in  /home/ertert/ohsome2label/result/other/raw corresponding to the tags designated in config.yaml is:

  • ohsome2label --config config/config.yaml - v vector

The corresponding commands:

  • ohsome2label --config config/config.yaml - v label

and

  • ohsome2label --config config/config.yaml - v image

create labels (as geojson files in /home/ertert/ohsome2label/result/other/tiles) and images (as .png files in /home/ertert/ohsome2label/result/image). 

To create  tiles, we find it more convenient to create a directory called say "out":

  • mkdir /home/ertert/ohsome2label/out 

and to replace the line: 

  • if not os.path.exists(dir)

in def visualize_overlay in

/home/ertert/ohsome2label/lib/python3.6/site-packages/ohsome2label/visualize.py

with the line

  • dir=os.path.join('/home/ertert/ohsome2label/out/' ,str(i))

The command:

  • ohsome2label visualize -t overlay

creates tiles for satellite images overlayed by labelled areas in the  /home/ertert/ohsome2label/result/preview  directory. The tiles' file names start with the zoom level specified in config.yaml (e.g., "15.16974.11587.png") , so the process needs to be repeated for several zoom levels. 

Tiles for each zoom level can be placed in separate directory for each zoom level and for each each coordinate ("16974" in the case of "15.16974.11587.png") and  copied to a directory of an apache2 web server (say /var/www/html/label) for serving as {z}{x}{y}.png web tiles using an appropriate index.html file (see maps.peterboswell.net/label showing labelled built-up areas in the Jura Mountains).

We find that ohsome2label is useful for creating geojson files for features such as buildings.

Updated 24 November 2020


Email: peter@peterboswell.com
Phone: +41792989666
Skype: live:petergboswell
WhatsApp: PeterBoswellcom
Messenger: petergboswell
Telegram: PeterBoswellBot
LinkedIn
Facebook
Twitter
OneName