DAMN micromapping tasking manager

¦ DAMN tasking manager (St-Cergue, Jura Mountains) ¦

For some reason, Chrome acquires tokens from the OpenStreetMap database server but Firefox does not. So use Chrome.

These are the commands to run DAMN.

Terminal 1

  • cd /home/user/damn-server/damn-deploy
  • dd if=/dev/urandom bs=8 count=8 | base64

This gives the 64 character long Postgres database <password>.

Paste this password in the .env file at /home/user/damn-server/.env

POSTGRES_PASSWORD= <password>

  • sed -i '/POSTGRES_PASSWORD=.*/d' .env
  • sudo docker-compose -f http.yml up db

Terminal 2 (server IP is 192.168.1.103)

  • cd /home/user/damn-server
  • source tve/bin/activate
  • sudo chmod 777 /var/run/docker.sock
  • export DB_HOST=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
  • gunicorn damn_server.api:app -w 1 -k uvicorn.workers.UvicornWorker -b "192.168.1.103:8000"

Open DAMN Manager in browser

  • https://192.168.1.103/damn-manager

There are many self-hosted HOT Tasking Manager instances which are used to coordinate OpenStreetMap (OSM) mapping. Actively supported alternatives are the Simple Tasking Manager (STM) and Divide & Map - Now (DAMN). All three are open-source client-server applications and each has its advantages and disadvantages.

Overall DAMN is preferred since it makes use of fairly straightforward components (i.e., several repositories) that are familar to those operating OSM tile servers and OSM websites used to edit an OSM database by means of well-established mapping clients such as JOSM. Moreover, DAMN is not necessarily heavily dependent on Docker images, and the manager and client front-ends constitute straightforward Javascript with little embelishment. There are obviously self-hosted DAMN server instances operating but a comparatively simple description as to how to set up the front- and back-ends seems to be missing.

The interest in a self-hosted DAMN tasking manager stems from the need to manage comprehensive micromapping of a village in Switzerland and, to some extent, of the surrounding area in the Jura Mountains.

DAMN server

For a DAMN server on Ubuntu 18.04 LTS, damn-server was cloned ("git clone https://git.sr.ht/~qeef/damn-server") and following the repository's summary, a Python virtual environment created in the directory "/home/user/damn-server". In the directory, damn-deploy was cloned ("git clone https://git.sr.ht/~qeef/damn-deploy") and the requirements installed:

  • cd /home/user/damn-server
  • python3 -m venv tve
  • source tve/bin/activate
  • pip install -r requirements.dev.txt

It was necessary to install Python 3.8, use update_alternatives to make sure that Python 3.8 was being used, and install the latest version of pip:

  • python3 -m pip install --upgrade pip

asyncpg also had to be installed

  • pip install asyncpg

docker-compose is needed:

  • sudo apt install docker-compose.

As it was planned to reverse proxy the DAMN server behind a web server (apache2 in the present case), gunicorn is needed:

  • pip install gunicorn

As summarised in the summary, in damn-deploy (terminal command: "cd damn-deploy") the ".env" file needs to be edited. In a terminal command:

  • dd if=/dev/urandom bs=8 count=8 | base64

is executed three times to generate hashes for POSTGRES_PASSWORD, JWT_SECRET and SESSION_SECRET.

Hashes are also needed for the OpenStreetMap website OAUTH_CONSUMER_KEY and OAUTH_CONSUMER_SECRET variables in "/home/user/dam-server/damn-deploy/.env". The procedure for both a self-hosted OSM website and the official OSM website is the same. Following login, under My Settings there is link to "oauth settings" where the DAMN client application can be registered in order to obtain a Consumer Key and Consumer Secret for the DAMN application.

The only other entry in ".env" that may need changing is DOMAIN_NAME.

The configuration file "conf.py" in "/home/user/damn-server/damn-deploy" also needs editing. The changed entries are:

  • DAMN_SERVER = "https://server.peterboswell.net"
  • OAUTH_CONSUMER_KEY and OAUTH_CONSUMER_SECRET as above;
  • OUATH urls (token, authorize, request token, and base) to refer to "https://edit.peterboswell.net", the OSM website

To create the postgresql database, a sed terminal command:

  • sed -i '/POSTGRES_PASSWORD=.*/d' .env

places the Postgresql postgres password in ".env" on output (and deletes the password from ".env");

  • docker-compose -f http.yml up db

creates the database "damndb". In principle, creating the database can be done manually, but since the routine is fairly standard, using Docker is reasonable.

As explained in the damn-server repository summary, the address of the damndb database must be set for all terminals that interact with damn-server:

  • export DB_HOST=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)

The file "acme.json" should be created in the "damn-deploy" directory with only read/write permissions for the user (called "user" in the present case).

In a separate terminal running in the virtual environment, after running tests (see the repository summary), the damndb address may need to be set (as above) and the DAMN server is started:

  • gunicorn damn_server.api:app -w 1 -k uvicorn.workers.UvicornWorker -b "192.168.1.100:8000"

which binds the server to port 8000 on a server with a local address ("192.168.1.100" in the present case). It is then useful to run the integration tests (see the summary).

Somewhat surprising is that the damndb Docker image was anonymised (located in the directory "/var/lib/docker/volumes/damn-deploy_dammndb-volume/_data/" and not in "/var/lib/postgresql/data" as would be expected from "/home/user/damn-server/damn-deploy/http.yaml"). Fundamentally this is not a problem.

The postgresql database can be accessed with the terminal command:

  • psql "postgresql://damnuser:$(cat .env | sed -n 's/^POSTGRES_PASSWORD=\(.\+\)$/\1/p')@$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)/damndb"

which asks for the password ("pass" for the user "damnuser").

The damndb Docker container name can be found with:

  • docker container ls -all
  • docker inspect --type container -f '{{range $i, $v := .Mounts }}{{printf "%v\n" $v}}{{end}}'

and the container (and database) can be deleted with:

  • docker-compose rm -sfv postgres

DAMN manager

For the manager front-end, instead of getting involved with more Docker images, the simplest is probably to git clone https://git.sr.ht/~qeef/damn-manager.js and copy the files in the directory "static" to a web (apache2) directory (e.g., "/var/www/html/damn-manager"). In the file "f.js", the DAMN_SERVER variable needs to be changed (to "http://192.168.1.100:8000" in the present case)

DAMN client

Similarly, for the client front-end https://git.sr.ht/~qeef/damn-client.js can be git cloned to say "/var/www/html/dam-client". Then:

  • the damn_api variable in "index.html" needs to be changed (to "https://server.peterboswell.net" in the present case case);
  • the variable "link" under "switch" "case" "id" in the file "fn.js" needs to be changed to the local address of the OSM server ("http://192.168.1.100:3000" in the present case);
  • references in the file "html.js" to www.openstreetmap.org and damn-project.org should be changed (to "edit.peterboswell.net" in the present case). It is also useful to add;

+ "&map=18"
+ "/" + ((mm.latmax - mm.latmin)/2 + mm.latmin)
+ "/" + ((mm.lonmax - mm.lonmin)/2 + mm.lonmin);

to the link address for the id editor, as is done for the Rapid OSM editor in the same file. Without this change the OSM website does not zoom in on the area or square that was selected by the DAMN client (or by the DAMN mappy client discussed below).

DAMN review ("mappy")

As pointed out by the DAMN project, managers and reviewers often use the DAMN mappy front-end to decide what should be done about the mapping of a particular square of a map area (or indeed of an entire area of a map). Mappy together with script to load a panel in the DAMN client are part of the DAMN quality assurance package.

For mappy, the simplest is to click the download link at the foot of the https://qa.damn-project.org/mappy/ page and then copy the downloaded file "index.html" to the web server directory ("/var/www/html/damn-qa/mappy" in the present case). Important to change are:

  • the link to the DAMN api;
  • D.s = dam_api("https://server.peterboswell.net","https://finished.peterboswell.net");
  • the "id" "edit" "case" link (as described above for the DAMN client);
  • the link called "imgurl" to a folder called "damn-osmsm" in a web directory that stores images of the areas that are being mapped and are displayed by mappy. In the present case: "imgurl : 'https://map.peterboswell.net/damn-osmsm' ".

These images are in fact created in a DAMN server directory ("/home/user/damn-server/osmsm") and are copied to the web directory.

Creating the map images requires the damn-server terminal command:

  • ./damn-client.py/osmsm_gen.py https://server.peterboswell.net

to give in "/home/user/damn-server/osmsm" .png files of areas that have been created for mapping using the DAMN manager ("damn-client.py" is a directory containing useful scripts that is included in the damn-server repository).

So that the map image is the background map used for the OSM website, the simplest is to edit "/tve/ lib/python3.8/site-packages/ststicmaps" files. Add say "tile_provider_New" to "tile_provider.py" and change "tile_provider_OSM" to "tile_provider_New" in "context.py" and "cli.py" and in "osmsm_gen.py".

DAMN client panel

The DAMN client has a link to a panel giving information about the area or square being mapped as well as additional links. Access to the panel is only available once the client has been authenticated (by clicking the authentication link that takes the client to the OSM website to authorise editing the area or square).

As for mappy above, the panel script can be displayed and copied in a browser by clicking a "View source" command in the browser. In the present case, the script is copied to the directory at "/var/www/html/damn-qa/panel". Several variables need changing such as:

  • the id editor link discussed above;
  • D.s = dam_api("https://server.peterboswell.net","https://finished.peterboswell.net") as discussed above;
  • "var dml = 'dam-manager' "; as discussed above.

JOSM DAMN plugin

An important component of the DAMN ecosystem is the JOSM DAMN plugin. For a self-hosted DAM server it is necessary to change the plugin so that a self-hosted OSM website can be used to authenticate access and store edits to an OSM database. Unfortunately, it was not immediately evident as to how to build the latest version (0.11.3) of the plugin from the SourceHut DAMN plugin repository.

Fortunately, an older version (version 0.8.1; fa87b01b of 26 December 2022) is available on GitLab corresponding to version 93f4f97c of the current SourceHut repository (see log).

Adjusting the plugin's authentication server is simply a matter of changing the "damn.server_url" variable in the DamnPlugin.java file in the repository's "src" directory from "https://www.openstreetmap.org" to "https://edit.peterboswell.net" in the present case.

Once the change is made, with the plugin cloned to the directory "/home/user/damn-plugin", since the JOSM plugin is based on the gradle-josm-plugin a new damn.jar file can be built using the terminal command:

  • ./gradlew localDist

A "damn.jar" is built and stored in the directory "/home/user/damn-plugin/build/dist" . The DAMN JOSM plugin is loaded into JSOM by copying "damn.jar" to the JSOM plugin directory (e.g., for Windows10: "\Users\admin\ AppData\Roaming\JOSM\plugins").

It is unclear why the latest DAMN JOSM plugin could not be built and which change since the 0.8.1 version might be responsible. The issue may be related to kotlin (error: "could not get unknown property kotlin_version 2") or translations (error: "PoCompile property outDir directory is missing").

Summary

In summary, setting up a self-hosted DAMN tasking manager is fairly straightforward, although it has to be said that there are several features that still need to be implemented and tested. The essentials however are available, and are turning out to be very useful in micromapping Saint-Cergue. Full credit for the Divide & Map - Now project is acknowledged.

Updated 1 July 2023

PeterBoswell.com