GIS Open Source versioning tool for a multi-user Distributed Environment (part 1)
Editors note: This article has been jointly written by Nicolas Gignac, Dami Sonoiki (eHealth Africa) and Samuel Aiyeoribe (eHealth Africa). These authors would like to thank Yves Moisan, Vincent Mora (Oslandia) and Vincent Picavet (Oslandia) for their support and input.
Cet article est aussi disponible en français.
In today’s digital world, organisations are effectively working to publish clean and up-to-date dynamic geospatial data. This data needs to be time-based in order to track changes in multiple versions within short periods. Finding the best tool that adapt seamlessly to one’s own organisation’s workflow and capability has been a challenge faced by many organisations.
Managing geospatial data processing within a large team of editors in a distributed work environment can be complex especially with regard to conflicting modifications related to geometries or attributes. When managing a centralised database or an open crowd-sourcing environments, editors may like to track changes and edit data while disconnected from the centralised master database, using source code versioning like tools.
A number of tools already exist in the market to tackle versioning, for instance ArcGIS versioning database in the ESRI world. Unfortunately, this option requires buying multiple Desktop and Server licences.
In the OpenStreetMap platform, Overpass queries can reveal modification history. However the tool cannot be used for an in-house database. There is also GeoGig and PGversion as Free and Open Source Software for Geospatial (FOSS4G) tools, but both lack functionality to fulfill the above described purposes. The setup involves using PostGIS as the main repository by having the ability to edit specific geographic sections of a data layer and to work offline without having to write any lines of code from the editor.
Solution development
To meet these requirements while reducing licensing costs two organisations: Valcea County in Romania and eHealth Africa (eHA) in Nigeria used FOSS4G and funded the development of a new QGIS versioning plugin in 2015: https://plugins.qgis.org/plugins/qgis_versioning/. The new improved and simple to use QGIS versioning plugin was developed and integrated into their production environment since 2016.
Figure 1. Versioning plugin workflow.
The technical contributions of this development came in two phases. The plugin was initially developed in 2014, by Oslandia, a company from France, for the GIS office of Apavil, Valcea County in Romania. In 2015 and 2016 to meet eHA’s business requirements, the plugin was further improved with support from eHA’s GIS team and Yves Moisan, a freelance Python GIS developer from Sherbrooke (Canada) contracted by eHA. The goal was to develop a tool to manage history of geodata stored in PostGIS using QGIS, but also enable the creation of geographic filtered working copies in SpatiaLite file for offline edition. This plugin is based mainly on ogr2ogr (GDAL) open source library and SQL command line.
A typical use case met by the plugin involves one or more users checking out a (local) working copy to edit features in (potentially) offline mode using SpatiaLite file (in roam, mobile apps or in QGIS) and synchronising (push commit) changes back to the main PostGIS server. All changes are tracked just like in source code management systems like CVS (Concurrent Versions System) so the data manager can see the work that has been done and view features showing the differences between revisions.
Target workflow
Before deciding to enhance the original Oslandia plugin, use cases were documented and different pre-existing solutions were assessed. It was decided that Oslandia would develop a new QGIS versioning plugin to meet the most critical needs of both the County of Valcea and eHealth Africa (having one of the biggest active and competent GIS department based in West Africa). Features included were allowed for the ease of use, user edit tracking, managing data conflicts and version control. The plugin, at its version 0.7 (April 2017), can be used with this workflow:
- Activate the PostGIS Database schema planning to receive edits and user management-permission.
- Select features before checkout, based on a geographical extent (ex. municipality, region, etc.) in QGIS (if needed), in SpatialLite file.
- Edit the SpatiaLite within a mobile app or directly in QGIS.
- Compare the new version to a current one in PostGIS by viewing differences before committing work back to the main PostGIS database.
- Synchronise SpatiaLite file to the main PostGIS database and save a new version by automatically storing information related to the Editor username and its commit timestamp.
Figure 2. QGIS Versioning plugin schema workflow.
This new tool and its use cases were presented at the last International FOSS4G conference held in Bonn, Germany in August 2016. Dami Sonoiki, eHA’s GIS Manager, presented this approach of multi-user distributed data management at the FOSS4G 2016 in Bonn.
Picture 1. eHealth Africa GIS department in action during massive edition Vaccine campaign.
In fact, the NGO’s GIS team working in Nigeria formerly employs proprietary database management tools and migrated to an hybrid open source solution during 2016. This was done by making iteration with GIS “champion” of the organisation and then to a final implementation across the GIS team in mid-2016. The plugin has recently been successfully tested for massive data updates to fulfill eHealth Africa’s requirement of being able to update many data tables (more than 30) and to synchronise large modifications (millions of complex polygons).
How to try it or contribute?
For more information on this new plugin, you can test and install the QGIS versioning plugin directly in QGIS (Menu: Plugins=Manage plugins: Versioning) or go on this documentation site: http://qgis-versioning.readthedocs.io/en/latest/ or download the source code on GitHub: https://github.com/Oslandia/qgis-versioning. You can also contribute to the source code by sending pull request or open issues if you have any comments or bug to report.
