On-demand transformation of Open Street Map Data into common GIS formats

Back to GSoC2012 Projects page.

End result: An addition to the 52°North WPS to transform OSM data in various GIS formats on demand.

Student: Sarah Harvey

Mentor: Benjamin Proß

Subversion repository: https://svn.52north.org/svn/geoprocessing/incubator/osmtransform/

Weekly Report

Files

Original Proposal


Installation

Installation of osmtransform component is rather easy given an existing installation of WPS. There are several scenarios for installation, from source and from binary.

Installation from binary

Installation from full source

  • Grab a copy of WPS (e.g. from repository):
    svn export https://svn.52north.org/svn/geoprocessing/main/WPS/trunk/WPS/
  • Check out the latest osmtransform source from repository:
    svn export https://svn.52north.org/svn/geoprocessing/incubator/osmtransform/trunk/osmtransform/
  • Move the osmtransform/ folder into WPS/
    mv osmtransform/ WPS/
  • Edit the WPS pom.xml file and add the following in the dependencies section:
    <dependency>
    <groupId>org.n52.wps</groupId>
    <artifactId>osmtransform</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    </dependency> 
  • Edit wps_config.xml as per the Configuration section.
  • Run "maven install" to build the war file for deployment
  • Deploy the war

Installation from partial source

  • Check out the latest osmtransform source from repository:
    svn export https://svn.52north.org/svn/geoprocessing/incubator/osmtransform/trunk/osmtransform/
  • Run "mvn install -Dmaven.test.skip=true" to build the jar file for deployment
  • Copy the jar file into WEB-INF/lib directory:
    cp osmtransform/target/osmtransform-0.0.1-SNAPSHOT.jar /path/to/tomcatdir/WEB-INF/lib/
  • Edit wps_config.xml as per the Configuration section.

Configuration

Before the osmtransform component can be used, the WPS has to be configured to use it. To do this, you need to edit the wps_config.xml file and add some components.

Parsers

The core component of the osmtransform project is the parser. Add the following somewhere between the ParserList tags:

<Parser name="OSMVectorParser" className="org.n52.wps.osm.io.OSMVectorParser" active="true">
<Format mimetype="text/xml" schema="https://svn.52north.org/svn/geoprocessing/incubator/osmtransform/trunk/xsd/src/main/resources/osm.xsd" />
</Parser>

Algorithms

Some helper classes have been integrated into WPS to allow for on-the-fly conversion between formats. Additionally, some helper classes exist to interact with Overpass servers and converting their returned output into GML, KML, and Shapefiles. Add the following somewhere between the AlgorithmRepositoryList tags:

<Repository name="OSMAlgorithmRepository" className="org.n52.wps.server.osm.OSMProcessRepository" active="true">
<Property name="Algorithm"  active="true">org.n52.wps.server.osm.converter.OSM2Vector</Property>
</Repository>
<Repository name="OverpassAlgorithmRepository" className="org.n52.wps.server.osm.OverpassProcessRepository" active="true">
<Property name="OSM3S" active="true">http://www.overpass-api.de/api/interpreter</Property>
<Property name="Algorithm" active="true">org.n52.wps.server.overpass.algorithm.BBox</Property>
<Property name="Algorithm" active="true">org.n52.wps.server.overpass.algorithm.GenericQuery</Property>
</Repository>

As you can see here, we really have two types of conversion facilities; one supports direct conversion, and the other supports interaction with the Overpass servers. The OSM3S property is configurable to be set to any compatible Overpass server. For more information on the Overpass API, check the OpenStreetMap API.

Usage

Some sample queries exist in the WPS test client for osmtransform. For convenience of the user, some are posted here.

Inputs/Outputs for OSM format conversion

The algorithm org.n52.wps.server.osm.converter.OSM2Vector is really mostly a pass-through system that exists for convenience in converting between various formats. The main parameter that needs to be set is the "data" parameter, which should simply contain the contents of the OSM file. It is important to specify the schema and mimeType, as otherwise it may not be detected at all as a valid input format. The response parameter "result" should contain settings on what sort of format is returned. The response format then can basically be anything vector-based, as for the most part, the geometries should be converted.

The schema should be set to the following for now (subject to change): https://svn.52north.org/svn/geoprocessing/incubator/osmtransform/trunk/xsd/src/main/resources/osm.xsd

Inputs/Outputs for Overpass interaction

The algorithms in org.n52.wps.server.overpass.algorithm.* are mostly Overpass querying algorithms, with the option to return the output in either a WPS-supported vector format, or OSM. Each algorithm has its own parameters. There exists a GenericQuery algorithm that will passthrough Overpass queries (specified using the parameter "query") directly to the server, then return the output in a user-specified format. The BBox algorithm exists mostly as a model on how to create future algorithms.

Common to these algorithms are the response or output parameters. There are currently three:

  • result - the response from Overpass in any vector-based format supported by WPS
  • osmresult - the original osm response from Overpass
  • osmquery - the osm query that was sent to Overpass
Any or all of these values may be used.

GML conversion using inline OSM data

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<wps:Execute service="WPS" version="1.0.0" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:ogc="http://www.opengis.net/ogc" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsExecute_request.xsd">
   <ows:Identifier>org.n52.wps.server.osm.converter.OSM2Vector</ows:Identifier>
   <wps:DataInputs>
      <wps:Input>
      <ows:Identifier>data</ows:Identifier>
      <wps:Data>
      <wps:ComplexData schema="https://svn.52north.org/svn/geoprocessing/incubator/osmtransform/trunk/xsd/src/main/resources/osm.xsd" mimeType="text/xml">
        <osm version="0.6" generator="Osmosis 0.40.1">
          <bound box="12.93200,103.54300,13.73500,104.47900" origin="http://www.openstreetmap.org/api/0.6"/>
          <node id="88118827" version="6" timestamp="2009-02-15T21:47:05Z" uid="39889" user="Worldtravler" changeset="464707" lat="13.3479685" lon="103.9740925"/>
          <node id="88118833" version="4" timestamp="2009-02-15T21:47:06Z" uid="39889" user="Worldtravler" changeset="464707" lat="13.3479772" lon="103.977999"/>
          </osm>
      </wps:ComplexData>
    </wps:Data>
    </wps:Input>
   </wps:DataInputs>
   <wps:ResponseForm>
    <wps:RawDataOutput>
      <ows:Identifier>result</ows:Identifier>
    </wps:RawDataOutput>
  </wps:ResponseForm>
</wps:Execute>

GML conversion using externally referenced OSM data

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<wps:Execute service="WPS" version="1.0.0" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:ogc="http://www.opengis.net/ogc" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsExecute_request.xsd">
   <ows:Identifier>org.n52.wps.server.osm.converter.OSM2Vector</ows:Identifier>
   <wps:DataInputs>
      <wps:Input>
         <ows:Identifier>data</ows:Identifier>
         <wps:Reference schema="https://svn.52north.org/svn/geoprocessing/incubator/osmtransform/trunk/xsd/src/main/resources/osm.xsd"
            xlink:href="http://csc.metaether.net/siem-reap.osm" mimeType="text/xml" method="GET"/>
      </wps:Input>
   </wps:DataInputs>
   <wps:ResponseForm>
    <wps:RawDataOutput>
      <ows:Identifier>result</ows:Identifier>
    </wps:RawDataOutput>
  </wps:ResponseForm>
</wps:Execute>

On-the-fly GML conversion of results returned by Overpass; a simple bounding box algorithm

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<wps:Execute service="WPS" version="1.0.0" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:ogc="http://www.opengis.net/ogc" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsExecute_request.xsd">
   <ows:Identifier>org.n52.wps.server.overpass.algorithm.BBox</ows:Identifier>
   <wps:DataInputs>
   <wps:Input>
         <ows:Identifier>north</ows:Identifier>
         <wps:Data>
            <wps:LiteralData dataType="xs:double">51.251</wps:LiteralData>
         </wps:Data>
      </wps:Input>
   <wps:Input>
         <ows:Identifier>south</ows:Identifier>
         <wps:Data>
            <wps:LiteralData dataType="xs:double">51.249</wps:LiteralData>
         </wps:Data>
      </wps:Input>
   <wps:Input>
         <ows:Identifier>east</ows:Identifier>
         <wps:Data>
            <wps:LiteralData dataType="xs:double">7.152</wps:LiteralData>
         </wps:Data>
      </wps:Input>
   <wps:Input>
         <ows:Identifier>west</ows:Identifier>
         <wps:Data>
            <wps:LiteralData dataType="xs:double">7.148</wps:LiteralData>
         </wps:Data>
      </wps:Input>
   </wps:DataInputs>
   <wps:ResponseForm>
    <wps:RawDataOutput>
      <ows:Identifier>result</ows:Identifier>
    </wps:RawDataOutput>
  </wps:ResponseForm>
</wps:Execute>

On-the-fly KML conversion of results returned by Overpass; custom Overpass query

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<wps:Execute service="WPS" version="1.0.0" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:ogc="http://www.opengis.net/ogc" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsExecute_request.xsd">
  <ows:Identifier>org.n52.wps.server.overpass.algorithm.GenericQuery</ows:Identifier>
  <wps:DataInputs>
  <wps:Input>
      <ows:Identifier>query</ows:Identifier>
      <wps:Data>
        <wps:LiteralData dataType="xs:string">(
  node(50.746,7.154,50.748,7.157);
  <;
);
out meta;</wps:LiteralData>
      </wps:Data>
    </wps:Input>
  </wps:DataInputs>
  <wps:ResponseForm>
    <wps:RawDataOutput mimeType="application/vnd.google-earth.kml+xml">
      <ows:Identifier>result</ows:Identifier>
    </wps:RawDataOutput>
  </wps:ResponseForm>
</wps:Execute>
​

Extension

Extending this project to add custom Overpass algorithms is fairly easy. There are a few steps to doing this:

  • Create a class that extends AbstractSelfDescribingAlgorithm that describes how and what to query the Overpass server. Look at BBox.java and GenericQuery.java for examples.
  • Create a corresponding xml file that describes the inputs and outputs of the algorithm. Look at BBox.xml and GenericQuery.xml for examples.
  • Compile the project
  • Edit wps_config.xml to add your specified algorithm to the OverpassProcessRepository:
    <Property name="Algorithm" active="true">org.n52.wps.server.overpass.algorithm.YourQuery</Property> 
  • Test your algorithm

Comments and Future Work

There probably exists a lot of future work to be done in this area. The entirety of this GSoC experience has taught me that 2-3 months is barely enough to understand all there is to know about the various GIS formats. There is a lot more that I would like to do with this extension and with WPS in general, which, thankfully I had the foresight not to suggest in the original proposal, as I know I would not have been able to get them done in time.

The questions stand:

Did I learn a lot from this project?

Definitely.

Did I get everything done as I'd hope?

Some, not so much, due to my limited understanding of the field.

Would I like to continue developing?

For sure.

Some research has shown that the ability to ensure continued interoperability between formats is highly beneficial to the GIS community, and I think any step in this direction is the right direction for ensuring this.

With that said, here are some things I would consider interesting additions to the project:

  • Support for PBF format
  • Adding more algorithms to the Overpass algorithm component (What would be some useful requests that would be made)
  • Verification that particular tools can use the exported GML/KML/Shapefiles from osmtransform
  • Adding support for bzipped/gzipped OSM files
  • Prod Geotools to integrate some of the OSM parsing component
  • Making a more easily subclassable Overpass algorithm class for easier extension; perhaps an auto-generating tool?

Metadata

  • Topic created by: DanielNuest
  • Topic created on: 2012-05-07
Topic attachments
I Attachment Action Size Date Who Comment
PNGpng osmui_mockup.png manage 38.1 K 2012-06-27 - 03:11 SarahHarvey osmtransform ui mockup
PNGpng osmui_mockup_rough.png manage 2341.3 K 2012-06-27 - 04:37 SarahHarvey osmtransform ui mockup rough
PNGpng osmui_mockup_rough_small.png manage 316.4 K 2012-06-27 - 04:36 SarahHarvey osmtransform ui mockup rough (small)
PNGpng osmui_workflow.png manage 42.4 K 2012-06-27 - 02:50 SarahHarvey osmtransform ui workflow
Unknown file formatPNG workflow.PNG manage 61.0 K 2012-05-28 - 15:16 SarahHarvey workflow
Tags:
create new tag
, view all tags
Topic revision: r11 - 2012-08-24 - 09:50:38 - SarahHarvey

 
  • Search: 
This site is powered by the TWiki collaboration platform Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback