Simple Features for protobuf and others


Introduction

The main goal of my GSoC 2017 project “Simple Features for protobuf and others” is to define and implement a serialization API for spatial vector data, which will transparently serialize geometries based on the Simple Feature specification using Protobuf into binary encoding. This serialization API not only works for binary encoding, but is also extensible to support other language specific geometric models such as JTS and others. In addition, Protocol Buffers will be utilized as the serialization framework because of their unique features, such as flexibility, efficiency and automated mechanism for serializing structured data. The serialization API will be integrated in common 52°North Open Source Software, e.g. the Sensor Observation Service or the Web Processing Service and should serve as a common framework for providing efficient encoding of spatial data.

About Me

My name is Malith Withanage Don. I am currently reading M.Sc. Informatics at the Technical University of Munich. In addition, I am an open source enthusiastic focusing on Distributed Computing and Java. You can follow the project development work on my Github.

Weekly Reports

Week 01

Status
  • Setted up the environment including git. Please find the git repo at [1].
  • Added the initial project structure with maven build.
  • Added maven automated build support for protobuf files by integrating maven plugin [2].
  • Added the initial protobuf schema and generated code to serialize Point geometric type.
  • Did a experiment on available JTS libraries and selected a stable one.
Problems
  • Choosing an official JTS library was little bit problematic as the orginal company (vividsolutions) has discontinued support and now it has been taken over by locationtech. They are in 1.15.0-SNAPSHOT version but it looks more stable. Documentation and code is avaialble.
Next Steps
  • Add unit test for Point serialization.
  • Add common interface to write data (Serialize).
  • Implement common interface and add support for Protobuff serialization for Point geometric type.
  • Utilize the common API and write (serialize) JTS Geo Point data with protobuff
[1]. https://github.com/52North/topology-serialization-framework

[2]. https://github.com/os72/protoc-jar-maven-plugin

Week 02

Status Problems
  • Providing serilization for Polygon type was quite complex as it contains holes. Solved it by including hierarchical structure for Geometry type in protobuf schema.
Next Steps [1]. https://github.com/52North/topology-serialization-framework/commit/2395957b3961e7aaf79575b177ed7da8f300435b

[2]. https://github.com/52North/topology-serialization-framework/commit/7c1b50b964f36b811838605c984560a1ff15e4a4

Week 03

Status
  • Added deserialization handler
  • Implemented deserialization for the geometric types Point, LineString, Polygon, MultiPoint and MultiLineString
  • Implemented Protobuf serialization/deserialization for MultiPolygon, Line, LinearRing, Triangle
  • Added Unit tests
  • Reading and understanding Avro and trying out serialization samples
Problems
  • Discussed the verdict of embedding complete Simple feature access functionalities in protobuf bindings. But to make the serialization layer more robust and high performance, embedding all the functionalities of Simple feature access will obviously be a performance impact and over complex the serialization layer. And those functionalities are already provided by existing libraries such as JTS, Esri etc.
  • There is no JTS model for TIN. So had to exclude that from implementation for the moment and will be considered later to extend the JTS library and model TIN.
  • Line and Triangle do not inherit from Geometry abstraction in JTS. So had to use overloading to support those serialization options.
Next Steps
  • Define avro schemas for Simple Access features
  • Implement serialization handler for avro
  • Implement avro serializations for Point, Linestring, Polygon
  • Understand Coverage Type (ISO Coverage Type, GML Coverage) and how it can be integrated to serialization layer

Week 04

Status
  • Implemented initial avro schema fro simple features
  • Implement serialization handler for avro
  • Implemented avro serializations for Point, Linestring and Polygon
  • Added Unit tests
  • Understanding GeoTIFF spec and analyzing how to model Coverage Type with protobuff
Problems
  • Lack of proper documentation on avro for complex use cases makes it difficult to proceed. Need more self learning compared to protobuf
  • Have some issues in Serialization without code generation. Passing Enum value is one such. Still in progress that area
Next Steps
  • Continue on remaining simple features for avro serialization
  • Avro deserialization
  • Integrating coverage type to protobuff bindings
  • Document about avro serialization options(Dynamic and Static)

Week 05

Status
  • Trying out different libraries to extract raster data from geotiff files
  • Analyzed geotiff file reader support by GDAL java library, Apache commons Imaging, Geotools geotif plugin and selected Geotools geotif plugin.
  • More reading on geotiff spec and understanding the concepts (this is needed to understand the Geotools geotif plugin API)
  • Implemented protobuf schema for coverage data
  • Scratch implementation of avro serialization without code generation - sample added
  • Scratch implementation of extracting coverage data from geotiff file using geotools plugin - sample added
Problems
  • Understanding GDAL java API for extracting raster data is hard due to lack of documentation.
  • Geotools geotif plugin has adequate documentation but needs thorough understanding of geotiff spec to read the API and identify useful functions which consumes lot of time.
Next Steps
  • Use Geotools to serialize Geotiff data with created coverage protobuf model
  • Continue on remaining simple features for avro serialization
  • Avro deserialization for simple features

Week 06

Status
  • Reading on CRS and EPSG codes and how to extract it from geotif via geotools
  • Added serialization handler for coverage data serialization with initial implementation for storing tfw data and raster grid data
  • Added unit test
  • Extracting more information such as pixel color code and CRS data such as EPSG code from geotif source using geotools
  • Analyzing the geotools library on how to transform the coverage data based on given CRS config data
  • Added serialization implementation for geoetif with meta data ( added unit test)
  • Added deserialization handler to deserialize coverage data with initial implementation to deserialize simple coverage grid (added unit test)
Problems
  • Implemented deserializer to return grid coverage object. Do we need to support to convert it to tif file and tfw file as well?
Next Steps
  • Use goetools library to transform the coverage data based on given CRS config data
  • Continue on coverage data deserialization implementation
  • Continue on remaining simple features for avro serialization

Week 07

Status
  • Added a test case for desirialization coverage model. But still it is not capable of generating the grid.
  • Analyze geotiff file to to find out what are the necessary meta data needs to be serialized and how to extract them from geotif using geotools
  • Extract tie point and pixel scale data from geotif file and serialize them.
  • Extend the protbuf schema to support required meta data
  • Debug the geotools library to find out how to inject tfw meta data to deserialized coverage model
  • Writing mid term blog post [1]
Problems
  • By debuging the geotools code I found out that GeoTiffReader automatically pics up the twf file and and parse it and generates MathTransform object out of it. Thats the functionality which is needs to to transform raw,column to real life coordinates. MathTransform object can be generated from ProjectiveTransform utility in geotools by passing Metrix of meta data in tfw file.
Next Steps
  • Deserializing coverage data by injecting tfw meta data to coverage model
  • Deserializing coverage data by injecting meta data stored in Geotiff image file header
  • Use goetools library to transform the coverage data based on given CRS on the fly
[1] http://blog.52north.org/2017/07/20/simple-features-for-protobuf-and-others-mid-term-blog-post/

Week 08

Status
  • Deserializing coverage data by injecting tfw meta data to coverage model
  • Deserializing coverage data by injecting meta data stored in Geotiff image file header
  • Unit test for both above implementations
  • Completing Midterm blog post
Problems
  • As of now grid to world transformation is done based on avilabality of meta data in geotiff file or data given with tfw file. There is another option which is using model transformation meta data. Need to look in to how utilize that.
Next Steps
  • Transform real world coordinates based on given CRS on the fly
  • Continue on Avro serialization/deserialization
  • Discuss other priorities in next meeting
Topic revision: r9 - 24 Jul 2017 11:16:28, MalithWithanageDon
This site is powered by FoswikiCopyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Wiki? Send feedback