Streaming Based WPS

This page is intended to show technical details about Streaming based WPS. Further documentation can be found in a couple of blog posts (Streaming based WPS and Near real-time Geoprocessing), in the tutorials (ALERT! Implementing an Output Streaming WPS process and Implementing a Full Streaming WPS process), and in the code itself.

Design (Diagrams)

The following are the class diagrams for Output and Full Streaming WPS, as well as the sequence diagram for Ouput Streaming WPS.

Class diagram for Output Streaming WPS

os_classDiagram.png

This diagram represents the vector class diagram. The raster case is slightly different because it takes two inputs as streamable, so lists of IData parameters are used instead of single IData parameters.

Class diagram for Full Streaming WPS

fs_classDiagram.png

This diagram represents both vector and raster cases.

Sequence diagram for Output Streaming WPS

os_sequence.png

This diagram was taken from (Foerster et al., 2012).

Changes introduced to the 52°North WPS framework

The implementation of Streaming based WPS introduced some changes to the framework. Changes to the core of the framework are minimal, though. The following is the list of modifications:

  • 2 new Abstract classes for Output Streaming (1 Vector and 1 Raster)
    • New parameters:
      • Vector: NumberOfChunks
      • Raster: ChunksByRow, ChunksByColumn
  • 2 new Abstract classes for Full Streaming (1 Vector and 1 Raster)
    • New parameters:
      • maxTimeIdle
  • 2 new algorithms:
    • SnapPointsToLines based on GeoTools User Guide (Vector)
    • NDWI (Normalized Difference Water Index) calculation (Raster)
  • 3 new Streaming based WPS processes:
    • Vector
      • OutputStreamingSimplifyDouglasPeucker
      • FullStreamingSnapPointsToLines
    • Raster
      • OutputStreamingNDWI
  • 2 new DataBindings: VectorPlaylistBinding/RasterPlaylistBinding
  • 2 new generators: VectorPlaylistGenerator/RasterPlaylistGenerator
  • 2 new parsers: VectorPlaylistParser/RasterPlaylistParser
  • 2 new data handlers: PlaylistOutputHandler/PlaylistInputHandler
  • 1 new Database: FolderDatabase
  • Method updateComplexValue added to FlatFileDatabase for updating the Playlist.
  • LocalAlgorithmRepository sets the ExecuteRequest when Algorithm is instanceof Full/Output streaming abstract classes. This is done for accessing the input data formats from the execute request in the algorithm.
  • InputHandler: interception of Streaming based WPSs.
  • Observer pattern: The observer doesn't get the ISubject object but the (generic) state in the update() method. This is because one is not interested in the ISubject object but in the state (sending the ISubject is still possible, though). This also helps to avoid problems with synchronization when getting many notifycations and then having to call the getState method, which could give another state.
  • Tweaks in OutputDataItem.
  • Tweaks in RetrieveResultServer.
  • New dependency (AsyncHTTPClient) for asynchronous HTTP requests.

Known issues

There are some known issues in this implementation:

  • Closing the AsyncHttpClient lets at least one Callback thread alive, whereas not closing it lets a Reaper thread alive. Callback and Reaper threads are both daemons. Since there can be more threads when closing the client, I decided not to close it. The code to close it is commented out in InputPlaylistHandler.java, though.
  • Exceptions due to parameter validation of the base process are currently notified via the playlist, not via the response document.
    • This could be solved by reading the process description of the base process in the streaming based abstract class' Run method and checking that all mandatory parameters come in inputData.
  • getInputFormat and getOutputFormat methods were implemented in every streaming based abstract algorithm. They could be offered directly from the ExecuteResponseBuilder class. The methods are intended to provide the format (mime type, schema, encoding) of a specific input or ouput parameter.

References

  • T. Foerster, B. Baranski, and H. Borsutzky. (2012). Live Geoinformation with Standardized Geoprocessing Services. In J. Gensel, D. Josselin, and D. Vandenbroucke (Eds.), Bridging the Geographic Information Sciences (pp. 99–118). Berlin, Heidelberg: Springer Berlin Heidelberg. 2012. Retrieved from http://www.springerlink.com/index/10.1007/978-3-642-29063-3_6

Metadata

I Attachment Action Size Date Who Comment
fs_classDiagram.pngpng fs_classDiagram.png manage 173 K 07 Sep 2012 - 09:02 UnknownUser Class diagram for Full Streaming WPS
os_classDiagram.pngpng os_classDiagram.png manage 97 K 07 Sep 2012 - 08:55 UnknownUser Class diagram for Output Streaming WPS
os_sequence.pngpng os_sequence.png manage 64 K 07 Sep 2012 - 09:07 UnknownUser Sequence diagram for Output Streaming WPS
This topic: Geoprocessing > StreamingBasedWPS
Topic revision: 09 Jun 2015, EikeJuerrens
Legal Notice | Privacy Statement


This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Wiki? Send feedback