Streaming Based WPS

This page is intended to show technical details about Streaming based WPS. Further documentation can be found in a couple of blog posts (Streaming based WPS and Near real-time Geoprocessing), in the tutorials ( Implementing an Output Streaming WPS process and Implementing a Full Streaming WPS process), and in the code itself.

Design (Diagrams)

The following are the class diagrams for Output and Full Streaming WPS, as well as the sequence diagram for Ouput Streaming WPS.

Class diagram for Output Streaming WPS

This diagram represents the vector class diagram. The raster case is slightly different because it takes two inputs as streamable, so lists of IData parameters are used instead of single IData parameters.

Class diagram for Full Streaming WPS

This diagram represents both vector and raster cases.

Sequence diagram for Output Streaming WPS

This diagram was taken from (Foerster et al., 2012).

Changes introduced to the 52°North WPS framework

The implementation of Streaming based WPS introduced some changes to the framework. Changes to the core of the framework are minimal, though. The following is the list of modifications:

• 2 new Abstract classes for Output Streaming (1 Vector and 1 Raster)
  • New parameters:
    • Vector: NumberOfChunks
    • Raster: ChunksByRow, ChunksByColumn
• 2 new Abstract classes for Full Streaming (1 Vector and 1 Raster)
  • New parameters:
    • maxTimeIdle
• 2 new algorithms:
  • SnapPointsToLines based on GeoTools User Guide (Vector)
  • NDWI (Normalized Difference Water Index) calculation (Raster)
• 3 new Streaming based WPS processes:
  • Vector
    • OutputStreamingSimplifyDouglasPeucker
    • FullStreamingSnapPointsToLines
  • Raster
    • OutputStreamingNDWI
• 2 new DataBindings: VectorPlaylistBinding/RasterPlaylistBinding
• 2 new generators: VectorPlaylistGenerator/RasterPlaylistGenerator
• 2 new parsers: VectorPlaylistParser/RasterPlaylistParser
• 2 new data handlers: PlaylistOutputHandler/PlaylistInputHandler
• 1 new Database: FolderDatabase
• Method updateComplexValue added to FlatFileDatabase for updating the Playlist.
• LocalAlgorithmRepository sets the ExecuteRequest when Algorithm is instanceof Full/Output streaming abstract classes. This is done for accessing the input data formats from the execute request in the algorithm.
• InputHandler: interception of Streaming based WPSs.
• Observer pattern: The observer doesn't get the ISubject object but the (generic) state in the update() method. This is because one is not interested in the ISubject object but in the state (sending the ISubject is still possible, though). This also helps to avoid problems with synchronization when getting many notifycations and then having to call the getState method, which could give another state.
• Tweaks in OutputDataItem.
• Tweaks in RetrieveResultServer.
• New dependency (AsyncHTTPClient) for asynchronous HTTP requests.

Known issues

There are some known issues in this implementation:

• Closing the AsyncHttpClient lets at least one Callback thread alive, whereas not closing it lets a Reaper thread alive. Callback and Reaper threads are both daemons. Since there can be more threads when closing the client, I decided not to close it. The code to close it is commented out in InputPlaylistHandler.java, though.
• Exceptions due to parameter validation of the base process are currently notified via the playlist, not via the response document.
  • This could be solved by reading the process description of the base process in the streaming based abstract class' Run method and checking that all mandatory parameters come in inputData.
• getInputFormat and getOutputFormat methods were implemented in every streaming based abstract algorithm. They could be offered directly from the ExecuteResponseBuilder class. The methods are intended to provide the format (mime type, schema, encoding) of a specific input or ouput parameter.

References

• T. Foerster, B. Baranski, and H. Borsutzky. (2012). Live Geoinformation with Standardized Geoprocessing Services. In J. Gensel, D. Josselin, and D. Vandenbroucke (Eds.), Bridging the Geographic Information Sciences (pp. 99–118). Berlin, Heidelberg: Springer Berlin Heidelberg. 2012. Retrieved from http://www.springerlink.com/index/10.1007/978-3-642-29063-3_6

Metadata

• Topic created by: GermanCarrillo
• Topic created on: 2012-09-07