SOS Extension Ideas

A page with first ideas on what useful extensions could be developed for the OGC SOS standard version 2.0.

Existing Extensions (in all stages)

Get Data Availability

Get more detailed information about when and where data is available by providing a specific algorithm and encoding to provide time intervals

Status: "stuck in design phase", the latest documents are available with an OGC Portal (requires login) here, UML diagrams are available in the OGC SWE Umbrella SWG SVN.

A summary of the status and next steps was given by Johannes Echterhoff at the OGC TC in Exeter, the presentation can be downloaded here (Thanks, Johannes): 2012_06_OGC_TC_Exeter_SOS_SWG.pdf

Contact: Johannes Echterhoff

Result Filter

Getting back the old filtering functionality for results.

Status: Currently under development (design, not implementation). A summary of the status and next steps was given by Johannes Echterhoff at the OGC TC in Exeter, the presentation can be downloaded here (Thanks, Johannes): 2012_06_OGC_TC_Exeter_SOS_SWG.pdf

Contact: Johannes Echterhoff

Wildcard Offerings

See WildcardOffering.

Status: Implemented, not standardized.

Split Inserts

The extension SplitDataArrayIntoObservations forces the SOS to split the incoming SweArrayObservation into single observations - one for each block in the data array - resulting in observations similar to sending multiple InsertResults.

See https://wiki.52north.org/bin/view/SensorWeb/SensorObservationServiceIVDocumentation#SplitDataArrayIntoObservations.

Status: Implemented, not standardized.

Un-implemented Extension Ideas

Latest Observation Extension

Support getting the first and latest observation available as a temporal filter, using the already implemented keyword latest in a time period. Possible extension to also get the latest X values, for example by providing a markup such as latest{10} and first{10}.

Comments:

• Suggested by -- DanielNuest - 2012-11-23
• Very good. But when thinking about this specific things, we could consider the idea of introducing a simple expression language which may compete with the normal filter mechanisms. Perhaps we could find a sharp separation where this makes sense.
• Should be easily doable based on existing implementation and sounds like a useful extension. -- Main.Staschc - 2012-11-26
• Do not make this a SOS only solution but extend OGC Filter Encoding, e.g. utilizing Fes:SortyBy in combination with paging and a new projection. -- Johannes Echterhoff (via Email)

Paging Extension

Paging for GetObservation responses. There's probably many best practices for paging around that could be re-used. Paging could include "navigational" links such as "get previus/next 100 observations".

Comments:

• Suggested by -- DanielNuest - 2012-11-23
• paging could be interesting, and possible via the latest{10} idea by adding an offset like latest{100,1000} for the last 1000 values showing 100 until 199 (server may cache the values for later calls). This may be an alternative to streaming responses.
  • This issue has been discussed in the SOS SWG. There were some volunteers who were planning to work on this in the SOS SWG, but seems like there hasn't been work on an extension yet. So would be useful, if starting with an implementation and proper documentation of the approach. In a former version of SOS, paging was included and had attributes like startObservation=n1, endObservation=n2 and orderBy=asc in the GetObservation request. -- Main.Staschc - 2012-11-26
• The ability to count the number of observations in a request before actually downloading the observations could be usefull for client software to inform the user about downloading progress.
  • Note that this functionality is part of the GetDataAvailability operation that is already defined as an extension of the SOS 2.0. -- Main.Staschc - 2012-11-26

Contents Filtering Extension

Filter the capabilities document not only after high level elements, but also introduce filtering for the contents section, e.g. only with certain keywords or observed properties... This could be a way to limit the size of capabilities documents.

Comments:

• Suggested by -- DanielNuest - 2012-11-23
• There is already an inheritance mechanism in the SOS 2.0 for redundant information in the Capabilities. If this is implemented properly by servers and clients, I don't think an extension for filtering the Capabilities is needed. In case there are thousands of heterogenuous sensors, IMO it is more appropriate to think about best practices how to split the content on several SOS instances and use a catalog for discovering particular sensors/observations - maybe with help of the SOR/SIR. -- Main.Staschc - 2012-11-26
• This is related to the Wildcard offering extension, see WildcardOffering, by Shane StClair.
• Make this more generic on the OWS Capabilities level. Filtering could be based on generic XML features such as XSLT. -- Johannes Echterhoff (via Email)
  • Or XPath -- DanielNuest - 2012-11-26
• Discussed in more detail here one the sensor web mailing list: http://sensorweb.forum.52north.org/How-to-find-observed-properties-being-served-from-a-sos-server-SEC-UNCLASSIFIED-td4026320.html

NoData Values Enumeration

Suggested on the mailing list by Derek Hohls: Extend the functionality to allow multiple "no data" values with a certain semantic meaning (not sampled, non detect, machine out of order, ...).

See http://sensorweb.forum.52north.org/InsertObservation-different-types-of-missing-data-tc4025493.html for the suggestion and dicussion.

Comments:

• ...

Bulk Download Extension

An extra servlet regularly (last hour, last days, last week, last month, last year) the database as a dump in (a) database format, or (b) CSV file (though flattening the structure could be hard - information loss acceptable! The files are deleted so that all information is available but might have to be collected in several files, i.e. no "hourly" downloads that are older than 24 hours, because then one should download the "last day" file, no daily download older than seven days because then the data is in the "weekly" file. Delivery is done fast and easy as compressed files.

Potentially this could be extended for specific timeseries combinations, might also be able to integrate this into the timeseries API.

API sketch

• .../SOS/bulk/2013.csv.tar.gz > a year (only if the year is over)
• .../SOS/bulk/201203.csv.tar.gz > a month (only if the month is over)
• .../SOS/bulk/20120301_20120307.tar.gz > a week (only if the week is over)
• .../SOS/bulk/201203.tar.gz > Daten für die ganze Woche
• .../SOS/bulk/offeringXY/all.tar.gz

Background

• Do you really need an API to provide Open Data? Issues and arguments for bulk file downloads > http://www.directionsmag.com/podcasts/podcast-sharing-data-via-apis-or-downloads/241138
  • An API might limit the use of the data (since it supports "your way"/the developers way of using the data), have bulk download and API both together
• http://sunlightlabs.com/blog/2012/government-do-you-really-need-an-api/
  • Geodata might require APIs, could also be a better choice for real-time data and collaborative processes

Comments:

• Suggested by -- DanielNuest - 2013-11-19

General Discussion

• We should also think: What features are in the 52N SOS ("low hanging fruits") that are not in the standard, and could they be useful for others so that the extension is worth the work. -- DanielNuest - 2012-11-23

Metadata

• Topic created by: DanielNuest
• Topic created on: 2012-11-15