SOS 4.x Documentation
Quick Links
For Information about the
new 52°North implementation,
SOS 5.x, please see
SensorObservationServiceV for general information and for documentation.
Introduction
Scope
This document describes the architecture and the install process of the 52°North Sensor Observation Service (SOS).
Changes compared to previous versions
The 52°North SOS 4.x is a reimplementation of the 52°North SOS. Here are the biggest changes from the previous version 3.5.x:
- SOS API
- new database model: uses numerical identifiers, required to match the OGC SOS 2.0.0 spec properly
- persistence framework Hibernate and Hibernate spatial
- Java ServiceLoader concept for binding, de-/encoder and operations
- binding/encoding/operation: develop a new feature against the SOS API without changing the core
- Admin GUI: change the settings, de-/activate encodings and bindings
- Installer GUI: configure the database connection, create database model, configure the settings
Some Words on OGC SOS
The information about the OGC SOS specifications are described
SensorObservationServiceDocumentation
The
OGC SOS version 2.0 specification defines a core and five extensions (in SOS 1.0.0 called profiles). The current release implements these operations:
= implemented
= not implemented
= not specified
Bindings
[A binding] describe[s] how SOS 2.0 clients and servers can communicate with each other. (OGC-006)
- KVP - Key Value Pair encoding of requests via HTTP GET.
- SOAP - SOAP based encoding of all extensions via HTTP POST.
- POX - XML based encoding of all extensions via HTTP POST ( without SOAP wrapping).
- REST - RESTful binding for all useful operations of resources being available in the SOS. Link to documentation.
- JSON - JSON based encoding of all extensions via HTTP POST. The JSON implementation is based as much as possible on the abstract definitions of the specifications but it is not yet specified and not all features are supported.
- EXI - Efficient XML Interchange (EXI) Format 1.0 based encoding of all extensions via HTTP POST and POX encoded request. SOAP requests are currently not supported via the EXI binding.
Architecture
This image describes the basic architecture ot the 52°North SOS 4.x.
Click to enlarge.
Installation
Requirements
This tutorial is based upon the following software which have to be downloaded and installed.
- Java runtime environment (JRE) 7.0 or higher (or JDK 7.0 or higher if you want to build the SOS from source): Download
- Java Servlet-API 2.5 compatible application server. E.g.:
- Tomcat 6 or higher: Download
- Do not use Tomcat versions 8.0.8, 7.0.54 and 6.0.41 because is a known bug in these releases whereby the SOS installation fails. Please use an newer Tomcat release.
- Jetty 6 or higher Download
- Glassfish 3 or higher Download
- Running Database Managment System
- Apache Maven 3 or higher (if you want to build the SOS from source): Download
Supported Database Managment System
Storage Space requirements
The values presented here are experienced data. If you have any additional observations, please share them with the community using the
mailing list, for example.
Definition
time series: A time series is a group of observations having the following parameters in common:
- feature of interest
- procedure/sensor
- observed property/observable property
- observation type (e.g. numeric)
Requirement per observation 1,36 mio observations = 750MB
1,36 mio observations = 786432000B
1 observation = 578B
⇒ A time series with a frequency of 1 observation per minute results in a storage requirement of
578B * 60 * 24 * 365 = 303796800B (= 289MB) per year per time series.
A1) Download the SOS WAR-File
If you want to build the SOS from source you can skip this point and proceed with
B1) Building from Source.
Download package including the WAR file:
Since version 4.3 of the 52°North SOS we also provide a bundle package that includes the
SWC REST-API and the
JavaScript SOS Client Helgoland. This package has
bundle
in the name.
Unzip the package and browse to
UNZIPPED_PACKAGE/bin/target
where the WAR FILE is located.
B1) Building from Source
If you want to use the binary distribution of the SOS you can skip this step and proceed with
2) Installing the Webapp.
Make shure you have all required Software installed.
- Checkout the sources directly from our subversion repository
- change to the directory where you've just checked out the source code
- run
mvn package
to build the webapp.
- The following profiles are not included because they require dependencies which are not publicly available (licensing) and must be independently activate. See Oracle and SQL Server support
-
oracle
: enable the Oracle support (see below)
-
sqlserver
: enable the Microsoft SQL Server support (see below)
- copy
webapp/target/52n-sos-webapp.war
to a directory of your choice
- Or the bundle can be found in her
webapp-bundle/target/52n-sos-webapp.war
Oracle support
52n SOS supports either
thin (pure java) or
OCI (more performant, platform specific) Oracle 11g drivers. The
thin driver is easier to use for building the project.
These drivers are not freely distributable, so they have to be downloaded from Oracle's website and installed manually.
Download the latest 11.2
driver (
ojdbc6.jar) and install the jar in your local repository:
mvn install:install-file -Dfile=ojdbc6.jar -DgroupId=com.oracle -DartifactId=ojdbc6 -Dversion=11.2.0 -Dpackaging=jar
Then you can build using the Oracle profile:
mvn package -P oracle
If the Oracle driver is not installed in Maven, the build will fail.
Also, the Oracle datasource project (hibernate/datasource/oracle) is ready for some unit testing related to database connectivity. Since it requires an existing Oracle database, tests are executed
only if the environment variable
SOS_TEST_CONF is defined. The value of this variable must be a valid path to a Java properties file. This file must contain the following keys:
- oracle_host: Oracle DB host
- oracle_port: Oracle DB port
- oracle_user: Oracle DB user. This user must have permissions to create tables and sequence and, in general, be ready to use it for the 52N SOS server successfully.
- oracle_pass: Oracle DB pass for the previously explained user.
- oracle_user_no_rights: Oracle DB user. This user must NOT have permissions to create tables or sequences.
- oracle_user_no_rights: Oracle DB pass for the previously explained user.
This is a sample file:
oracle_host=localhost
oracle_port=1521
oracle_user=oracle
oracle_pass=oracle
oracle_user_no_rights=sos_test_no_rights
oracle_pass_no_rights=sos
Microsoft SQL Server support
Since 4.4.x the SQL Server JDBC is available via Maven repository!
The following discription is obsolete.
The Microsoft SQL Server driver is not freely distributable, so it has to be downloaded from Microsoft website and installed manually.
Download the
latest driver (sqljdbc4.jar) and install the jar in your local repository:
mvn install:install-file -Dfile=sqljdbc4.jar -DgroupId=com.microsoft -DartifactId=sqljdbc4 -Dversion=4.0 -Dpackaging=jar
Then you can build using the SQL Server profile:
mvn package -P sqlserver
If the Microsoft SQL Server driver is not installed in Maven, the build will fail.
If you don't want the built-in installer you also can configure the SOS while building it:
- To define the database connection settings copy
conf/datasource.properties.template
to conf/datasource.properties
and adjust it to your needs.
- typically only the user name/password and URL properties have to be changed. While user name and password are quite self explanatory the URL is constructed like the following:
jdbc:postgresql://$HOST:$PORT/$DATABASE_NAME
- To adjust service settings open the copy the file
db/default_settings.sql
and edit to your needs and recreate conf/default_settings.db
(e.g sqlite3 misc/conf/default_settings.db -init misc/db/default_settings.sql
).
- To build the project run
mvn package -Pconfigure-datasource,use-default-settings[,additional-profiles]
- to choose a different file than
conf/datasource.properties
pass it to Maven by specifying -DdatabaseConfigurationFile=ABSOLUTE_PATH_TO_FILE
at the command line
- to choose a different file than
conf/default_settings.db
pass it to Maven by specifying -DsqliteSettingDatabase=ABSOLUTE_PATH_TO_FILE
at the command line
- any profile of the standard build process (see above)
- After building the preconfigured webapp is located at
webapp/target/52n-sos-webapp.war
2) Installing the Webapp
PostgreSQL installation
- Make sure your Tomcat and PostgreSQL are running.
- Create a new PostgreSQL database with e.g. pgAdmin3 using the PostGIS template created during the PostGIS installation.
- (for a short tutorial see for example here)
- If you update your SOS from Beta1 to Beta2 it is mandatory to delete clear the old database because there are changes in the database model.
- Navigate to the Tomcat Manager, e.g. http://localhost:8080/manager/html
- Scroll down to the section “WAR file to deploy” and select the WAR file which you've build or downloaded in the previous step.
- Click on “deploy”. The new service instance will be shown in the application list.
- Click on the service instance to navigate to your newly deployed SOS.
- Follow the steps on the screen to configure your SOS instance (you don't have to do this if you've build the preconfigured webapp). More information about the settings can be found in the Configure/Administrate the SOS section of this page.
- If you have installed the preconfigured webapp change the administrator password using the admin GUI. The default password is
password
and the default user name is admin
.
Oracle installation
We assume that an Oracle 11g instance is already running and accessible through the network. You can find some instructions on how to install Oracle 11g on Ubuntu 12.04
here.
1. Create an Oracle user
We need to create an Oracle user with the following configuration:
- Connect permission
- Create table permission
- Create sequence permission
- Unlimited tablespace
- Quota unlimited on users
To create a sos user with this configuration, you can use the following sqlplus commands:
> create user sos identified by *password*;
> alter user sos identified by *password* quota unlimited on users;
> grant connect, create table, create sequence to sos;
> grant unlimited tablespace to sos;
2. Get the Oracle Instant Client (OCI)
52n SOS supports either
thin (pure java) or
OCI (more performant, platform specific) Oracle 11g drivers. The
OCI driver is recommended for production.
Get the platform specific Oracle Instant Client (
OCI) from Oracle's website:
http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html
Note: If the full Oracle 11g has been installed in the same machine, the Instant Client is already there.
3. Install and configure Tomcat
We'll need Tomcat 6 or above. Download it from Tomcat's website, or use the package manager from your Operating System.
-
Make sure that Tomcat will have access to the OCI driver files.
- LD_LIBRARY_PATH (unix) or PATH (windows) environment variable should point to the location where Oracle Instant Client is installed.
- The ojdbc6.jar file in this same location should be included in tomcat's classpath. For instance, make a symbolic link to it in <TOMCAT_BASE>/lib.
- Windows since Vista with OCI installed under C:\ and Apache Tomcat 7. You need administration rights
-
mklink "c:\Program Files\Apache Software Foundation\Tomcat 7.0\lib\ojdbc6.jar" c:\instantclient_12_1\ojdbc6.jar
- Copy your 52n-sos-webapp.war file into <TOMCAT_BASE>/webapps and start the Tomcat server. For Linux instance:
sudo service tomcat6 start
4. First run: setup SOS server
Now you have 52N SOS running in your Tomcat server. To access it, open this URL in your browser:
http://localhost:8080/52n-sos-webapp
You will find the 52N SOS home page with the following message:
You first have to complete the installation process! Click here to start it.
Click
here, press Start and select Oracle Spatial from the drop-down list. A database configuration form will appear below. Fill it with your database settings. For example, for the "sos" user created above, these will be the configuration values:
- User Name: sos
- Password: sos
- Database: sos
- Host: localhost
- Database port: 1521
- Schema: sos
- Database concept: Select from dropdown
- Old concept
- Series concept (default)
- eReporting concept (extended Series concept)
- Multilingualism support: Unchecked
- Transactional profile: Checked
- Provided JDBC drive: Checked
- Minimum ConnectionPool size: 10
- Maximum ConnectionPool size: 30
- Batch size: 20
- Create tables: Checked
- Delete existing tables: Unchecked
- Force updating existing tables: Unchecked
Click Next and fill the Settings. For testing purposes it is possible to simply skip this process by clicking Next again.
Finally, set a user name and a password for the 52N SOS administrator and click Install.
Microsoft SQL Server installation
We assume that an Microsoft SQL Server 2012 instance is already running and accessible through the network.
1. Create an Microsoft SQL Server user
We need to create an Microsoft SQL Server user with permissions to create a new database and to write into the new database.
2. Install and configure Tomcat
Since 4.4.x the SQL Server JDBC is available via Maven repository!
The following discription is obsolete.
We'll need Tomcat 7 or above. Download it from Tomcat's website, or use the package manager from your Operating System.
-
Make sure that Tomcat will have access to the Microsoft SQL Server driver files.
- LD_LIBRARY_PATH (unix) or PATH (windows) environment variable should point to the location where Microsoft SQL Server driver is installed.
- The sqljdbc4.jar file in this same location should be included in tomcat's classpath. For instance, make a symbolic link to it in /lib.
-
mklink "c:\Program Files\Apache Software Foundation\Tomcat 7.0\lib\sqljdbc4.jar" "C:\SqlServer\Microsoft JDBC Driver 4.0 for SQL Server\sqljdbc_4.0\enu\sqljdbc4.jar"
- Copy your 52n-sos-webapp.war file into /webapps and start the Tomcat server. For Linux instance:
sudo service tomcat6 start
3. First run: setup SOS server
Now you have 52N SOS running in your Tomcat server. To access it, open this URL in your browser:
http://localhost:8080/52n-sos-webapp
You will find the 52N SOS home page with the following message:
You first have to complete the installation process! Click here to start it.
Click here, press Start and select SQL Server from the drop-down list. A database configuration form will appear below. Fill it with your database settings. For example, for the "sos" user created above, these will be the configuration values:
- User Name: sos
- Password: sos
- SQL Server instance: SQLEXPRESS
- Database: sos
- Host: localhost
- Database port: 1433
- Schema: dbo
- Database concept: Select from dropdown
- Old concept
- Series concept (default)
- eReporting concept (extended Series concept)
- Multilingualism support: Unchecked
- Transactional profile: Checked
- Provided JDBC drive: Checked
- Minimum ConnectionPool size: 10
- Maximum ConnectionPool size: 30
- Batch size: 20
- Create tables: Checked
- Delete existing tables: Unchecked
- Force updating existing tables: Unchecked
Click Next and fill the Settings. For testing purposes it is possible to simply skip this process by clicking Next again.
Finally, set a user name and a password for the 52N SOS administrator and click Install.
This section contains only the default settings of the SOS which can be supplemented by settings of an
extensions.
Installation
During the installation you can configure the SOS by changing the following setting. These settings can be changed later with the administration interface.
- Informative settings
- Service Provier: change the provider information (Capabilities) of this service
- Service Identification: change the identification infomation (Capabilities) of this service.
- other
- Transactional Security : Simple security for transactional operations
- Service: Service relevant settings like URL, cache update interval, ...
- Miscellaneous: Not assignable settings like separators, exception output
- CRS: Default EPSG codes, axis order of the stored geometries, supports the datasource spatial queries.
- RESTful Binding: Configure the RESTful binding
- Procedure Description: Automatic procedure description generation and enrichment, e.g. featureOfInterest, discovery profile
- EXI Bindind: Configure the EXI binding
- Streaming : En-/disable XML response and/or datasource streaming
- eReporting: Define prefixes, global namespace for AQD e-Reporting support
- FlexibleIdentifier: En-/disable flexible identifier
- I18N: Multilingualism defintion
- INSPIRE: Define metadata for SOS as INSPIRE download service
Administration
With the SOS administration interface you can change the settings you've made during the installation and you can make additional settings, e.g. logging configuration, supported bindings. To access the administration interface go to
[SOS_URL]/admin
, e.g.
http://localhost:8080/52n-sos-webapp/admin.
On the administration interface start page you can perform a reloading of the Capabilities cache (internal cache) or select the folloing submenues:
- Datasource Maintenance: Get some raw SQL queries, clear the database or delete deleted observaitions (marked by DeleteObservation operation)
- Cache summary: Get some statistics about the internal cache, e.g. number of procedure or temporal and spatial bboxes.
- Reset: Resets the datasource configuration.
- Settings: Change the settings you have made during the installation process.
The
Settings submenue contains further submenues which are briefly explained:
- Logging: Change the log level or show the latest log statements
- Operations: En-/Disable supported operations
- Encodings: En-/Disable supported encodings for observations and procedures
- Bindings: En-/Disable supported bindigs, see Bindings above
- Extensions: En-/Disable supported extensions for ExtendedCapabilities or Offering extensions, e.g. see the INSPIRE
- Datasource: See the datasource configuration
- Procedure Descriptions: Change or delete procedure descriptions
- Capabilities Settings: Add/Change/remove static Capabilities, Capabilities and Offering extensions.
- Observable Properties: Rename ObservableProperties.
- I18N Settings: Create/Update/Delete multilingual data, see Multilingual support
Detailed information about the administration interface:
SensorObservationServiceIVAdministratorInterface
Transactional operations
To avoid the unwanted manipulation of SOS data via the transactional operations in the default 52°North SOS, we have disabled the transactional operations and enabled the
Transactional Security by default since 52°North SOS 4.2.
This affects the following operations:
- InsertSensor
- InsertObservation
- InsertResultTemplate
- InsertResult
- UpdateSensorDescription
- DeleteSensor
- DeleteObservation
The transactional operations can be enabled via the admin GUI of the 52°North SOS. Go to the
Operations submenue of the Admin -> Settings in the 52°North SOS web interface.
The
Transactional Security is enabled by default and the
Transactional allowed IPs is set to _127.0.0.1.
Updating the SOS
Save the settings
The 52°North SOS 4.x offers the possibility to export the settings.
- Open a browser and enter the 52°North SOS 4.x URL, e.g.
http://localhost:8080/52n-sos-webapp
- Click the Admin tab and enter username and password
- Move to the Admin tab and select Settings
- Click the Export Settings button
- A new browser tab opens with the JSON encoded settings, if the browser supports JSON files
- Save the JSON file.
NOTE: The database settings are currently not exported. But there are plans to add this in the future.
Undeploy the old instance
There are two ways to undeploy the old instance:
- Via the Apache Tomcat HTML Manager by selecting
undeploy
- Stopping the Aache Tomcat Server and delete the war file and the folder from
[TOMCAT_HOME]/webapps
Deploy the new instance
Deploy the 52°North SOS 4.x war file as described in the
Installation section.
On the installation page with the title
52°North SOS Installation Wizard you can find at the bottom the heading
Upload a previous configuration file.
Here you can select the previously exported settings file to import the settings into the new instance.
Force updating existing tables - checkbox
The
Force updating existing tables is an
experimental functionality to update the exiting database model if needed. If this chekbox is selected, the 52°North SOS 4.x updates the database model during the installtion process.
There are also update scripts for the supported DBMS (PostgreSQL, Oracle, MySQL) available. These scripts are located in /misc/db/[DBMS]/[CONCEPT] of the sources.
We recommend to use the provided update scripts!
Changed Observation database model
Since SVN revision 22188 the default observation concept/database model has been changed!
The new default observation concept/database model is the series concept which means that the featureOfInterest, procedure and observableProperty references have moved from the observation table to a series table and the series table is referenced in the observation table.
The old observation concept/database model is still supported by the 52°North SOS 4.x and can be selected during the installtion process by checking the
Old observation concept checkbox.
There are also SQL scripts for PostgreSQL, Oracle and MySQL available to convert the old database model to the new series database model.
Database model changes from 4.0 to 4.1
Since the 52°North SOS 4.0 release have the following changes occurred:
- The series table was extended with additional columns to integrate the REST-API in the future.
- The procedure table was extended with an additional column to integrate the REST-API in the future.
- The spatialFilteringProfile table was deleted and instead a samplingGeometry column was added to the observation table.
You can find an update script for these changes in /misc/db/[DBMS]/[CONCEPT] of the sources.
The
update_spatialfilteringprofile_handling.sql script in /misc/db/[DBMS] copies the geometries from the spatialFilteringProfile table to the
samplingGeometry column in the observation table and then deletes the spatialFilteringProfile table.
Database model changes from 4.1 to 4.2
Since the 52°North SOS 4.1 release have the following changes occurred:
- The numericvalue.value, series.firstnumericvalue and series.lastnumericvalue types have been changed from BigDecimal to double.
- The series table was extended with additional column published
- Series with published = false are not returned in GetDataAvalability and GetObservations responses.
- The tables observation, offering, procedure, observableproperty, and featureofinterest have been unified to support the multilingualism. The following columns have been unified:
- identifier: unique identifier, used as request parameter
- codespace: codespace of the identifier
- name: object name
- codespacename: codespace of the name
- description: object description
- The tables i18noffering, i18nprocedure, i18nobservableproperty, and i18nfeatureofinterest have been added to support multilingualism.
Due to the changes of the column types and the renaming of some columns the automatic database model update functionality of the installation wizard does not work. Hence we recommend to run the provides update scripts according to the used DBMS and the used database model/concept. The update scripts contain the string
update_41_to_42.sql and can be found in /misc/db/[DBMS].
Bundle
Available since version: 4.3.0
The bundle contains in addition to the 52°North SOS the
SWC REST-API and the
JavaScript SOS Client.
The SWC REST-API uses the database settings defined during the installation of the 52°North SOS and has direct access to the database tables. This allows a fast access to the data for the JavaScript SOS Client.
The JavaScript SOS Client preconfigured to connect to the SWC REST-API provided by this bundle at
localhost:8080/52n-sos-webapp
.
This works with the default WAR file name "52n-sos-webapp##
.war", because Tomcat can use the file name for context configuration.
If you change the name of the war file you have to update the REST-API URL of the JavaScript SOS Client in the settings file: [TOMCAT_HOME]\webapps\[SOS_NAME]\static\client\jsClient\settings.json
of the deployed SOS.
A precompiled bundle package can be downloaded from the 52°North download page.
To build the bundle package from sources, see B1) Building from Source section.
Administrator Interface
SensorObservationServiceIVAdministratorInterface
Endpoints
The 52°North SOS 4.x supports different endpoints to which the client can send requests. In the earlier 4.x versions each binding had is own endpoint, e.g /sos/soap for SOAP requests or /sos/kvp for KVP requests. This behaviour was discussed on SensorObservationServiceIVEndpointsAndBindings and reported as a bug https://bugzilla.52north.org/show_bug.cgi?id=848.
Since the version 4.1 the 52°North SOS 4.x supports sending requests directly to the endpoint /sos and the additional endpoint /service. In this case the HTTP ContentType should be set. In the following matrix you can find the valid ContentType for each binding and the supported endpoints.
Where /xyz
stands for /service
or /sos
Further documention
Further documentation of the SOS features and functionality can be found here SensorObservationServiceDocumentation
Frequently Asked Questions
See SosFAQ