SOS interface for Rasdaman data storage
This project should connect Rasdaman
as a data storage backend to the 52\xB0North Sensor Observation Service (SOS) implementation
by adding Rasdaman as a data source in the 52\xB0North SOS.
The project description can be found on the first blog post about Sensor Data Access for Rasdaman
Weekly report no. 1 - Sensor Data Access for Rasdaman
- Build locally and display timeseries in SOS.js test client from sos-js project
- Formulate user stories and choose the most important ones for the first two sprints
- Learn about rasql
Weekly report no. 2 - Sensor Data Access for Rasdaman
- Build sos-js project
- Deploy SOS using the develop profile. This is useful for making successful request from sos-js(using SOS 1.0.0 version)
- Formulate user stories and choose the most important one for the first sprints
- Formulate tasks for the first sprint based on the selected user story
- Learn about rasql
- Read about JDBC and Hibernate
- Try the ASQLDB project. The ASQLDB project adapts HSQLDB(a Java embeded relational database) to support Rasdaman as a backend for array queries. HSQLDB has an extensive JDBC driver already, which could easily be extended to support multidimensional array results from Rasdaman and this would be a pretty elegant solution for our project(it is supposed to take less time than implementing a JDBC driver). Dimitar Misev - one of the mentors - is working on this project. He is also a Rasdaman developer and he considers using this project instead implementing a JDBC driver a more elegant solution.
- Can not display the timeseries in SOS.js test client from sos-js project
- Test if the ASQLDB can be used with Hibernate.
- Test and adapt ASQLDB's JDBC driver for multidimensional arrays
- Create Hibernate Rasdaman dialect.
Weekly report no. 3 - Sensor Data Access for Rasdaman
- Create Sensor Data Acces for Rasdaman wiki page. From now on, the weekly reports will be published here.
- Publish the first blog post.
- Write sos-js step by step guide. This guide will be published on the wiki as soon as we solve a little problem with the proper functioning of sos-js with the SOS data.
- Update tasks for the next sprint and set estimated time for each task. The first user story we have chosen is: "As a user, I want to query multidimensional Rasdaman data using the SOS GetObservation operation". In order to accomplish this complex task, we have splitted it in multiple smaller tasks which are going to be accomplised on the following two or three weeks.
- The sos-js problem is still not solved
- Test if ASQLDB works with Hibernate
- Test and adapt ASQLDB's JDBC driver for multidimensional arrays. The ASQLDB implementation is not finished yet and we may encounter some problems while using it, so this task may take some more time than estimated.
- Create Hibernate dialect for Rasdaman
- Write Java code to access Rasdaman through ASQLDB
- Write some java examples to the ASQLDB's github readme page.
Weekly report no. 4 - Sensor Data Access for Rasdaman
- Test if/how HSQLDB/ASQLDB works with Hibernate. In order to do that, I have created a new package in the asqldb project, included the hibernate dialect and wrote a test class. I have implemented the connection with the database, a method to insert data and a method for select queries. This test was useful because I didn't use hibernate before and I learned the basic actions - which are going to be really useful in implementing the hibernate dialect and especially in writing the test classes for the hibernate dialect. I've also made some little changes to adapt the dialect to Hibernate 4.5.1. I have moved the hibernate tests into the new project I describe below.
- Try eGit and Maven by creating a new project in eclipse. This project is supposed to implement the asqldb's JDBC driver multidimensional array support tests. In order to use the JDBC driver I exported the asqldb project as jar and installed it in maven's local repository. I will publish the repository here as soon as we have some relevant tests to show :).
- Start JDBC driver implementation. The JDBC driver for the HSQLDB already had array support and when an array request was received, the ResultSet would have returned the JDBCArray type. Now, the ASQLDB adapts the HSQLDB to access Rasdaman data by storing the object identifiers(OID) provided by Rasdaman to uniquely identify arrays, in an HSQLDB table. You can find more details about the OID from the ql-guide. The JDBCArray result is no longer relevant, so I have implemented a JDBCMArray class which casts the multidimensional array data to a RasGMArray object. I have also modified the getArray method from the JDBCResultSet class in order to return a JDBCMArray object.
- Here is a diagram on how the asqldb works using the hsqldb in order to get multidimensional array data from Rasdaman
- The changes I've made until now don't provide full multidimensional array support for the JDBC driver and I still have to discuss details about implementation with the mentors.
- Finish implementing the JDBC driver
- Implement tests for the JDBC driver multidimensional array support
- Published the asqldbTest project on gitHub. The JDBC driver tests will also be published in this project.
- On the asqldb repository I will be published the temporary JDBC driver updates that will be made to adapt it to multidimensional array queries.
- Here you can always find the final working version of the asqldb project.
Weekly report no. 5 - Sensor Data Access for Rasdaman
- The JDBC driver adaptation to return multidimensional arrays is finished. We don't have any issues that we are aware of right now. The result is that using the asqldb's JDBC driver to query multidimensional array data returns a multidimensional array object - instance of RasGMArray.
- Apply some little changes on the JDBC driver in order to pass all the select tests that were passing before.
- Implement tests for the multidimensional array queries. Until now, the select tests were only testing if the queries are executed without throwing an exception. The new tests are using a set of asqldb queries and an equivalent set of rasql queries. The asql and the rasql queries are executed and the result are compared. If the results are identical, then the test passes, otherwise the test fails. We have a set of 62 queries - 16 of them are failing because the asqldb doesn't offer full support until now for the multidimensional array queries. The number of tests is 104 but the other 40 tests are not translated to rasql yet.
- Read the hibernate dialect for hsqldb and started using it for a multidimensional array query. This query should work after I adapt the dialect to asqldb.
- Edit the asqldb's GUI in order to write the multidimensional array results to a file instead of writing this to the interface. The result returned for the column that contains a multidimensional array will be the name of the file where the result is written. The same change will be made for the command line queries because it wouldn't be useful to display o huge amount of data on the console or on the GUI.
- Here you can see the changes added to the asqldb. The JDBCMArray should provide a getResultSet method but this is not implemented yet and this is not a priority now. The GUI modification is not yet merged. I have modified the formatResultSet method from the DatabaseManagerSwing class.
- Below is the screenshot of the asqldb GUI returning the name of the file on the coll column:
- No major problems for this week
- Implement the Hibernate dialect for the asqldb
- Test the dialect by making some multidimensional array queries
- Complete the rasql tests with the other 40 tests that are not available yet.
- If the Hibernate dialect will not raise other problems, I will start to connect the SOS to Rasdaman using Hibernate.
Weekly report no. 6 - Sensor Data Access for Rasdaman
- Implement the Hibernate dialect. It proved that I have nothing to change in the Hibernate dialect. So, in order to use Hibernate with asqldb I had to implement two user types and register these types in the AppFactory.
- Test hibernate dialect with the select asql queries used to test the jdbc multidemensional array adaptation. We have 104 select test from which 22 are failing but this is because the hsqldb doesn't support all the rasdaman queries yet. The test were adapted to hibernate - little changes were needed and we splitted these into four folders(depending on the types returned by the rasql query) because the hibernate mapping types have to be precisely specified and we needed a clue in order to know what java class to be mapped to the result returned by Hibernate.
- Here is the commit with all the changes for the hibernate select tests. Some temporary files were added by mistake and I removed them.
- Write the blog post
- Extend the documentation for the asqldb project
- Adapt command line queries to return the ouptut into a file instead of writing the result to the console. This behavior was already implemented for the asqldb's GUI.
Weekly report no. 7 - Sensor Data Access for Rasdaman
Weekly report no. 9 - Sensor Data Access for Rasdaman
- Extend the documentation for the AslqdbTest project.
- Create the concept for the SOS in order to support multidimensional arrays. This concept splits the work into three main use cases:
- A user downloads multi-dimensional arrays using the SOS interface. The arrays will be encoded in O&M, for which a profile is not defined yet.
- A user explores time series data, then clicks on a specific cell. The data from that cell is displayed in the time series viewer and an existing SOS encoding is used.
- We didn't define an use case yet, but, instead of providing timeseries of cells - like the second use case, the SOS will provide timeseries data.
- Create the tasks definition for the first step presented in the concept: multi-dimensional arrays support
- Create the hibernate mappings - this is closely related to the existing database concept and model. We started adapting the SOS without using multi-dimensional arrays. We started using the ASQLDB mappings like this is not supposed to support multi-dimensional arrays. When everything will work well considering our supposition, we will pass to the second part: add a new "arrayValue" observation type to the SOS data model and test again by using array data this time. The code for the mappings can be found on GitHub, on the branch feature-rasdaman.
- Add the service loader file. This will be used by the SOS in order to view the ASQLDB datasource.
- Build and deploy the SOS.
- The installer from the interface does not display the Rasdaman datasource as an option.
- Fix the problem with the installer and run the installer
- Fix bugs
- Update the config file to get a large test dataset and insert the test data using the batch operation.
Weekly report no. 10 - Sensor Data Access for Rasdaman
- Solved the problem with the web interface installer that didn't display the rasdaman datasource into the datasources select list. The problem came from some undeclared maven dependencies and modules.
- Since the rasdaman datasource can be configured from the SOS web interface, I started the configuration. The default credentials for ASQLDB are:
- For the moment, the ASQLDB stores the data into a local file - not on a server. A server version will be available after we finish the work with the hibernate mappings and the multidimensional array support.
- Configure tomcat and eclipse in order to run the SOS web application using remote debug. This is extremely useful and I noticed an interesting behavior which was also realy helpful:
- The SOS web interface doesn't see the asqldb's jdbc driver when it runs without using debug. This is a problems that is still unsolved but, fortunately, I could debug and test the application using the debug mode.
- The SOS web interface doesn't see the asqldb's jdbc driver, only when this is run using the debug mode
- I had problems with the default password for the user "SA" because this is seen like an empty field by the SOS web interface. As a result, I created a new user - SIMONA, with password SIMONA. This way, this problem was solved.
- I encountered some problems with the ASQLDB database because is was giving an error every time I tried to access it. This problem occured out of the blue and the only solution I found was to recreate the file where the database was stored.
- When I reach the end of the datasource intallation and the core tables are actually created into the database, I get the error that can be seen in the image below. This error occurs when the FeatureOfInterest table is created. This table has a field that doesn't have a primitive type - Geometry - and I think it needs hibernate to associate the column type with the Geometry type.
- Solve the problem with the featureOfInterest table.
- Insert data with transactional SOS operation - The first steps will use only ASQLDB to insert and query data. Only after the database structure will be created and the connection with ASQLDB will be tested, we will start inserting multidimensional array data.
- Update config file to get the large test dataset.
- Query data from the SOS.
- Add new "arrayValue" observation type to SOS data model.
Weekly report no. 11 - Sensor Data Access for Rasdaman
- Trying to solve the problem with the featureOfInterest table that uses the Geometry Type from Hibernate Spatial. Because the ASQLDB doesn't support the Geometry Type, we've searched an alternative to use this type in our database. After some research and discussions we have two alternatives:
- The solution that uses the geomajas project was a priority and I started to integrate it into the ASQLDB project. Because ASQLDB is not a Maven project yet, I had to search all the necessary jar files. Testing how this project works with HSQLDB was not as easy as I thought - I started reading all the documentation and the final conclusion is that I only have to change some xml files in order to make it run with ASQLDB.
- I also build the geomajas project on it's own, using maven, and it succesfully builds. I did this because I discovered that integrating geomajas into ASQLDB was not what we needed, because we have to integrate this project into the SOS if we are going to use it.
- But the geomajas project seems to complex for what we need and we finally decided to implement a custom type (as we did for the RasGMArray type in the AsqldbTest project). This new type will use the JTS library to serialize the Geometry object to a String and back - this way we can store the Geometry type as VARCHAR and receive a Geometry object when executing queries using Hibernate.
- I had problems with the versions of the libraries used for the geomajas project, especially with the spring libraries.
- Solve the problem with the Geometry type and continue with the tasks that were listed the previous week.
Weekly report no. 12 - Sensor Data Access for Rasdaman
- Solved the problem with the Geometry type that was not supported by ASQLDB. Here is the code written in order to register the new type in Hibernate. I've also tried to create a new SessionFactoryProvider for Rasdaman that was supposed to register the user type into the Hibernate configuration, but this was not what we needed. You can find here the custom user type for the Geometry type.
- Run datasource configuration for Rasdaman using the SOS web interface. It creates all the necessary tables in the ASQLDB. Below you can see the tables displayed in ASQLDB's GUI.
- Add user field for full database path configuration. Before, only the name of the database could be configured. The full database path was configured for Linux and this would have been a big issue for Windows users. The inalterable database path was a problem for anyone because it couldn't be set to whatever location the user prefers. The "Database" field can be modified in order to change the path to the database. You can see it in the image below.
- Remove Host and Port fields because ASQLDB uses a file for storing the database and it doesn't need this kind of information.
- Remove hardcoded values from the source code.
- The batch example from the Test Client of the SOS web interface throws an exception when run.
- Re-install using a database that was used in a previous installation (i.e. The database already contains the SOS tables) throws a NullPointerException.
- Solve the problems presented above
- Add an "arrayValue" observation type to SOS data model in order to start using ASQLDB with Rasdaman and store multidimensional arrays.
- Establish array test dataset for transactional feeding and use it for testing the Rasdaman datasource.
Weekly report no. 13 - Sensor Data Access for Rasdaman
- Create a SQL script for ASQLDB that inserts all the data that would have been inserted using the batch example. I had to create this script because the problem with the batch example was not solved and the time is limited. Now we have a java class that executes all the queries written into the SQL script. I used a java class to run the queries because we also need to populate the ArrayValue table and this was easier to implement a Java class for this task. Why would be this task more complex than the other table insertions? - because the ArrayValue table stores array values, which means it must connect to rasdaman, put the data in there and save the necessary information to access this data when the user asks for it. Because the array insertion is not implemented yet for ASQLDB, we have to use create rasdaman collections, populate the collections, get the OIDs for these and store the OIDs in ASQLDB.
- Started to add an "arrayValue" observation type to SOS data model in order to start using ASQLDB with Rasdaman and store multidimensional arrays:
- Fix the clear datasource problem from the Admin web interface by overriding the parseDatasourceProperties method in RasdamanDatasource.
- Solve datasource configuration problem when not running in tomcat debug mode. This problem occured only because the ASQLDB's database file location didn't provide access for the user used by tomcat when not running in debug mode.
- Test the example operations for inserting data using SOS 2.0 examples. InsertSensor is one of the operations that returns success. The insertObservation operations throw an unexpected exception.
- Started writing installation instructions for the Sensor Data Access for Rasdaman project.
- Couldn't solve the problem that doesn't let the user configure the same database twice. I tried to debug and I got rid of the NullPointerException but other problems occured.
- Add encoders for the ArrayValue observation type and make all the other changes that are necessary in order to use this new type within the SOS
- Complete the installation instructions
- Create a demonstration for the project