GSoC 2013 WPS Admin Web Application Weekly Reports

This page will contain the weekly progress reports for the new WPS admin application GSoC project.

Week 14 (16th September - 22nd September)

This is the last weekly reports update. The past week has been spent mostly on fixes, more testing, updates, and clean up.

Status

Database backup fix

After digging through HSQLDB documentations, I found a solution to how to gracefully backup the database. Using an SQL command, the database will be locked, all memory data is written to a file, and the file is backed up in a tar.gz file to the path specified. Perfect backup, and without restart.

For the restore, I have added a command to shutdown the database and then overwrite the database files with the backup files. This is done through a special class provided by HSQLD. However, in order to restart the database and resync the configurations, a restart of the application is necessary.

I've updated the backup & restore interface with more information for the user. In the backup page, I've added a note that the database will be locked during backup. In the restore page, I've added a warning that the application must be restarted after restoring the database.

UploadController & BackupService Tests

I've added some tests for these classes, in addition to test files and folders in the test resources folder.

HSQLDB Conflict Fix

My mentors found a solution to the conflict that was happening between the new HSQLDB version used in the admin module, and an old version used in the parent module. The solution was to find an alternative to HSQLDB which provided the same functionality. I have added their solution to the project.

File Cleanup

Continued to delete unused files and folders from the old module. Some folders still remains as my mentors are still investigating if they're still needed.

Documentation

I've revisited almost all of the classes and updated the Javadoc with a brief description of each class, its purpose, and how does it fit in the overall application.

Blog Post

The final blog was posted which included quite a few diagrams and screenshots: Meet The New 52° North WPS Admin Web Application

Week 13 (9thSeptember - 15th September)

Status

R Script Upload

screen-21.png

Figure 1

The R script uploading is implemented as can be seen from Figure 1 above. However, the function requires a working R Module that uses the new configuration manager, and since the current module still use the old WPSConfig, the function doesn't recognize the LocalRAlgorithmRepository, so it will through an error message to the user to that effect.

In addition to adding RScript uploading, I have improved the uploading function to provide better exceptions and error handling. However, one shortcoming is recognizing file type. Currently, the module will check for the extension and whether the file is empty, while this is sufficient for now, a more robust way will require the use of external libraries to check for the mimetype and try to handle the many variations in how operating systems and browsers define mimetypes.

Backup Module

Most of the time was spent on this module. The module has two functions, backup and restore. The following screenshots shows the backup function with description:

screen-17.png

Figure 2

screen-18.png

Figure 3

Figure 2 shows the initial interface where the user can select which items to backup. Figure 3 shows the generated backup file after the user clicks the Backup button. The module will find the items required to backup, zip them together, and return a download URL to the user.

screen-19.png

Figure 4

Figure 4 shows the restore function. The user select the zip backup file created previously, the function will overwrite current files with the backup files inside the zip archive.

File Cleanup

Deleted a few files and folders that are no longer used. There are still many more, which I'll need to discuss them with my mentors before deleting, as I cannot figure out if they're used or not.

Third Blog

The last blog entry which discusses the results achieved has been submitted, and should be published tomorrow on the main blog.

Problems

Database backup was not as straight forward as I thought. While I was able to backup & restore the data folder for the database, the backup files are not consistent. To do that, we need to shut down the application and then do the backup. This is because HSQLDB will write the data in memory to actual files during restart or shutdown. So, I need to find a seamless way to achieve this, or find an alternative way to backup the database.

Tasks for next week

  • Tests for backup & upload modules.
  • Remove unused files & folders.
  • Last minute revisions, improve comments, logs, error messages.
  • Update the wiki page with blog links and new information.
  • Setup the demo. I was planning to do this by today, but I ran into problems uploading the WAR file due to the large size. I'll try again tomorrow.
  • Find a solution to the database backup problem.

Week 12 (2nd September - 8th September)

Status

Standard Modules Tag

In addition to the form input tag I created last week for custom forms, I wanted to use the same concept in creating the standard or pluggable modules. This led me to the following:
  • Created a custom JSP tag for standard or pluggable modules ( standardModule.tag). This dramatically improved maintainability through code reuse.
  • All pluggable modules (repositories, generators, and parsers) will have to only include two lines to import and use the shared tag.
JavaScript

JavaScript code was scattered and repeated throughout the JSP files. I have grouped it and structured it into three shared files.
  • commonjs.js is for all modules. It adds the document ready function for jQuery, and handle alert messages.
  • standard.module.js is for standard or pluggable modules. It provides AJAX form processing, error reporting, and module status toggle.
  • custom.module.js is for custom made forms. It also provide AJAX form processing and error reporting.
  • Small fragments that are only used once are located in the appropriate files (e.g. toggle algorithm status AJAX code in repositories.jsp).
  • All JavaScript code has been revisited and improved.
Configuration Service

I have revisited the configuration service, which can be seen as the core of the configuration API, and made an overhaul to how it process configuration module values:
  • Now configuration modules are validated, processed, and saved as one unit, instead of entry per entry. What this mean is that before this improvement, when a form is submitted, the values are evaluated and treated individually, this can cause inconsistency in the module state. Now all values are checked together, if they all pass the validation, they're all saved in the database together, if that succeeded, then they're passed to the setter methods.
  • Improved the way configurations are synchronized during startup. Different behavior and methods for existing and new modules.
  • Overall improvements and rewriting of methods, hiding methods from the public API since they're not going to be used by outside clients, and better comments and log messages.
  • Simplify methods calling. Some methods that are called by clients (e.g. set module status) require an instance of the module. I wanted to make it easier, so now the client needs to only pass the fully qualified name of the module and the new status. This is already been done in setting configuration values.
  • Although configuration entries are now processed as a unit, algorithm entries are processed individually and can be activated/deactivated on an algorithm by algorithm basis as this is a requirement for the application.
Exceptions

Improved the custom exception WPSConfigurationException.java. Now the exception will register the entry that caused the exception to be thrown, in addition to the error message. This will be passed to the client where the client can react to the error.

ValueParser

Updated the ValueParser.java to provide more user friendly error messages, since these messages will appear in the user interface when there is a validation error with the entry.

The combination of improvements in the service, the exception, and the value parser allowed us to have the following clear and exact error message in Figure 1:

error-01.jpg

Now this might look easy, and it would be if this is a standard module with straight forward simple types, but being specially designed configuration module where each input is a special class, the whole validation and error reporting has to be built very much from scratch.

R Script Upload

I have finished most of it, however, it is still not ready, so I'll show it next week.

ConfigurationDAO

Added debug messages to the configurations database operations to help in detecting data manipulation problems.

Problems

I was unable to achieve error code resolving at the client side. I was able to set up the errors.properties file and configuration, I was able to set the error code, and I was able to pass the code to the presentation layer. However, the code is resolved by Spring at that level, and since I'm returning errors as JSON object for AJAX processing, the errors.properties file is not resolved and default error messages are displayed. I need to dig a little deeper, however, this is not a high priority.

Tasks for next week

  • Backup and restore: Implement configurations backup and restore. This is a little tricky as we have several types of configuration. First we have configurations saved in the database, then we have the logback.xml and wpsCapabilitiesSkeleton.xml, both contains configurable values. So I need to figure out a way to back up and restore all.
  • R Script upload: finish the function.
  • Demo server: Setup a demo server for the application.
  • Cleanup: There are many left over files in from the previous webapp module, I will investigate them and delete unneeded ones.

Week 11 (26th August - 1st September)

Status

Upload Process

Uses jQuery plugin for a seamless file uploading on the client side. On the server side, the UploadController will parse the uploaded Java file, and the package name will be identified. The Java file will be saved in an upload folder under a directory structure that matches the fully qualified name.

For example, uploading a process with the FQN of org.example.Process.java will be saved as:

/WEB-INF/classes/uploaded/org/example/Process.java

Since the file will be parsed and the package name is read, there is no need for the user to write the FQN manually.

The process description file will be uploaded to the same location as the Java file.

The following figures show the upload function.

upload1.jpg

Figure 1: The upload interface

upload2.jpg

Figure 2: Successful upload

upload3.jpg

Figure 3: Incorrect file format

upload4.jpg

Figure 4: No file uploaded

Modules Updates

I keep going back to update previous modules and improve them. Three modules have been updated and improved. The modules are: User, ServiceIdentification, and ServiceProvider. The updates included:
  • Update the controllers to return a ValidationResponse object as a JSON object.
  • Update the view to read the returned JSON object and highlight fields’ errors.
  • Update the view to use the custom input tag.
  • Update beans to use annotated validation rules.
Input Custom Tags

Last week I have created an input tag to simplify writing HTML forms. This week, I have updated and combined the tags. Now supported inputs include text inputs, checkboxes, password fields, and submit buttons. Using custom inputs tags dramatically reduced form codes and improved it's maintenance.

The Big Picture

At this point, I wanted to highlight the approach in developing the configuration modules. Basically, we have two types of modules:

1. Pluggable modules (Repositories, Generators, Parsers, and Server)
  • Allow backend developers to plug-in configurations for their modules easily by implementing the ConfigurationModule interface.
  • The backend developer is able to define configurations entries in a type save manner.
  • The values are validated, parsed, passed, and saved in a database using ConfigurationService and ConfigurationDAO.
  • The module will appear automatically in the appropriate page according to the category selected by the backend developer.
  • Display, view, and error highlights are all handled automatically for all pluggable modules. All pluggable modules share the same view code to allow for automatic display, therefore, customization is not possible here.
Note: While you cannot plug in any modules in the Server, it still uses the same structure as other pluggable modules (no special storage or display needs), so I included it here.

2. Custom modules ( LogConfiguration, ServiceIdentification, ServiceProvider, and User)
  • Custom modules developed for special requirements. These modules cannot use the pluggable modules infrastructure because they have different requirements and structure.
  • These modules are handled manually one by one through special services and DAOs classes (e.g. LogConfigurationService, LogConfigurationDAO).
  • The validation and view for each module is created manually using annotation validation, custom input tags, and JSON response objects.

Problems

None.

Tasks for next week

  • Centralized errors file: currently, errors are written in different beans, this is not ideal. I would like to use error codes instead, and translate these errors code from a centralized errors.properties file.
  • Revisit pluggable modules: I will investigate further improvements to the pluggable modules infrastructure, mostly in the view side, and if see if I can develop more custom tags to simplify them.
  • Upload R script.

Week 10 (19th August - 25th August)

Status

Server Configuration Module

A module was created to configure and manage the server. Figure 1 shows the module.

fig1.jpg

Figure 1

Log Configuration Module

A module was created to manage log configurations from the interface. The module writes the new configcuration directly to the logback.xml file. Figure 2 shows the module.

fig2.jpg

Figure 2

Test Client

The test client have been moved from the old admin module. However, a lot of improvements have been added to it which included:
  • Improved the scripting using jQuery and AJAX.
  • Cleaned up and merged the HTML forms and inputs. Now it has one form.
  • Display the output in the same page.
  • Merged the design with the rest of the interface.
  • Removed the styling from the page to the main CSS file for more centralized control.
  • Moved the requests and codemirror folder and files under the resources folder.
The following figures show the test client.

fig3.jpg

Figure 3

fig4.jpg

Figure 4

Custom tags

Created custom JSP tags to make dealing with HTML forms much cleaner and easier. For example, here is a fragment before using the tags (two text input fields):

<div class="form-group">
<form:label class="col-lg-3 control-label" path="wpsfileAppenderFileNamePattern">File Name Pattern</form:label>
<div class="col-lg-6">
<form:input class="form-control" type="text" path="wpsfileAppenderFileNamePattern" />
</div>
<div class="col-log-1">
<span class="label label-danger">Required</span>
</div>
</div>
<div class="form-group">
<form:label class="col-lg-3 control-label" path="wpsfileAppenderEncoderPattern">File Encoder Pattern</form:label>
<div class="col-lg-6">
<form:input class="form-control" type="text" path="wpsfileAppenderEncoderPattern" />
</div>
<div class="col-log-1">
<span class="label label-danger">Required</span>
</div>
</div>

Will become after using custom tags:

<input:textInput label="File Name Pattern" field="wpsfileAppenderFileNamePattern" />

<input:textInput label="File Encoder Pattern" field="wpsfileAppenderEncoderPattern" />

textInput is a custom tag that allows us to reuse the input code in the first fragment. This is the content of textInput:

<div class="form-group">
<form:label class="col-lg-3 control-label" path="${field}">${label}</form:label>
<div class="col-lg-6">
<form:input class="form-control" type="text" path="${field}" />
<span class="help-block">${desc}</span>
</div>
<c:if test="${empty required || required eq 'true'}">
<div class="col-log-1">
<span class="label label-danger">Required</span>
</div>
</c:if>
</div>

All we need is to pass the field name and the label, and we will have our field. Other custom input types include checkbox. Currently, the custom tag is used in the log module, but I plan to bring the old modules up to speed.

Improve Error Handling

AJAX is being used to submit forms, but when we get field errors, it is not possible to know which field causes the error, we only get a bad HTTP status (e.g. 400). To solve this, and to provide better error reporting to the user, a ValidationResponse class is created. The class will hold all fields errors and will be returned as JSON object in case of an error. The client then loop through this object and highlight the fields with errors. Figure 5 shows an example:

fig5.jpg

Figure 5

Problems

In the test client, the returned XML is not well formatted or indented. I think somewhere upstream we need to set the output to pretty print.

Tasks for next week

  • Upload processes.
  • Continue to improve error reporting and messaging.
  • Bring the rest the forms and modules up to speed in terms of the improved error handling and custom form tags.

Week 9 (12th August - 18th August)

Status

User and Access Management
  • Created two levels, admin and user. Both have the same privileges, except admins can manage and add new users to the system.
  • Added the ability to change password (Figure 1 & 2).
  • Passwords are now hashed in SHA-256 with added salt.
  • User management (add, edit, and delete users. "Figure 3 & 4").
  • Implemented the remember me function. Logins are remembered for a month.
ui-001.jpg

Figure 1

ui-002.jpg

Figure 2

ui-003.jpg

Figure 3

ui-004.jpg

Figure 4

HSQLDB Relative Path

I've solved the problem of the relative path for the database files. I have created a listener which define a system property that contains the root of the web application. I have then used this system property in setting the relative path of the HSQLDB files. Now, the files are deployable and portable.

Integration Tests

These tests provide end to end testing starting from user requests and form submissions in the web layer, all the way down to the DAO layer. The integration initialize the entire Spring context and verify all of the wiring, settings, and configurations.

The following integration tests were implemented: RepositoriesControllerIntegrationTest, ParsersControllerIntegrationTest, GeneratorsControllerIntegrationTest, ServiceIdentificationControllerIntegrationTest, ServiceProviderControllerIntegrationTest, and UsersControllerIntegrationTest

Problems

None.

Tasks for next week

  • Server configuration module.
  • Log configuration module.
  • Test client.

Week 8 (5th August - 11th August)

Status

Service Identification Module

A service identification module ( ServiceIdentification, ServiceIdentificationController, service_identification.jsp) is created to update the wpsCapabilitiesSkeleton.xml document. The module uses the same configuration manager infrastructure as the main modules (repositories, generators, and parsers). However, after saving the values to the database, the CapabilitiesService is called to update the wpsCapabilitiesSkeleton.xml from the values entered by the user and saved in the database.

Figure 1 shows the module.

ui-08.jpg

Figure 1

Service Provider Module

Uses the same flow and structure as the service identification module. Figure 2 shows the module.

ui-01.jpg

Figure 2

Generators

The generators controller has been completed. It’s very close to the repositories controller except it doesn’t have methods to handle algorithms. Figure 3 shows some sample generators.

ui-03.jpg

Figure 3

Parsers

Similar to generators. Figure 4 shows example parsers.

ui-04.jpg

Figure 4

Interface Functionality

I’m trying to provide as much feedback to the user as possible. AJAX is used to provide a dynamic behavior and clear visual cues for successful and erroneous flow. The following figures shows some examples.

ui-02.jpg

Updating the status of a configuration module, overlay and fading alert on top.

ui-05.jpg

Updating the status of an algorithm, in-place and fading alert after successful execution.

ui-10.jpg

Successfully updating a module, alert and fading green fields.

ui-09.jpg

Unsuccessful module update, permanent error alert and fading red fields.

Exceptions & Errors Handling

A lot of time is spent planning and revising the approach to exceptions and error handling in the admin module. I finally arrived at a plan to follow. The approach is:
  1. For exceptions which have no possibility of recovery by the client, I will wrap the checked exception into unchecked exception and throw it. Example is if the XML file is corrupt and cannot be parsed, there is nothing the user or the client can do at runtime, so no need to have a checked exception. The same with FileNotFoundException, since the paths to files are hard coded and not provided by the user, there is no possibility to recover if the path is wrong, so the exception is wrapped in a RuntimeException.
  2. For user input in the configuration modules forms, I will throw a custom exception and bubble it up the layers until it reaches the web layer. There, I’ll catch the exception, log it, and display an error message to the user and allow him to correct the input. Figure shows an example of this:
  3. For unchecked and uncaught exceptions, I have setup a default exception page to display the error code and the exception message. Figure shows an example of an uncaught dummy RuntimeException:
ui-07.jpg

Problems

I'm having some difficulties in setting a relative and portable path to save HSQLDB files. Currently I'm using an absolute path to a my development folder. I think the solution is to define a system property which identify the webapp path, and then use that in setting HSQLDB file path. I'll explore this approach and see how it goes.

Tasks for next week

  • User management and access module.
  • Improve service calling. Right now, when a user submit a configuration provider module for example, the code manually calls the service method “configurationManager.getCapabilitiesServices().updateServiceIdentification();” which will update the XML file from the database values entered by the user. This is fine, and it’s working, however, I would like to have a less coupled approach. Perhaps I’ll have a custom event that is triggered whenever a special module that requires further actions is submitted.
  • Integration tests. Provide integration tests from the web layer -> service layer -> DAO layer. For these, I’ll use a test profile with in-memory database and test modules.

Week 7 (29th July - 4th August)

Status

The user interface has been implemented.

ui-1.jpg

Figure 1: Main page

ui-2.jpg

Figure 2: Admin pages are protected

ui-3.jpg

Figure 3: Repositories configuration pages with two example modules

ui-4.jpg

Figure 4: Viewing the configurations and algorithms of a repository

ui-5.jpg

Figure 5: Successfully saving configurations

ui-6.jpg

Figure 6: Error in saving the configurations. The error message might need some improvements. For now, it shows the exception message with the key of the entry causing the exception

The User interface

The user interface has been implemented “early version”. What is seen in the figures is actual working code using what have been accomplished in the configuration manager and services.

My goal in designing the interface is to have most, if not all, options accessed in one click, so I avoided dropdown lists and nested menus.

The interface uses:
  • Apache Tiles: Tiles is a powerful and centralized templating framework. I have created the following fragments for the template: header, footer, sidebar, and a content pan.
  • Bootstrap: A robust front-end development framework with CSS and JS components.
Controllers

Controllers have been implemented to process requests and responses via RESTful methods. Currently, we have four controllers: MainController, RepositoriesController, GeneratorsController, and ParsersController. The RepositoriesController is the one I have worked on the most, the others are placeholders. More controllers will be added for other pages and features.

ConfigurationManager

As I discussed last week, the configuration manager was repetitious, so I have updated it to retrieve all the services, and enable the calling code to access the services.

General Improvements

I keep going back to refactor and improve previous code, things I keep learning about and improving:
  • Exceptions: Checked vs. Unchecked and when to use each, when to log an exception and when to rethrow it.
  • Logs: What to log, what level is best for each log entry.
  • Tests: Create smaller and faster tests; eliminate external dependency as much as possible.

Problems

We had a conflict between the new HSQLDB version and an old one used in the project. I have communicated this problem to the mentors, and we’re working on a solution.

Tasks for next week

User interface

Continue work on the interface, implement other pages.

Controllers

Finalize the RepositoriesController and start work on the GeneratorsController and ParsersController.

Week 6 (22nd July- 28th July)

Status

Figure 1 shows the current status.

class-diagram-3.png

Figure 1

UserDAO, JdbcUserDAO, UserService, UserServiceImpl

The UserDAO interface allows for traditional CRUD operations on the user table. To support this, a user table has been created. The DAO return and receive users information as a User object.

LogConfigurationsDAO, XmlLogConfigurationsDAO, LogConfigurationsService, LogConfigurationsServiceImpl

The LogConfigurationsDAO interface is implemented to parse the logback.xml file. The parsed data is sent back to the service as a LogConfigurations object. The DAO also provide the ability to write the LogConfigurations object to the logback.xml file to update the log configurations.

CapabilitiesDAO, XmlCapabilitiesDAO, CapabilitiesService, CapabilitiesServiceImpl

Parses the wpsCapabilitiesSkeleton.xml and return the service identification and service provider information back to CapabilitiesService. It also takes a ServiceIdentification and a ServiceProvider and update the wpsCapabilitiesSkeleton.xml file.

ConfigurationManager

Updated to provide access to the new services.

Security configurations

Spring security configurations have been added which replaced previous 52°North security configurations. Currently, I have setup two roles, an admin and a user. The exact authorities and access privileges of each is yet to be determined. This will be decided sometimes down the road after we are at a more mature stage of development.

Right now, I setup a default admin user with ‘wps’ as a user and a password to match the documentations and tutorials.

Problems

None.

Tasks for next week

ConfigurationManager

The current state of the ConfigurationManager is not ideal. Currently, it performs direct calls to services. There are a lot of repetitions and violations of the DRY principle between the manager and the services. Therefore, I intend to come up with an improved approach, probably a way to simply get access to service operations, instead of calling it directly. For example, instead of:

configurationManager.getAllConfigurationModules();

It will be:

configurationManager.getConfigurationServices().getAllConfigurationModules();

Another example:

configurationManager.getUserServices().insertUser(user);

So we call the getAllConfigurationModules() and insertUser(user) service methods directly instead of relying the call from the manager.

Web and client side

A lot of progress has been made in the backend side, so I would like to establish the foundation for the web and client side:
  • Choose a templating framework and create a template and pages.
  • Start working on controllers to handle RESTful methods.
  • Create an initial user interface.

Week 5 (15th July- 21th July)

This week involved a lot of improvements on the configuration manager, and a move to a new storage medium.

Figure 1 shows the current status.

class-diagram-1.png

Figure 1

Status

ConfigurationManager, ConfigurationManagerImpl

The configuration manager has been changed to work as a Façade over the application services. Thus, the configuration manager will provide an application wide access to all configurations retrieval, storage, and management.

ConfigurationService, ConfigurationServiceImpl

The service will perform the actual logic of configurations syncing and operations. The reason for this architecture is to reduce coupling and complexity for clients. We can have multiple services covered by the manager, and where clients depend only on the manager instead of multiple services.

ConfigurationDAO, JdbcConfigurationDAO

The configuration DAO have been updated and implemented in JDBC instead of XML to reflect the move from XML storage to database. Having a separate DAO interfaces and implementations allow us to add new DAOs for configuration that have special requirements (e.g. storing to log configuration files).

Figure 2 shows the future direction, and how we can add new services and DAOs.

class-diagram-2.png

Figure 2


ConfigurationCategory

A configuration category enum has been added to be used by the configuration manager and service. This will allow us to retrieve configuration modules based on categories. For example:

configurationManager.getConfigurationModulesByCategory(ConfigurationCategory.PARSER);

New HSQLDB database

A new database has been added to store configurations. Currently, the database has two tables, one to store configuration values, the other to store algorithm entries. More tables will be added in the future to cover other types of configurations (e.g. users).

The database has been setup to work as light, embedded in-memory database, but with persistency between startups. All schemas and data are recreated and loaded in memory during startup. The footprint is very small, the jar itself is a little over 1MB, and during testing, the database files were less than 10KB (I expect it to be around 200KB to 300KB during production).

After a few experiments, I decided to go with JDBC using Spring JDBC template instead of using Hibernate. Hibernate mapping added unnecessary complexity for simple configurations storage and retrieval.

Improved Testing

Tests have been updated and improved for the manager and service. In addition, new test configuration modules have been created to have a more thorough testing. Furthermore, a test profile with a test database and sample test data has been setup. Finally, some of the previous unit tests feels more like integration tests rather than unit tests, therefore, many of the tests have been updated and improved to be more isolated and independent. This review will cover the rest of the tests in the coming weeks.

Simplification

As it was our goal last week, the ConfigurationModules class has been deleted as we now build directly to the database. RepositoryService and RepositoryServiceXml have been replaced by the more generic and versatile ConfigurationService and ConfigurationServiceImpl which covers multiple categories of configurations.

Second Blog

I wrote the second blog where I explained what have been done in the past five weeks and what’s next. The blog should be published next week.

Problems

None.

Tasks for next week

  • User access and management:
    • Replace current security configurations with Spring security.
    • Create user tables and access roles.
    • Create and implement a UserDAO and a UserService.
  • Log Configurations:
    • Update LogPropertiesDAO implementation to the new logback.xml.
    • Create a Log bean to hold log configurations.
    • Create and implement a LogService.
  • Capabilities:
  • ConfigurationManager
    • Wrap the previous services under the configuration manager.
  • Testing
    • Continue to review and improve previous unit tests.

Week 4 (8th July- 14th July)

Status

The following have been committed to the development repository:
  • ConfigurationManager, ConfigurationManagerImpl
    • Initialize configurations during startup.
    • Get all configuration modules.
    • Get a configuration module by class or by name.
    • Set a new configuration value after parsing it to the correct type.
    • Sync a configuration value from the datastore (during initialization).
    • Get a configuration entry value (after parsing to the caller's required type).
    • Pass a configuration value to configuration modules setter methods during initialization and whenever a new value is set.
  • ConfigurationModules
    • Build a map for all modules that implement the ConfigurationModule interface. This class uses Spring’s ListableBeanFactory to get all classes of type ConfigurationModule.
    • For each module, build the configuration entries map (Key, Entry), and ensure unique entry keys for each module (skip the entry and log an error if the key already exists).
    • Provide configuration modules to the ConfigurationManager.
  • ValueParser
    • Helper class used by the ConfigurationManager to parse values (form strings) to the configuration entry’s type. Throw an exception if the parse fails.
  • ConfigurationModule
    • An interface used by configuration classes to identify themselves as a configuration module. Implementing this class will require the module to provide a module name, status, a list of configuration entries, and a list of algorithms (for repositories).
  • ConfigurationEntry<T>
    • An abstract class which is implemented as a String, Integer, Double, Boolean, File, and URI configuration entries types.
  • AlgorithmEntry
    • To hold an algorithm entry in a repository. It has a name and active status fields.
  • WPSConfigurationException
    • To wrap and unify all configuration exceptions.
  • RepositoryService, RepositoryServiceXml
    • Works directly with the current XML configuration classes such as WPSConfigurationImpl to sync, update, and store values in the wps_config.xml file. Currently, the implementation stores configurations as a property (same as algorithms). This is inflexible; therefore, a new approach is needed (see next week tasks section).
      Half way through this, I realized that this not the way forward as I’ll need to also sync and store algorithms, generators, parsers, and server information. A unified approach is better than hard coded manipulation (e.g. getAlgorithmRepositoryList() , getGeneratorList()) and comparing them manually with the declared configuration modules.
      Therefore, this XML implementation is not complete, as it will most likely not going to be used.
  • HSQLDB and Hibernate
    • Setup and configure HSQLDB database and Hibernate to be used for configurations storage.
  • Adjust to slf4j
    • Update previous work to slf4j logger API to match the rest of the project.

Problems

  • The project has started using slf4j which uses new configuration file (logback.xml). This render previous work to read and write to log4j.properties obsolete. An update to LogPropertiesDAO implementation is needed to work with slf4j configuration file.

Tasks for next week

Week 3 (1st July- 7th July)

Status

The past week was spent in designing the configuration manager for WPS. Before I go into the details of the design, I will talk a little bit about the requirements which affected the design decisions. The configuration manager needs to do the following:
  • Enable developers to create configuration modules to appear in the front end.
  • These modules need to publish their configurations entries and parameters with supporting details such as the title, description, whether it’s a required or optional, and the default value.
  • The configuration values should be type safe.
  • These modules need to automatically get the values when entered by the user in the front end.
  • Configurations modules, entries, and values need to be persisted and stored.
  • Synchronization is needed between default values identified by the module, new values entered by the user, and values previously persisted.
The design that addresses those requirements is shown in Figure 1 (work in progress).

Figure 1

  1. A class identifies itself as a configuration module by implementing the ConfigurationModule interface. This will require the module to provide the following:
    - A module name (e.g. Grass Repository Manager).
    - Whether the module is active by default or not.
    - The list of configuration entries.
    - The list of algorithms (for repositories configuration modules).
  2. Spring will instantiate and load all classes that implement the ConfigurationModule interface.
  3. ConfigurationModules will build a map of maps for the configuration modules where each module will have a map of key/configuration entry. The method of building the map is annotated as @PostConstruct which will be invoked by Spring after instantiating the class.
  4. The ConfigurationManager will handle most of the work. It will do the following:
    - Get all configuration modules and their configuration entries.
    - Set/get values for configuration entries.
    - Sync ConfigurationModules map of maps with stored configurations, and update/add/delete modules and configuration entries as needed.
    - Parse values to their correct types (using the helper class ValueParser).
    - Pass values to configuration modules during initialize, and whenever a new value is added.
    - Call DAO classes to persist configurations.
    - All the dependencies for the ConfigurationManager (e.g. ConfigurationModules, ValueParser) will be injected by Spring.
  5. From the ConfigurationManager, a ConfigurationModule can receive configuration values in two ways.
    - The first is manual like this:
    configurationManager.getAllConfigurationModules().get(this).get (“config.key.1”);
    - The second is by annotating a setter method with the configuration key, the value will be passed automatically by the ConfigurationManager every time a new value is added to the entry:
    @ConfigurationKey(key = " config.key.1")
There is no committed code at this point as it’s not stage ready. By next week, the code will be finalized and committed.

Problems

  • I’m finding the current configuration file (wps_config.xml) rather limiting. For example, the current schema for repositories is:
    <Repository name="..." className="..." active="...">
    <Property name="..." active="...">value</ Property>
    </ Repository>

    This schema is not flexible enough to support configuration entries, and it doesn’t separate actual configurations from algorithms. I would like to have something like:
    <Repository ….>
    <Configuration key="..." title="..." description="..." required="...">value</Configuration>
    <Algorithm active="...">value</ Algorithm >
    </ Repository>

    Therefore, I’m considering creating a new schema; however, I’ll postpone this step a little bit until more progress is made.
  • Some concept problems with advanced generics. Need to do some reading.

Tasks for next week

  • Review and validate the design, and make any necessary corrections/changes.
  • Finlize the code for all classes in Figure 1.

Week 2 (24th June - 30th June)

Summary: The work focused on establishing the foundation level. A DAO layer, utility classes, and entity classes have been implemented and tested.

Status

  • A DAO layer has been implemented. This layer will be used by the configuration manager (indirectly through a service layer) to persist and load configurations and settings. The layer includes:
    • ConfigurationDAO, ConfigurationDAOXml, and ConfigurationDAOXmlTest: the implementation here is rather simple. It only calls existing classes and interfaces “WPSConfigurationDocument, WPSConfigurationImpl, WPSConfiguration, and WPSConfig” to store and reinitialize the configuration.
    • CapabilitiesDAO, CapabilitiesDAOXml, and CapabilitiesDAOXmlTest: parse the wpsCapabilitiesSkeleton.xml file and return a ServiceIdentification or ServiceProvider objects. In addition, it takes these objects and writes them to the wpsCapabilitiesSkeleton.xml.
    • LogPropertiesDAO, LogPropertiesDAOImpl, and LogPropertiesDAOImplTest: load and write log properties using Apache Common Configuration API. The implementation loads log4j.properties and returns a PropertiesConfiguration object, or write the object to the file.
    • UserDAO, UserDAOXml, UserDAOXmlTest: get and add users. The current implementation uses the users.xml as the data source for the users.
Note: as can be seen, data sources are distributed around multiple files. If times allow, I hope to create a unified datasource (SQLite?). The problem, however, is that changes to the datasources will require a lot of refactoring upstream to work with the new datasource. For now, this layer encapsulate and hide the underlining datasources, and it’s sufficient.

  • To support the loading and parsing of resources, two utility classes have been implemented:
    • ResourcePathUtil and ResourcePathUtilTest: uses Spring’s ServletContextResource and ClassPathResource to get the absolute path of resources and directories from the webapp and the classpath. The class takes any relative path and returns the absolute path.
    • JDomUtil and JDomUtilTest: parse XML resources to Document objects using JDom API. It also writes a supplied document to an XML file.

  • To hold the parsed and loaded data and to pass them around, three simple entity classes were implemented: ServiceIdentification, ServiceProvider, and User.
All of the above code can be found at: https://github.com/khalidq/WPS

Problems

  • Still working on the configuration manager. Several approaches were hindered by the fact that configuration modules are out of the scope of Spring (which resides in the webapp module). It seems a hybrid approach is needed.

Tasks for next week

  • Continue working on the configuration manager. Focus will be directed toward approaches for configuration modules to publish their configuration and retrieve their values.
  • Create a service layer that interact with the DAO layer. This layer will be called directly by the configuration manager to perform its operations (activate, add, get, etc.) *

    * Working on the second task depends on how much progress I make on the first task, which has higher priority.

Week 1 (17th June - 23rd June)

Status

  • Wrote the first blog: New 52° North WPS Admin Web Application
  • Updated the project's [[GSoC2013WPSAdminInterface] [wiki page]] with initial information such as:
    • Major phases to the development of the project.
    • Initial desired functionalists and features.
    • Other general information.
  • Setup the development environment, created a development repository, and performed first commit which included Spring dependencies and initial configurations.

Problems

  • Some concept problems with the configuration API.

Tasks for next week

  • Communicate with mentors to get a better understanding of the configuration API.
  • Design an approach to the API.
  • Implement initial code and tests for the approach.

· ConfigurationManager
Topic attachments
I Attachment Action Size Date Who Comment
Domain-Model.jpgjpg Domain-Model.jpg manage 382 K 08 Jul 2013 - 18:34 UnknownUser  
class-diagram-1.pngpng class-diagram-1.png manage 336 K 26 Jul 2013 - 21:17 UnknownUser  
class-diagram-2.pngpng class-diagram-2.png manage 275 K 26 Jul 2013 - 21:19 UnknownUser  
class-diagram-3.pngpng class-diagram-3.png manage 563 K 29 Jul 2013 - 20:00 UnknownUser  
error-01.jpgjpg error-01.jpg manage 186 K 09 Sep 2013 - 15:03 UnknownUser  
fig1.jpgjpg fig1.jpg manage 166 K 26 Aug 2013 - 07:57 UnknownUser  
fig2.jpgjpg fig2.jpg manage 217 K 26 Aug 2013 - 07:57 UnknownUser  
fig3.jpgjpg fig3.jpg manage 140 K 26 Aug 2013 - 07:58 UnknownUser  
fig4.jpgjpg fig4.jpg manage 377 K 26 Aug 2013 - 07:58 UnknownUser  
fig5.jpgjpg fig5.jpg manage 179 K 26 Aug 2013 - 07:58 UnknownUser  
screen-17.pngpng screen-17.png manage 35 K 16 Sep 2013 - 04:50 UnknownUser  
screen-18.pngpng screen-18.png manage 35 K 16 Sep 2013 - 04:50 UnknownUser  
screen-19.pngpng screen-19.png manage 32 K 16 Sep 2013 - 04:53 UnknownUser  
screen-21.pngpng screen-21.png manage 33 K 16 Sep 2013 - 04:47 UnknownUser  
ui-001.jpgjpg ui-001.jpg manage 187 K 19 Aug 2013 - 06:40 UnknownUser  
ui-002.jpgjpg ui-002.jpg manage 162 K 19 Aug 2013 - 06:40 UnknownUser  
ui-003.jpgjpg ui-003.jpg manage 140 K 19 Aug 2013 - 06:41 UnknownUser  
ui-004.jpgjpg ui-004.jpg manage 102 K 19 Aug 2013 - 06:41 UnknownUser  
ui-01.jpgjpg ui-01.jpg manage 195 K 12 Aug 2013 - 17:12 UnknownUser  
ui-02.jpgjpg ui-02.jpg manage 255 K 12 Aug 2013 - 17:12 UnknownUser  
ui-03.jpgjpg ui-03.jpg manage 231 K 12 Aug 2013 - 17:12 UnknownUser  
ui-04.jpgjpg ui-04.jpg manage 222 K 12 Aug 2013 - 17:12 UnknownUser  
ui-05.jpgjpg ui-05.jpg manage 124 K 12 Aug 2013 - 17:17 UnknownUser  
ui-06.jpgjpg ui-06.jpg manage 134 K 12 Aug 2013 - 17:17 UnknownUser  
ui-07.jpgjpg ui-07.jpg manage 121 K 12 Aug 2013 - 17:17 UnknownUser  
ui-08.jpgjpg ui-08.jpg manage 187 K 12 Aug 2013 - 17:17 UnknownUser  
ui-09.jpgjpg ui-09.jpg manage 241 K 12 Aug 2013 - 17:17 UnknownUser  
ui-1.jpgjpg ui-1.jpg manage 270 K 05 Aug 2013 - 17:08 UnknownUser  
ui-10.jpgjpg ui-10.jpg manage 217 K 12 Aug 2013 - 17:18 UnknownUser  
ui-2.jpgjpg ui-2.jpg manage 205 K 05 Aug 2013 - 17:08 UnknownUser  
ui-3.jpgjpg ui-3.jpg manage 261 K 05 Aug 2013 - 17:08 UnknownUser  
ui-4.jpgjpg ui-4.jpg manage 196 K 05 Aug 2013 - 17:08 UnknownUser  
ui-5.jpgjpg ui-5.jpg manage 104 K 05 Aug 2013 - 17:08 UnknownUser  
ui-6.jpgjpg ui-6.jpg manage 129 K 05 Aug 2013 - 17:09 UnknownUser  
upload1.jpgjpg upload1.jpg manage 125 K 02 Sep 2013 - 16:17 UnknownUser  
upload2.jpgjpg upload2.jpg manage 127 K 02 Sep 2013 - 16:17 UnknownUser  
upload3.jpgjpg upload3.jpg manage 134 K 02 Sep 2013 - 16:18 UnknownUser  
upload4.jpgjpg upload4.jpg manage 133 K 02 Sep 2013 - 16:18 UnknownUser  
Topic revision: r20 - 23 Sep 2013 13:36:17, KahlidAlqinyah - This page was cached on 18 Aug 2016 - 09:27.

This site is powered by FoswikiCopyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Wiki? Send feedback