WPS4R Backlog

Legend

Priorities: 1 = highest priority, n = lower priorities

Costs: "Felt" development costs, relative, no actual estimation neccessary, going from = "low" to = "very high"

Statuses: = open task, = somebody is working on the task, the task is "done". We have a very loose definition of done: "Contributor did some tests and it worked."

Group: Can be used freely to associate items together, but is optional.

Items for Version 1.1 of the Backend

Group	Item	Priority	Costs	Contributor/Status
feature	As a user I want to use the INTAMAP functionalities based on WPS4R.	1
doc	As a user I want the the documentation to be up to date, especially the tutorials. All documentation must be checked and partially updated. Transfer backend documentation to Wiki from here to here	1
dev	Fix warninigs in integration tests (issue with RServe), email by MH.	1
dev	As a developer I want the integration tests to run on the build server (using remote RServe?)	3
doc	As a user I want a demonstration of the capabilities with spatial input and output data, best based on INTAMAP functionality and common input formats, also demonstrating the data conversion facilities of the WPS, e.g. shp input to INTAMAP process with GML output.	4

Items for later Version

Group	Item	Priority	Costs	Contributor/Status
	As a script developer I want to be able to send different log statements to the runtime environment (as part of a wps4R extension package) which are added with log levels ERROR, INFO and DEBUG to the log file of the WPS.
	As a developer I want to use/develop/test the annotation parsing mechanism standalone as well, e.g. for integration into MovingCode.	1
	As a new user I want a very short (no scrolling) description of WPS4R in website R-backend description and more details moved to wiki pages.	1
	Check before running a script if all required R packages are installed. If not do not attempt to start the script but throw an error (option: autoinstall?).	1
	As an adminstrator I want to ensure that no harmful code is executed in my runtime environments based on a simple blacklist of commands.	1
	As a user I want to be informed about annotation validation errors then I call a process.	1
	Check https://wiki.52north.org/bin/view/Geostatistics/WPS4RDevelopment#WPS4R_Source_Setup and update the tutorial (https://wiki.52north.org/bin/view/Processing/TutorialBackendR) with new configuration options.	2
	As a user I want to be notified about missing dependencies (by scanning the require and library statements of my script) before the process is started.	2
	As a user I want to validate R scripts locally as part of a wps4R R extension package.	2	%ICON{led-aqua}
	As a script developer I want to run the same script that works in WPS4R in a moving code package.	2
	Include sessionInfo() as text in every execute response	3
	As a user I want to upload several versions of scripts and have this included in provenance information documents.	3
	Create WPS4RExamples and WPS4RAnnotations wiki page.	4
	As a script developer I want to submit new R script using the transactional WPS interface (WPS-T)	4
	As a user I want to use extended INTAMAP functionalities with a temporal component.	5
	As an adminstrator I want to ensure that no harmful code is executed in my runtime environments based on a sandbox.	5
	As an administrator I want to run WPS4R in a public cloud environment (Microsoft, Amazon, ...)	5
	As a user I want to use a generic WPS4RforSweave process so that I can upload a Sweave file and get pdf/dvi/latex as output. Potentially the last one could be used as input for the first two proceses using a process chain. I want to run this process from an wps4R extension package and have a conveience function that let's me check if the process is finished yet and if so it download the output file.	5
	As a user I want to run asynchronous R processes so that I... can query the status of a process. can abort a process.	7
	As a script developer I want to use an R API (as part of a wps4R extension package) to communicate the progess of the process to the runtime environment.	8
	As an administrator I want to install R scripts interactively in the web admin backend. Or even have a full console there.	10
	As a user I want to see the R version and available packages in the capabilities file of the WPS. This should be done via a generic mechanism in the process repository based on the ows:AbstractMetaData element.	10
	As a user I want to outsource parts of my R script by upload RData files to a WPS and getting RData files back. The wps4R package should allow me seamless integration of these resources. R Datentypenals input und output unterstützen, siehe http://geoprocessing.forum.52north.org/problem-with-annotating-the-R-script-td4025097.html [Ist: etwa 5-6 Stunden] Konfigurationsdatei Datentypen (mapping extension <> mimetype) .RData (damit geht ja dann quasi "alles") als Input [4 Stunden] mime type analog zu best practices discussion paper selbst definieren https://portal.opengeospatial.org/files/?artifact_id=47860, application/rData .RData als output [4 Stunden] Demo mit spatial class gewrappt in .RData (and dadurch auch die davon abgeleiteten SpatialPoints, SpatialPolygons, SpatialLines, SpatialGrid und deren ~DataFrame Varianten) [4 Stunden] application/rData+Spatial und application/rData+SpatialPoints und application/rData+SpatialPolygons > Benennung der Variablen Beispielskript für Points und auf WPS4RExamples basierend auf Zeilen 19, 26, 27, 28 von http://www.asdar-book.org/book/geos_mod.R aus dem ASDAR Buch (bitte verlinken im Beispiel) [1 Stunde] Inputs: SpatialPointsDataFrame (in Code: meuse), sollte im Beispiel dann einen generischeren Namen bekommen (z.B. inputSPDF). Dies gilt auch für die Zwischenergebnisse, genauer gesagt zn.lm dass besser linearModel oder so heißen sollte. x-Koordinatenname als Text (in Code: "x") y-Koordinatenname als Text (in Code: "y") linear model als Text (hier: "log(zinc)~sqrt(dist)") Outputs: SpatialPointsDataFrame mit linear model (in Code: meuse) Text: Feldname für Modelergebnis (in Code, und vermutlich auch im Beispiel: fitted.s) Text: Feldname für Modelresiduen (in Code, und vermutlich auch im Beispiel: residuals)	10
Gen.	As a user I want to upload/manage (CRUD) resources for specific processes using a process, or using specific operations (e.g. from Spacebel WPS?!)	11
	As a user I want to use compiled R code (http://dirk.eddelbuettel.com/blog/2011/04/12/) to increase performance - maybe this could even be applied automatically in WPS4R?	15

Other Tentative Tasks and Ideas

• Research about security mechanisms in INTAMAP, RWeb and OpenCPU [DN]
• WPS4R for open, transparent climate science might be a great usecase: http://www.easterbrook.ca/steve/?p=2865
• wps4R client extension package for R [DN]
  • test annotated scripts offline
  • upload scripts
  • wpsSessionInfo()
  • wpsResourceByName()
• Possible features based on User Stories
• • O&M Parser for R [4] (could be re-used in sos4R)
  • WPS client in R which also allows local execution of scripts (via boolean flag, or using a wrapper function that contains inputs and returns a list with outputs) [6, 7]
  • Automatic deployment of processes from R (simple switch between local and remote execution) [1, 2, 5]
  • errors and warnings in execute response, including tracebacks [8]
  • 1:n > one maintainer (script uploader), many users [1, 2, 5, 8]
  • n:m > many users who can upload scripts: WPS user management to allow script upload, but not service configuration [3, 6]
  • multiple processing sessions using one script (distinct environments)
  • A framework or environment which allows to share WPS processes in the Web [1, 5, 6, 8]
    • moving code paradigm, app-store model > dependencies must be included in annotations (documented, "tested with versions")
  • Compatibility assurance / testing WPS clients for WebGIS with WPS4R [1, 8]
  • documentation of tested package versions of a script (metadata) [1, 3, 5, 7]
  • downloading whole scripts and the package versions it was tested/deployed with [1, 7, 8]
    • should it be possible to disallow this? Intelectual property on an algorithm?
    • should there be annotations that specifiy (when uploading) which package versions were used, and what should the service response if the version do not match the ones installed?
  • process sandbox (handle things like conflicting package versions?) [3, 5]
• Could WPS4R help in reproducible research?
  • Provide a mechanism to outsource the heavy lifting of maintaining multiple R versions and packages to a server
  • Good discussion on R-devel about reproducibility with R http://r.789695.n4.nabble.com/enabling-reproducible-research-amp-R-package-management-amp-install-package-version-amp-BiocLite-td4660277.html
    • http://cran.r-project.org/web/packages/rbundler/index.html
  • Challenge: not only R packages are versioned, but also compilers, (C) libraries, ...
  • What are reasonable trade-offs?

Metadata

• Topic created by: DanielNuest
• Topic created on: 2013-04-14