Dynamic and Interactive Execution and Visualization of Distributed Workflows
The Generic Workflow Execution Service (GWES) is the workflow enactment engine originally implemented for the Fraunhofer Resource Grid, redesigned within the K-Wf Grid project, and enhanced for Enterprise Grids, Instant-Grid, MediGRID, BauVOGrid, TextGrid, and PneumoGRID. The Generic Workflow Execution Service is formally known as "Grid Workflow Execution Service". With the implementation of a plugin concept for arbitrary workflow activities, it has now a broader area of application, not limited to the orchestration of Grid and Web Services. The GWES coordinates the composition and execution process of workflows in arbitrary distributed systems, such as SOA, Cluster, Grid, or Cloud environments. It implements a highly dynamic workflow concept by means of the Generic Workflow Description Language (GWorkflowDL), which is based upon the theory of high-level Petri nets. It provides interfaces to Web Portal frameworks and to a command line client for user interaction and to the low-level Middleware for the invocation of application operations.
The main purpose of the Generic Workflow Execution Service and its client interfaces is to enable users to automatically execute workflows on distributed resources without being bothered with implementation-specific details, putting more attention to the functionality of the workflow itself. Therefore the Generic Workflow Execution service provides methods to initiate and analyze technical workflows, and to coordinate and optimize the execution of these workflows on distributed and inhomogeneous resources regarding the control as well as the data flow. Abstract operations are automatically mapped onto matching software and hardware resources, triggering web service operations, remote executions of programs, or file transfers. The workflow service supports pure Web Services and Globus Toolkit 4, and it is easily extensible for further execution platforms, such as UNICORE, Condor, GRIA, and SUN Grid Engine, allowing reusing existing workflows on different Grid middleware.
The following GWES plugins are available or in preparation:
Using the plugin concept, GWES application developers are now able to implement own activity plugins without touching the GWES core code. Please refer to the GWES Activity Plugin Development Tutorial (pdf) for details.
For further information about the Generic Workflow Execution Service please refer to:
The main scientific achievements include the definition of a theoretically well founded Workflow Description Language, which is very simple but expressiveness at the same time, using the well-known concept of high-level Petri nets. With this approach we are able to model arbitrary algorithms, including parallel branches or loops. Transitions can possess conditions that we express using the XPath 1.0 syntax.
In order to make the workflows persistent, the execution service uses a native XML database for storing snapshots of the workflows including the workflow's state, which is represented by the marking of the corresponding Petri net. The system provides a simple and robust checkpoint-restore functionality and the user may initiate each of these snapshots as a new workflow instance. Another scientific achievement is the overall concept of iteratively mapping abstract workflows onto concrete resources, taking into account the most current monitoring information. Each of the abstraction levels uses the same workflow description language, so it is possible to mix several levels of abstraction in one workflow.
The Generic Workflow Execution Service is deployed as a standard Web Service, so it seamlessly integrates into common system architectures. There is a strict separation of the enactment machine and its client interfaces. The graphical user interface is realized by a set of Java Applets and Servlets which can be easily integrated in generic or application-specific web portals. In addition there exists a command line client program that implements the full set of workflow management features and which simplifies the integration into legacy frameworks.
These are the basic steps for deploying the GWES as a Web Service:
The GWES supports TLS/SSL security with client authorization based on X509 certificates. The security layer is optional, so if the GWES clients and server are behind a firewall you may prefer the default configuration without security layer. Please note, that the documentation about the GWES security is not part of the open source distribution. If you need assistance in setting up the security layer, please contact andreas.hoheisel@first.fraunhofer.de.
Depending on your requirements, you may need to deploy and configure additional services, such as the eXist database, the Resource Matcher, the Resource Updater and the Linux Toolbox. For details please refer to the GWES User Manual or contact Fraunhofer FIRST to get an offer for the configuration of your system.
With the default configuration, the GWES uses a simple scheduling technique, that prioritizes operations on resources with good quality. For more advanced scheduling, including automatic prioritization of operations using a HEFT algorithm, you may switch to the two-tier workflow scheduling approach developed by Dietmar Sommerfeld / GWDG. In order to do so, please edit the file gwes.properties and include the line: gwes.operationmapper.blue2green.class=net.kwfgrid.gwes.prorater.TwoTierProrater.
The GWES possesses several interfaces for remote access:
The GWES uses Apache Axis2 for providing a SOAP interface to its clients. If the GWES is deployed at the URL http://localhost:8080/gwes/services/GWES, then Axis2 generates the corresponding WSDL at http://localhost:8080/gwes/services/GWES?wsdl or http://localhost:8080/gwes/services/GWES?wsdl2 for WSDL version 2.
From this WSDL, you can either compile your own stubs in your favorite programming language or you can use the pre-compiled Java stubs from the GWES client package net.kwfgrid.gwes.client with its main entry class net.kwfgrid.gwes.client.RemoteGWES (JavaDoc). If you want to integrate the pre-compiled Axis2 Java stubs in your own Apache Maven2 project, use the following dependency:
<dependency>
<groupId>firstgrid</groupId>
<artifactId>gwes-client-cli</artifactId>
<version>2.1.1.rc2</version>
</dependency>
[...]
<repositories>
<repository>
<id>gridworkflow.org</id>
<url>http://www.gridworkflow.org/maven2</url>
</repository>
</repositories>
In addition to the SOAP interface, the GWES also provides a RESTful interface for client implementations with small footprint. Here some examples of remote invocation of GWES methods using HTTP GET, POST, or DELETE:
The Java package net.kwfgrid.gwes.restfulclient contains a full client implementation using the GWES REST interface. The main entry classes are net.kwfgrid.gwes.restfulclient.RestfulClient (JavaDoc) and net.kwfgrid.gwes.restfulclient.RestfulStructureGWES (JavaDoc).
If you want to integrate the pre-compiled REST client in your own Apache Maven2 project, use the following dependency:
<dependency>
<groupId>firstgrid</groupId>
<artifactId>gwes-client-rest</artifactId>
<version>2.1.1.rc2</version>
</dependency>
[...]
<repositories>
<repository>
<id>gridworkflow.org</id>
<url>http://www.gridworkflow.org/maven2</url>
</repository>
</repositories>
The GWES command line interface is a Java application that uses the SOAP Java Stubs to connect to a remote GWES. These are the basic steps for installing the GWES client command line interface:
-------------------------------------------------------------------------------
Usage: java net.kwfgrid.gwes.GWESClient -gwes | -g <gwesURL>
-initiate | -i <GWorkflowDL>
-initiateStart | -is <GWorkflowDL> [-monitor]
-start | -s <workflowID> [-monitor]
-suspend <workflowID>
-resume <workflowID>
-store <workflowID>
-remove | -r <workflowID> <level>
-restore <workflowID>
-restart <workflowID> [-monitor]
-abort | -a <workflowID>
-getWorkflowDescription | -gwd <workflowID>
-getStatus | -gws <workflowID>
-getWorkflowIDs | -gwi <level>
-getWorkflowStatusArray | -gwa <level>
-getWorkflowProperties | -gwp <workflowID>
-getWorkflowActivities | -gwc <workflowID>
-getData <workflowID> <placeID>
-getDescription <workflowID> | -gd <workflowID>
-setDescription <workflowID> <description> | -sd <workflowID> <description>
-monitor | -m <workflowID>
-credential | -c
-------------------------------------------------------------------------------
If you define several commands in one call, only the last one will be
invoked!
<gwesURL> : The URL where the GWES is located, e.g.
http://server:8080/gwes
<GWorkflowDL> : The filename of the GWorkflowDL document containing the
workflow to enact
<workflowID> : The identifier of the workflow
<description> : The human-readable description of the workflow
<level> : The storage level: 1=memory, 2=database, 4=temporary workflow data
-monitor | -m : The GWESClient monitors the running job until it
terminates or completes
-credential | -c : delegate local GSS credential to GWES
-------------------------------------------------------------------------------
For the graphical interaction with humans the GWES provides a set of servlets that generate HTML pages for uploading and monitoring workflows. Users can access the servlets targeting their Web browser to the basic GWES deployment URL, such as, http://localhost:8080/gwes. For the authorization and authentication you need to include the user login/password (HTTP) or certificate DN (HTTPS) with the role gwes to the corresponding tomcat configuration file tomcat-users.xml.