Gateway Developer Guide

Transcription

1 Gateway Developer Guide Apache Airavata's Programming API is the API which is exposed to the Gateway Developers. Gateway Developers can use this API to execute and monitor workflows. The API user should keep in mind that a user can not compose an Airavata workflow (.xwf) using the API. Inorder to do that a user can use the XBaya User Interface. Therefore, other than creation of the workflow; Client API supports all other workflow related operations. The main motivation behind, having a Client API is that to expose the user to an API that will let him/her access to a the persistent information stored in the Registry. The information persisted in the Registry can be; Descriptors Workflow information Workflow provenance information Airavata configuration Following are the high level usecases which uses Airavata API. Airavata API Usecases Registry Operations #* Retrieve registry information Update registry information Delete registry information Search registry information Execute workflows#* Run workflow Set inputs Set workflow node IDs Workflow Monitoring Provenance User Management (This is not yet implemented. It's currently in our Road Map and this is added as a place holder.) User roles Administration Airavata API Components The Airavata API consists of 5 main components. 1. Airavata API

2 It is an Aggregator API which contains all the base methods for Airavata API. Airavata Manager This exposes config related information on Airavata. This currently contains Service URLs only. Application Manager This will handle operations related to descriptors. Namely; a. Host description b. Service description c. Application description Execution Manager This can be used to run and monitor workflows. Provenance Manger This provides API to manage provenance related information. ie. Keeps track of inputs, outputs, etc related to a workflow. User Manger User management related API is exposed through this. Currently, Airavata does not support User management but it is in Airavata roadmap. Workflow manager Every operation related to workflows is exposed through this. ie: a. saving workflow b. deleting workflow c. retrieving workflow Registry Operations From AiravataClient, you can access methods of Airavata Regisrty API. Following code snippets shows how you can access registry API via Airavata Client. Once you have registry instance available, you can do many registry operations that are exposed by Registry API. Please note that this code snippet will only works if you are using airavata 0.4 release.

3 String user = airavataclient.getregistry().getusername(); List<WorkflowExecution> workflowexecutionbyuser = airavataclient.getregistry().getworkflowexecutionbyuser(user); //Get workflow execution data stored in the Registry WRT user for (WorkflowExecution next : workflowexecutionbyuser) { System.out.println(" Experiment ID : " + next.getexperimentid()); System.out.println(" Topic : " + next.gettopic()); System.out.println(" Meta data : " + next.getmetadata()); Thread.sleep(10); With airavata 0.5 release, Airavata Registry API has changed since we replace backend Jackrabbit registry with Mysql/Derby database. However Airavata Registry API is now hidden behind Airavata API. If you are using Airavata API 0.5 or later, you can follow the below code snippet. //get current registry user String user = airavataapi.getcurrentuser(); //get all the experiments run by the given user List<ExperimentData> experimentbyuser = airavataapi.getprovenancemanager().getexperimentdatalist(); //If you are using 0.5 //List<ExperimentData> experimentbyuser = airavataapi.getprovenancemanager().getworkflowexperimentdatalist(); for (ExperimentData next : experimentbyuser) { System.out.println(" Experiment ID : " + next.getexperimentid()); System.out.println(" Topic : " + next.gettopic()); System.out.println(" Meta data : " + next.getmetadata()); Airavata configuration related information can be retrieved using Airavata Manager. // Get Airavata configuration information using Airavata Manager AiravataManager airavatamanager = airavataapi.getairavatamanager(); System.out.println("Message Box Service URL : " + airavatamanager.getmessageboxserviceurl()); System.out.println("Eventing Service URL : " + airavatamanager.geteventingserviceurl()); System.out.println("Registry Service URL : " + airavatamanager.getregistryurl()); Registry can be searched for descriptors using regular expressions. Please note that, this is only valid for Airavata 0.4 release as well.

4 // Search the registry for a host ApplicationManager applicationmanager = airavataapi.getapplicationmanager(); List<HostDescription> hostdescriptions = applicationmanager.searchhostdescription("ranger"); Iterator<HostDescription> hostdescriptioniterator = hostdescriptions.iterator(); while(hostdescriptioniterator.hasnext()) { HostDescription hostdescription = hostdescriptioniterator.next(); System.out.println("Host Name : " + hostdescription.gettype().gethostname()); System.out.println("XML : " + hostdescription.toxml()); Can retrieve the descriptors saved in the registry using Application Manager. // Retrieve all the host descriptions in the registry List<HostDescription> allhostdescriptions = applicationmanager.getallhostdescriptions(); Iterator<HostDescription> descriptioniterator = allhostdescriptions.iterator(); while(hostdescriptioniterator.hasnext()) { HostDescription hostdescription = descriptioniterator.next(); System.out.println("Host : " + hostdescription.gettype().gethostname()); Saving the host descriptions to the registry using Application Manger. // Save host description HostDescription host = new HostDescription(); host.gettype().sethostname("cutom-host"); host.gettype().sethostaddress(" "); applicationmanager.savehostdescription(host); Execute Workflows Inputs can be set to the workflow using Airavata Client.

5 // Set the input values to the nodes AiravataAPI airavataapi = AiravataClientUtils.getAPI(config); Workflow workflow = airavataapi.getworkflowmanager().getworkflow(templateid); List<WorkflowInput> workflowinputs = workflow.getworkflowinputs(); int count = 10; for (WorkflowInput workflowinput : workflowinputs) { if ("int".equals(workflowinput.gettype())) { workflowinput.setvalue(count--); Airavata Client can be used to set CPU counts to the workflow nodes. int cpucount = 1; int nodecount = 1; // Set the CPU count to the node Property workflowasstring = airavataclient.getworkflowasstring(templateid); Workflow workflow = new Workflow(workflowAsString.getString()); List<NodeImpl> nodes = workflow.getgraph().getnodes(); for (Node node : nodes) { if (node instanceof WSNode) { ApplicationSchedulingContextDocument.ApplicationSchedulingContext applicationschedulingcontext = airavataclient.getbuilder().getcontextheader().getworkflowschedulingcontext().addnewap plicationschedulingcontext(); applicationschedulingcontext.setserviceid(node.getid()); if (cpucount!= -1) { applicationschedulingcontext.setcpucount(cpucount); if (nodecount!= -1) { applicationschedulingcontext.setnodecount(nodecount); if ("SimpleMathServicePortType_multiply".equals(node.getID())) { applicationschedulingcontext.setnodecount(2); applicationschedulingcontext.setcpucount(2); applicationschedulingcontext.setqueuename("normal"); Workflow can be executed using Airavata Client by dynamically setting the input parameters. // Run the workflow airavataclient.runworkflow("sampleworkflow", workflowinputs, "admin", null, "SampleWorkflow");

6 [EDIT] A workflow is identified using the name of the workflow called the workflowtemplateid. Given this workflow is present in the Airavata system we can invoke a workflow as follows, //setup inputs for the workflow List<WorkflowInput> workflowinputs = new ArrayList<WorkflowInput>(); workflowinputs.add(new WorkflowInputs("parameter1","value1")); workflowinputs.add(new WorkflowInputs("parameter2",42)); To run a workflow we use the execution managers in the Airavata API. We call this as running an experiment. An experiment is defined as running one or more workflows. Right now Airavata supports running one workflow per experiment as follows, //run the workflow airavataapi.getexecutionmanager().runexperiment(templateid, workflowinputs); Workflow Monitoring Workflow monitoring can be done through the Execution Manger. ExecutionManager executionmanager = airavataclient.getexecutionmanager(); Monitor workflowintancemonitor = executionmanager.getworkflowintancemonitor("sampleworkflow_ f4a-9265-e6d 1135d3b80"); MonitorConfiguration monitorconfiguration = workflowintancemonitor.getconfiguration(); System.out.println("Topic : " + monitorconfiguration.gettopic()); System.out.println("Broker URL : " + monitorconfiguration.getbrokerurl()); System.out.println("Message Box URL : " + monitorconfiguration.getmessageboxurl()); if(monitorconfiguration.getinteractivenodeids()!=null) { List<String> interactivenodeids = monitorconfiguration.getinteractivenodeids(); for (String nodeid : interactivenodeids) { System.out.println("Node ID : " + nodeid);

7 Once an experiment is launched a unique id is returned to identify that experiment. You can use this unique id (here onwards called the experiment Id) to monitor the progress of the experiment through Airavata API, String experimentid=airavataapi.getexecutionmanager().runexperiment(templateid,workflowinputs ); Monitor experimentmonitor = airavataapi.getexecutionmanager().getexperimentmonitor(experimentid, new MonitorEventListener() { public void notify(monitoreventdata eventdata, MonitorEvent event) { System.out.println(event.getMessage()); ); experimentmonitor.startmonitoring(); Provanance Provenance related information can be accessed through the Provenance Manager. Please note that this code snippet works with airavata 0.4 release. ProvenanceManager provenancemanager = airavataclient.getprovenancemanager(); // Get all experiments if (provenancemanager.getexperiments()!=null) { List<String> experiments = provenancemanager.getexperiments(); for (String next : experiments) { System.out.println("Experiment : " + next); With airavata 0.5 release to get all experiment IDs, use the following code snippet. ProvenanceManager provenancemanager = airavataapi.getprovenancemanager(); // Get all experiments if (provenancemanager.getexperimentidlist()!=null) { List<String> experiments = provenancemanager.getexperimentidlist(); for (String next : experiments) { System.out.println("Experiment : " + next); User related provenance information can be retrieived through Provenance Manager. Please note that this code snippet works with airavata 0.4 release.

8 // Get workflow instance information List<WorkflowInstance> workflowinstances = provenancemanager.getworkflowinstances("admin"); for (WorkflowInstance next : workflowinstances) { System.out.println("Workflow Name : " + next.getworkflowname()); System.out.println("Experiment ID : " + next.getexperimentid()); System.out.println("Topic ID : " + next.gettopicid()); With airavata 0.5 release, you can retrieve user related provenance data from the below code snippet. ProvenanceManager provenancemanager = airavataapi.getprovenancemanager(); // Get all workflow experiments for admin user List<ExperimentData> experimentdatalist = provenancemanager.getworkflowexperimentdatalist("admin"); for (ExperimentData experimentdata: experimentdatalist){ System.out.println("****************************************************"); System.out.println("exp name : "+ experimentdata.getexperimentname()); System.out.println("topic name : "+ experimentdata.gettopic()); System.out.println("user name : " + experimentdata.getuser()); List<WorkflowInstanceData> workflowinstancedata = experimentdata.getworkflowinstancedata(); for (WorkflowInstanceData workflowinstance : workflowinstancedata){ System.out.println("Workflow Name : " + workflowinstance.getworkflowname()); System.out.println("Experiment ID : " + workflowinstance.getexperimentid()); System.out.println("Topic ID : " + workflowinstance.gettopicid()); Airavata Rest Services If you are using REST service instead of Airavata Registry, you will need to call the following method. airavataapi = AiravataClientUtils.getAPI(<REST service URI>, getregistryusername(), passwordcallback); passwordcallback class is the implementation of a Callback interface which will be used to retrieve the password for the current user.