1. Single Jobs

• Create your proxy, e.g.:

• Create a simple Job Template (called jt), such as:

• Use gwsubmit command to submit the job, specifying the job template created:

• Use GridWay CLI for monitoring jobs and resources. Alternatively, a GridWay Web interface (see Section 5) can be used for monitoring.

• Use gwhost command to see available resources:

• and get more detailed information specifying a <HOST_ID>:

• Check the resources that match job requirements with gwhost -m <JOB_ID>:

• Follow the evolution of the job with gwps command:

• Use gwps -c <seconds> for continuous output.
• See the job history with gwhistory command, specifying a <JOB_ID>:

• Once execution is done … it is time to check the results. The results of the execution of the job are in stdout.<JOB_ID> by default.

2. Array Jobs

• Array jobs are described in the following section with the π example.
• π is calculated with the integral of the function f(x) = 4/(1+x2). So, π will be the integral of f(x) in the interval [0,1]. In order to calculate the whole integral, it's interesting to divide the function in several sections and compute them separately. π will be more accurate as more sections you make. Therefore, it can be calculated by giving all the nodes a section to compute.

• Here is the serial source code. Create a pi.c file and copy inside these lines:

• Their arguments are:

• Task ID: The identifier of the current task.

• Total tasks: The number of tasks the computation should be divided into.

• Integral intervals: The number of intervals over which the integral is being evaluated.

• Compile the serial code:

• Create a job template (pi.jt), such as:

• Submit the array with gwsubmit command and specifying the number of task with option -n:

• Use the gwwait command to wait for the jobs to complete. The <ARRAY_ID> is given by option -A.

• At the end, the execution of these jobs has returned some output files with the result of each execution:

• Sum the partial results to get the value of π:

$ awk 'BEGIN {sum=0} {sum+=$1} END {printf "Pi is %0.12g\n", sum}' stdout_file.*

Pi is 3.1415926536

3. MPI Jobs

• Previous problem can be computed as an MPI job.

• Here is the MPI application for computing π. Create a mpi.c file and copy inside these lines:

• Use mpicc to compile it:

• Create a job template (mpi.jt) for the MPI job:

• and then, submit it to GridWay as any other job:

• Check the results in stdout.${JOB_ID}.

4. Workflow Jobs

• GridWay can handle workflows with the following functionality:

• Sequence, parallelism, branching and looping structures.

• The workflow can be described in an abstract form without referring to specific resources for task execution.

• Quality of service constraints and fault tolerance are defined at task level.

• Job dependencies are specified by using the -d option of gwsubmit command

• Next an example to illustrate how working with array jobs. Consider the following job structure:

  
  

  

• Create the following job templates:

• Submit job A:

• Submit jobs B and C which depend of A, whose <JOB_ID> is 5:

• Finally, submit job D which depend of B and C, whose IDs are 6 and 7:

• Check the the partial results out.A, out.B and out.C, and the final result in out.D.

5. DRMAA API

• Users can submit jobs to GridWay, and control and monitor the execution of jobs through DRMAA. GridWay supports DRMAA v1, and  GridWay 5.12 comes with a technology preview of DRMAA v2.

• Create a DRMAA application, such as this one (ls_drmaa.c) that, for the sake of simplicity, does not include error checking:

• Compile the code using DRMAA library v1:

$ gcc -L $GW_LOCATION/lib -I $GW_LOCATION/include -o ls_drmaa ls_drmaa.c -ldrmaa

• Run the executable:

$ ./ls_drmaa
 Job successfully submitted ID: 6
 Job finished with exit code 0, usage: 6
     start_time=12:53:54
     exit_time=12:54:24
     cpu_time=00:00:29
     xfr_time=00:00:00

• Check the results in stdout.${JOB_ID}, as usual.

• DRMAA V2 is supported by GridWay since version 5.12! Here is an example of DRMAA2 program (ls_drmaa2.c):

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "drmaa2.h"
#include <unistd.h>

int main(int argc, char *argv[])
{
    char          cwd[80]; 
    drmaa2_string jid;
    drmaa2_jstate jstate;
    drmaa2_string substate = (char*) malloc(50);
    drmaa2_string statestr;

    printf("==== Create a job session with given session name.\n");
    drmaa2_jsession js = drmaa2_create_jsession("mysession", NULL);

    printf("==== Creating the job template.\n");
    drmaa2_jtemplate jt = drmaa2_jtemplate_create();
    jt->jobName = strdup("ht");
    jt->remoteCommand = strdup("/bin/ls");
    getcwd(cwd, DRMAA2_ATTR_BUFFER);
    jt->workingDirectory = strdup(cwd);
  
    jt->args=drmaa2_list_create(DRMAA2_STRINGLIST,DRMAA2_UNSET_CALLBACK);
    drmaa2_list_add(jt->args,"-l");
    drmaa2_list_add(jt->args,"-a");
    drmaa2_list_add(jt->args,"/tmp");

    jt->outputPath=strdup("stdout."DRMAA2_GW_JOB_ID);
    jt->errorPath =strdup("stderr."DRMAA2_GW_JOB_ID);

    printf("==== Submitting the job.\n");
    drmaa2_j job = drmaa2_jsession_run_job(js, jt); 
    jid = drmaa2_j_get_id(job);

    drmaa2_jinfo jinfo = (drmaa2_jinfo) malloc(sizeof(drmaa2_jinfo_s));
    jstate = drmaa2_j_get_state(job,&substate);
    statestr = drmaa2_gw_strstatus(jstate);
    printf("     Job %s released.\n", jid);
    printf("     Job DRMAA2 state is: %s\n", statestr);
    printf("     Job Gridway substate is: %s\n",substate);
    printf("     Wait for job %s to finish.\n", jid);

    drmaa2_j_wait_terminated(job, DRMAA2_INFINITE_TIME);
    jinfo = drmaa2_j_get_info(job);
    printf("     Info about the job %s\n", jid);
    printf("\tjob->jobId=%s\n", jinfo->jobId);
    printf("\tjob->exitStatus=%d\n", jinfo->exitStatus);
    printf("\tjob->queueName=%s\n", jinfo->queueName);
    printf("\tjob->wallclockTime=%lld\n", (long long)jinfo->wallclockTime);
    printf("\tjob->cpuTime=%lld\n", jinfo->cpuTime);
    printf("\tjob->submissionTime=%lld\n", (long long)jinfo->submissionTime);
    printf("\tjob->dispatchTime=%lld\n", (long long)jinfo->dispatchTime);
    printf("\tjob->finishTime=%lld\n", (long long)jinfo->finishTime);

    drmaa2_jinfo_free(&jinfo);
    printf("==== Destroying job template and job session.\n");
    drmaa2_jtemplate_free(&jt);
    drmaa2_destroy_jsession("mysession");
    drmaa2_jsession_free(&js);
    printf("==== Exiting now.\n");
    return 0;
}

• Compile the code using DRMAA library v2:

$ gcc -L $GW_LOCATION/lib -I $GW_LOCATION/include -o ls_drmaa2 ls_drmaa2.c -ldrmaa2 

• Run the executable:

$ ./ls_drmaa2
==== Create a job session with given session name.
==== Creating the job template.
==== Submitting the job.   
==== Your job has been submitted with id: 0
     Job 0 released.    
     Job DRMAA2 state is: DRMAA2_QUEUED
     Job Gridway substate is: GW_JOB_STATE_PENDING
     Wait for job 0 to finish.
     Info about the job 0.
        job->jobId=0
        job->exitStatus=0
        job->queueName=default
        job->wallclockTime=16
        job->cpuTime=2
        job->submissiontime=1353581783
        job->dispatchTime=1353581783
        job->finishTime=1353581799
==== Destroying job template and job session.
==== Exiting now.

• Check the results in stdout.${JOB_ID}, as usual.
  

6. BES Interface

• Users can submit jobs to GridWay, and control and monitor the execution of jobs through a BES interface, which is based on GridSAM.

• First, delegate your credentials to a MyProxy server.
  
• Create a simple JSDL (called ls.jsdl), such as:

• Users just need a BES client, such as the one provided by GridSAM.
• Submit a single job:

$ ./gridsam.sh BESCreateActivity -s "https://gridway.fdi.ucm.es:8443/gridsam/services/bes" -j ls.jsdl >
job_id

$ cat job_id
  <?xml version="1.0" encoding="UTF-8"?>
  <EndpointReference xmlns="http://www.w3.org/2005/08/addressing">
  ...<ID>urn:gridsam:13e0999c3848481f0138484838b80001</ID>...
  </EndpointReference>

• Follow the evolution of the job:
  

$ watch -n 5 ./gridsam.sh BESGetActivityStatuses -s "https://gridway.fdi.ucm.es:8443/gridsam/services/bes" -file job_id
  ...
  <?xml version="1.0" encoding="UTF-8"?>
   <GetActivityStatusesResponse xmlns="http://schemas.ggf.org/bes/2006/08/bes-factory">
   ...<ActivityStatus state="Running"/>...</GetActivityStatusesResponse>

• Once execution is done… it is time to check the results.

7. Monitoring Web Interface

• Jobs and resources can also be monitored through GridWay Web interface.

• Monitoring information about the IGE GridWay instance is available at http://gridway.fdi.ucm.es:8080/gwmap.

• Information about hosts can also be shown clicking on “Hosts” tab from Web interface.

• Information about jobs can also be shown clicking on “Jobs” tab from Web interface.

• Web interface also shows information about job histories and registered users clicking on “History” and “Users” tabs respectively.

8. Live demos

• Submitting a MPI job through GridWay CLI:

• Monitoring resources and jobs through GridWay Web Interface: