Submit Grid Job¶

1. Before you can submit Grid jobs you need to prepare your setup.

Finish first the steps described here: https://p55cc-redmine.utu.fi/projects/user-s-page/wiki/Prepare_For_Grid_Usage

2. Other preparation

You use the Grid resources via the ARC (Advanced Resource Connector) middleware, developed by the Nordugrid community.

There are basically two ways to submit Grid job using ARC:

1. From ANY other computer but Pleione or Titan Login frontend
2. From Pleione or Titan Login frontend. (i.e. pleione.utu.fi or titan.utu.fi)

2.1 Using the Grid from other computers¶

Using the Grid from other computers than Pleione or Titan you need to install ARC client software and configure it. Such tasks
are documented here: https://research.csc.fi/fgci-arc-middleware#1.3.2

2.2 Using the Grid from Pleione or Titan frontend¶

The rest of this guide is just short version of this guide: https://research.csc.fi/fgci-using-arc-middleware

So you might want to read the comprehensive guide before running the example.

3. Running self made binaries and FGCI software via runtime environment

In principle there are two ways to use the Grid resources:
- running self made binaries
- running FGCI pre-installed software via runtime environment

4. Example of running a simple self made binary as a Grid job from Titan login node

To run your self made binary as a Grid job you need the following files:

- compiled program (the binary)
- job description file
- batch file
- possibly input file(s)

The c-source of an example program gtest.c :

#include <stdio.h>

int main(void) {

char *line = NULL;
size_t size;

  printf("Hello UTU.\n");

  while (getline(&line, &size, stdin) != -1)
    printf("%s", line);

  return(0);

}

Compile the source:

gcc -Wall -o gtest gtest.c

The job description file gtest.xrsl :

&(executable=gtest.sh)
(jobname=g_test)
(runtimeenvironment>="ENV/FGCI")
(join="yes")
(stdout=std.out)
(cpuTime="1 hours")
(count="1")
(memory="1000")
(inputfiles=
   ("gtest" "" )
   ("gtest.txt" "" )
)
(outputfiles=
   ("gtest.tgz" "" )
)

The batch file gtest.sh :

#!/bin/sh
echo "Running gtest" 
module load OpenMPI
chmod u+x gtest
mpirun ./gtest < gtest.txt > gtest.out
tar czf gtest.tgz gtest.out
echo "Done" 
exit 0

Input file for this test run gtest.txt :

Hello FGCI.

Now you are ready to run the job.

Get proxy to run the job:

$ arcproxy

Enter pass phrase for private key:

Your identity: /DC=org/DC=terena/DC=tcs/C=FI/O=Turun yliopisto/CN=Timo Eronen tke@utu.fi
Proxy generation succeeded
Your proxy is valid until: 2016-12-08 19:46:02

Run the job:

$ arcsub gtest.xrsl

Job submitted with jobid: gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm

Query status of the job:

$ arcstat gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm

Job: gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm
 Name: g_test
 State: Finishing

Status of 1 jobs was queried, 1 jobs returned information

Job was not yet finished but after awhile:

$ arcstat gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm

Job: gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm
 Name: g_test
 State: Finished
 Exit Code: 0

Status of 1 jobs was queried, 1 jobs returned information

Get the results:

$ arcget gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm

Results stored at: 3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm
Jobs processed: 1, successfully retrieved: 1, successfully cleaned: 1

And then you are ready to view the results:

$ ls -l 3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm/
total 8
-rw------- 1 tke admin 148 Dec  8 09:14 gtest.tgz
-rw------- 1 tke admin  19 Dec  8 09:14 std.out

$ cd 3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm/

$ cat std.out
Running gtest
Done

$ tar xf gtest.tgz

$ cat gtest.out

Hello UTU.

Hello FGCI.

5. Example of running a Grid job via FGCI runtime environments from Titan login node

Comprehensive info:

https://research.csc.fi/fgci-using-software-through-runtime-environments

[an example will be added later]