Skip to content. | Skip to navigation

Sections
Personal tools
You are here: Home → Computational Resources → HPC Resources → PBS at CSE - User's Guide
 

PBS at CSE - User's Guide

User's Guide for Portable Batch System (PBS) in the Penn State University's Department of Computer Science and Engineering

 

About

Systems

User Guide

Software

Links

 User Guide

 

 

 

Overview

CSE HPC resources are available for use via PBS Pro 7.0 (Portable Batch System).

Jobs are submitted to the PBS Server and from there are distributed to one of the various clusters in the department.  Most clusters in the department are restricted use for specific research groups and many are used for kernel development and testing.

 

Cluster access is controlled by a users UNIX group. To determine what groups you are in issue the command “groups”.

 

PBS Quick Start

Make sure you are in the correct groups

Check you groups by using the command  groups

  you may see something like this

   % groups
   gcse inti titan surya

 

If you need to access the inti cluster make sure you have the inti group. Same with surya cluster.

 

Setting up your account

PBS requires you to be able to SSH between nodes follow the following steps.

1.      SSH to a department sun machine (lab218)

2.      % chmod 700 ~/.ssh

3.      % chmod go-w ~/

4.      edit or create the file ~/.ssh/config to have the following

StrictHostKeyChecking no

BatchMode yes

5.      Generate  RSA2 key with “ssh-keygen –t rsa –f ~/.ssh/id_rsa” DO NOT put in a password at the prompt leave it blank.

6.      Create your authorized_keys file: “cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys”

7.      Test by connecting between hosts “ssh borg” then “ssh cylon” you should be able to SSH without password or error.

 

An .rhosts file must now be created. This file needs to include the host from which you will be submitting your jobs from, it may include multiple hosts

The file should look something like this with the hostnames reflecting the machines you normally use.

borg.cse.psu.edu    <username>
inti.cse.psu.edu    <username>
surya-0.cse.psu.edu <username>

 PBS Commands

Here are the most used commands for users.  If you don’t see what you need here download the PBS User Guide.

 

qstat –Q                         Displays a list of queues (refer to queue

                                                     description to for availability)

qstat                              List all jobs in the queues

qstat –q <queuename>    List jobs in a specific queue

qstat –u <username>       List jobs for a particular user

qsub <script>                   Submit a job to the default queue (submit)

qsub –q <queue> <script> Submit a job to a particular queue.

qsub –I <script>               Submit a job for an interactive run (will return a prompt when  job is run)

qsub –h <script>               Submit a job and put a hold on it (useful for

                                                      starting jobs with dependencies)

qrls <jobid>                      release a hold on a job

qdel <jobid>                     stop and remove a job from pbs

xpbs &                             run the X-windows batch monitoring application

 

 

examples:

This command submits a job to the queue inti.

% qsub –q inti myjob.pbs

This command will delete job number 13452.surya-0

 % qdel 13452.surya-0


PBS Scripts

PBS uses a flat text file to contain environment settings. A simple script to submit a parallel job:

Submit.pbs:

#!/bin/sh
#PBS -l nodes=2:ppn=2
#PBS -l walltime=00:30:00
#PBS -q inti@surya-0.cse.psu.edu
#PBS -N test_program
#PBS -W group_list=inti
##   Other options for group list are fcse, gcse, ucse,
##     sparse, inti, surya
#PBS -m abe
#PBS -M user@cse.psu.edu
#
PROG=/home/users/prescott/mpi/hello-world
ARGS=""
#
cd $PBS_O_WORKDIR
# Get number of nodes allocated (using the magic shell script)
#
NO_OF_NODES=`cat $PBS_NODEFILE | egrep -v '^#'\|'^$' | wc -l | awk '{print $1}'`
# run the program
#
/usr/local/topspin/mpi/mpich/bin/mpirun_ssh -hostfile $PBS_NODEFILE -np $NO_OF_NODES $PROG $ARGS
exit

Explanation:

First we call the shell (any shell can be used but the path must exist on the execution nodes and syntax must match the shell called).

The next 6 lines are PBS options (They may also be specified on the command line) they are not comments dispite the #

   -l nodes=2:ppn=2         specifies 2 nodes and 2 processors per node

   -l walltime=00:30:00     says how long a job will run (when time runs out your job is killed.) ALWAYS SPECIFY A WALLTIME

   -q <queuename>          directs job to a specific queue. By default jobs will go to the route queue submit@surya-0 and pbs will try to
determine what queue your job should go to.

   -N <name of job>        sets the name of the job for output files.

   -W group_list=i3c        sets the group that your job is submitted as. This is necessary for restricted queues (all but 

                                       lab218)  common groups are  "inti" "surya" for respective queues. 

   -m <abe>                     Mail options for when PBS will send you mail (begin, aborted, terminated)

    -M <addresses>           Where mail should be sent (defaults to the user who submits the job)

Next we set some variables to try to keep the script clean

            PROG= full path to the binary you want to run

            ARGS= and arguments you want to send to $PROG

 

$PBS_O_WORKDIR is the directory from which the job was submitted from. We cd here because otherwise jobs are started from the users /home/users1/<user>/  directory.

Serial job scripts are a bit less complicated most can use something as simple as this:

 Submit.pbs:

#!/bin/sh
#PBS -l nodes=1:ppn=1
#PBS -l walltime=00:30:00
#PBS -q inti@surya-0.cse.psu.edu
#PBS -W group_list=inti
##   Other options for group list are 
##     inti, surya, 
##     fcse, gcse, ucse, 
##     sparse, scalpHPC cg477
#PBS -N test_program
#PBS -m abe
#PBS -M user@cse.psu.edu
#
PROG=/home/users/prescott/hello-world
ARGS=""
#
cd $PBS_O_WORKDIR
# run the program
#
$PROG $ARGS
exit 0

 

Once you have created your submit.pbs script it can be submitted with qsub and PBS will return a job number:

% qsub submit.pbs 
2344.surya-0
 

To view the status of your job use the command qstat. While you will receive mail from your job you ay not want to receive mail from each one if you are submitting 2000 jobs.

 

% qstat -a
Job id            Name              User             Time Use S Queue
----------------  ----------------  ---------------- -------- - -----
399407.surya-0    254.gap.ref       jhu              13:11:34 R surya          
399423.surya-0    175.vpr.ref       jhu              10:07:14 R surya          
399441.surya-0    submit.pbs        prescott         00:00:00 R inti

To kill a job in the queue the command qdel <jobid> is used;

% qdel 399441.surya-0

Out put from all jobs that finish are placed in the PBS_O_WORKDIR (directory where job was submitted from) either using the name of the script or the job name as specified using the –N option in the PBS script. A pair of files will be moved into the directory with the names of  “test_program.o399411” and “test_program.e399411” as per the script above .o<jobid> files contain the standard output and .e<jobid> contain the standard error.

 

Examples

Here are some examples and how the output should look.

Parallel hello-world

Serial matlab

Specific resources:

 

PBS looks for tags in the node line that may specify specific resources requested. These resources can be connection type, operating system, and cluster.

Valid options are:

Clusters:            surya, inti, pr, hhd, lab218
Interconnect:        ib (infiniband)
Operating system:    solaris, linux, FC1, FC2, solaris9, solaris10
 


And example of this would be:

#PBS –l nodes=1:ppn=1:ib

A job sent to the submit queue would be routed to the Inti Linux cluster queue as this is the only cluster that currently has Infiniband interconnect at this time.

 

#PBS –l nodes=1:ppn=1:solaris9

A job sent to the submit queue would be routed to the Surya cluster/queue as cluster/queue lab218 is running solaris10.

The advantage of letting PBS determine where it can run is that if you have a large number of jobs and have compiled your code for multiple systems you can setup your scripts to determine what system you have been assigned and run the appropriate code. By doing this you open your self to more computing resources.

 

If you know your queue and want to make sure your job will only go to that queue specify it with the option.

 #PBS –q <queuename>@surya-0

 

Inti Cluster and Infiband

 

To compile parallel programs on the inti cluster to take advantage of the TopSpin Infiniband interconnect users will need to compile their programs on the head node of the cluster  inti.cse.psu.edu and use the top spin libraries.  Running the following (in tcsh format) will set the environment:

set MPICH = /usr/local/topspin/mpi/mpich
set MPICH_LIB = $MPICH/lib64
set MPICH_PATH = $MPICH/bin
set path = ($path $MPICH_PATH )

 

Compile your programs using mpicc (/usr/local/topspin/mpi/mpich/bin/mpicc) and using the topspin mpi.h (/usr/local/topspin/mpi/mpich/include/mpi.h)

 

You should then tell PBS that you want your job to run using the inti cluster and use mpirun_ssh (/usr/local/topspin/mpi/mpich/bin/mpirun_ssh). To require infiniband in your pbs script add “ib” to the nodes line:

 #PBS –l nodes=2:ppn=2:ib

 

Matlab and PBS

 

To run matlab using pbs you first need to create a .m file that will hold all of your matlab instructions (It is important that the .m file have a quit as the last line!).

 

Matlab polar-plot.m:

angle = 0:.1*pi:3*pi;
   radius = exp(angle/20);
   polar(angle,radius),...
   title('An Example Polar Plot'),...
   grid
print -deps polar-plot.ps;
quit;

 

The “print –deps polar-plot.ps” is necessary to see the graphical output.  ALWAYS have a “quit;” as your last line or matlab will not shutdown.

 

Submit file for PBS matlab.pbs:

 

#!/bin/sh
#PBS -l nodes=1:ppn=1:solaris
#PBS -l walltime=00:10:00
cd $PBS_O_WORKDIR
#Be sure to include “-nodisplay” to stop X display.
/usr/local/bin/matlab
  –nodisplay < polar-plot.m
exit 0

 

FAQ

 

Q: Who manages PBS?

A: The manager of the CSE PBS system and cluster machines is Eric Prescott. All requests for help should be submitted via helpdesk www.cse.psu.edu/support

 

Q: How do I get an account?

A: First you must have a CSE account. All CSE users have access to the queue lab218. Other queues are restricted to research groups (see table). If you need access contact the researcher in charge of the group and have them contact IT support.

 

Q: How do I connect to the clusters?

A: Access is via SSH protocol (version 2). Connect to any machine in lab218 or to surya-0 or to the head node of a specific cluster such as inti.cse.psu.edu.  DO NOT connect to individual nodes in clusters. Access to cluster nodes is granted only via PBS.

 

Q: Why can I not SSH between nodes?

A: PBS requires you to be able to SSH between nodes follow the following steps.

  • SSH to a department sun machine (lab218)
  • chmod 700 ~/.ssh
  • chmod go-w ~/
  • edit or create the file ~/.ssh/config to have the following
StrictHostKeyChecking no
BatchMode yes
  • Generate  RSA2 key with “ssh-keygen –t rsa –f ~/.ssh/id_rsa” DO NOT put in a password at the prompt leave it blank
  • Create your authorized_keys file: “cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys”
  • Test by connecting between hosts “ssh borg” then “ssh cylon” you should be able to SSH without password or error.

 

Q: How do I submit batch jobs?

A: Read the PBS tutorial (this full document and follow the examples).

 

Q: I keep getting the following error when I submit jobs “qsub: Bad UID for job execution”

A: Check that the host you are submitting from is in your ~/.rhosts file.  The host you plan on smiting your jobs from must be in the .rhosts file it should be in the format of:

borg.cse.psu.edu

<username>

inti.cse.psu.edu

<username>

surya-0.cse.psu.edu

<username>

 

Q: My host is in .rhosts but I still get the error “qsub: Bad UID for job execution”

A: remove your .rhosts and rebuild with just the host your submitting from and try again. A control character in the wrong place can cause it to fail.

 

Q: I still get the error "qsub: Bad GID for job execution"

A: Make sure your "PBS -W group_list=" has the proper group for the queue your submitting to.

           Queue                 Group

            inti@surya          inti

            surya@surya      surya

Q: Can I just run MPI jobs without PBS?

A: NO. All MPI jobs to clusters must use PBS. For testing purposes several nodes will be made available on request in such cases use of a –hostfile is necessary.

 

Q: My jobs are not being terminated or I started the on the wrong nodes by mistake.

A: Sometimes MPI jobs don’t terminate properly or can be assigned to the wrong nodes. If you notice this happening, contact the system manager as soon as possible.

 

Q: What is with all the serial jobs? This person is hogging the cluster!?

A: This is a shared resource. Sometimes users will need to submit upwards of 40 thousand jobs! Users must think of others when submitting jobs. Make sure if you are submitting a large number of jobs that you script your submission so that you give time for others to get their jobs in as well.  Currently our scheduling is first come first serve, but if we need to regulate submission we will.

 

Document Actions