2007-10-12T18:05:47Z

Plazonic: added matlab usage

== Introduction ==
In Math Dept./PACM we use the Sun Grid Engine (SGE from now on) for job submission and management on various clusters. This page contains information on SGE/Sun Grid Engine usage on those clusters.

All jobs on the cluster have to be submitted through the SGE. SGE will queue up your job and then choose free node(s) on which it will be run. If there are no free nodes, or not enough of them, your job will wait in the queue until appropriate resources are available and then the job will be executed.

Before proceeding you may want to first read [[Documentation_and_Information:Modules|documentation about modules]] because you are likely to have to use them if you will be using MPI or if you will be using compilers different from gcc (like PGI or Intel). We will also refer to modules and show them in examples below.

== Basic SGE usage ==
When submitting a job you will first have to create a submission script that will, when executed, launch your actual computation. The submission script can also contain various options that will be interpreted by SGE and that will influence how your job is executed.

=== Serial jobs/qsub ===
We will begin with a serial job, i.e. a job that will run on only one processor. Create a submission script, for example call it myjob.sh (.sh extension because this is going to be a bash/sh script but the extension is not necessary - you can choose any name). We will be running myjobexecutable located in myjobdir:
#!/bin/sh
# following option makes sure the job will run in the current directory
#$ -cwd
# following option makes sure the job has the same environmnent variables as the submission shell
#$ -V

# this executable was compiled with intel compiler so we need to load the intel module so that all the libraries will work and be found
module load intel
# and now the actual executable
$HOME/myjobdir/myjobexecutable option1 option2
This job can then be submitted with qsub command and we will call this job run "Job_name":
qsub -N Job_name myjob.sh
SGE will queue up the job and assign it a number (say 3233 - as in 3233rd job). Then on you can refer to this job by either the name you used during submission ("-N" option) or else by its number (3233 in this case).

If the job, i.e. myjobexecutable, outputs anything on the terminal SGE will redirect that output (stdout) and errors (stderror) into files called like Job_name.o3233 (for stdout) and Job_name.e3233 (for stderror) located in the same directory where the job was submitted. These files should be the first place to look at if you need to debug errors in your program or the submission script.

=== Basic qsub options ===
We've already seen "-N" option but there are two other options that were placed in the submission script itself, instead of specifying them on the command line. Any option that qsub understands and are used on the command line can also be specified in the submission script. You put such option(s) in a line of its own whose beginning is "#$". For example, instead of specifying "-N Job_name" we could've added the following line to the above script and submitted the job with just "qsub myjob.sh":
#$ -N Job_name
"-cwd" and "-V" options were already seen in myjob.sh sample script.

"-cwd" makes it so that the job is executed in the directory where it was submitted. If this option is missing the job will be executed in your home directory. You will almost always want this option which is why it is convenient to place it in your submission scripts. The main reason why it is useful is if the job reads input files (say initial conditions from a file INPUT) and/or creates output files (say OUTPUT) in the current working directory then you will want to create different directories for each of your runs and submit your jobs from those directories.

"-V" option makes sure that your job has the same environment variables as the shell in which you submit the job. Again, this is an option that is prudent to always have though it shouldn't be depended on completely (because in MPI case the slave jobs might not actually respect this option, unlike the master node that always will).

There are numerous other options that you can use - some are listed below and others can be found on qsub's man page.

=== Cluster/job status ===
Now that you know how to submit the job you will want to also know how to check on its status as well as on the status of the cluster.

"qstat" will show you the status of Grid Engine jobs and queues. For example:
[mathuser@comp01 mathuser]$ qstat
job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
232629 0.51000 IMAGE005 mathuser r 06/18/2006 14:29:48 all.q@comp-02 4
231554 0.52111 Pt_Al_vac student r 06/16/2006 14:47:32 all.q@comp-04 3
232626 0.51000 IMAGE002 professor r 06/18/2006 14:29:33 all.q@comp-11 1
232597 0.52333 O_img3 someoneelse r 06/18/2006 13:16:48 all.q@comp-16 6

If you type "qstat -f" you will get a detailed list of queues (on each host) and jobs in that queue.

You can get extensive details about a job with "qstat -j jobname/jobnumber". This might also be useful to find out why a job is still waiting to be executed (especially when you have submitted the job with some requirements, like large memory).

You can get a general picture of how busy the cluster really is by typing "qstat -g c":
[mathuser@comp01 mathuser]$ qstat -g c
CLUSTER QUEUE CQLOAD USED AVAIL TOTAL aoACDS cdsuE
-------------------------------------------------------------------------------
all.q 0.31 14 1 16 0 1

Finally, you can get a quick view on the status of cluster nodes by running "qhost":
[mathuser@comp01 mathuser]$ qhost
HOSTNAME ARCH NCPU LOAD MEMTOT MEMUSE SWAPTO SWAPUS
-------------------------------------------------------------------------------
global - - - - - - -
comp01 lx26-x86 1 1.00 1011.1M 164.2M 1024.0M 86.4M
comp02 lx26-x86 1 1.04 503.5M 491.7M 1024.0M 628.2M
comp03 lx26-x86 1 2.04 503.6M 334.5M 1024.0M 175.1M
comp04 lx26-x86 1 1.12 503.6M 184.7M 1024.0M 169.6M
. . . . . . . .
. . . . . . . .
. . . . . . . .
comp16 lx26-x86 1 0.00 1011.1M 92.9M 1024.0M 0.0

=== Cancel/modify jobs ===
If you decide to cancel/delete one of your jobs (or of others if you have been designated as a cluster administrator) you can do it with "qdel" command by using jobs name(s) or job IDs. You can also delete all jobs for a particular user:
[mathuser@comp01 mathuser]$ qdel job_name1
[mathuser@comp01 mathuser]$ qdel 33245 33246 33246
[mathuser@comp01 mathuser]$ qdel -u smith
If a job is already running and the regular qdel is not working try forcing the removal with "-f" option. E.g. "qdel -f job_name1".

"qmod" command allows you to modify a job - e.g. you can suspend it, reschedule it, clear error states and so on.

=== Job statistics ===
After the job execution has ended you can ask SGE for its statistics - e.g. CPU time and memory used during exection:
[mathuser@comp01 mathuser]$ qacct -j 232741

=== Parallel jobs (MPI - mpich) ===
Submit script for MPI parallel jobs has to contain a very specific mpirun command. This is because mpirun needs to be given a list a machines that SGE will reserve for job use. We also want mpirun to use SGE's rsh command which ensures that the job can be properly monitored and controlled by SGI. In particular we can then cancel it or view how many CPU cycles it used. Example submission script for myjobdir/myparallel.exe MPI job compiled with mpich:
#!/bin/sh
# following option makes sure the job will run in the current directory
#$ -cwd
# following option makes sure the job has the same environmnent variables as the submission shell
#$ -V
# VERY IMPORTANT: load appropriate environment module
# in this case this program was compiled with mpich intel version
module load mpich/intel
# and now run the program
mpirun -np $NSLOTS -machinefile $TMPDIR/machines -rsh $TMPDIR/rsh $HOME/myjobdir/myparallel.exe param1 param2
This is how we submit this job to be executed on 10 hosts with job name Job_name:
qsub -N Job_name -pe mpich 10 mympijob.sh
The key option is "-pe" which accepts 2 parameters, the parallel environment (table of available ones follows) and the number of processors you want to reserve for your job. Number of processors to use can be also specified with a range, e.g. 10-20, and the SGE will give you as many as are available in that range. A table describing various options for parallel environment follows.

The next example is for MPI executables compiled with openmpi - note that the file is different from the one we use for mpich and aside from loading a different module we also use mpiexec instead of mpirun
#!/bin/sh
# following option makes sure the job will run in the current directory
#$ -cwd
# following option makes sure the job has the same environmnent variables as the submission shell
#$ -V
# VERY IMPORTANT: load appropriate environment module
# in this case this program was compiled with openmpi pgi version
module load openmpi/pgi
# and now run the program
mpiexec -np $NSLOTS $HOME/myjobdir/myparallel.exe param1 param2
You would submit the above job with a line resembling:
qsub -N Job_name -pe openmpi 10 mympijob.sh

== More advanced SGE usage ==
=== Request a node with lots of memory ===
If your job will require a lot of memory you can request from SGE to assign you nodes with a minimum amount of free memory by specifying a job resource requirement '''mem_free'''. E.g.
qsub -l mem_free=1G testjob.sh
would ask for nodes with at least 1GB of free memory. Similarly, if you want to see which nodes match your requirements you can query for the same resource:
qhost -l mem_free=1G
The output should contain all the nodes that currently have at least 1GB of free memory.

Note that the job will wait in the queue until a host with enough memory is available, in other words until all of your requirements can be met. To check why a job is waiting just ask for its details with qstat -j jobnum.
=== Run SGE Mathematica jobs ===
The simplest way is to create a file with your mathematica commands, say math-input.m, and just input that in the batch file.
#!/bin/sh
# following option makes sure the job will run in the current directory
#$ -cwd
# following option makes sure the job has the same environmnent variables as the submission shell
#$ -V
math < math-input.m
If we name the above file as math-job.sh and place it in the same directory as math-input.m we can submit it from the same directory with
qsub math-job.sh
The output will be left in file math-job.sh.o#JOBNUM# and any errors in math-job.e#JOBNUM#.

You could also incorporate mathematica commands in the job file itself, rather then have them in a separate file:
#!/bin/sh
# following option makes sure the job will run in the current directory
#$ -cwd
# following option makes sure the job has the same environmnent variables as the submission shell
#$ -V
math <<END_MATH_COMMANDS
1+1
3*3
END_MATH_COMMANDS
The above notations means that everything between "<<END_MATH_COMMANDS" and "END_MATH_COMMANDS" will be used as math program's input. You can again submit this job with qsub.
=== Run SGE Matlab jobs ===
You can run matlab jobs similar to the way mathematica jobs, just use "matlab -nodisplay -nodesktop -nojvm -nosplash" for the command. E.g. you could do:
#!/bin/sh
# following option makes sure the job will run in the current directory
#$ -cwd
# following option makes sure the job has the same environmnent variables as the submission shell
#$ -V
matlab -nodisplay -nodesktop -nojvm -nosplash < math-input.m
or you could do
#!/bin/sh
# following option makes sure the job will run in the current directory
#$ -cwd
# following option makes sure the job has the same environmnent variables as the submission shell
#$ -V
matlab -nodisplay -nodesktop -nojvm -nosplash -r math-input.m

Documentation and Information:SGE

2007-10-09T13:54:30Z

Plazonic: How to submit mathematica jobs

2007-03-08T14:48:51Z

Plazonic: link to HowTo for tex on webpages

Here you will find instructions on how to do some of the more common computing tasks.

== Certificates ==
If you are presented with warnings about unsigned certificate when visiting Math webmail or e-mail you will have to import our security certificate. Please follow instructions for your operating system and/or program:
* [[HowTos:Certificate importing for Mozilla applications|Certificate import for Mozilla, Thunderbird of Firefox]] (on any operating system)
* [[HowTos:Certificate importing for Windows|Certificate import for Windows]] (used by Internet Explorer, Outlook or Outlook Express)
* [[HowTos:Certificate importing for MacOSX|Certificate import for Mac OS X]] (used by Safari and Mac Mail program)

== Connect to Math/PACM systems remotely ==
There are a number of different ways to access Math/PACM systems and services - login servers, computational machines, E-mail, files on file server and others. Here are some of these ways:
* [[HowTos:Access your files on Math/PACM file server via cifs/samba|Access your files on Math/PACM file server via cifs/samba on Windows, Mac OS X or Linux]] - directly access your files on the file server, on campus or after connecting via VPN
* [[HowTos:Connect to login servers via ssh|Connect to login servers via ssh from Windows, Mac OS X or Linux]] (also copy files back and forth by using ssh/scp)
* [[HowTos:Remote Linux Desktop access|Remote Linux Desktop access]]
For E-mail reading/access only please read below.

== E-mail access and configuration ==
* [[HowTos:E-mail configuration for Thunderbird on Math Linux machines|Configure Thunderbird on Math Linux workstations]]
* [[HowTos:E-mail configuration for Thunderbird|Configure Thunderbird or Mozilla in general]]
* [[HowTos:Read E-mail with webmail|Read your e-mail in your web browser by using Horde/IMP webmail]]

== File restore/undelete/backup/snapshots ==
* [[HowTos:Restore files from snapshots on Linux from home directory on Math file server|How to restore deleted files or previous versions on Linux from home directory on Math/PACM file server]] (for files deleted or changed within last 4 days)
* [[HowTos:Restore files from snapshots on Windows from home directory on Math file server|How to restore deleted files or previous versions on Windows from home directory on Math/PACM file server]] (for files deleted or changed within last 4 days)
* [[HowTos:Restore files from snapshots on Mac OS X from home directory on Math file server|How to restore deleted files or previous versions on Mac OS X from home directory on Math/PACM file server]] (for files deleted or changed within last 4 days)
* [[HowTos:Restore files from backups|How to obtain files from backups]] (for files deleted or changed more than 4 days ago and usually not more than 3-4 months ago)

== Printing ==
* [[HowTos:Configure MacOSX for Dell W5300n|How to configure your Macintosh for printing with the Dell printers on 11th and 5th floor (W5300n)]]
* [[HowTos:Configure Windows Printing for Fine Hall|How to configure your Microsoft Windows computer for printing to public printers in Fine Hall]]

== TeX ==
* [[HowTos:Install TeX on a Microsoft Windows computer|A quick HowTo about installing TeX on a Microsoft Windows computer]]
* [[HowTos:Add TeX to your webpage|How to add good looking TeX code to your webpages on Math webserver]]

MediaWiki:Sidebar

2007-03-07T21:34:43Z

Plazonic:

HowTos:Configure Windows Printing for Fine Hall

2006-10-16T16:07:34Z

Plazonic:

In order to print to one of the public printers in Fine Hall from a Windows computer (for example your laptop) you can use the windows print server '''<tt>finehallprint</tt>'''. Note that you can access and use Fine Hall printers only within Fine Hall.

These are publicly available printers:
{|
| '''Windows printer name'''
| '''Printer location'''
| '''Printer type'''
|-
| ''<tt>\\finehallprint\fine205</tt>''
| 205 Fine Hall
| HP LaserJet 4250 Duplex
|-
| ''<tt>\\finehallprint\fine219</tt>''
| 219 Fine Hall cluster
| HP LaserJet 4300 Duplex
|-
| ''<tt>\\finehallprint\fine305</tt>''
| 305 Fine Hall (restricted access outside business hours)
| HP LaserJet 4350 Duplex
|-
| ''<tt>\\finehallprint\fine305pcl</tt>''
| 305 Fine Hall (restricted access outside business hours)
| HP LaserJet 4350 Duplex PCL driver
|-
| ''<tt>\\finehallprint\fine511</tt>''
| 5th floor Fine Hall, outside offices 504 and 505
| Dell W5300 Duplex printer
|-
| ''<tt>\\finehallprint\fine811</tt>''
| 8th floor Fine Hall, outside offices 804 and 805
| HP LaserJet 4300 Duplex
|-
| ''<tt>\\finehallprint\fine1111</tt>''
| 11th floor Fine Hall, outside offices 1104 and 1105
| Dell W5300 Duplex printer
|}
While you may find other printers on <tt>finehallprint</tt> they are private printers reserved for use by their owners so please do not try to use them.

Note also that all of Fine Hall printers default to duplex (double sided) printing so you may have to change your printer settings if you want to print single sided.

== Detailed instructions ==
Example instructions on how to setup one of these printers, for example fine305, on your computer follow. First click on "Start" button (1) and then on "Run" (2):
[[Image:Fs-startrun.jpg|center]]
In the "Run" window that will come up type the printer address from above table in "Open" (1). In our case for printer 305 it is ''<tt>\\finehallprint\fine305</tt>''. Then click on "OK" (2):
[[Image:Finehallprint-run305.jpg|center]]
Your computer will then attempt to connect and it may ask you to confirm the installation with a dialog that resembles the following where you should click on "Yes" (1):
[[Image:Finehallprint-confirm.jpg|center]]
That's it - you should now be able to use this printer.

Documentation and Information:Public printers