Documentation and Information:Computational clusters in Fine Hall - Revision history

Plazonic at 17:19, 5 August 2010

2010-08-05T17:19:00Z

Plazonic: /* Submitting Multiple Parametrized Jobs */

2010-08-03T14:52:15Z

Submitting Multiple Parametrized Jobs

Plazonic at 14:51, 3 August 2010

2010-08-03T14:51:41Z

Plazonic at 15:52, 26 July 2010

2010-07-26T15:52:48Z

Plazonic at 15:43, 26 July 2010

2010-07-26T15:43:42Z

Plazonic at 15:34, 26 July 2010

2010-07-26T15:34:41Z

Plazonic at 15:32, 26 July 2010

2010-07-26T15:32:42Z

Plazonic: /* Comp computational cluster */

2007-09-26T20:45:49Z

Comp computational cluster

Plazonic: added macomp notes

2005-11-15T16:16:15Z

added macomp notes

Plazonic: wiffin description

2005-10-31T19:03:44Z

wiffin description

@@ Line 112: / Line 112: @@
   cd $HOME/my_serial_directory
   # and run it
-  ./myprogram $PBS_ARRAY_ID
+  ./myprogram $PBS_ARRAYID
 ==== Useful Scheduler Tools ====

@@ Line 113: / Line 113: @@
   # and run it
   ./myprogram $PBS_ARRAY_ID
- ==== Useful Scheduler Tools ====
 * showbf - shows how many nodes are available and for how long. The wall clock limit of a job should be less than the duration reported by showbf, otherwise the job will not run before the next scheduled maintenance period.
 * diagnose -p - shows the priority assigned to queued jobs

@@ Line 98: / Line 98: @@
   qsub parallel.cmd
-==== Useful Scheduler Tools ====
+==== Submitting Multiple Parametrized Jobs ====
 * showbf - shows how many nodes are available and for how long. The wall clock limit of a job should be less than the duration reported by showbf, otherwise the job will not run before the next scheduled maintenance period.
 * diagnose -p - shows the priority assigned to queued jobs

@@ Line 23: / Line 23: @@
   ssh newcomp
 Login should proceed without the need to enter any passwords.
 === Scheduling/Running Jobs ===
 No jobs/computations, expect maybe very short test runs, should be run on the head node.  Any other jobs will be terminated without prior notice.
@@ Line 48: / Line 51: @@
 Please take a look at the below examples - you absolutely have to specify how many nodes you need and how many cores/node as well as the wall clock.  Make sure you specify enough time for your job to finish while trying to be close to the actual run time.  The scheduler will use that information to fit your job best and requesting much more time then you actually need might make your jobs wait too long to be scheduled for running.
 ==== Submitting Single Core/Serial Jobs ====
-To run a single core program with executable called, say, myprogram, you will need to write a job script for torque. Here is a sample command script, serial.cmd, which uses (of course) 1 core:
+To run a single core program with executable called, say, myprogram compiled with intel 10.1 compiler, you will need to write a job script for torque. Here is a sample command script, serial.cmd, which uses (of course) 1 core:
   cd my_serial_directory
@@ Line 61: / Line 64: @@
   #PBS -m abe
+  #
   cd $HOME/my_serial_directory
   ./myprogram

@@ Line 45: / Line 45: @@
 * 24 max processes total (of all users together)
 * 12 max processes per user
+==== Job Submission Gotchas ====
 ==== Submitting Single Core/Serial Jobs ====
 To run a single core program with executable called, say, myprogram, you will need to write a job script for torque. Here is a sample command script, serial.cmd, which uses (of course) 1 core:
@@ Line 89: / Line 90: @@
   qsub parallel.cmd

← Older revision		Revision as of 15:34, 26 July 2010
Line 66:		Line 66:

	qsub serial.cmd		qsub serial.cmd
		+	==== Submitting Parallel Jobs ====
		+	To run your parallel/MPI processing executable called myparallelprog a job script will need to be created for torque. Here is a sample command script, parallel.cmd, which uses 8 cores total (4 cores per node).
		+
		+	cd my_mpi_directory
		+	cat parallel.cmd
		+
		+	#!/bin/bash
		+	# parallel job using 2 nodes and 16 CPU cores, and runs
		+	# for 4 hours (max).
		+	#PBS -l nodes=2:ppn=8,walltime=4:00:00
		+	#
		+	# sends mail if the process aborts, when it begins, and
		+	# when it ends (abe)
		+	#PBS -m abe
		+	#
		+	module load openmpi
		+	cd /u/username/my_mpi_directory
		+	numprocs=`wc -l <${PBS_NODEFILE}`
		+	mpiexec -np $numprocs ./a.out
		+
		+	To submit the job to the batch system, use:
		+
		+	qsub parallel.cmd

@@ Line 1: / Line 1: @@
-Fine Hall machine room is currently hosting 3 different computational clusters:
+Fine Hall machine room is currently hosting 1 mini computational cluster
-== Comp computational cluster ==
+== NewComp computing cluster ==
 === Description ===
-Comp cluster is an older cluster consisting of 16 single CPU AMD Athlon machines with speeds around 1.6Ghz and memory per node ranging from 1GB to 512MB.  Nodes are connected together with 100Mb ethernet networking and have 20GB-40GB hard drives.
+NewComp mini computational cluster consists of 4 nodes, each with 2 Xeon X5680 CPUs - 6 cores each, 12 cores total, running at 3.33GHz.  Each node has 96GB of memory - 8 GB/node.  Head node is equipped with one Intel Xeon X5650 CPUs (6 cores total, running at 2.67GHz) and with 12 GB of memory.
 === Configuration ===
-The cluster is integrated into Fine Hall Math/PACM network and all the cluster machines mount Math/PACM home directories and run the same operating system version as the rest of Fine Hall Linux machines - PU_IAS/Elders 5 Linux (clone of RHEL5).  The software set also closely matches Fine Hall Linux workstations though some of the graphical/desktop applications with no computational use have not been installed.
+The cluster is integrated into Fine Hall Math/PACM network and all the cluster machines mount Math/PACM home directories.  The operating system used on these machines is a clone of RHEL 6.
-For temporary storage, besides /tmp, one can use also /scratch - with no quotas.  It must be emphasized that both /scratch and /tmp cannot be used for permanent data storage and no crucial data should be stored there, e.g. use it for intermediate computational results.  /tmp and /scratch are '''NOT''' backed up and can be erased at any time, especially if a reinstall of one or more machines is required or if one of these directories is full and other users need space. /tmp is also regularly cleaned up by a system job and any file in /tmp that hasn't been accessed in last 10 days will be deleted.
+For temporary storage, besides /tmp, one can use also /scratch - with no quotas. It must be emphasized that both /scratch and /tmp cannot be used for permanent data storage and no crucial data should be stored there, e.g. use it for intermediate computational results. /tmp and /scratch are NOT backed up and can be erased at any time, especially if a reinstall of one or more machines is required or if one of these directories is full and other users need space. /tmp is also regularly cleaned up by a system job and any file in /tmp that hasn't been accessed in last 10 days will be deleted.
-The scheduling software used on the cluster is the Sun Grid Engine.
+Head node /scratch is approximately 3TBs and its subdirectory /scratch/network is exported to all nodes (as /scratch/network). Therefore if you need to access/write temporary data from all nodes create a subdirectory of /scratch/network (like /scratch/network/username) and read/write there.
 === Access ===
-This cluster is fully accessible to all members of Math/PACM.
+At this time the cluster is open to all Math and PACM members.
 === How to connect ===
-Comp cluster's head node name is comp (comp01).  You can connect to it with ssh but only from '''math.princeton.edu''' and '''pacm.princeton.edu'''.  E.g. '''<tt>ssh comp</tt>'''.
+In order to connect to NewComp cluster you will have to login first to <tt>math.princeton.edu</tt> and from there you can:
-== How to Use ==
+=== Scheduling/Running Jobs ===
-No computations/jobs should be ran on the cluster without the use of the scheduling software, SGE.  Any jobs not using SGE might be removed at any time.
+No jobs/computations, expect maybe very short test runs, should be run on the head node.  Any other jobs will be terminated without prior notice.
-== Macomp computing cluster ==
+All jobs have to be submitted to the scheduler which will take care of assigning the necessary resources and running the job.  Any computations found running without being submitted through the scheduler or that were submitted incorrectly (e.g. if the job consumes more cores then allocated or runs after it was supposed to complete) will be terminated without prior notice.
-For temporary storage, besides /tmp, one can use also /scratch - with no quotas. It must be emphasized that both /scratch and /tmp cannot be used for permanent data storage and no crucial data should be stored there, e.g. use it for intermediate computational results. /tmp and /scratch are NOT backed up and can be erased at any time, especially if a reinstall of one or more machines is required or if one of these directories is full and other users need space. /tmp is also regularly cleaned up by a system job and any file in /tmp that hasn't been accessed in last 10 days will be deleted.
+The scheduler in use on newcomp is torque/maui.
-The scheduling software used is Sun's Grid Engine version 6.0 and all the jobs '''have to''' be submitted with SGE.  Once logged in please check <tt>/usr/finehall/computing/sge/samples/readme.txt</tt> for basic instructions on how to submit jobs to SGE and in particular how to submit MPIch jobs. You can find sample submission scripts in <tt>/usr/finehall/computing/sge/samples</tt>
+==== Torque/Maui Queues ====
-=== Access ===
+The scheduler will automatically place your job in one of the following queues.  Here are their names and their current limits:
-At this time access is restricted to grant applicants/contributers.
+===== Short Length Queue =====
-=== How to connect ===
+==== Submitting Single Core/Serial Jobs ====
-In order to connect to MaComp cluster you will have to login first to <tt>math.princeton.edu</tt> and from there you can:
+To run a single core program with executable called, say, myprogram, you will need to write a job script for torque. Here is a sample command script, serial.cmd, which uses (of course) 1 core:
-=== Configuration ===
+ cd my_serial_directory
-The cluster is running a version of RedHat Linux.
+ cat serial.cmd
-Scheduling software used is Sun's Grid Engine version 5.3 and all the jobs '''have to''' be submitted with SGE.
+To submit the job to the scheduling system, use:
-=== Access ===
+ qsub serial.cmd

← Older revision		Revision as of 19:03, 31 October 2005
Line 24:		Line 24:

	== Wiffin computing cluster ==		== Wiffin computing cluster ==
		+	=== Description ===
		+	Wiffin computational cluster consists of 20 dual Xeon 2.2Ghz nodes. Half of the nodes have 2GB and the other half 4GB memory. Nodes are connected with gigabit ethernet networking.
		+
		+	=== Configuration ===
		+	The cluster is running a version of RedHat Linux.
		+
		+	Scheduling software used is Sun's Grid Enging version 5.3 and all the jobs '''have to''' be submitted with SGE.
		+
		+	=== Access ===
		+	The access to this cluster is restricted to members of Prof. Emily Carter's research group and it is not otherwise part of Fine Hall network of Math/PACM Linux machines.