[Rocks-Discuss] pbs ppn not behaving

[Donald Tripp]

Maui will look at all resources at a large pool, and will put jobs  
anywhere they fit. In default mode, for nodes=X:ppn=Y, maui will find  
(X*Y) number of available processors and put them there. In general  
this works fine, but if you need more control over the execution, say  
if you have a application that takes a large amount of memory, you  
would indeed want to specify a minimal number of processes per node to  
preserve maximum RAM.

Try this in your maui.cfg file:

JOBNODEMATCHPOLICY          EXACTNODE
NODEACCESSPOLICY            SINGLEJOB

Its even a bit more restrictive in that it allows only one job to run  
per node. Even with EXACTNODE enabled, multiple users jobs could run  
per node, defeating the purpose of specifying a particular number of  
processors per node. Since you are being a bit more restrictive with  
SINGLEJOB, the queue may slow down a bit, so also make sure:

BACKFILLPOLICY        FIRSTFIT

is enabled, so smaller jobs will run and not get backlogged.


- Donald Tripp
   dtripp at hawaii.edu
----------------------------------------------
HPC Systems Administrator
High Performance Computing Center
University of Hawai'i at Hilo
200 W. Kawili Street
Hilo,   Hawaii   96720
http://www.hpc.uhh.hawaii.edu


On Oct 7, 2008, at 8:29 PM, Thomas Hamel wrote:

> Hi,
>
> Simple, not really, but there is one. When you do a nodes=X:ppn=Y  
> you're
> asking maui to give you X groups of Y CPUs where the Y CPUs are on  
> the same
> node. Maui can then put multiple groups on the same node. So, in  
> your case
> you have 2 groups of 1 cpus per node.
>
> You can change this behavior by setting JOBNODEMATCHPOLICY to  
> EXACTNODE in
> /opt/maui/maui.cfg :
>
> http://www.clusterresources.com/products/maui/docs/a.fparameters.shtml#jobnodematchpolicy
>
> Regards,
>
> Thomas HAMEL
>
> On Wed, Oct 8, 2008 at 3:43 AM, Ross Ishida  
> <ishida at soest.hawaii.edu> wrote:
>
>> Hi All,
>>
>>
>>
>> I have an issue that seems very simple, but I can't seem to figure  
>> it out.
>>
>>
>>
>> I want to submit a job to the default queue for 4 nodes using 1  
>> processor
>> per node.   Each node has 2 processors.  I use the following line  
>> in my
>> submission script.
>>
>>
>>
>> #PBS -l nodes=4:ppn=1
>>
>>
>>
>> When I submit this script, I cat the $PBS_NODEFILE to see what  
>> nodes were
>> allocated.  It seems to always come back with 2 nodes with 2 ppn  
>> for a
>> total
>> of 4 cpus.
>>
>>
>>
>> I just tried this on my other cluster with a similar setup except  
>> with 8
>> cores per node.  With "-l nodes=3:ppn=3" I get 3 nodes with 3  
>> proccessors
>> on
>> each one.  When I use nodes=3:ppn=1 I get 1 node with 3 cores  
>> allocated.
>>
>>
>>
>> Is there a simple explanation for this?
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Ross
>>
>>
>>
>> /---------------------------------------------------------------------\
>>
>> |  Ross J. Ishida                 University of Hawaii,  
>> Manoa         |
>>
>> |  Network Administrator          School of Ocean and  
>> Earth           |
>>
>> |  Research Computing Facility    Science and  
>> Technology(SOEST)       |
>>
>> |  1680 East West Road            Phone:  
>> (808)956-2558                |
>>
>> |  POST #820                      FAX:    
>> (808)956-9256                |
>>
>> |  Honolulu, HI     96822         email:  
>> ishida at soest.hawaii.edu      |
>>
>> \---------------------------------------------------------------------/
>>
>>
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20081007/78254812/attachment.html
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20081008/a9e474d4/attachment.html

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20081007/c49e1768/attachment.html