[Rocks-Discuss] pbs ppn not behaving
Donald Tripp
uhhhpc at gmail.com
Tue Oct 7 23:44:39 PDT 2008
Maui will look at all resources at a large pool, and will put jobs
anywhere they fit. In default mode, for nodes=X:ppn=Y, maui will find
(X*Y) number of available processors and put them there. In general
this works fine, but if you need more control over the execution, say
if you have a application that takes a large amount of memory, you
would indeed want to specify a minimal number of processes per node to
preserve maximum RAM.
Try this in your maui.cfg file:
JOBNODEMATCHPOLICY EXACTNODE
NODEACCESSPOLICY SINGLEJOB
Its even a bit more restrictive in that it allows only one job to run
per node. Even with EXACTNODE enabled, multiple users jobs could run
per node, defeating the purpose of specifying a particular number of
processors per node. Since you are being a bit more restrictive with
SINGLEJOB, the queue may slow down a bit, so also make sure:
BACKFILLPOLICY FIRSTFIT
is enabled, so smaller jobs will run and not get backlogged.
- Donald Tripp
dtripp at hawaii.edu
----------------------------------------------
HPC Systems Administrator
High Performance Computing Center
University of Hawai'i at Hilo
200 W. Kawili Street
Hilo, Hawaii 96720
http://www.hpc.uhh.hawaii.edu
On Oct 7, 2008, at 8:29 PM, Thomas Hamel wrote:
> Hi,
>
> Simple, not really, but there is one. When you do a nodes=X:ppn=Y
> you're
> asking maui to give you X groups of Y CPUs where the Y CPUs are on
> the same
> node. Maui can then put multiple groups on the same node. So, in
> your case
> you have 2 groups of 1 cpus per node.
>
> You can change this behavior by setting JOBNODEMATCHPOLICY to
> EXACTNODE in
> /opt/maui/maui.cfg :
>
> http://www.clusterresources.com/products/maui/docs/a.fparameters.shtml#jobnodematchpolicy
>
> Regards,
>
> Thomas HAMEL
>
> On Wed, Oct 8, 2008 at 3:43 AM, Ross Ishida
> <ishida at soest.hawaii.edu> wrote:
>
>> Hi All,
>>
>>
>>
>> I have an issue that seems very simple, but I can't seem to figure
>> it out.
>>
>>
>>
>> I want to submit a job to the default queue for 4 nodes using 1
>> processor
>> per node. Each node has 2 processors. I use the following line
>> in my
>> submission script.
>>
>>
>>
>> #PBS -l nodes=4:ppn=1
>>
>>
>>
>> When I submit this script, I cat the $PBS_NODEFILE to see what
>> nodes were
>> allocated. It seems to always come back with 2 nodes with 2 ppn
>> for a
>> total
>> of 4 cpus.
>>
>>
>>
>> I just tried this on my other cluster with a similar setup except
>> with 8
>> cores per node. With "-l nodes=3:ppn=3" I get 3 nodes with 3
>> proccessors
>> on
>> each one. When I use nodes=3:ppn=1 I get 1 node with 3 cores
>> allocated.
>>
>>
>>
>> Is there a simple explanation for this?
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Ross
>>
>>
>>
>> /---------------------------------------------------------------------\
>>
>> | Ross J. Ishida University of Hawaii,
>> Manoa |
>>
>> | Network Administrator School of Ocean and
>> Earth |
>>
>> | Research Computing Facility Science and
>> Technology(SOEST) |
>>
>> | 1680 East West Road Phone:
>> (808)956-2558 |
>>
>> | POST #820 FAX:
>> (808)956-9256 |
>>
>> | Honolulu, HI 96822 email:
>> ishida at soest.hawaii.edu |
>>
>> \---------------------------------------------------------------------/
>>
>>
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20081007/78254812/attachment.html
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20081008/a9e474d4/attachment.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20081007/c49e1768/attachment.html
More information about the npaci-rocks-discussion
mailing list