[Rocks-Discuss] questions on rocks cluster hardware setup

Leers, Frank Frank.Leers at lifetech.com
Thu Sep 30 18:41:11 PDT 2010


On Sep 30, 2010, at 4:03 PM, Tim Carlson wrote:

On Thu, 30 Sep 2010, Philip Papadopoulos wrote:

On Thu, Sep 30, 2010 at 9:39 AM, Gijs Koppers <gijs at koppers.se<mailto:gijs at koppers.se>> wrote:

Hi Philip,

Since most IB card come with 2 interfaces i was wondering if i could apply
the same thinking (no switch) to a 3 node cluster:
1 head node (that I do NOT use for computations) with
* eth0: public network
* ib0 <-> node1-ib0
* ib1 <-> node2-ib0
Node 1
* ib0 <-> head-ib0
* ib1 <-> node2-ib1
Node 2
* ib0 <-> head-ib0
* ib1 <-> node1-ib1

I think, in theory, this should work, but I don't really know if the IB
software stack was ever tested for a
fully connected graph (triangle in your case).
You should look at the string of messages in
http://www.mail-archive.com/linux-rdma@vger.kernel.org/msg03533.html

I think you want to run a subnet manager on the head node bound to the first
IB port (guid).
(this should enable traffic between H <--> N1,  H <--> N2
Then run another subnet manager on either node 1 or node 2 and bind it to
the second IB port.
(this should enable traffic (N1 <--> N2).

Any IB wizards out there know if this will work?

Not so sure about the binding of opensm. It is all connected so I think
that as long as opensm is running somewhere you will be fine.

OpenSM instances bind to one port and one port only, you'll need to specify in the conf file.  Multiple instances will be required, but you will end up with multiple fabrics/subnets in your scenario.

I don't think you can do much towards building a fabric with >2 nodes and no switch.

The line

Node 2
* ib0 <-> head-ib0

Needs to be replaced with
   * ib0 <-> head-ib1

assuming my napkin is correct :)

I'm going to try this out on one of my clusters tomorrow for grins.

Tim

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20100930/1e4bd509/attachment.html 


More information about the npaci-rocks-discussion mailing list