To 40 Gb/s Infiniband QDR and Beyond! HOW-TO

In this article I will show you how to install and configure two 64-bit Linux servers with a 40Gb/s Infiniband interconnect. The two servers are connected directly together with a 7mm QDR Cable. No switch is used in this configuration or setup document.

The Inifinband cards used for the How-To have the Mellanox® ConnectX®-2 IB QDR chip MT26428. They are a dual Infiniband QSFP port with a 40Gb/s data rate per port. The cards use a PCI-E 2.0 x8 (5GT/s) slot in a UIO low-profile half-length form factor designed by SuperMicro.

Step 1: Install OpenFabric software – CentOS

  • Select OpenFabrics Enterprise Distribution

f1

  • Next, install the openib and mstflint
    • Select Optional Packages

f2

      • Select openib
      • Select mstflint
      • Click Close
    • Click Apply
    • Allow download and install to complete

Step 2: Verify that you have the most recent firmware for the Infiniband cards.

  • Go to the manufacture website and locate the firmware update bin file for your Infiniband card.
    • While you are at the site you also may want to download the latest software drivers for your specific Infiniband card.
    •  www.mellanox.com – Mellanox Technologies
  • Next, update the firmware for our Mellanox Technologies MT26428 Infiniband card
  • Open a Terminal session and type the following commands: your output will look similar to the output listed below. lspci

f3

Please note: the device id of your Infiniband card. In the above screen shot you will see that the pci device id for the card is 41:00.0

 

  • Next, begin the firmware update:
    • Make sure that your terminal window session is in your firmware update location. I saved the update.bin file on the Desktop of the root user.
    • Next type: mstflint –d 41:00.0 -i AOC.bin b
      • Checkout the screen shot below, amake sure within your file name there are no are in the file name of the update.bin file. If in the below command line entry the bin file read AOC-1.bin the firmware update would fail.

f4

  • After successful firmware update, restart the server and move on to Step 3.

Step 3: Configuring the network-scripts on Linux

  • In a terminal window change your path location to /etc/sysconfig/network-scripts/
  • Type vi ifcfg-ib0 to enter into text editor mode.

f5

  • Next, enter in the following text for the Infiniband card.

DEVICE=ib0
ONBOOT=yes
IPADDR=192.168.0.x
NETMASK=255.255.255.0
TYPE=Ethernet

PLEASE NOTE: Pertaining to the IPADDR entry above, a good rule of thumb rule that I use is to add an additional digit to the first IP number of your server. For example, if your server IP is 10.0.0.100 I would make the Infiniband IP address for IB port 0: 11.0.0.100 as well as using the correct subnetmask for your IP address range.

  • Finally, save the ifcfg-ib0 file by performing the following keystroke commands
    • Press shift and then :
    • Next type wq
      • Here is link to a blog article that I wrote listing several vi commands that I use the most. (You will need to scroll down they are under Step 2 )

http://www.padtinc.com/blog/post/2011/01/07/IT-Tips-Setting-Up-FreeNX-for-Remote-Access-to-Linux-Machines-from-Windows.aspx

Step 4: Verify that the following Infiniband services startup on reboot:

  • Open Service configuration manager off of the file menu click >System >Administration >Services
    • Select start on boot
  • Select opensm & openib on MASTER node
    • Please Note: Only one subnet manager needs to be running on any given Infiniband interconnect network
  • openib on each additional server within the Infiniband interconnect network

f6

  • Select the check box. Click Start and click Save
  • Restart servers

Done – The installation and configuration of the Infiniband card is now completed.

Important Linux Infiniband commands that I used during the installations

  • lspci – Lists all of the device ids
  • Ibv_devinfo – checks that the IB driver is running on all nodes and shows port status
  • Sminfo – check to see if the subnet manager is running.
  • Ibchecknet – checks the network connectivity status
  • Ibdiagnet – performs a set of tests on the IB network
  • Ibhosts – simple discover IB hosts
  • Ibstat – checks state of local IB port
  • Ibnodes – discovery of nodes