In this article I will show you how to install and configure two 64-bit Linux servers with a 40Gb/s Infiniband interconnect. The two servers are connected directly together with a 7mm QDR Cable. No switch is used in this configuration or setup document.
The Inifinband cards used for the How-To have the Mellanox® ConnectX®-2 IB QDR chip MT26428. They are a dual Infiniband QSFP port with a 40Gb/s data rate per port. The cards use a PCI-E 2.0 x8 (5GT/s) slot in a UIO low-profile half-length form factor designed by SuperMicro.
Step 1: Install OpenFabric software – CentOS
- Select OpenFabrics Enterprise Distribution
- Next, install the openib and mstflint
- Select Optional Packages
- Select openib
- Select mstflint
- Click Close
- Click Apply
- Allow download and install to complete
Step 2: Verify that you have the most recent firmware for the Infiniband cards.
- Go to the manufacture website and locate the firmware update bin file for your Infiniband card.
- While you are at the site you also may want to download the latest software drivers for your specific Infiniband card.
- www.mellanox.com – Mellanox Technologies
- Next, update the firmware for our Mellanox Technologies MT26428 Infiniband card
- Open a Terminal session and type the following commands: your output will look similar to the output listed below. lspci
Please note: the device id of your Infiniband card. In the above screen shot you will see that the pci device id for the card is 41:00.0
- Next, begin the firmware update:
- Make sure that your terminal window session is in your firmware update location. I saved the update.bin file on the Desktop of the root user.
- Next type: mstflint –d 41:00.0 -i AOC.bin b
- Checkout the screen shot below, amake sure within your file name there are no – are in the file name of the update.bin file. If in the below command line entry the bin file read AOC-1.bin the firmware update would fail.
- After successful firmware update, restart the server and move on to Step 3.
Step 3: Configuring the network-scripts on Linux
- In a terminal window change your path location to /etc/sysconfig/network-scripts/
- Type vi ifcfg-ib0 to enter into text editor mode.
- Next, enter in the following text for the Infiniband card.
DEVICE=ib0
ONBOOT=yes
IPADDR=192.168.0.x
NETMASK=255.255.255.0
TYPE=Ethernet
PLEASE NOTE: Pertaining to the IPADDR entry above, a good rule of thumb rule that I use is to add an additional digit to the first IP number of your server. For example, if your server IP is 10.0.0.100 I would make the Infiniband IP address for IB port 0: 11.0.0.100 as well as using the correct subnetmask for your IP address range.
- Finally, save the ifcfg-ib0 file by performing the following keystroke commands
- Press shift and then :
- Next type wq
- Here is link to a blog article that I wrote listing several vi commands that I use the most. (You will need to scroll down they are under Step 2 )
https://www.padtinc.com/blog/post/2011/01/07/IT-Tips-Setting-Up-FreeNX-for-Remote-Access-to-Linux-Machines-from-Windows
Step 4: Verify that the following Infiniband services startup on reboot:
- Open Service configuration manager off of the file menu click >System >Administration >Services
- Select start on boot
- Select opensm & openib on MASTER node
- Please Note: Only one subnet manager needs to be running on any given Infiniband interconnect network
- openib on each additional server within the Infiniband interconnect network
- Select the check box. Click Start and click Save
- Restart servers
Done – The installation and configuration of the Infiniband card is now completed.
Important Linux Infiniband commands that I used during the installations
- lspci – Lists all of the device ids
- Ibv_devinfo – checks that the IB driver is running on all nodes and shows port status
- Sminfo – check to see if the subnet manager is running.
- Ibchecknet – checks the network connectivity status
- Ibdiagnet – performs a set of tests on the IB network
- Ibhosts – simple discover IB hosts
- Ibstat – checks state of local IB port
- Ibnodes – discovery of nodes
