To 40 Gb/s Infiniband QDR and Beyond! HOW-TO

Categories:

In this article I will show you how to install and configure two 64-bit Linux servers with a 40Gb/s Infiniband interconnect. The two servers are connected directly together with a 7mm QDR Cable. No switch is used in this configuration or setup document.

The Inifinband cards used for the How-To have the Mellanox® ConnectX®-2 IB QDR chip MT26428. They are a dual Infiniband QSFP port with a 40Gb/s data rate per port. The cards use a PCI-E 2.0 x8 (5GT/s) slot in a UIO low-profile half-length form factor designed by SuperMicro.

Step 1: Install OpenFabric software – CentOS

  • Select OpenFabrics Enterprise Distribution

f1

  • Next, install the openib and mstflint
    • Select Optional Packages

f2

      • Select openib
      • Select mstflint
      • Click Close
    • Click Apply
    • Allow download and install to complete

Step 2: Verify that you have the most recent firmware for the Infiniband cards.

  • Go to the manufacture website and locate the firmware update bin file for your Infiniband card.
    • While you are at the site you also may want to download the latest software drivers for your specific Infiniband card.
    •  www.mellanox.com – Mellanox Technologies
  • Next, update the firmware for our Mellanox Technologies MT26428 Infiniband card
  • Open a Terminal session and type the following commands: your output will look similar to the output listed below. lspci

f3

Please note: the device id of your Infiniband card. In the above screen shot you will see that the pci device id for the card is 41:00.0

 

  • Next, begin the firmware update:
    • Make sure that your terminal window session is in your firmware update location. I saved the update.bin file on the Desktop of the root user.
    • Next type: mstflint –d 41:00.0 -i AOC.bin b
      • Checkout the screen shot below, amake sure within your file name there are no are in the file name of the update.bin file. If in the below command line entry the bin file read AOC-1.bin the firmware update would fail.

f4

  • After successful firmware update, restart the server and move on to Step 3.

Step 3: Configuring the network-scripts on Linux

  • In a terminal window change your path location to /etc/sysconfig/network-scripts/
  • Type vi ifcfg-ib0 to enter into text editor mode.

f5

  • Next, enter in the following text for the Infiniband card.

DEVICE=ib0
ONBOOT=yes
IPADDR=192.168.0.x
NETMASK=255.255.255.0
TYPE=Ethernet

PLEASE NOTE: Pertaining to the IPADDR entry above, a good rule of thumb rule that I use is to add an additional digit to the first IP number of your server. For example, if your server IP is 10.0.0.100 I would make the Infiniband IP address for IB port 0: 11.0.0.100 as well as using the correct subnetmask for your IP address range.

  • Finally, save the ifcfg-ib0 file by performing the following keystroke commands
    • Press shift and then :
    • Next type wq
      • Here is link to a blog article that I wrote listing several vi commands that I use the most. (You will need to scroll down they are under Step 2 )

https://www.padtinc.com/blog/post/2011/01/07/IT-Tips-Setting-Up-FreeNX-for-Remote-Access-to-Linux-Machines-from-Windows

Step 4: Verify that the following Infiniband services startup on reboot:

  • Open Service configuration manager off of the file menu click >System >Administration >Services
    • Select start on boot
  • Select opensm & openib on MASTER node
    • Please Note: Only one subnet manager needs to be running on any given Infiniband interconnect network
  • openib on each additional server within the Infiniband interconnect network

f6

  • Select the check box. Click Start and click Save
  • Restart servers

Done – The installation and configuration of the Infiniband card is now completed.

Important Linux Infiniband commands that I used during the installations

  • lspci – Lists all of the device ids
  • Ibv_devinfo – checks that the IB driver is running on all nodes and shows port status
  • Sminfo – check to see if the subnet manager is running.
  • Ibchecknet – checks the network connectivity status
  • Ibdiagnet – performs a set of tests on the IB network
  • Ibhosts – simple discover IB hosts
  • Ibstat – checks state of local IB port
  • Ibnodes – discovery of nodes
Categories

Get Your Ansys Products & Support from the Engineers who Contribute to this Blog.

Technical Expertise to Enable your Additive Manufacturing Success.

PADT’s Pulse Newsletter

Keep up to date on what is going on at PADT by subscribing to our newsletter.


By submitting this form, you are consenting to receive marketing emails from: . You can revoke your consent to receive emails at any time by using the SafeUnsubscribe® link, found at the bottom of every email. Emails are serviced by Constant Contact

Share this post:

Upcoming Events

04/03/2024

Low Frequency Updates in Ansys 2024 R1 - Webinar

04/03/2024

Venture Madness Conference Reception + Expo

04/03/2024

Stratasys F3300: Game Changing Throughput - Webinar

04/08/2024

39th Space Symposium

04/09/2024

39th Space Symposium

04/10/2024

Discovery Updates in Ansys 2024 R1 - Webinar

04/10/2024

39th Space Symposium

04/11/2024

39th Space Symposium

04/22/2024

Experience Stratasys Truck Tour: Houston, TX

04/24/2024

Structures Updates in Ansys 2024 R1 (2)

04/24/2024

Experience Stratasys Truck Tour: Houston, TX

05/07/2024

Experience Stratasys Truck Tour: Albuquerque, NM

05/08/2024

Fluent Materials Processing Updates in Ansys 2024 R1 - Webinar

05/09/2024

Experience Stratasys Truck Tour: Los Alamos, NM

05/14/2024

Simulation World 2024

05/15/2024

Simulation World 2024

05/16/2024

Simulation World 2024

05/22/2024

Optics Updates in Ansys 2024 R1 - Webinar

06/12/2024

Connect Updates in Ansys 2024 R1 - Webinar

06/26/2024

Structures Updates in Ansys 2024 R1 (3) - Webinar

06/27/2024

E-Mobility and Clean Energy Summit

07/10/2024

Fluids Updates in Ansys 2024 R1 - Webinar

08/05/2024

2024 CEO Leadership Retreat

10/23/2024

PADT30 | Nerdtoberfest 2024

Search in PADT site

Contact Us

Most of our customers receive their support over the phone or via email. Customers who are close by can also set up a face-to-face appointment with one of our engineers.

For most locations, simply contact us: