To 40 Gb/s Infiniband QDR and Beyond! HOW-TO

By: David Mastel
– March 4, 2011
Categories:

In this article I will show you how to install and configure two 64-bit Linux servers with a 40Gb/s Infiniband interconnect. The two servers are connected directly together with a 7mm QDR Cable. No switch is used in this configuration or setup document.

The Inifinband cards used for the How-To have the Mellanox® ConnectX®-2 IB QDR chip MT26428. They are a dual Infiniband QSFP port with a 40Gb/s data rate per port. The cards use a PCI-E 2.0 x8 (5GT/s) slot in a UIO low-profile half-length form factor designed by SuperMicro.

Step 1: Install OpenFabric software – CentOS

  • Select OpenFabrics Enterprise Distribution

f1

  • Next, install the openib and mstflint
    • Select Optional Packages

f2

      • Select openib
      • Select mstflint
      • Click Close
    • Click Apply
    • Allow download and install to complete

Step 2: Verify that you have the most recent firmware for the Infiniband cards.

  • Go to the manufacture website and locate the firmware update bin file for your Infiniband card.
    • While you are at the site you also may want to download the latest software drivers for your specific Infiniband card.
    •  www.mellanox.com – Mellanox Technologies
  • Next, update the firmware for our Mellanox Technologies MT26428 Infiniband card
  • Open a Terminal session and type the following commands: your output will look similar to the output listed below. lspci

f3

Please note: the device id of your Infiniband card. In the above screen shot you will see that the pci device id for the card is 41:00.0

 

  • Next, begin the firmware update:
    • Make sure that your terminal window session is in your firmware update location. I saved the update.bin file on the Desktop of the root user.
    • Next type: mstflint –d 41:00.0 -i AOC.bin b
      • Checkout the screen shot below, amake sure within your file name there are no are in the file name of the update.bin file. If in the below command line entry the bin file read AOC-1.bin the firmware update would fail.

f4

  • After successful firmware update, restart the server and move on to Step 3.

Step 3: Configuring the network-scripts on Linux

  • In a terminal window change your path location to /etc/sysconfig/network-scripts/
  • Type vi ifcfg-ib0 to enter into text editor mode.

f5

  • Next, enter in the following text for the Infiniband card.

DEVICE=ib0
ONBOOT=yes
IPADDR=192.168.0.x
NETMASK=255.255.255.0
TYPE=Ethernet

PLEASE NOTE: Pertaining to the IPADDR entry above, a good rule of thumb rule that I use is to add an additional digit to the first IP number of your server. For example, if your server IP is 10.0.0.100 I would make the Infiniband IP address for IB port 0: 11.0.0.100 as well as using the correct subnetmask for your IP address range.

  • Finally, save the ifcfg-ib0 file by performing the following keystroke commands
    • Press shift and then :
    • Next type wq
      • Here is link to a blog article that I wrote listing several vi commands that I use the most. (You will need to scroll down they are under Step 2 )

http://www.www.padtinc.com/blog/post/2011/01/07/IT-Tips-Setting-Up-FreeNX-for-Remote-Access-to-Linux-Machines-from-Windows.aspx

Step 4: Verify that the following Infiniband services startup on reboot:

  • Open Service configuration manager off of the file menu click >System >Administration >Services
    • Select start on boot
  • Select opensm & openib on MASTER node
    • Please Note: Only one subnet manager needs to be running on any given Infiniband interconnect network
  • openib on each additional server within the Infiniband interconnect network

f6

  • Select the check box. Click Start and click Save
  • Restart servers

Done – The installation and configuration of the Infiniband card is now completed.

Important Linux Infiniband commands that I used during the installations

  • lspci – Lists all of the device ids
  • Ibv_devinfo – checks that the IB driver is running on all nodes and shows port status
  • Sminfo – check to see if the subnet manager is running.
  • Ibchecknet – checks the network connectivity status
  • Ibdiagnet – performs a set of tests on the IB network
  • Ibhosts – simple discover IB hosts
  • Ibstat – checks state of local IB port
  • Ibnodes – discovery of nodes
Categories

Certified Elite Channel Partner

Get Your Ansys Products & Support from the Engineers who Contribute to this Blog.

Product Development
Diamond Partner

Technical Expertise to Enable your Addictive Manufacturing Success.

PADT’s Pulse Newsletter

Keep up to date on what is going on at PADT by subscribing to our newsletter.


By submitting this form, you are consenting to receive marketing emails from: Phoenix Analysis and Design Technologies, 7755 S. Research Dr., Tempe, AZ, 85284, https://www.padtinc.com. You can revoke your consent to receive emails at any time by using the SafeUnsubscribe® link, found at the bottom of every email. Emails are serviced by Constant Contact

Share this post:

Share on twitter
Share on facebook
Share on linkedin
Share on pinterest

Upcoming Events

05/25/2022

05/24/2022

SMR & Advanced Reactor 2022

05/19/2022

RAPID + tct 2022

05/19/2022

Venture Cafe Roundtable: AI & Healthcare

05/18/2022

Tucson after5 Tech Mixer: World View

05/18/2022

RAPID + tct 2022

More Info

05/18/2022

Signal & Power Integrity Updates in Ansys 2022 R1 - Webinar

05/18/2022

Simulation World 2022

05/17/2022

RAPID + tct 2022

05/11/2022

Experience Stratasys Manufacturing Virtual Event

05/04/2022

Mechanical Meshing Updates in Ansys 2022 R1 - Webinar

04/27/2022

04/22/2022

12TH ANNUAL TUCSON GOLF TOURNAMENT

04/21/2022

04/20/2022

Additional Fluids Updates in Ansys 2022 R1

04/20/2022

Experience Stratasys Tour – Tempe Arizona

04/18/2022

Experience Stratasys Tour - Flagstaff Arizona

04/14/2022

D&M West | MD&M West

04/13/2022

D&M West | MD&M West

04/13/2022

Experience Stratasys Tour - Albuquerque New Mexico

04/12/2022

D&M West | MD&M West

04/12/2022

Experience Stratasys Tour - Los Alamos New Mexico

04/12/2022

Optimizing Engineering Workflows f​​​​or Propulsion System Design

04/07/2022

Experience Stratasys Tour - Austin Texas

04/07/2022

37th Space Symposium - Arizona Space Industry

04/06/2022

Transforming Digital Engineering with Ansys Discovery 2022 R1

04/06/2022

37th Space Symposium - Arizona Space Industry

04/05/2022

37th Space Symposium - Arizona Space Industry

04/04/2022

37th Space Symposium - Arizona Space Industry

03/30/2022

Simulation Best Practices for Vehicle Engineering - Webinar

03/23/2022

03/23/2022

High & Low Frequency Electromagnetics Updates in Ansys 2022 R1

02/24/2022

Arizona Technology Council After 5 Tech Mixer "Pandemic Pivot Pizza Pa

02/23/2022

SciTech Festival: Spend an Hour with 3D Printing Experts

02/11/2022

Webinar: Mechanical overview for Ansys 2022 R1

More Info

02/09/2022

Webinar: Product Development 101 (FAKE)

02/08/2022

Webinar: Navigating the Additive Landscape

01/27/2022

Arizona Technology Council 1st Quarter VIP Tech Mixer

More Info

01/26/2022

Simulation Best Practices for Gas Turbine Design & Development - Webin

More Info

01/19/2022

Arizona Photonics Days

More Info

11/04/2021

ExperienceIT, New Mexico

More Info

11/03/2021

Additive Manufacturing & Structural Optimization in Ansys 2021 R2 - We

More Info

11/03/2021

Optics Valley Technical Series: The Future of Simulation in the Optics

More Info

11/02/2021

SBIR Liftoff AZTC Virtual Breakfast Series

More Info

10/10/2021

Stratasys Mobile Truck Stop - Tucson Arizona

More Info

Search in PADT site

Contact Us

Most of our customers receive their support over the phone or via email. Customers who are close by can also set up a face-to-face appointment with one of our engineers.

For most locations, simply contact us: