CUBE Systems are Now Part of the ANSYS, Inc. HPC Partner Program


The relationship between ANSYS, Inc. and PADT is a long one that runs deep. And that relationship just got stronger with PADT joining the HPC Partner Program with our line of CUBE compute systems specifically designed for simulation. The partner program was set up by ANSYS, Inc. to work:

CUBE-HVPC-512-core-closeup3-1000h_thumb.jpg“… with leaders in high-performance computing (HPC) to ensure that the engineering simulation software is optimized on the latest computing platforms. In addition, HPC partners work with ANSYS to develop specific guidelines and recommended hardware and system configurations. This helps customers to navigate the rapidly changing HPC landscape and acquire the optimum infrastructure for running ANSYS software. This mutual commitment means that ANSYS customers get outstanding value from their overall HPC investment.”


PADT is very excited to be part of this program and to contribute to the ANSYS/HPC community as much as we can.  Users know they can count on PADT’s strong technical expertise with ANSYS Mechanical, ANSYS Mechanical APDL, ANSYS FLUENT, ANSYS CFX, ANSYS Maxwell, ANSYS HFSS, and other ANSYS, Inc. products, a true differentiator when compared with other hardware providers.

Customers around the US have fallen in love with their CUBE workstations, servers, mini-clusters, and clusters finding them to be the right mix between price and performance. CUBE systems let users carry out larger simulations, with greater accuracy, in less time, at a lower cost than name-brand solutions. This leaves you more cash to buy more hardware or software.

Assembled by PADT’s IT staff, CUBE computing systems are delivered with the customer’s simulation software loaded and tested. We configure each system specifically for simulation, making choices based upon PADT’s extensive experience using similar systems for the same kind of work. We do not add things a simulation user does not need, and focus on the hardware and setup that delivers performance.


Is it time for you to upgrade your systems?  Is it time for you to “step out of the box, and step in to a CUBE?”  Download a brochure of typical systems to see how much your money can actually buy, visit the website, or contact us.  Our experts will spend time with you to understand your needs, your budget, and what your true goals are for HPC. Then we will design your custom system to meet those needs.


The 10 Coolest New Features in R15 of ANSYS Mechanical APDL

On Tuesday we posted on what I thought were the 10 coolest features in ANSYS Mechanical for R15. Now it is time to take a good look at ANSYS Mechanical APDL, or MAPDL (classic ANSYS, black window ANSYS, or my favorite: ANSYS).  The developers have been very busy and added a lot of useful features, and there are a large number of “Oh Yes!” capabilities in this release that different groups of users will be very excited about.  For this posting though, we are going to stay focused on the things that impact larger groups of users and/or expand capability in the code.  As always, you can learn more by attending one of the many upcoming ANSYS webinars or reading the release notes in the help.

image1: Rezoning Enhancements and Additions

This is my favorite change in R15, a mix of some improvements and some new capabilities. The whole idea of rezoning is that when you have a part that sees a large amount deformation, the mesh often gets very distorted. It often gets so distorted that the elements are no longer accurate and crazy strains are calculated and the element literally blows up.  Or it turns inside out and generates an error in the solver.

Rezoning has been around for a while but at this release some holes are plugged and some big advances are made.  The first change was a hole plug, you can now rezone areas that contain surface effect  (SURF153/154) elements.  It is very common to have that type of load on highly distorted geometry, so this is welcomed.

The next change was adding mesh splitting for 3D  tetrahedral elements. This is used for manual rezoning with the REMESH command. It was available before with 2D elements.  The 2D example from the manual shows it best:


The advantage of this approach is that the subsequent stress field that is placed upon the new mesh is already accurate at the nodes that existed for the original mesh, and are fairly accurately interpolated for the new nodes.  When you read in a completely new mesh, you have to interpolate the stress field and then iterate till the stresses are accurate.  This approach can be much faster.

The third and best addition is Automatic Rezoning or Mesh Nonlinear Adaptivity.  This process is completely automatic and does not require the user interaction that rezoning does.  Both splitting and remeshing are used. You can turn remeshing on based upon position, energy levels, or contact conditions.

Here is an example from the user manual:


And here is an example that ANSYS, Inc. is showing on the new :


image2: Bolt Thread Modeling

Modeling bolt threads.  Classic newby mistake right?  They model threads on 37 bolts and then try and set up contact on all of them.  Never goes well does it. ANSYS MAPDL has had a bolt modeling capability for some time that allows you to simulate a bolt as a cylinder with preload and everything. But what that approximation missed was the fact that the contact is at an angle that is not normal to the cylinder surface.

At R15 you can now specify your thread geometry and the contact algorithm will calculate the proper normal and contact pattern for the contact forces. Much more efficient. There is a great example in the Technology Demonstration Guide, Section 39, showing all three approaches: model the threads in the mesh, use the new contact threads, or just bond the threaded area.


Needless to say we will be doing an in-depth posting on this one in the future.

image3: Mode-Superposition for Harmonic Analysis of Cyclic Structures

I started my career in turbomachinary and from day one, one of the holy grails was to be able to do a harmonic analysis using blade pressure loads from a CFD run: getting the actual stresses in the blades caused by the varying aerodynamic load as they spun around and dealt with variations caused by passing frequencies and resonance in the flow itself.  It was always doable as a full 360 model on both the CFD and structural side. And you could have done it using the full method for a few releases.  But now we can use cyclic symmetry and mode-superposition.

The ANSYS MAPDL side of things is released in R15.  You can take your complex loading info from CFD and apply that as a load on your blades using the new /MAP pre-processor (see below), a bit of a pain to do in the past.  The other big change was making it all work with modal-superposition.  The grail is almost complete.

image4: Arc Length

This is one of those things buried in the code that the user really doesn’t have to do anything to benefit from. If you are not familiar with the method, it is an approach used on non-linear problems using the Newton-Raphson method. Most solves use other methods,  but for things like non-linear buckling it is a better method. Check out 14.12 in the Theory Reference for the math and all that.

The bottom line at R15 is that they changed the algorithm to use the Crisfield Method and to avoid Driftback.  What it means to you the user is those nasty non-linear buckling problems that always seem to have a hard time converging, or that require really small steps to converge, should converge now or converge faster.

image5: Mode Selection

This is another one of those advanced options that users of other solvers have been asking for in the past. When you do a modal analysis that produces a ton of modes, you often want to ignore the majority of them and focus on the few modes that are strong or that get excited.  In the past you could specify a range only, and only one range. At R15 you can now select which modes to use in modal-superposition analysis.  You can decide which modes to use based on the modal effective mass, the mode coefficient, or the DDAM Procedure. Or, if you have your own criteria, you can use APDL to create a table that specifies a 1 for keep, and a 0 for toss. Very handy.

image6: Acoustics Enhancements

This is really not one new feature, but an overall continuation of adding functionality to the acoustics capability in ANSYS MAPDL. For decades, this capability was not really focused on advanced acoustic modeling. But over the last couple of releases we have seen added functionality that put the functionality on a par with specialty acoustics codes.

The key enhancements at R15 are:

  • Frequency-dependent acoustic material properties
  • Surface impedance can be frequency-dependent
  • A new boundary layer impedance (BLI) model is available for visco-thermo fluids modeling
  • A wider range of units are now supported for acoustics, including support for user defined units (/UNITS)
  • Many enhancements for coupling acoustics with CFD for FSI
  • New postprocessing commands for calculating acoustics specific information like sound power level, A-weighted sound pressure level (dBA), and return loss, and transmission loss, amongst others.


image7: Shape Memory on Beam, Shell and Plane Strain

For whatever reason shape memory alloys have always fascinated me, and being able to simulate them accurately is very important for those that use the material in their products. Development has been adding more and more functionality in this area for many releases.  The material has two unique properties: it is super elastic and it has a memory effect.

With R15 development rounds out the capabilities with full support for beam and shell elements, adding the memory behavior.  This is important, and warrants top 10 status for me, because many of the geometries we have worked on that use Nitinol (the most common shape memory alloy) are made with wires.  In the past you had to model them in 3D, now we can use beams.  Faster, more accurate, etc…


image8: RSTMAC to Experimental Data

Something we should all be doing more is comparing our FEA results to experimental data. One excuse we often use is that it is too hard to compare the data from a vib test to our modal analysis results. Well, that is no longer to. The RSTMAC command has been modified to not only compare to ANSYS result files, but to also read the old Universal file format for results. Yipee!

Why that format? Because back in the days when SDRC was SDRC and IDEAS was their prep/post tool, they had some awesome result comparison tools. So a lot of test software out there writes to the file that IDEAS read, the Universal file.  If your software does not write to a Universal file, the key info you need is in the user manual: Basic Analysis Guide, section  and here is a link to some documentation on it.


image9: Mapping Processor

It has been a long time since ANSYS, Inc. has added a new processor to ANSYS Mechanical APDL.  /PREP7, /SOLVE, /POST26.  So it was kind of cool to see that they are creating a new mapping processor called /MAP that will be a place for you to do load mapping. At this initial release, it maps surface pressures as a point cloud from a CFD model onto your mechanical model.

Under the hood it is actually just the algorithms used in the *MOPER APDL function.  But now it is exposed in through its own set of commands so that users don’t have to script their load mapping. And it supports imaginary loading for that fancy cyclic-symmetry stuff some of us need to do. As you can imagine, this needs its own article, but here are the high points:

  • Enter the processor with /MAP
  • Your model must have surface effect elements (SURF154) elements paved onto the outside of where you want the pressures.


  • Specify the nodes you want to map the pressures on to with the TARGET command
  • You can provide pressures in a variety of formats (specified with the FTYPE command):


  • CFX Transient Blade Row format is made by CFX and contains real and imaginary terms
  • The standard output file from ANSYS CFDPost
  • A fixed format file that has x, y, z, and pressure, and yes, you can specify the actual format in FORTRAN using the READ command
  • And of course the trusty comma delimited file format: x, y, z, pressure.


  • The READ command specifies some other parameters and reads in the point based pressure data.
  • They have given us a PLGEOM command to view the target nodes and the point cloud on top of each other so you can see if things are aligned
  • A whole slew of /PREP7 like commands to edit and move your point cloud data. Basically they are treated as nodes and you manipulate them like nodes. They are just nodes with a pressure assigned to them.
  • When everything is good, use MAP to actually do the interpolation.
  • View the results with PLMAP
  • When you are happy, save the pressures as SFE commands using WRITEMAP


There is no GUI interface for this yet.  It was put into place to support some advanced FSI modeling of turbomachinery, but it benefits all users.  We hope to see more in this new module in future releases.  Here is an example we were playing with at PADT:



Image10: Performance Enhancements, Including GPU Stuff

Last but certainly not least are the enhancements to solver performance that we have come to expect. New compilers, optimized code, and new hardware all come together to deliver better bang for your ANSYS buck.  There is a ton in there, all documented in section 2.3.1 of the ANSYS, Inc. Release Notes part of the help.  The highlights are:

  • The sparse solver now has some sophisticated error detection for handling singular or near singular matrices.  This should keep you from solving poorly constrained models, or models with really messed up elements. Do note, some models that ran in the past, maybe with a warning, will now not solve. This is a good thing since the matrices are not good
  • Better domain decomposition for distributed ANSYS, especially for larger core counts.
  • The subspace method has been added for solving modal analysis.  It is well suited for larger problems and runs well distributed.
  • Switching to the new Intel compiler has resulted in a 30% faster solve time on some problems when using Sandy Bridge Intel processors.
  • Harmonic analyses solves using the full method have been improved, resulting into up to 40% improvements in solve time.
  • The Intel Xeon Phi coprocessor is now supported – we have no data yet on the performance but will try and get some as soon as we can
  • The latest NVIDIA Kepler GPU’s are now supported and the sparse solver has been improved again to take advanged of the Kepler GPU’s.

CUBE HVPC 512 Core System


The hard part for me in writing this posting was picking the top 10. There are a lot of significant enhancements but few are world changing. Most improve existing technologies, provide functionality for a subset of users, or fill a hole in capability.  Taken as a whole though, they show ANSYS, Inc.’s strong commitment to core technology: new elements, new material models, faster solves, expanded advanced capability, etc…

The end result is giving greater power to the user through greater depth and breadth.  And in the end, isn’t this why we use ANSYS Mechanical APDL in the first place – the incredible breadth and depth of capability it offers?  Scrolling back up through the images you have to admit, this is some pretty cool stuff.

The 10 Coolest New Features in R15 of ANSYS Mechanical

It’s that time of year again, time for a new release of ANSYS, Inc’s products.  R15 is being released in stages to the user community this week so we thought we would take some time to point out ten features in R15 of ANSYS Mechanical that we find useful, important, or just plain cool.  There are a ton of new features and we will try and cover most of them in the coming months, but these are the ones we felt every user should know about.

This posting will focus on features in ANSYS Mechanical that are unique to ANSYS Mechanical.  Later this week or next week we will do the same for ANSYS Mechanical APDL, and we will cover solver changes that impact ANSYS Mechanical there as well.

image1: Mesh Based Geometry

This is by far the most far-reaching enhancement in R15.  A fundamental limitation of ANSYS Mechanical from the beginning was the requirement that you had to have a valid BREP geometry that can be correctly meshed in ANSYS Mechanical. For most problems this is fine, you have a CAD model, you mesh it, and you move on.  But there are often situations where you have a legacy model or a mesh from another source that you want to use. And in such cases you were just stuck. Most things in ANSYS Mechanical work on geometry and if you just have a mesh, and no geometry, there area not a lot of options.

No longer.  Yipidee yapidee dooooooo daaaa!!!!  That gets a leap for joy gif:

It works through an External Model system in the Workbench project page.  There are a slew of options to copy, translate, change units, etc.. for the model. However, the most important option is the tolerance angle. The way the mesh geometry import works is that it takes the external faces of your mesh and treats them as facets. Any facets that have an angle less than the tolerance angle are considered to be on the same face. Any angle greater than the tolerance treats the edge between facets as an edge on a face.  This is key to understanding how using an mesh in Mechanical works.

I’ll resist the temptation to get into the details and save that for a future posting where we can dig deep. 

To prove it out I searched my hard drive for old *.DB files and found one from 2001. A generic turbine blade ad disk model I made for some cyclic-symmetry testing ANSYS was doing.  Here is what is looks like in ANSYS Mechanical APDL:


And here it what it looks like in ANSYS Mechanical:


That is a sight for sore eyes.  It’s not perfect, the trailing edge muffs things up a big because the mesh is a bit coarse there. But a little work playing with the tolerance angle and/or named selections in MAPDL and that can be cleaned up. More on that in that promised post on this capability.  Here is a wireframe where you can see the internal cooling passages, and the funky elements.


Changing the tolerance from the default 45deg to 55deg cleans up most of the problems:


image2: Assembly Mechanical Models

A related capability to the mesh geometry mentioned above is the ability to create an assembly of multiple external models (mesh based) and other Mechanical Models.

You can take as many External Model, Mechanical Model, or various Analysis systems as you want, and feed them all into a new mechanical model or any analysis system you want to use them in. ANSYS, Inc. provided this really cool example of a model of a pad lock:


As much information as is possible is transferred over, depending on what makes sense.  Also note that you can apply transformations and units conversions to each model, so if you build your parts in different coordinate systems, you can move them around and get them set up when you build the assembled model.

image3: Parallel Meshing by Part

The first step in making meshing fully parallel in ANSYS Mechanical has been released in R15: parallel part meshing.  Basically, if you have more than one part, the program will mesh each part on its own CPU.  So if you have 8 cores and 6 parts, it will mesh on 6 cores at one time.  The default setting is 0, which tells the program to run on as many cores as it can.  Most users will want to keep it there, but if you do a lot of models with lots of large parts, you may want to set it at N-1 where N is the number of cores on your machine.  Leave one open to watch YouTube on while you are meshing.

The settings are in Tools->Options->Meshing->Meshing->Meshing->Number of CPUs for Meshing Methods

image4: Mechanical Shortcut Keys

You may love this one, or it may generate a “Meh.”  Shortcut Keys are almost a religious thing.  If you are on the “true believer” side then you now can use function and control keys to do the following actions:

Tree Outline Actions

F1: opens the Mechanical User’s Guide.
F2: rename a selected tree object.
Ctrl + S: save the project.

Graphics Actions

F6: toggles between the Shaded Exterior and Edges, Shaded Exterior, and Wireframe views
F7: executes Zoom to Fit option
F8: hide selected faces.
F9: hide selected bodies.
Ctrl + A: selects all entities based on the active selection filter (bodies, faces, edges, vertices, nodes).

Selection Filters

Ctrl + B: activate Body selection.
Ctrl + E: activate Edge selection.
Ctrl + F: activate Face selection.
Ctrl + P: activate Vertex selection.

Even if you are not a Shortcut Key acolyte, the selection filters and the F7 options look pretty useful.

image5: Element Selection and Grouping

In the last release the developers at ANSYS, Inc. gave users full access to nodes in ANSYS Mechanical. At R15 they have opened up access to elements.  Using the term “opened up access” is a bit misleading, they did not just change a parameter from FALSE to TRUE and boom, you have elements.  It was a major change to how data is stored and accessed in the program. 

Selecting works as you would expect, just like nodes. You choose “Select Mesh” from the Select Type icon:


Then you choose the Select “Body/Element” from the type choices (the green cube).  Here is where you can use those shortcuts: Ctrl + B selects it for you. Now you can pick elements or use box select to get what you want.

Names selections for elements work just like nodes.  Very useful indeed. And they do get converted to components in Mechanical APDL, avoiding that annoying snippet where you had to convert a nodal component into an element component.

image6: Mapping Enhancements

One feature set in ANSYS Mechanical that makes users of other ANSYS products jealous are the mapping tools. And at this release they got even better, adding more functionality, feedback, and making some beta features released features. If you are not familiar with mapping tools, they allow you to take a load specified on points in space, and interpolate that load on to your model. Again, this is a topic worthy of its own posting, but here are the highlights:

  • Support for Velocity
  • Support for Initial Stress or Strain
  • Support for Body Force Density (forces from an electromagnetic solution)
  • Pressure can now be applied to nodes as well as to elements
  • Acoustic loads from a Harmonic Response Analysis can be mapped as velocities
  • UV Mapping is now available for surfaces that don’t really line up.

That last one, UV Mapping, may be one of the more powerful. It is not that uncommon for you to get pressures on a surface that does not really sit on your model surface.

image7: Better Control of Hydrostatic Pressure

This is pretty specialized, but if you work on parts that see hydrostatic loading you always had to play around with APDL snippets to get the control you needed.  At R15 they have added those controls into the program for us.


The first addition is that you can turn the pressure on and off for a given load step.  This is not necessarily done in an intuitive way, but it works.  Select the step in the graph or in the table when you have the load selected in the tree. Then Right Mouse Button to get the context menu and activate or deactivate the load. It doesn’t show anything in the table, but it does show on the graph that the load is turned off.  Note, you can’t change the acceleration, you can only turn it on or off.

The second addition is simply that the values for fluid density and magnitude are parameters.


image8: Multiple GPU Support, and Intel Phi Support

GPU usage for ANSYS Mechanical solves is growing.  We are seeing good enhancements in performance at every release, and R15 is no exception.  But that is solver stuff and I said we would not talk about solver stuff…  What is important in this area for ANSYS Mechanical users is that you can now specify more than one GPU for a solve, and at R15 the new Intel Phi coprocessor, which is really not a GPU, is supported. You access the control, and all settings for HPC, under Tools-> Solve Process Settings-> Advanced.  Note that there are different settings for solving interactive and in the background.


image9: Follower Load for Rigid Body Dynamics

Because Rigid Body Dynamics are, well rigid body dynamics, they are generally inherently large deflection.  When you apply a load to an object you usually want that load to move with the objects motion, to follow it.  In the past, you had to create a dummy rigid part and apply a pressure to get this.  Now at R15 you can set “Follower Load” to yes in the details view for a Remote Force and it will go along for the ride.  If you do RBD, this is critical.


image10: Bearing Enhancements

The last item on our top ten list for this release are improvements to modeling bearings in ANSYS Mechanical. The ANSYS Mechanical APDL solver supports a wide range of bearing capabilities, and with this release most of them are now exposed in ANSYS Mechanical. 

The big change is that all of the solvers that support bearings are now supported in ANSYS Mechanical. In the past, it was only modal analysis. Now you can simplify your model and get the proper stiffness and damping of your bearing for transient, static, and any other type of run you want to do.

As you would expect with the support of the bearing joint on the pre-processing side, they have added a probe that allows you to get key information out of your bearing after the solve. Since a bearing joint is basically two spring-dampers, you can get spring type info for each spring: elastic force, damping force, elongation, and velocity (for transient runs).


If you look at this list you should notice one common thread, that most of these changes are not general, they are for specific analysis types.  As time has gone by ANSYS Mechanical has grown and matured, and there are less and less basic or general features that need to be added. So now we are in to the fun stuff, digging down into the nitty gritty and exposing more and more of the most powerful solver available (ANSYS Mechanical APDL), through what is the most powerful user interface for structural mechanics, ANSYS Mechanical.

This May Be the Fastest ANSYS Mechanical Workstation we Have Built So Far

The Build Up

Its 6:30am and a dark shadow looms in Eric’s doorway. I wait until Eric finishes his Monday morning company updates. “Eric check this out, the CUBE HVPC w16i-k20x we built for our latest customer ANSYS Mechanical scaled to 16 cores on our test run.” The left eyebrow of Eric’s slightly rises up. I know I have him now I have his full and complete attention.

Why is this huge news?

This is why; Eric knows and probably many of you reading this also know that solving differential equations, distributed, parallel along with using graphic processing unit makes our hearts skip a beat. The finite element method used for solving these equations is CPU intensive and I/O intensive. This is headline news type stuff to us geek types. We love scratching our way along the compute processing power grids to utilize every bit of performance out of our hardware!

Oh and yes a lower time to solve is better! No GPU’s were harmed in this tests. Only one NVIDIA TESLA k20X GPU was used during the test.

Take a Deep Breath and Start from the Beginning:

I have been gathering and hording years’ worth of ANSYS mechanical benchmark data. Why? Not sure really after all I am wanna-be ANSYS Analysts. However, it wasn’t until a couple weeks ago that I woke up to the why again. MY CUBE HVPC team sold a dual socket INTEL Ivy bridge based workstation to a customer out of Washington state. Once we got the order, our Supermicro reseller‘s phone has been bouncing of the desk. After some back and forth, this is how the parts arrive directly from Supermicro, California. Yes, designed in the U.S.A.  And they show up in one big box:


Normal is as Normal Does

As per normal is as normal does, I ran the series of ANSYS benchmarks. You know the type of benchmarks that perform coupled-physics simulations and solving really huge matrix numbers. So I ran ANSYS v14sp-5, ANSYS FLUENT benchmarks and some benchmarks for this customer, the types of runs they want to use the new machine for. So I was talking these benchmark results over with Eric. He thought that now is a perfect time to release the flood of benchmark data. Well some/a smidge of the benchmark data. I do admit the data does get overwhelming so I have tried to trim down the charts and graphs to the bare minimum. So what makes this workstation recipe for the fastest ANSYS Mechanical workstation so special? What is truly exciting enough to tip me over in my overstuffed black leather chair?

The Fastest Ever? Yup we have been Changed Forever

Not only is it the fastest ANSYS Mechanical workstation running on CUBE HVPC hardware.  It uses two INTEL CPU’s at 22 nanometers. Additionally, this is the first time that we have had an INTEL dual socket based workstation continue to gain faster times on and up to its maximum core count when solving in ANSYS Mechanical APDL.

Previously the fastest time was on the CUBE HVPC w16i-GPU workstation listed below. And it peaked at 14 cores. 

Unfortunately we only had time before we shipped the system off to gather two runs: 14 and 16 cores on the new machine. But you can see how fast that was in this table.  It was close to the previous system at 14 cores, but blew past it at 16 whereas the older system actually got clogged up and slowed down:

  Run Time (Sec)
Cores Used Config B Config C Config D
14 129.1 95.1 91.7
16 130.5 99 83.5

And here are the results as a bar graph for all the runs with this benchmark:


  We can’t wait to build one of these with more than one motherboard, maybe a 32 core system with infinband connecting the two. That should allow some very fast run times on some very, very large problems.

ANSYS V14sp-5 ANSYS R14 Benchmark Details

  • Elements : SOLID187, CONTA174, TARGE170
  • Nodes : 715,008
  • Materials : linear elastic
  • Nonlinearities : standard contact
  • Loading : rotational velocity
  • Other : coupling, symentric, matrix, sparse solver
  • Total DOF : 2.123 million
  • ANSYS 14.5.7

Here are the details and the data of the March 8, 2013 workstation:

Configuration C = CUBE HVPC w16i-GPU

  • CPU: 2x INTEL XEON e5-2690 (2.9GHz 8 core)
  • GPU: NVIDIA TESLA K20 Companion Processor
  • RAM: 128GB DDR3 1600Mhz ECC
  • HD RAID Controller: SMC LSI 2208 6Gbps
  • HDD: (os and apps): 160GB SATA III SSD
  • HDD: (working directory):6x 600GB SAS2 15k RPM 6Gbps
  • OS: Windows 7 Professional 64-bit, Linux 64-bit
  • Other: ANSYS R14.0.8 / ANSYS R14.5

Here are the details from the new, November 1, 2013 workstation:

Configuration D = CUBE HVPC w16i-k20x

  • CPU: 2x INTEL XEON e5-2687W V2 (3.4GHz)
  • GPU: NVIDIA TESLA K20X Companion Processor
  • RAM: 128GB DDR3 1600Mhz ECC
  • HDD: (os and apps): 4 x 240GB Enterprise Class Samsung SSD 6Gbps
  • OS: Windows 7 Professional 64-bit, Linux 64-bit
  • Other: ANSYS 14.5.7

You can view the output from the run on the newer box (Configuration D) here:

Here is a picture of the Configuration D machine with the info on its guts:


What is Inside that Chip:

The one (or two) CPU that rules them all:

Intel® Xeon® Processor E5-2687W v2

  • Status: Launched
  • Launch Date: Q3’13
  • Processor Number: E5-2687WV2
  • # of Cores: 8
  • # of Thread: 16
  • Clock Speed: 3.4 GHz
  • Max Turbo Frequency: 4 GHz
  • Cache:  25 MB
  • Intel® QPI Speed:  8 GT/s
  • # of QPI Link:  2
  • Instruction Se:  64-bit
  • Instruction Set Extension:  Intel® AVX
  • Embedded Options Available:  No
  • Lithography:  22 nm
  • Scalability:  2S Only
  • Max TDP:  150 W
  • VID Voltage Range:  0.65–1.30V
  • Recommended Customer Price:  BOX : $2112.00, TRAY: $2108.00

The GPU’s that just keep getting better and better:





Number and Type of GPU


Kepler GK110

Kepler GK110

Peak double precision floating point performance

515 Gflops

1.31 Tflops

1.17 Tflops

Peak single precision floating point performance

1.03 Tflops

3.95 Tflops

3.52 Tflops

Memory Bandwidth (ECC off)

144 GB/sec

250 GB/sec

208 GB/sec

Memory Size (GDDR5)




CUDA Cores





Ready to Try one Out?

If you are as impressed as we are, then it is time for you to try out this next iteration of the Intel chip, configured for simulation by PADT, on your problems.  There is no reason for you to be using a CAD box or a bloated web server as your HPC workstation for running ANSYS Mechanical and solving in ANSYS Mechanical APDL.  Give us a call, our team will take the time to understand the types of problems you run, the IT environment you run in, and custom configure the right system for you:,
or call 480.813.4884

Press Release: Free Thermal-Fluid Simulation Training Offered to Mark Growing Usage in the US and Demonstrate Advantages of Flownex Simulation Environment

987786-flownex_simulation_environment-11_12_13PADT is getting the word out about growing usage of the Flownex Simulation Environment in the US, and marking that growth with some free training in January. If our previous avalanche of marketing did not embed it in your memory, Flownex is a simulation tool used to model thermal-fluid systems.  PADT is the distributor for Flownex in the US and we really like this tool.  It is powerful, easy to use, and easily integrates with other tools like ANSYS, FLUENT, Excel, Matlab/Simulink, etc…

As part of a real marketing effort (I was being sarcastic about the avalanche), we have sent out the following press release:


We also created a new video that gives a brief introduction to Flownex. If you are still wondering what exactly Flownex is, this is a great place to start:

987786-flownex_multistage_compressor-11_12_13As is mentioned in the release, we are offering two free training classes as part of this effort.  These two day classes are a bit different than the standard Flownex introduction training in that they are more focused on giving you the skills you need to understand and try the Flownex out on your own – so a little more breadth and a little less depth.  After completing the class you will receive a 45 day licence. Our technical support team will also be available to help you as you try the tool out on your real problem.

The first class is being held in our Littleton, Colorado office on January 13 and 14, 2014 (REGISTER) and the second is at our main office in Tempe, Arizona on January 23 and 24, 2014 (REGISTER).   Space is limited so make sure you sign up early.

987786-flownex_powerplant_thermal-fluid-model-1-11_12_13We can honestly say that everyone that has seriously looked at Flownex has been pleased and has quickly learned that this tool is easy to learn, easy to use, and very capable.

987786-flownex_two_phase_flow-11_12_13Contact Roy Haynie ( to learn more.

Recommendations to Avoid ANSYS Mechanical Database Corruption

It’s late. The report for the project that you have spent over 140 hours on in the past two weeks is due in the morning. It is crunch time. Your computer resources are maxed out while you are running a final test scenario, post-processing another Workbench Mechanical module, and grabbing screenshots while you finish up your report formatting. Then, the unutterable occurs, ok, well maybe isn’t utter-able since I’m writing it, but, in short, your run is complete, you hit save, your computer locks up, you have to force quit, but you are sure that your save was successful. And it was…mostly.

Upon re-opening your project you find that all but one of your Mechanical databases are healthy and happy. But that one, the one that you needed a final image from, is corrupted. You know this because of the error messages that pop up with the slew of text that might look something like this:


Your frustration is building. You have already used results from that Mechanical model and reflected it in your report, so you do not want to lose it. I feel your pain.

Since this error message pin-points the SYS.mechdb file as the problem, it is unlikely that you can recover it. I know, not what you wanted to hear. But there is a chance that the database is not corrupt. To verify that, follow the steps Ted Harris outlined in a post he made earlier this year here.

If your Mechanical model is, indeed, corrupt and you were not able to recover it from steps outlined by Ted, make note of the following list of guidelines to help avoid database corruption in the future. I received this list of recommendations from ANSYS Inc. after one of our customers experienced a similar scenario as described above.

  1. Open your project from a Local mounted disk drive
  2. Do not work off of a network drive. It is OK to save to it after you are done
  3. Do not work off of a portable USB flash drive. It is OK to save to it after you are done
  4. Software backup programs can often lock a file and prevent WB from writing to the file
  5. Virus scan programs can also lock the file, and prevent WB from writing to the file
  6. Virus scan program can sometimes find a false positive in the file, and “disinfect” it, causing corruption
  7. Determine if the problem is related to the particular computer. ANSYS has seen bad memory or failing disk drives cause problems with saving files
  8. Use Windows Update regularly
  9. Update graphics drivers as needed

Bullet points 4, 5, and 6 are items that can possibly cause corruption while running, so be aware of the times they are generally run. In addition, ANSYS has recommended that disabling the Pre-Load of the Mechanical (and Meshing) editors can reduce the risk of database corruption. Here are the steps to do that:

  1. Reboot the computer (or Close/kill all AnsysFWW.exe and AnsysWBU.exe processes)
  2. Start a new instance of Workbench to change the settings:
    Tools > Options > Mechanical > Pre-Load the Mechanical Editor (disable)
    Tools > Options > Meshing > Pre-Load the Meshing Editor
  3. Exit Workbench
  4. Start a new instance of Workbench and work normally

As a disclaimer, even if you follow the above guidelines, there is still the chance of losing data. To avoid losing all of your data, follow the motto: save early, save often, and with backups! You can create backups by archiving your project as you make progress so that there is always a version to fall back on. Or, if you have the disk space to handle it, you can simply “Save As.” We hope following these recommendations will save you from headache down the road.

Making Charts and Tables in ANSYS Mechanical

imageOne of the nicer features in ANSYS Mechanical is the fact that when you enter in any type of tabular data, or look at any type of tabular results, you can view it as a table or as a graph.  But what if you want to make your own graph, maybe even viewing values from two different solutions?  ANSYS Mechanical has a little used feature called “New Chart and Table” that will allow you to make a table or a graph (chart) of quantities in your model tree that make sense when displayed as a graph or table: Time, loads applied over time, and results over time.


I have found myself exporting data to excel and making graphs all the time. And this is OK if you just do it once. But if you make a change to the model, you need to export again and redo your graph.  The Chart and Table function makes this an automatic step, right there in your model tree.

For this posting, we will just use a simple plasticity bending example. We hold the bottom of a round bar with a grove cut in the bottom part and push on the top with forces.

In its simplest form the “Chart and Table” duplicates what you see in the graph and Tabular Data windows when you click on a load or a result. Here is what you get when you click on a displacement:


And if you select the probe in the tree and click on the “New Chart and Table” icon you get:


No woop.  But even if I want to just plot one value, I can now customize the look of the graph a bit.  Take a look at the Details for the Chart:


With the Chart Controls you can define what is shown on the X axis; if you want lines, points or both with Plot Style, log or linear scale, and if you want horizontal, vertical, neither, or both gridlines.


This is what it looks like if I turn on both gridlines and use a log scale for the Y Axis.

Next, we can add axis labels with “Axis Labels:”


The “Report” Section tells the program what to do when a report is generated. By Default you get a table and a graph.  But you can do either, both, or you can suppress it in the report.  You can give the plot and/or table in the report a caption by filling in the Caption field.  It comes out nice:


Note that it actually includes a legend in the report. If you want the legend when you are looking at a graph interacively, just Right Mouse Button on the graph and choose “Show Legend” to turn it on:



Note that the legend shows the name of the branch in the tree. That is not very informative. So I change it to something useful and now the legend is useful:



So even with a basic graph, we can do a lot. But the real power is when you want to look at more. Let’s say I want to plot the force and the stress over time. I create a new chart with the icon then select the force and the stress results as my “Outline Selection”


I get a lot of stuff on my graph. That is because the program starts by plotting all the components for the load, and all max and min stress over time for the result. I simply change the ones I don’t want from “Display” to “Omit.”  Then I get:


Much more useful.  Note that it does not create two separate Y axis. Instead it normalizes the values between the min and max for each. This is not ideal, and hopefully in the future they will support multiple axis, but it still works for most cases when you want to compare things. Note that I renamed the branches in my tree so they show up in the legend correctly.  Next I will add some labels and turn on gridlines.


We have been neglecting the table. It also gets created:


As with any table in ANSYS Mechanical, it can be exported to Excel. So if you find yourself grabbing data from multiple input or result tables and pasted them into Excel, make a Chart and Table item to grab all the data you want in one place, then export it once.  To be honest, the quality of the graphs that are made are good enough for engineering, but maybe not good enough for a presentation. By making a Chart & Table of what you need, then exporting to Excel or some other graphing tool, you can still save a lot of time.

Next, let us look at plotting values from multiple simulations.  If you look at the tree, you will notice that the charts are a child of the model, not the simulations.  This signals that we can show data form the same model, but different simulations:


In our example I’ve simply made one with a tip force in the Y direction, and one with a tip force in the X direction. And I can show that by making a chart:


And I get a table:


HINT: If you want to make a single table or chart that shows all your input loads over time, in a single simulation or across multiple simulations, this is the way to do it.  If I add a third simulation where I vary the load in all three directions, I can capture all three cases in one table:


These examples show loads. Here is what it looks like if we review the deflection on the tip probe over time for two simulations:


Or mash it all up, and show stress and deflection for both cases:


In every case so far we have used time (Load Step for static) as our X axis. But you can put any value you want on the X axis.  Here is Force applied vs Tip Deflection:


Make sure you turn off Time and loads you don’t want to see.  This is a great way to plot hysteresis effects.

You may notice the plots in this posting are nice and big and have a good aspect ratio. And your screen looks like this:


Every window in ANSYS Mechanical can be dragged out of the frame and positioned/sized however you want. So I pull off the Graph window by itself and resize it to the aspect ratio I want. Now when I want to save the image all I have to do is select that window and hit Alt-Print Screen. The image is now stored in the clipboard and I can past it where I want.


To get the normal window configuration back, click View>Windows>Reset Layout.

As always, play with it to figure more out. I’ve included my simple test case in case you want to play with it first:

Video Tips: DesignXplorer – Single Objective Parameterization

This video gives an example of using DesignXplorer to automate the optimization of a tuning fork to achieve a particular desired frequency

ANSYS 14.5.7 Quick Install Instructions for Windows

ANSYS-14-5-7_Install-GuideAs part of our support for our existing ANSYS customers we prepare a quick install guide that we can send users when they get the new releases. The contents are based upon our own install experience, and what we have learned helping our customers install on a large number of different networks.  We just updated the 14.5.7 guide for Windows and thought we would share it with the community.

This particular release is fairly straight forward, so this is a short doc.

We hope you find it useful:

ANSYS 14 5 7_Install_Quick_Guide

/HBC: One of Those Little Known Commands

The other day we received a tech support call requesting a way to remove the space between the element faces on a pressure plot.  He wanted this so that he could get a contour plot without seeing the contours of the elements on the back side of the part. So I built my trusty test block and applied a pressure. By turning on the pressure load symbols with  the /PSF command, also under PlotCrtls > Symbols, you can get plots like this.


Face Outlines (/PSF,1,1)



Arrows (/PSF,1,2)



Contours (/PSF,1,3)

Of course the customer was using this last contour plot option, but as you can see below, if you have pressure on both sides of the model, then the backside pressures show through the gaps. The plot can get a bit confusing. So after some digging, starting with the /PSF command, and not finding any reference on how to change the plot behavior, I asked around if anyone else had a way to do it, other than my first inclination which was to write a macro. So as I reverted to creating a macro, to do what should be a simple task, I thought, “No, there HAS to be an easier way.” Of course there is.


The one thing I’ve learned over the years… Well, yes, I’ve learned more than ONE thing, but I’m trying to make a point here… The one thing I’ve learned over the years, is that no matter how much I learn, there is always someone who know more than me.  So I asked Sheldon! (Not the Sheldon on Big Bang Theory; ANSYS, Inc’s very own Sheldon Imaoka.) I thought, “Surely he will know some undocumented command to save me time.  It took him all of three minutes to get back to me with the /HBC command. It is a fully documented, but seldom used, command that is hidden in the recesses of the Command Reference that determines how boundary condition symbols are displayed. When turned on, it will “use an improved pressure contour display.” So you go from the picture on the top, to the picture on the bottom.


So I learned two new things. One is the /HBC command can give you nicer looking plots. The other, and even more useful thing, is to click the links on the help page at the upper right corner.


For if I did, I would have found the /HBC command on my own.


It looks like I need to sit down with a nice cup of hot chocolate* and the Command Reference and just scan the listing for commands that I don’t recognize and learn what they do.  Oh, what I go through for you people. Well, I’ll just make sure that it’s really good hot chocolate*.   I’ll write a new post from time to time on cool commands I find useful.

Have a great day!!!

*It’s 85 degrees here this week and I really meant iced tea, but I didn’t want to rub it in. Smile

Part 2: ANSYS FLUENT Performance Comparison: AMD Opteron vs. Intel XEON

AMD Opteron 6308, INTEL XEON e5-2690 & INTEL XEON e5-2667V2 Comparison using ANSYS FLUENT 14.5.7

Note: The information and data contained in this article was complied and generated on September 12, 2013 by PADT, Inc. on CUBE HVPC hardware using FLUEN 14.5.7.  Please remember that hardware and software change with new releases and you should always try to run your own benchmarks, on your own typical problems, to understand how performance will impact you.

By David Mastel

Due to the response to the original article on this subject,  I thought it would be good to do a quick follow-up using one of our latest CUBE HVPC builds. Again, the ANSYS Fluent standard benchmarks were used in garnering the stats on this dual socket INTEL XEON e5-2667V2 configuration.

CUBE HVPC Test configurations (Same as in last comparison)

  • Server 1: CUBE HVPC c16
  • CPU: 4, AMD Opteron 6308 @ 3.5GHz (Quad Core)
  • Memory: 256GB (32x8G) DDR3-1600 ECC Reg. RAM (1600MHz)
  • Hardware RAID Controller: Supermicro AOC-S2208L-H8iR 6Gbps, PCI-e x 8 Gen3
  • Hard Drives: Supermicro HDD-A0600-HUS156060VLS60 – Hitachi 600G SAS2.0 15K RPM 3.5″
  •  OS: Linux 64-bit / Kernel 2.6.32-358.18.1.e16.x86_64
  • App: ANSYS FLUENT 14.5.7
  • MPI: Platform MPI
  • HCA: SMC AOC-UIBQ-M2 – QDR Infiniband
    • The IB card installed however solves were run distributed locally
  • Switch: MELLANOX IS5023 Non-Blocking 18-port switch

Server 2: CUBE HVPC c16i (Intel server from last comparison)

  • CPU: 2, INTEL XEON e5-2690 @ 2.9GHz (Octa Core)
  • Memory: 128GB (16x8G) DDR3-1600 ECC Reg. RAM (1600MHz)
  • RAID Controller: Supermicro AOC-S2208L-H8iR 6Gbps, PCI-e x 8 Gen3
  • Hard Drives: Supermicro HDD-A0600-HUS156060VLS60 – Hitachi 600G SAS2.0 15K RPM 3.5″
  • OS: Windows 7 Professional 64-bit
  • App: ANSYS FLUENT 14.5.7
  • MPI: Platform MPI

Server 3: CUBE HVPC c16ivy (New “Ivy” based Intel server)

  • CPU: 2, INTEL XEON e5-2667V2 @ 3.3 (Octa Core)
  • Memory: 128GB (16x8G) DDR3-1600 ECC Reg. RAM (1600MHz)
  • RAID Controller: Supermicro AOC-S2208L-H8iR 6Gbps, PCI-e x 8 Gen3
  • Hard Drives: Supermicro HDD-A0600-HUS156060VLS60 – Hitachi 600G SAS2.0 15K RPM 3.5″
  • OS: Linux 64-bit / Kernel 2.6.32-358.18.1.e16.x86_64
  • App: ANSYS FLUENT 14.5.7
  • MPI: Platform MPI
  • HCA: SMC – QDR Infiniband
    • The IB card installed however solves were run distributed locally

ANSYS FLUENT 14.5.7 Performance using the ANSYS FLUENT Benchmark suite provided by ANSYS, Inc.

ANSYS Fluent Benchmark page link:

Release ANSYS FLUENT 14.5.7 Test Cases
(20 Iterations each)

  • Reacting Flow with Eddy Dissipation Model (eddy_417k)
  • Single-stage Turbomachinery Flow (turbo_500k)
  • External Flow Over an Aircraft Wing (aircraft_2m)
  • External Flow Over a Passenger Sedan (sedan_4m)
  • External Flow Over a Truck Body with a Polyhedral Mesh (truck_poly_14m)
  • External Flow Over a Truck Body 14m (truck_14m)

Here are the results for all three machines, total and average time:



Summary: Are you sure? Part 2

So I didn’t have to have the “Are you sure?” question with Eric this time and I didn’t bother triple checking the results because indeed, the Ivy Bridge-EP Socket 2011 is one fast CPU! That combined with a 0.022 micron manufacturing process  the data speaks for itself. For example, lets re-dig into the data for the External Flow Over a Truck Body with a Polyhedral Mesh (truck_poly_14m) benchmark and see what we find:

























Current Pricing of INTEL® and AMD® CPU’s

Here is the up to the minute pricing for each CPU’s. I took these prices off of NewEgg and IngramMicro’s website. The date of the monetary values was captured on October 4, 2013.

Note AMD’s price per CPU went up and the INTEL XEON e5-2690 went down. Again, these prices based on today’s pricing, October 4, 2013.

AMD Opteron 6308 Abu Dhabi 3.5GHz 4MB L2 Cache 16MB L3 Cache Socket G34 115W Quad-Core Server Processor OS6308WKT4GHKWOF

  •  $501 x 4 = $2004.00

Intel Xeon E5-2690 2.90 GHz Processor – Socket LGA-2011, L2 Cache 2MB, L3 Cache 20 MB, 8 GT/s QPI

  • $1986.48 x 2 = $3972.96

Intel Xeon E5-2667V2 3.3 GHz Processor – Socket LGA-2011, L2 Cache 2MB, L3 Cache 25 MB, 8 GT/s QPI,

  • $1933.88 x 2 = $3867.76


INTEL XEON e5-2667V2

INTEL XEON e5-2690

AMD Opteron 6308


PADT offers a line of high performance computing (HPC) systems specifically designed for CFD and FEA number crunching aimed at a balance between cost and performance. We call this concept High Value Performance Computing, or HVPC. These systems have allowed PADT and our customers to carry out larger simulations, with greater accuracy, in less time, at a lower cost than name-brand solutions. This leaves you more cash to buy more hardware or software.

Let CUBE HVPC by PADT, Inc. quote you a configuration today!


Quick Tip: Tool for Putting LaTex Equations in PowerPoint

I’m not a big fan of LaTex. That is probably more a reflection of the fact that I don’t have an advanced degree and didn’t have to write a dissertation than anything else.  But one thing that is worse than LaTex is the equation editor in PowerPoint. If you are like us, you use PowerPoint as your primary reporting tool and dread putting equations in.

Matt Sutton was doing just that the other day and thought “there has got to be a better way!”  He found one. A tool called IguanaTex.

There is not much to it, it is free, and it works well.

Iguana Tex screen shot

Example of IguanaTex output

Here are some equations from a presentation Matt just did:





Making APDL Parameters Available in the ANSYS Parameter Manager or DesignXplorer: Prep, Solve, and Post

This is one of those questions that comes up every once in a while that is not so obvious at first glance, but that is simple once you understand how ANSYS Mechanical interacts with ANSYS Mechanical APDL.  After a couple of email exchanges around a tech support question, we thought it would be good to share with everyone.

Before we get started, if you need a refresher on Command Objects in ANSYS Mechanical, the way in which you send APDL commands to the ANSYS Mechanical APDL solver, here is a seminar from a couple of years ago that covers the whole deal:

The basic problem is this: you have an APDL script you execute as a command object that does some sort of model interrogation or stores the result of some calculation, and you want to use that parameter in the parameter manager or in DesignXplorer.  If you look at the details view for a command object you will notice that it only supports input parameters: ARG1-ARG9.


If you look at the example (silly) macro you will see that it:

  1. Grabs component (named selection) END1
  2. Figures out how many nodes are attached to END1 (NMND)
  3. Takes ARG1 as the total load applied load
  4. Calculates the per node load by dividing the total load by the number of loads.
  5. Applies that per node load
  6. Reselects all the nodes

If I want to know how many nodes I put the load on and what the per node load is I’m kind of stuck here.  Any command object you add to the tree above the Solution branch only allows input parameters.

But a command snippet applied in the Solution branch is different, it allows you to pull parameters back and share them through the parameter manager.

When you first insert a command object you only get input parameters (ARG1-ARG9) as usual, and an empty section called “Results”


The way you get result parameters, or what I think should be called “Output Parameters” is you create a parameter in the command object’s APDL script that starts with “my_”  When you click outside the text input window the program parses you script and if it finds any “my_” parameters in the text, it sticks them in the Results section:


Note, the default is “my_” but you can change it n the “Output Search Prefix” line in the Definition block.

Initially they will show up pinkish because the model has not been run and they are not defined. Click on the box to make them parameters that get passed outside of the program and then run:


If you pop back out to the project view you will see that we now have a Parameter Set bar with both input and output parameters:


And if you open the parameter manager up you can see the input and output parameters:


This works because all ANSYS mechanical is doing is making one big batch input file for ANSYS MAPDL.  That file contains any command objects you insert into the tree and extracts any parameters that you tagged in a post processing command object for return to ANSYS Mechanical.

ANSYS FLUENT Performance Comparison: AMD Opteron vs. Intel XEON

AMD Opteron 6308 & INTEL XEON e5-2690 Comparison using ANSYS FLUENT 14.5.7

Note: The information and data contained in this article was complied and generated on September 12, 2013 by PADT, Inc. on CUBE HVPC hardware using FLUEN 14.5.7.  Please remember that hardware and software change with new releases and you should always try to run your own benchmarks, on your own typical problems, to understand how performance will impact you.

A potential customer of ours was interested in a CUBE HVPC mini-cluster. They requested that I run benchmarks and garner some data on a two CPU’s. The CPU’s were benchmarked on two of our CUBE HVPC systems. One mini-cluster has dual INTEL® XEON e5-2690 CPU’s and another mini-cluster has quad AMD® Opteron 8308 CPU’s. The benchmarking was only run on a single server using a total of 16 cores on each machine. The same DDR3-1600 ECC Reg RAM, Supermicro LSI 2208 RAID Controller and Hitachi SAS2 15k RPM hard drives were used on each system.


CUBE HVPC Test configurations:

Server 1: CUBE HVPC c16
  • CPU: 4, AMD Opteron 6308 @ 3.5GHz (Quad Core)
  • Memory: 256GB (32x8G) DDR3-1600 ECC Reg. RAM (1600MHz)
  • Hardware RAID Controller: Supermicro AOC-S2208L-H8iR 6Gbps, PCI-e x 8 Gen3
  • Hard Drives: Supermicro HDD-A0600-HUS156060VLS60 – Hitachi 600G SAS2.0 15K RPM 3.5″
  • OS: Linux 64-bit / Kernel 2.6.32-358.18.1.e16.x86_64
  • App: ANSYS FLUENT 14.5.7
  • MPI: Platform MPI
  • HCA: SMC AOC-UIBQ-M2 – QDR Infiniband
    • The IB card installed however solves were run distributed locally
  • Stack: RDMA 3.6-1.el6
  • Switch: MELLANOX IS5023 Non-Blocking 18-port switch
Server 2: CUBE HVPC c16i
  • CPU: 2, INTEL XEON e5-2690 @ 2.9GHz (Octa Core)
  • Memory: 128GB (16x8G) DDR3-1600 ECC Reg. RAM (1600MHz)
  • RAID Controller: Supermicro AOC-S2208L-H8iR 6Gbps, PCI-e x 8 Gen3
  • Hard Drives: Supermicro HDD-A0600-HUS156060VLS60 – Hitachi 600G SAS2.0 15K RPM 3.5″
  • OS: Windows 7 Professional 64-bit
  • App: ANSYS FLUENT 14.5.7
  • MPI: Platform MPI

ANSYS FLUENT 14.6.7 Performance using the ANSYS FLUENT Benchmark suite provided by ANSYS, Inc.

The models we used can be downloaded from the ANSYS Fluent Benchmark page link:

Release ANSYS FLUENT 14.5.7 Test Cases  (20 Iterations each):
  • Reacting Flow with Eddy Dissipation Model (eddy_417k)
  • Single-stage Turbomachinery Flow (turbo_500k)
  • External Flow Over an Aircraft Wing (aircraft_2m)
  • External Flow Over a Passenger Sedan (sedan_4m)
  • External Flow Over a Truck Body with a Polyhedral Mesh (truck_poly_14m)
  • External Flow Over a Truck Body 14m (truck_14m)
Chart 1: Total Wall Clock Time in seconds: (smaller bar is better)


Chart 2: Average wall-clock time per iteration in seconds: (smaller bar is better)




Are you sure?

That was the question Eric proposed to me after he reviewed the data and read this blog article before posting. I told him “yes I am sure data is data, and I even triple checked.” I basically re-ran several of the benchmarks to see if the solve times came out the same on these two CUBE HVPC workstations. I went on to tell Eric , “For example, lets dig into the data for the External Flow Over a Truck Body with a Polyhedral Mesh (truck_poly_14m) benchmark and see what we find.”

Quad socket Supermicro motherboard

4 x 4c AMD Opteron 6308 @3.5GHz

Dual socket Supermicro motherboard

2 x 8c INTEL e5-2690 @2.9GHz

clip_image002[1] clip_image004[1]

The INTEL XEON e5-2690 INTEL CPU dual socket motherboard is impressive; it may have been on the Top500 list of some of the fastest computers in the world ten years ago. Anyways, so after each solve I captured the solve data and as you can see below. The AMD Opteron wall clock time was faster than the INTEL XEON wall clock time.

So why did the AMD Opteron 6308 CPU pull away from the INTEL for the ANSYS FLUENT solve times? Lets take a look at couple of reasons why this happened. I will let you make your own conclusions.

  • Clock Speed, but would a 10.4GHz difference in total CPU speed make a 100% speedup in ANSYS Fluent wall-clock times?
  • Theoretical total of:
  • AMD® OPTERON 6308 = 16 x 3.5GHz = 56.0 GH
  • INTEL® XEON e5-2690 = 16 x 2.9GHz – 46.4 GHz
  • The floating point argument? The tic and tock of the great CPU saga continues.
  • At this moment in eternity, it is a known fact that the AMD Opteron 6308 and many of its brothers, have one floating point unit per two integer cores. INTEL has one integer core per one floating point core. However what this means to ANSYS CFD users in my MIS/IT simpleton terms is the AMD CPU was simply able to handle and process more data in this example.
  • It’s possible that there were more integer calculations required than floating point? If that is the case then the AMD CPU would have had eight pipelines for integer calculations. The AMD Opteron is able to process four floating point pipelines. While the INTEL CPU can process eight floating point pipelines.

Let us look at the details of what is on the motherboards as well.  4 data paths vs 2 can make a difference:

Dual socket Supermicro motherboard

2 x 8c INTEL e5-2690 @2.9GHz

Quad socket Supermicro motherboard

4 x 4c AMD Opteron 6308 @3.5GHz

Processor Technology 32-Naometer 32-Naometer SOI (silicon-on-insulator) technology
HyperTransport™ Technology Links

Quick Path Interconnect Links

Two links at up to 8GT/s per link up to 16 GB/s direction peak bandwidth per port Four x16 links at up to 6.4GT/s per link
Memory Integrated DDR3 memory controller – Up to 51.2 GB/s memory bandwidth per socket
Number of Channels and Types of Memory Four links at up to 51.2GB/s per link Four x16 links at up to 6.4GT/s per link
Number of Channels and Types of Memory Quad channel support Quad channel support
Packaging LGA2011-0 Socket G34 – 1944-pin organic Land Grid Array (LGA)
Current pricing of the CPU’s

Here is the up to the minute pricing for each CPU’s. I took these prices off of NewEgg and IngramMicro’s website. The date of the monetary values was captured on September 12, 2013.

  • AMD Opteron 6308 Abu Dhabi 3.5GHz 4MB L2 Cache 16MB L3 Cache Socket G34 115W Quad-Core Server Processor OS6308WKT4GHKWOF
    • $499.99 x 4 = $1999.96
  • Intel Xeon E5-2690 2.90 GHz Processor – Socket LGA-2011, L2 Cache 2MB, L3 Cache 20 MB, 8 GT/s QPI,
    • $2010.02 x 2 = $4020.40


PADT offers a line of high performance computing (HPC) systems specifically designed for CFD and FEA number crunching aimed at a balance between cost and performance. We call this concept High Value Performance Computing, or HVPC. These systems have allowed PADT and our customers to carry out larger simulations, with greater accuracy, in less time, at a lower cost than name-brand solutions. This leaves you more cash to buy more hardware or software.

Let CUBE HVPC by PADT, Inc. quote you a configuration today!

Efficient Engineering Data, Part 1: Creating and Importing Material Properties in Workbench

Note: This is part 1 of a two-part series in Engineering Data customization and default settings. This article essentially serves as a foundation for my next one, which will cover how to set up default material choices and assignments in Workbench.

As you’ve probably noticed, the Workbench installation comes with an extensive set of material libraries. If you haven’t noticed, then open a Workbench session, go into Engineering Data, and click that button on the upper right that looks like a stack of books: image

Click on one of the libraries, say, General Materials, and take a look at the selection of materials.


So you see things like Stainless Steel, Aluminum Alloy, Titanium Alloy, etc. but which alloys exactly? 301 1/2-hard steel? 17-4PH? 6061-T6 aluminum? Or cast C355? Titanium 6-4? Or 6-2-4-2? Obviously you’re going to have your own material properties in mind, and you’ll probably use them frequently enough to where you’d like to have them readily accessible. Maybe store them in a library, or something.

As it turns out, you’re not confined to the libraries ANSYS provides with the Workbench installation. You can create your own libraries too. To start this off, first click in the first blank line in the top Engineering Data Sources section, where it says, “Click here to add a new library” (seems pretty straight-forward, doesn’t it?) and type a unique name for the library. I’ll call mine “Jeff’s Materials” because I’m incredibly original that way. Then hit Enter.



You’ll be prompted for a location and xml file name for the library. Specify these and click Save. All of your material names and properties will be stored in this file.


Notice that the new library is checked. That means it is unlocked and able to be edited.


At that point you can add material names, insert properties from the left side Toolbox, etc.


Type in some material names


Then define their properties

Once you’re finished adding and editing materials, uncheck the column B box of the library to lock it up. Click Yes to accept changes. If you want to add or edit materials to your library at a later date, simply unlock it by checking the column B check box again.


Now, let’s say you want to share your awesome material library with your co-workers, or maybe you’ve installed a new version of ANSYS and you want to include it, or maybe your library was deleted by gnomes during the night. How do you bring it back into Workbench? Simple. First make sure the xml file is available (you’ll want to email it to your co-workers and have them save it to their disks if you’re sharing it with them). Toggle the libraries on by clicking on the stack of books button. Then simply click the little ellipsis button on the “Click here to add a new library” line.


Browse to the appropriate xml file and open it.


And now you have your library back.


I was too lazy to define all the materials for this article, hence the question marks

This is all well and good, but wouldn’t it be nice if we could change the materials that are immediately available in Engineering Data upon opening Workbench, and set the default material assignment to something besides Structural Steel? As it turns out, you can do both of these, and I’ll show you how in the next installment of Efficient Engineering Data.