Why do my ANSYS jobs take days and weeks to finish? Well it depends…

Real World Lessons on How to Minimize Run Time for ANSYS HPC

Recently I had a VP of Engineering start a phone conversation with me that went something like this. “Well Dave, you see this is how it is. We just spent a truckload of money on a 256 core cluster and our solve times are slower now than with our previous 128 core cluster. What the *&(( is going on here?!”

I imagine many of us have heard similar stories or even received the same questions from our co-workers, CEO’s & Directors. I immediately had my concerns and I truly thought carefully as to what I should say next. I recalled a conversation I had with one of my college professors. He had told me that when I find myself stepping into gray areas that a good start to the conversation is to say. “Well it depends…”

Guess what, that is exactly what I said. I said “Well it depends…” followed by going into explaining to him two fundamental pillars of computer science that have plagued most of us since computers were created: I said “Well you may be, CPU bound (compute bound) or I/O bound. He told me that they had paid a premium for the best CPU’s on the market and some other details about the HPC cluster. Garnering some of other details about the cluster my hunch was that his HPC cluster may actually be I/O bound.

I/O Bound

Basically this means that your cluster’s $2,000 worth of CPU’s are basically stalled out and sitting idle. The CPU’s are waiting for new data to process and move on. I also briefly explained that his HPC cluster may be compute bound. I quickly reassured him that the likelihood of his HPC cluster being compute bound was about 10% possible and very unlikely. I knew the specifications on the CPU’s in this HPC cluster and the likelihood that they were the issue of his ANSYS slow run times was low on my radar. These literally were the latest and greatest CPU’s ever to hit this planet (at that moment in time). So, let me step back a minute, to refresh our memories on what it means when a system is compute bound.

Compute Bound

Being compute bound means that the HPC cluster’s CPU’s were sitting at 99 or 100% for long periods of time. When this happens very bad things begin to happen to your HPC cluster. CPU requests to peripherals are delayed or infinitely lost to the ether. The HPC cluster may become unresponsive and even lock up.

All I could hear was silence on the other end. “Dave, I get it, I understand, please find the problem and fix our HPC cluster for us. ” I happily agreed to help out! I concluded our phone conversation asking that he send me the specific details, down to the nuts and bolts of the hardware! I also requested operating system and software that was installed and used on the 256 core HPC cluster.

What NOT to do when configuring an ANSYS Distributed HPC cluster.

Seeking that perfect balance!

After a quick NDA signing, a few dollars exchange and a sprinkle of some other legal things that lawyers get excited about. I set out to discover the cause. After reviewing the information provided to me I almost immediately saw three concerns:

To interconnect what?

Let Merriam-Webster describe it:

Definition of INTERCONNECT

transitive verb
: to connect with one another
intransitive verb
: to be or become mutually connected

— in·ter·con·nec·tion noun
— in·ter·con·nec·tiv·i·ty noun

1. The systems are interconnected with a series of wires.
2. The lessons are designed to show students how the two subjects interconnect
3.  A series of interconnecting stories

First Known Use of INTERCONNECT: 1865

Concern numeral Uno!!! Interconnect me

Though the company’s 256 core HPC cluster had a second dedicated GigE interconnect. Distributed ANSYS is highly bandwidth and latency bound often requiring more bandwidth than a dedicated NIC (Network Interface Card) may provide. Yes, the dedicated second GigE card interconnect was much better than trying to use a single NIC for all of the network traffic which would also include the CPU interconnect. I did have a few of the MAPDL output files from the customer that I could take a peek at. After reviewing the customer output files it became fairly clear that interconnect communication speeds between the 16 core x 16 server in the cluster was not adequate. The master Message Parsing Interface (MPI) process that Distributed ANSYS uses requires a high amount of bandwidth and low latency for proper distributed scaling to the other processes. Theoretically the data bandwidth between cores solving local to the machine will be higher than the bandwidth traveling across the various interconnect methods (see below). ANSYS, Inc. recommends Infiniband for CPU interconnect traffic. Here are a couple of reasons why they recommend this. See how the theoretical data limits increase going from Gigabit Ethernet up to FDR Infiniband.

Theoretical lane bandwidth limits for:

  • Gigabit Ethernet (GigE): ~128MB/s
  • Signal Data Rate (SDR): ~ 328 MB/s
  • Double Data Rate (DDR): ~640 MB/s
  • Quad Data Rate (QDR): ~1,280 MB/s
  • Fourteen Data Rate (FRD): ~1,800 MB/s

GEEK CRED: A few years ago companies such as MELLANOX started aggregating the Infiniband channels. The typical aggregate modifiers are 4X or even a 12X increase. So for example the 4X QDR Infiniband switch and cards I use at PADT and recommended to this customer, would have a (4X 10Gbit/s) or 5,120 MB/s of throughput! Here is a quick video that I made of a MELLANOX IS5023 18-port 4X QDR full bi-directional switch in action:

This is how you do it with a CUBE HVPC! MAPDL output file from our CUBE HVPC w16i-GPU workstation. This is running the ANSYS industry benchmark V14sp-5. I wanted to show the communication speeds between the master MPI process and the other solver processes to see just how fast the solvers can communicate. With a peak communication speed of 9593 MB/s this CUBE HVPC workstation rocks!

Chassis Profile 4u standard depth or rackmountable
CPU 1 x One Dual Socket
Chipset INTEL 602 Chipset
Processors 2 x INTEL e5-2690 @ 2.9GHz
Cores 2 x 8
Memory 128GB DDR3-1600 ECC Reg RAM
OS Drives 2 x 2.5″ SATA III 256GB SSD Drives RAID 0
DATA/HOME Hard Disk Drives 4 x 3.5″ SAS2 600GB 15kRPM drives RAID 0
SAS RAID (Onboard, Optional) RAID 0 (OS RAID)
SAS RAID (RAID card, Optional) LSI 2208 (DATA VOL RAID)
Networking (Onboard) Dual GigE (Intel i350)
Video (Onboard) NVIDIA QUADRO K5000
GPU (Optional) NVIDIA TESLA K2000
Operating System Windows 7 Professional 64-bit
Optional Installed Software ANSYS 14.5 Release

imageStats for CUBE HVPC Model Number : w16i-KGPU

Learn more about this and other CUBE HVPC systems here.

Concern #2: Using RAID 5 Array for Solving Disk Volume

The hard drives that are used for I/O during a solve, the solving volume, were configured in a RAID 5 hard disk array. Some sample data below showing the minimum write speed of a similar RAID 5 array. These are speeds that are better off seen in your long-term storage volume not on your solving/working directory.

Qty / Type / Size / RAID Qty 8 x 3.5″ SAS2 15k 600GB RAID 5
TEST # p1
min Read 204 MB/s
max Read 395  MB/s
Avg Read N/A
min Write 106 MB/s
max Write 243.5 MB/s
Avg Write N/A
Access Time N/A

Concern #3: Using RAID 1 for Operating System

The hard drive array for the OS was configured in a RAID 1 configuration. For a number cruncher server having RAID 1 is not necessary. If you absolutely have to have RAID 1. Please spend the extra money and go to a RAID 10 configuration.

I really don’t want to get into the seemingly infinite details of hard drives speeds, latency. Or even begin to explain to you if I should be using an onboard RAID Controller, dedicated RAID controller or a software RAID configuration completed within the OS. There is so much information available on the web that a person gets overloaded. When it comes to Distributed ANSYS, think fast hard drives and fast RAID controllers. Start researching your hard drives and RAID controllers using the list provided below. Again, only as a suggestion! I have listed the drives in order based on a very scientific and nerdy method. If I saw a pile of hard drives, what hard drive would I reach for first?

  1. I prefer using SEAGATE SAVVIO or HITACHI enterprise class drives. (Serial Attached SCSI) SAS2 6Gbit/s 3.5”15,000 RPM spindle drives (best bang for your dollar of space, more read & write heads over a 2.5” spindle hard drive).
  2. I prefer using Micron or INTEL SSD enterprise class SSD. SATA III Solid State Drive 6 Gbit/s (SSD sizes have increased however you will need more of these for an effective solving array and they still are not cheap).
  3. I prefer using the SEAGATE SAVVIO 2.5” enterprise class spindle drives. SAS2 6Gbit/s 2.5” 15,000 RPM spindle drives (if you need a small form factor, fast and additional storage. But the 2.5” drives do not have as many read & write heads as a 3.5” drive. In a situation where I need to slam 4 or 8 drives into a tight location.
    Right now, SEAGATE SAVVIO 2.5” are the way to go!  Here is a link to a data sheet.
    Another similar option is the HITACHI ULTRASTAR 15k600.  It’s spec sheet is here.
  4. SATA II 3Gbit/s 3.5” 7,200 RPM spindle drives are also a good option.  I prefer Western Digital RE4 1TB or 2TB drives. There spec sheet is here.

LSI 2108 RAID Controller and Hard Drive data/details:


How a CUBE HVPC System from PADT, Inc. balanced out this configuration and how much would it cost?

I quoted out the below items, installed and out the door (including my travel expenses, etc.) at: $30,601

The company ended up going with their own preferred hardware vendor. Understandable, one good thing is that we are now on the preferred purchasing supplier list. They were greatly appreciative of my consulting time and indicated that they will request a “must have” quote for a CUBE HVPC system the next refresh in a year. They want to go over 1,000 cores the next refresh.

I recommended that they install the following into the HPC cluster based: (note they already had blazing fast hard drives)

  • 16 – Supermicro AOC-S2208L-H8iR LSI 2208 RAID controller cards.
  • 32 – Supermicro CBL-0294L-01 cabling to connect the LSI RAID cards to the SAS2 hard drives.
  • 1 – MELLANOX IS5023 18-port 4X QDR Infiniband switch
  • 16 – Supermicro AOC-UIBQ-M2 Dual port 4X QDR Infiniband card
  • 16 – Supermicro QSFP Infiniband cables in a couple different lengths

A special thanks and shout out to Sheldon Imaoka of ANSYS, Inc. for inspiring me to write this blog article!

Video Tips: Automatic Cross-Sections in ANSYS DesignModeler

A quick video showing how to create a script that will automatically generate standard cross-sections in DesignModeler to be used for, as an example, line bodies.

ANSYS Help Link:  // DesignModeler User Guide // Scripting API // Script Features // Features within Script Features // Cross Section Feature

Ready to go for Turbo Expo 2013 in San Antonio

booth2The PADT and Flownex teams have our booth set up and ready to go for the next three days at Turbo Expo 2013.

This is always one of our favorite events because most of us came from this industry, and in fact all four of the founders were turbine-engine-engineers before we started PADT.  A special part of this years event is that we are introducing Flownex to the North American Turbo community as well as our CUBE HVPC computer systems.  So lots of new things to talk about along with our established offerings of ANSYS, Inc software consulting, customization, and training.

If you are there, please make sure you stop by our booth. We would love to see you and chat.


Here is our press release on the event:



Trusted Partners for Turbomachinery Simulation


The ASME Turbo Expo is the industry show where PADT feels at home the most.  Founded by experienced turbine engine simulation, design, and manufacturing engineers, the company has a true understanding of the real world needs of those who are focused on simulation for Turbomachinery.

Our primary focus for this year’s conference will be the full introduction of the Flownex Simulation Environment to North America.  This thermal-fluid system simulation tool started life as a solver for combustor analysis, and has grown up to be a full featured toolset that can model any fluid-thermal network in your engine or pump.  Flownex is ideal simulation software for the quick thermo-fluid analysis of gas turbine performance.

It provides aircraft engine design and system engineers with the ability to simulate complicated air and gas flow patterns through fans, compressors and turbines; match compressor and turbine power and compile maps; calculate thrust, shaft power, combustion calculations with convection, conduction and radiation heat transfer; and determine fuel consumption.  If you are using an in-house tool or software written for other applications to model your flow networks, please come by to see how Flownex can reduce the amount of time you spend modeling your systems while increasing the fidelity of your models.s grown up to be a full featured toolset that can model any fluid-thermal network in your engine or pump.  Flownex is ideal simulation software for the quick thermo-fluid analysis of gas turbine performance.


PADT’s reputation in the Turbomachinery industry is built on our expertise selling, using, supporting, and customizing the complete suite of ANSYS FEA and CFD.  Turbo companies come to us for training on ANSYS software, customization of analysis tools, FEA and CFD outsourcing, and HPC hardware because they know we know their business and how to maximize the return on their investment in simulation.  We can help anyone doing simulation on Turbomachinery in a variety of ways, stop on by to find out how.


Another new area the PADT provides this type of help to turbo companies is by offering a complete line of High Value Performance Computer systems specifically designed for the simulation user.  From workstations to large clusters, PADT can custom design a system that hits the sweet spot between cost and performance, delivering faster turnaround of CFD and FEA runs for considerably less than systems offered by general purpose computer suppliers.

Stop by our booth to look at the hardware, software, training, and consulting that we offer to companies around the world to help them make their studies more efficient and effective.

Video Tips: Automatic Contact Generation in ANSYS Mechanical

A quick video showcasing the automatic contact generation feature in ANSYS Mechanical.  This feature automatically selects the faces that are in contact or are close to contact and assigns a contact definition.

Video Tips: Importing SolidWorks Geometry into ANSYS Mechanical

TheFocus-Video-Tips-2We are pleased to introduce a new feature in The Focus blog, video posts.  With this entry we are putting up our first “The Focus Video Tips, Examples, and Demonstrations”  Sometimes a video just works better, especially when showing how to do something in a Graphical User Interface.

So we have put some basic infrastructure in place and that lets us quickly record something on one of our computers, stick a title and end slide on it, and then upload to YouTube.

In this first entry, we show how easy it is to read in geometry from SolidWorks to ANSYS Mechanical.

CFX Expression Language–Part 5: Using CEL for Solution Monitoring

In four previous entries we introduced CFX Expression Language, CEL:

After a break to teach some ANSYS classes in beautiful northern Utah we’re back to conclude our series on CFX Expression Language.


In our fifth and final installment on CEL we will discuss the use of CEL in monitoring items of interest while the CFX solution is progressing. Back in the first installment in this series, we showed how to create expressions for results quantities in CFD Post. By creating expressions in similar fashion for results quantities in CFX Pre, we can use these expressions to monitor items during solution.

Here is an example. In CFX Pre we have defined three expressions which are really extracting and operating on results data.

forceX1 = resulting force on one face of the structure
forceX2 =resulting force on another face of the structure
fdiffx = the difference between these two values or the net force acting on the structure

This shows these three expressions in the CFX Pre outline tree:


Still within CFX Pre, click on Insert > Solver > Output Control. That will add an Output Control tab on the left side of the CFX Pre window. Click on the Monitor tab, expand Monitor Points and Expressions, and then click on the button near the right of the window below that to add a monitor point:


Set the Option to Expression and in the Expression Value box left click then right click to select from your list of defined expressions:


The CFX Pre tree will now have one or more Monitor Points listed under Output Control:


When we initiate the solution, these user-defined monitor points will be available for real time display in addition to the normal Momentum and Mass and Turbulent kinetic energy monitors. All we need to do is click on the User Points tab during solution to view our expressions as monitors. In the example shown below, Monitor 1 is forceX1, Monitor 2 is forceX2, and Monitor 3 is fdiffx, the difference between the first two quantities. These could have been renamed during their definition to make it easier to understand the monitor plot. Here is a snapshot of the quantities being monitored during the solution:


So, as we have seen in the last 5 CFX Expression Language blog entries, CEL exposes a lot of powerful capability to CFX users (and to Fluent users within CFD Post). In this case we have seen how to add additional items to monitor during the solution process. The advantages of this are to help us determine on the fly if the solution is progressing as expected and to give us an idea of the values of certain results quantities before the solution is fully completed.

We certainly hope you now have a better understanding for how CEL can be used to increase the capabilities and useful information available from CFX and CFD Post.

CFX Expression Language–Part 4: Applying Ramped and Stepped Boundary Conditions Using CEL

In three previous entries we introduced CFX Expression Language, CEL:

In this fourth installment we will demonstrate the use of CEL to apply ramped or stepped boundary conditions. In certain circumstances we might want to ramp a load rather than apply it all at once. For example, convergence difficulties can sometimes arise when a fast rotation rate is applied initially in rotating machinery simulations. Starting off with a smaller value of load and ramping it to the final value can aid in convergence in these circumstances.

Note that the rate of load application can be manually changed during the solution in the solver manager, but why not take advantage of CEL and do it automatically? As we will see, this is fairly easy to do.

The ability to ramp a load makes use of a built-in CEL variable labeled “aiturn”, which is the accumulated value of the iteration number. If we assign an expression for the number of iterations we want, we can then create a combined expression for the ramped load:


In the above list of expressions, Flow999 is our desired full amount of flow at the end of ramping. Iter is defined to have a value of 100. Both of those are names that we picked. We then defined expression flowapplied, which is the value of Flow999 times the built-in expression aitern (the current solver iteration number) divided by the total number of iterations desired for the ramping, Iter. Once aitern = 100, then the value of flowapplied will equal Flo999 or 9.99 ft/sec in this case.

Here is a plot of the expression flowapplied for values of aitern between 0 and 100. The plot is in m/s but the peak works out to be 9.99 ft/sec.


As we have seen in prior entries in this series, we plug in the expression name for the input in the appropriate field. In this case, the name of the expression flowapplied is entered in the Normal Speed field in the Inlet boundary details.

After solution, we can verify that at the end of the solution the applied inlet velocity had reached the full value of 9.99 ft/sec. in CFD Post:


The next step might be rerun the solution while maintaining a constant value of the applied load for an extended period of time. This can be accomplished by modifying the expression which defines the load so that it has some additional values:


In the above expression we have added a step() function, which can either be typed in or added by right clicking, Functions > CEL > step. This causes the ramped load to peak at the value of Flow999 when aitern reaches the previously defined value of Iter at 100, then drop to zero after that. This happens because if Iter-aitern is greater than one, step=1, but if Iter-aitern is less than one, step=0. Here is the resulting plot in CFX Pre:


That’s not quite what we want, but if we tweak the expression a bit more, we can get it to ramp to the full value then remain constant.


Now we have another term involving the step() function, but with the expression names inside the step function reversed. This means that once aitern exceeds the value of Iter, the first term becomes zero and the second term takes over with a constant value of the load equal to Flow999, as shown here:


By using similar expressions involving time, we can create a load history that turns off and on at desired time points.

Hopefully by now you’re starting to see the value of CEL. We are just scratching the surface here, but once you start using it you will find that CEL has a lot of potential powerful uses. In the next installment we’ll cover some additional capabilities available in CEL.

CFX Expression Language–Part 3: Applying Boundary Conditions Using CEL

In two previous entries we introduced CFX Expression Language, CEL:

Part 1: Accessing CFD Simulation Information in CFX (and FLUENT)

Part 2: Augmenting Material Property Assignments in ANSYS CFX

In this third installment we will see how to use CEL to apply boundary conditions as equations rather than constant values. For example, if a non-constant velocity profile can be defined as an equation, we can use CEL to define as well as apply the profile.

Let’s look at an example in which the velocity profile is a function of y coordinate:

u(y) = 6 * Umax * y / H * (1 – y/H) (m/s)

Using the procedure we learned in part 1 of this series, in CFX Pre we have defined expressions for H and Umax. We then defined the equation for the velocity profile as Uprofile:


Next we go to the Plot tab within the Expressions editor to verify that our velocity profile matches expectations:


To use our new expression in CFX Pre, we just enter the expression name in the appropriate field when defining the inlet velocity:


Finally, this velocity plot from CFD Post shows that indeed our desired velocity profile was applied at the inlet.


Hopefully this demonstrates how easy it can be to use CFX Expressions to define non-constant boundary conditions. In the next part of the series, we will look at using expressions to ramp or step apply loads.

2000 Core Milestone Passed for CUBE HVPC Systems

IMG_9548As we put the finishing touches on the latest 512 core CUBE HVPC cluster, PADT is happy to report that there are now 2,042 cores worth of High Value Performance Computing (HVPC) power out there in the form of PADT’s CUBE computer systems.  That is 2,042 Intel or AMD cores crunching away in workstations, compute servers, and mini-clusters chugging on CFD, Explicit Dynamics, and good old fashioned structural models – producing more accurate results in less time for less cost.

When PADT started selling CUBE HVPC systems it was for a very simple reason: our customers wanted to buy more compute horsepower but they could not afford it within their existing budgets. They saw the systems we were using and asked if we could build one for them.  We did. And now we have put together enough systems to get to 2,042 cores and over 9.5TB of RAM.


Our Latest Cluster is Ready to Ship

We just finished testing ANSYS, FLUENT, and HFSS on our latest build, a 512 core AMD based cluster. IT is a nice system:

  • 512 2.5GHz AMD Opteron 6380 Processors: 16 cores per chip, 4 chips per node, 8 nodes
  • 2,048 GB RAM, 256GB per node, 8 nodes
  • 24 TB disk space – RAID0:  3TB per node, 8 nodes
  • 16 Port 40Gbps Infiniband Switch (so they can connect to their older cluster as well)
  • Linux

All for well under $180,000.

It was so pretty that we took some time to take some nice images of it (click to see the full size):

CUBE-HVPC-512-core-front1-1000h CUBE-HVPC-512-core-service1-1000h CUBE-HVPC-512-core-stairs-1000h

And it sounded so awesome, that we took this video so everyone can here it spooling up on an FLUENT benchmark:

If that made you smile, you are a simulation geek!

Next we are building two 64 core compute servers, for another repeat customer, with an Infiniband switch to hook up to their two existing CUBE systems. This will get them to a 256 core cluster.

We will let you know when we get to 5000 cores out there!

Are you ready to step out of the box, and step into a CUBE?  Contact us to get a quote for your next simulation workstation, compute server, or cluster.

CFX Expression Language – Part 2: Augmenting Material Property Assignments in ANSYS CFX

In a previous entry we introduced CFX Expression Language, CEL.  You can view that post here

Before we get started, there are some key things to remember:

  1. Expressions can be easily created by right-clicking in the Expressions tab after double clicking on Expressions in the CFX Pre object tree.
  2. Expressions and their contents are case sensitive.

In this next part of the series, we’ll show how to use CEL to augment your material property definitions in CFX. If material properties are constants then their input is straightforward. However, if the properties are defined as equations, we can use CEL to input those equations in CFX.

For example, if viscosity is defined as a function of shear strain rate, we need to define viscosity using an equation that captures that relationship, such as

m = K * gn-1

Below are shown two ways in which that equation can be captured using CFX Expression Language, visc1 and visc2. The second equation, visc2, is more flexible in that we have defined the constant terms as expressions themselves.


It’s always a good idea to verify the input. Most expressions can be easily plotted by clicking on the Plot tab in the Details view. Here is a plot of the viscosity vs. shear strain rates between 0 and 1, as calculated by expression visc2:


Similarly, the Evaluate tab can be used to evaluate the expression for desired values of the inputs.

So, we have defined an expression for a material property, viscosity in this case. How do we get CFX to use that expression? In the material property input, we click on the expression icon to the right of the particular material property we are defining, then enter the name of the expression, as shown here for expression visc2:


Summing it up, we can use CFX Expression Language to define material property equations for non-constant material values. In the next installment we will look at how to use CEL to define changing boundary conditions, such as a ramped load.

CFX Expression Language – Part 1: Accessing CFD Simulation Information in CFX (and FLUENT)

This week we are presenting an introduction to CFX Expression Language. If you’re not familiar with CFX, it is one of the two CFD tools available from ANSYS, Inc., the other being Fluent. CFX has been part of the ANSYS family of engineering tools since 2003. It is relatively easy to use and can be run stand-alone or tightly integrated with other ANSYS products within ANSYS Workbench. We have some general information on CFX available at this link.

CFX Expression Language, or CEL, is the scripting language that allows us to define inputs as variables, capture outputs as variables, and perform operations on those variables. Through the use of CEL we can be more efficient in our CFD runs and better capture results that we need. With CEL we can access and manipulate information without needing to recompile code or access separate routines besides the main CFX applications.

Also note that since CEL can be used in CFD Post, it is useful for postprocessing FLUENT solutions in addition to CFX, since CFD Post is common to both CFX and FLUENT. There are some things to be aware of regarding FLUENT In CFD Post. This link in to the ANSYS 14.5 Help system explains it:

// User’s Guide :: 0 // 7. CFD-Post File Menu // 7.15. File Types Used and Produced by CFD-Post // 7.15.10. Limitations with FLUENT Files

If you are a user of APDL, ANSYS Parametric Design Language, what I have written above about CEL should look familiar. One difference, though, is that while Mechanical APDL is dimensionless, CFX is not. Therefore, CEL definitions contain units where appropriate.

CEL is typically used in CFX-Pre and CFD-Post. A handy editor is available to assist in the definition of the expressions. Most of the activity is enabled by right clicking.

Virtually any quantity in CFX that requires a value input can make use of CEL, including boundary conditions and material properties. CEL can also be used to access and enhance results information. Expressions defined in CEL can be used in design point studies in ANSYS Workbench, either as input or output parameters.

So, what kind of things can you do in an expression? In addition to accessing simulation information and storing it as a variable, you can manipulate values using operators such as add, subtract, multiply, divide, and raise to a power. You can also use built-in functions such as sine, cosine, tangent and other trig functions, exponent, log, square root, absolute value, minimum, maximum, etc.

There are many predefined values, including some common CFD constants such as pi, the universal gas constant, and Avogadro’s number. The available options are different in CFX pre vs. CFD Post, with relevant choices for each.

In CFX Pre, expressions are accessed by double clicking on Expressions in the tree. That takes you to the expression editor, as shown here:


Notice how units are defined for each expression, but they can be mixed if desired.

Regarding CFD Post, the example below shows three expressions defined in CFD Post. The expressions within the box are user-defined. The other expressions listed are setup automatically.

The values for forceX1 and forceX2 are calculated by extracting X-direction forces on two different surfaces. The surface names were defined in ANSYS Meshing in this case, as Named Selections. The value fdiffx is calculated by subtracting forceX1 from forceX2. The resulting value, fdiffx, has been specified as an output parameter in Workbench; hence the P-> symbol next to the name.


New expressions are created by right-clicking in the Expressions tab. The new expression value is given a name, then the definition is input, typically by right clicking and selecting from the menus of available quantities, like this:


The location of application for an expression can also be selected by right clicking:


So we’ve got our variables defined using CEL. Now what? Here are some things we can do with CEL variables:

1. Use them as inputs such as material properties or boundary condition values in CFX. If we are running multiple cases, it is typically much easier to define quantities that we want to vary this way. The values can then be changes in the Expression Window, or if defined as a parameter in Workbench, in the parameters view as part of a parameter study.

2. Use them for reporting results quantities of interest, such as forces at a desired location.

3. Use them as input or output parameters in a design point study or design optimization.

Hopefully this brief introduction gives you a glimpse at the power of CEL. In a future article we will look at using CEL for more advanced functionality, such as applying ramped or time varying boundary conditions, using IF statements, and monitoring expression values during solution.

ANSYS Acquires EVEN, the Makers of the ANSYS Composite PrepPost Tool (ACP)

Good news out there in ANSYS land.  ANSYS, Inc.  just made the relationship with EVEN as close as possible – by acquiring them.  Here at PADT it was love at first sight when we first were introduced to the ANSYS Composite PrepPost (ACP) add-on.  The solver capabilities in ANSYS Mechanical APDL have been very strong for composite modeling for some time.  But the pain and suffering required to set up a complex composite geometry kept many users from accessing those fantastic elements.  ACP solved that problem by providing a tool that takes care of the bookeeping and geometry issues involved in building an accurate model of composite layups.

Here is the official press release.


With this acquisition ANSYS, Inc. has secured the future development of this tool and given all of us in the ANSYS world even better access to the consulting team at EVEN.  You can learn more about the ACP tool on our ACP page.  We also have an older blog posting on ACP when it came out.  We also did a seminar on the last release, here is the recording to that. [probably time to write an updated posting on newer capabilities…].

Learn more about EVEN on their web site.

This is great news, and we can not wait to see further improvements in the composite modeling capabilities for the ANSYS Product family.

Yes! Concurrent Design Point Solves Using New ANSYS HPC Parametric Pack Licensing



Design Optimization – Design Point Studies.

These are terms that for many years now have been tossed about as powerful simulation tools. Indeed they are powerful tools, but for anything but relatively small models, the computing resources and time involved to get solutions have been prohibitive in many cases.

We are now in the 2010’s and computing power is far greater than it was just a few years ago. To help us better take advantage of those horsepower increases, ANSYS, Inc. has released a new license product with version 14.5, called the ANSYS HPC Parametric Pack.

How does a six minute turnaround time for 4 design points look when compared to a two hour time for a single design point? If you find that intriguing, please keep reading.

Simply put, the Parametric Pack license allows us to solve simultaneous design points on multi-core systems. For the most part, design point runs have been serial up to now. With Parametric Packs, you can solve several design points at the same time, each running in parallel.

What ANSYS, Inc. has done with the Parametric Pack concept is to allow you to multiply your existing licenses for use in simultaneous solutions of design points. Each Parametric Pack license provides a multiplier on existing licenses. If you currently have one Mechanical or ANSYS CFD license, with a Parametric Pack license it now becomes equivalent to 4 licenses for the purposes of solving concurrent design points. The more parametric pack licenses, the greater the multiplier, as shown in the following table. Note that the maximum allowed number of Parametric Pack licenses for a given study is 5.

# Parametric Pack Licenses # Simultaneous Design Point Solves
1 4
2 8
3 16
4 32
5 64

The Parametric Pack license multipliers apply in two scenarios. With scenario one, a design point study has been setup in ANSYS Workbench in which there is a set of input parameters and a set of output parameters. A table of various values of the input parameters has been defined for which we want to track the outputs. An example of this is shown below. The other scenario in which Parametric Pack licenses can be used is with design optimization using an ANSYS DesignXplorer license. We will focus on scenario one in this article, while a future article will address scenario two.

The example we will use is a Fluent study. It could just as well be an ANSYS structural or thermal solution, CFX solution, coupled field solution, etc.


In this case, we just have one varying input parameter (inlet velocity) and one varying output parameter (mass flow at the outlet) for the sake of simplicity.


Design point updates with the Parametric Pack license work through the ANSYS Remote Solve Manager, RSM. The runs can be made either on the local machine or on a remote number cruncher, but either way they need to be submitted with RSM. RSM comes with ANSYS automatically, but needs to be configured the first time you use it.

For the example shown here, I set it up to run on one of our Linux PADT Cube systems. The submission to RSM was made from my local Windows box while the solving was done on the remote Cube on PADT’s cluster.

ANSYS has to be told to use an available Parametric Pack license. It also has to be told which licenses to be used on conjunction with the Parametric Pack license. This information is defined from within Workbench, by right-clicking on the Parameter Set box and displaying Properties. Once License Checkout is set to Reserved, we click on the Reserve Licenses link to select the desired licenses to be used:


In the window below you can see I have reserved 1 ANSYS CFD license which allows for 1 Fluent solve. I have also reserved one ANSYS HPC Pack which allows for up to 8 parallel tasks per solve. By also reserving one ANSYS HPC Parametric Pack license, the other two are amplified. As the last column shows, the reported number of concurrent licenses is 4 for the ANSYS CFD license and 4 for the ANSYS HPC Pack license (meaning 4*8 or 32 total cores for 4 simultaneous solves).


More HPC Parametric Packs would amplify the licenses further. It’s important to note that not all ANSYS licenses can be amplified by the Parametric Pack license. In general, the licenses that can’t are products that rely on a third party for some of the technology, such as DesignModeler which uses the Parasolid kernel. That doesn’t mean that DesignModeler can’t be part of a study that utilizes the Parametric Pack licenses, though. It just means that that the DesignModeler tasks will be automatically completed before the jobs are submitted for simultaneous solving.

Getting back to the example, we asked ANSYS Workbench to solve 4 design points. Without Parametric Pack licensing, that would have been done sequentially. On my local Workstation, solving on a single core each design point takes about 2 hours to solve. Using 8 cores on our Cube machine, each design point takes about 6 minutes to solve. What happens when I activate the simultaneous solution with the Parametric Pack license? All 4 design points solve in 6 minutes. This particular Cube has 64 cores, so solving a single design point on 8 cores or four design points concurrently using 32 total cores both take six minutes. That is a very significant speedup. I say it’s a game changing speedup.

Here is a graph of CPU utilization during the concurrent design point solution. 32 processors utilized and the elapsed time was about 6 minutes.


The resulting design point info including the as-solved output parameters:


The bottom line:

What do you need to be able to take advantage of this capability?

1. A regular license enabling the solver you need, such as ANSYS Mechanical, Multiphysics, ANSYS CFD, ANSYS Fluent, ANSYS CFX, etc.

2. ANSYS HPC or ANSYS HPC Pack licenses which allow you to solve on more than two processors/cores for each design point.

3. At least one ANSYS HPC Parametric Pack license which allows the simultaneous design point studies and the amplification of the existing licenses. Talk to your local ANSYS rep or ANSYS Channel Partner for more info.

4. A multi-core machine, such as one of PADT’s Cube systems. More info: http://www.padtinc.com/products/hardware/cube-hvpc/index.html

In a future article we will look at the use of the HPC Parametric Pack license in conjunction with a design optimization study.

Direct Coupled-Field Elements in Mechanical APDL

We received one of those tech support calls last week that you hate getting.  It was something like “I need to transfer my ANSYS model to this other FEA package, how do I do that?”  We of course asked “Why do you need to go to this other package?” The answer was “Because they have elements that solve for stress and thermal degrees of freedom in the same element.”  Well, so does ANSYS Mechanical APDL, and it has for years.  But as a Workbench user they had only been exposed to Multiphysics that uses Load transfer as the mechanism to solve different domains in the same run

Therefore, a The Focus posting is born.

In this posting we will go over the basics of direct coupled-field elements and simulation to make everyone aware of what is available.

Direct Coupled-Field vs. Load Transfer

When most people talk about Multiphysics they are talking about Fluid-Structural Interaction (FSI) or some other interaction between two different models where the program solves each physics by itself and transfers the resulting values from one physics as a load on the next physics.  This is called load transfer Multiphysics and it is very useful and powerful.  But it requires a solve for each physics for each step in your solving process, and often more because you have to iterate back and forth between physics till things converge before you can move to the next substep.

There is a whole other way to do Multiphysics if you have the same mesh for each physics: you can modify your finite element equations to cover all the different physics in one set of equations, therefore in one matrix, and therefore in one pass through the solver for each solve.  This capability has been in the ANSYS Mechanical APDL solver for a very long time and has been expanded over time to cover some surprising combinations of physics.

So when should you use one over the other? That depends. Here are some thoughts:

  • Load Transfer Approach:
    • Your meshes need to be or are different
    • Fluid flow with something other then heat-transfer
  • Direct Approach:
    • The interaction between two physics is strongly coupled
    • The interaction is non-linear
    • Acoustics is involved
    • Piezoelectric is involved
    • Porous fluid flow is involved
    • Diffusion is involved

In general, if you can use Direct Coupling and you know MAPDL well, it is the preferred way to go, it is just a lot easier to do. But if you are not familiar with MAPDL for running and post processing, you may be better off with the Load Transfer approach.

The Coupled-Field Elements

You access the coupled-field capabilities in the solver through the use of the coupled-field elements.  Although there are some legacy elements that can be used as well, we will focus on the three standard coupled-field elements. They all have the same capability, and just vary in topology:

  • PLANE223: 2D 8 Node Quad
  • SOLID226: 3D 20 Node Hex
  • SOLID227: 3D 10 Node Tet

All of these support the following physics, DOF’s, forces and reaction loads:

Field DOF Label Force Label Reaction Solution
Structural UX, UY, UZ FX, FY, FZ Force
Thermal TEMP HEAT Heat Flow
Electric Conduction VOLT AMPS Electric Current
Electrostatic/Piezo VOLT CHRG Electric Charge
Diffusion CONC RATE Diffusion Flow Rate

You use a combination of KEYOPTS and material properties to enable the various types of coupling.  Take a look at the element documentation to see how it all works.

In addition to these, there are some specialty elements worth discussion. The first are FLUID29/FLUID30. These are the Acoustic field elements. These solve for displacement and pressure. They also can share the displacement DOF’s with structural elements where they touch.

Unfortunately the electromagnetic coupled field elements have been put on legacy status, as ANSYS Maxwell is where the development effort is going in this area. But you can still use them for coupled-field simulation that involves the MAG degree of freedom.  The elements are: PLANE13, SOLID5, SOLID98. ANSYS MAPDL still has actively supported electromagnetic elements, but they are electromagnetic only and do no support displacement or thermal degrees of freedom.

Flow in a fully saturated porous media can be modeled with the Coupled Pore-Pressure elements. These elements: CPT212/213/215/216/217, solve for pressure and deflection and are used for things like modeling nuclear waste issues, soil subsidence, oil well stability, and bone deformation and healing.

We should also mention that ANSYS supports circuit simulation using the CIRCU124 element.  This element can be coupled to other elements that have VOLT, CURR, or EMF capability.


Running Direct Coupled-Field Multiphysics in ANSYS Mechanical APDL

When I wrote this section heading it seemed like a good idea. But this is supposed to be a short blog entry and not a full one day training class. So I will wimp out and share where you can find more information in the help:

There is a whole manual dedicated to coupled-field analysis: Mechanical APDL // Coupled Field Analysis Guide. Within that guide is the Direct Coupled-Field Analysis section, Chapter 2.  In it you will not only find discussions about how to do what you need to do, but also a whole bunch of simple examples that are very helpful.

In general, you run like any other simulation.  There is really nothing special or unique and you do not have to deal with managing the load transfer like you do with load transfer coupled field simulations.

Running Direct Coupled-Field Multiphysics in ANSYS Mechanical

This is a question that comes up a lot. Unfortunately only one type of direct coupling is supported, Thermal-Electric.  What we recommend people do is they build their models in ANSYS mechanical for one of the physics, then use code snippets to change the elements to the proper direct coupled-field type and to also do any post processing. It will run when you solve, but it will come back with an error, and you need to post processes via APDL code or you need to post process in MAPDL interactively.

NEW INFO:  Edward points out in the comment below that you can get this to work.  I’ll repeat it here:

“We’ve had some success post-processing U-TEMP-VOLT analyses in Mechanical. Mechanical seems to accept a model as solved, so long as it sees a result file of the correct type in the Solver Files directory. The coupled field analysis in this case output a .rst file, so we used a Static Structural object as the base model. 
We could access the structural results directly and used User-defined results to access most of the thermal and electric results.
I seem to recall that we also had success using a Thermal analysis as a base and then changing the result file extension from .rst to .rth, but I can’t find my test model to confirm this.”

I can verify that both of these approaches work. I added a /sys, copy file.rst to file.rth to a code segment for the thermal base.  But it was simpler to just use the structural as the base.  If you do this you can do your post processing for the most part in ANSYS Mechanical. [E. Miller 3/28/2013)


So this was, as promised, a very high level overview. The fact of the matter is that there are a significant number of users, especially in the MEMS industry, that use these direct coupled-field elements all the time.  They are powerful and robust with as many uses as you can dream up, truly expanding the reach of what you can model and the accuracy of those models.

Over the years we have found some good tricks for using these elements effectively:

  1. Pick one of the physics and get a static run of that physics by itself running first. Debugging your model this way is usually faster and clears out any issues before you deal with the direct coupling issues. If you have more than two physics, add them in one at a time.
  2. Pay attention to units. When you start mixing voltage and distance or what not, it is easy to get confused. If you are doing MEMS devices, you need to make sure you are using the MEMS units and that you are consistent.  Unlike ANSYS Mechanical, ANSYS Mechanical APLD is unitless and requires the user to make sure the are consistent across physics.
  3. Try not to use the legacy elements if you don’t have to. They may not be around in the future.
  4. If you are doing EMAG, you may want to look at using load coupling with Maxwell or MAPDL instead of using the legacy direct coupled elements.  Maxwell and the newer elements in MAPDL have more capabilities and are more efficient.
  5. Make sure you really understand how your physics interact. Go through the thought experiment of predicting the interaction on as simple of a problem as you can, while keeping it relevant. Think about what loads interact with what structures and what that interaction implies.

Introduction to the ANSYS Parametric Design Language (APDL) Book Now Available on Amazon!

PADT-Intro-APDL-Amazon-PagePADT’s popular “ANSYS Customization with the ANSYS Parametric Design Language Guide” Has been updated and reformatted as a book and published as “Introduction to the ANSYS Parametric Design Language”  in both softcover and Kindle formats.

This book started life as a class that PADT taught for many years. Then over time people asked if they could buy the notes.  And then they asked for a real book.  The bulk of the content came from Jeff Strain with input from most of our technical staff.  Much of the editing and new content was done by Susanna Young and Eric Miller.

Here is the Description from Amazon.com:

The definitive guide to the ANSYS Parametric Design Language (APDL), the command language for the ANSYS Mechanical APDL product from ANSYS, Inc. PADT has converted their popular “Introduction to APDL” class into a guide so that users can teach themselves the APDL language at their own pace. Its 12 chapters include reference information, examples, tips and hints, and eight workshops. Topics covered include:
– Parameters
– User Interfacing
– Program Flow
– Retrieving Database Information
– Arrays, Tables, and Strings
– Importing Data
– Writing Output to Files
– Menu Customization

At only $75.00 it is an investment that will pay for itself quickly.  Even if you are an ANSYS Mechanical user, you can still benefit from knowing APDL, allowing you to add code snippets to your models. We have put some images below and you can also learn more and purchase your copy on Amazon.com.  They can ship anywhere in the world.