Coordinating Coordinate Systems in ANSYS Mechanical

Coordinate systems are one of those things that are fundamental to Finite Element Analysis, but that most of us do not think about a lot.  They are there, but some users never fiddle with them. And some users are constantly futzing around with them.  We thought it would be a good idea to do a quick review of how they work in ANSYS Mechanical.  We will also go over the basics for  Mechanical APDL (MADL) in case you need to work with snippets.

Why Coordinate Systems Matter

ANSYS cares a lot about coordinate systems because they allow the program to solve in a standard, global, Cartesian system while allowing loads, constraints, material directions, layer information, beam sections, joints, result values, and a whole slew of other important aspects of the model to be specified in unique coordinate systems. This avoids making the user do coordinate system transformations.  At solve time, everything gets converted.

Since everyone reading this is an engineer, I’m going to assume that everyone already knows what a coordinate system is.

Coordinate Systems in DesignModeler and CAD Tools

It is important to start at the beginning.  Design Modeler and all CAD packages I’m aware of allow you to define some sort of coordinate system. Usually just Cartesian.  In workbench you can import those coordinate systems into ANSYS Mechanical by clicking “Import Coordinate Systems” from the “Advanced Geometry Options” properties for the Geometry cell in you systems.

For DesignModeler, there is an extra step.  Even if you turn on the import properties you need to dell DM which coordinate systems you want imported.  But first, be aware that there is no “coordinate system” entity in DM.  Instead it has planes, which is a coordinate system where you draw on the Z-normal plane.

To make these available in ANSYS Mechanical, you need to scroll down to the bottom of the details for the planes you want converted over to coordinate systems, and set “Export Coordinate System” to Yes.

The following three images show setting it in DM, setting the property on the Project page, and how it shows up in Mechanical:

Creating Coordinate Systems in ANSYS Mechanical

In ANSYS Mechanical, coordinate systems reside in the Model Tree between Geometry and Connections.  Once you define a coordinate system it becomes available for use with any other object that can use a coordinate system.  This allows you to define it once, and then use it many times.

You always get a Global Cartesian coordinate system, called Global Coordinate System.  It is Cartesian, has an ID of 0, and sits at 0,0,0.  You can not change any of these values. Any imported coordinate systems will show up underneath the global.

To create a coordinate system you Right Mouse Button (RMB) on the Coordinate Systems branch and Insert->Coordinate System.  Or, when you click on the branch you also get a Coordinate System toolbar:

Click on the three color triad icon and a new system will be inserted.

Let’s look at each of the options in the details view.  But note before you go there, that the first set of group define a starting location and orientation, then you apply transformations in the last detail group in order to modify those locations.

• Definition Group: This group specifies the type and MAPDL number for the Coordinate System
• Type:  You can have the default Cartesian or Cylindrical here. The resulting coordinate system triads show up on your model like so.  As you can see, Z is the rotational axis, Y is tangential an X is radial.

• Coordinate System: In my opinion this should say Coordinate System ID because this detail lets you decide if you want ANSYS Mechanical to assign the number that MAPDL will use, or if you will.  Program Controlled is the default and is fine in most cases.  If you need to wrote a snippet to work with a coordinate system then you should change it to “Manual” and Coordinate System ID will show up.  Set it to any number over 11.

• Origin Group:  This group defines where the center of the coordinate system is.
• Define By:  You can specify a Geometry Selection or Global Coordinates.
• Geometry Selection: The cool thing about using Geometry Selection is that as you update your CAD model, the origin will shift with it.  As with any geometry specification, you click on a surface, line, vertex or a collection of these.  Mechanical will calculate the geometric center of the entity of entities that you picked and place the origin at that centroid.  It will also shows the position in the global coordinate system below your geometry selection, but you can not change them.
• Global Coordinates: Here you simply put in an X, Y, and Z value in one of two ways. The easiest is to just type them in.  Or, you click “Click to Change” for Location and pick the coordinate picker icon and move your mouse over your model.  Mechanical calculates the point under the cursor (surface closest to camera) and displays it. When you click it will create a little blue cross on the geometry.  Choose apply on Location an it will enter that point in as the origin.  Kind of cool, if a bit inaccurate…
• Principal Axis Group: You need to tell Mechanical how to orient the coordinate system. By default it will align with the global.  But you can use Axis and Define By to specify that any of the three axis are aligned with a global axis, or with a piece of geometry.  Aligning with geometry is very useful because this is how you get coordinate systems aligned with your geometry. And when your geometry updates, that coordinate system aligns with the updated geometry.  This is especially useful when specifying a coordinate system in a cylinder because you can pick the cylinder face for your Z axis and it will move with the cylinder.
Note that you can not specify align with Global –Z. If you want to do that you need to align with Z and use the transformation below to flip that.
One option for “Define By” is “Fixed Vector” This uses the current orientation but disassociates it from the geometry.
• Orientation about Principal Axis Group:  One point and a vector does not a coordinate system define. You have to specify an orientation around that principal axis.  You do that just like the how you specify that principal axis.  Define a global X, Y, or Z or a piece of geometry.
• Direction Vectors Group:  These show the vectors for X, Y, and Z.  You can’t change them (I wish you could) and they are not a parameter. But they are useful.
• Transformations Group:  This area allows you to stack offsets, rotations, and mirrors.
You use the Coordinate System Toolbar to insert transformations into this group.  They are executed in order from top to bottom and the resulting orientation and position are shown in Directional Vectors and Transformed Configuration.
There is a “Move Transform Up” and “Move Transform Down” icon as well in the toolbar to move the transformations around. There is also a delete to remove one.
Note: When you click on a transformation in the list, the coordinate system on your model is shown AT THAT STEP, not at the final position.  This always confuses me.  So make your change, then click on the last step to see it.

Using Coordinate Systems

This is the easiest part. You simply choose one of your defined coordinate systems from a dropdown list when you create an object that is dependent on a coordinate system.  Usually this is when you can define a value based Components rather than on geometry:

Do note that you can also use coordinate systems to transform directional result values.  Simply pick the Coordinate system from the dropdown list.  This is especially important when looking at hoop or radial stresses in a cylindrical part.

Coordinate Systems in ANSYS Mechanical APDL

Coordinate systems are huge in MAPDL.  Nodes have them, elements have them, sections have them. Plus you can make a coordinate system active and every command you execute is done in that active coordinate system, and converted for you to the global. Very powerful.

If you do a search in help on “Coordinate System” you get hundreds of hits in the MAPDL manual.  Way too much to go over here.  We do recommend that you start with:

Mechanical APDL // Modeling and Meshing Guide // 3. Coordinate Systems // 3.1. Global and Local Coordinate Systems

It explains the types, the math, and the commands needed.  Read that, then move on to 3.3, 3.4, and 3.5 which talk about nodal, element and result coordinate systems.

Some key things every user should know are:

1. All coordinate systems are defined by a number.  0-10 are reserved by MAPDL for its use.  Users can do 11 or higher.
2. MAPDL has 6 default coordinate systems:
0 = Cartesian
1 is cylindrical down the Z axis
2 is Spherical
4 is Cartesian, same as 0
5 is Cylindrical down the Y axis.
6 is Cylindrical down the X axis (not shown)
I have no idea what happened to 3 or why 4 is the same as 0.
3. When you change the active coordinate system with CSYS, all commands that involve coordinates get transformed into that coordinate system.  So:
local,11,1,2.5,0,0,0,90,90
n,1,10,10,10
Actually makes a node at 12.5,9.8481,1.7365 in the global coordinate system.
4. You can show local coordinate system with /psymb,csys,1

5. You can list your coordinate system definitions with CSLIST:

6. Only Cartesian and cylindrical are supported in ANSYS Mechanical, so if you need to use spherical or Toroidal you need to use snippets

Thoughts

Make sure you understand how Mechanical is using coordinate systems by bringing your models up in MAPDL.  Look at your nodes and see if they are rotated and how.  Check the coordinate systems with a CSLIST. Make sure you feel comfortable, don’t take it for granted.

Writing Text files with *VWRITE

A very common need in the world of ANSYS FEA simulation is to write text to a text file from within Mechanical APDL. Sometimes you are running in MAPDL, sometimes you are using ANSYS Mechanical but you still need to write stuff out using APDL with a code snippet. The way most people do that is with *VWRITE.

Originally written to write out data in arrays, it is a very flexible and powerful command that can be used to write pretty much any type of formatted output. Something that every ANSYS user should have in their back pocket.

The Command

*VWRITE, Par1, Par2, Par3, Par4, Par5, Par6, Par7, Par8, Par9, Par10, Par11, Par12, Par13, Par14, Par15, Par16, Par17, Par18, Par19

Looks pretty simple right, just *vwrite and list what you want printed. But there is a lot more to this command.

A Lot More

First off you need to open up a file to write to.  You have a couple of options.

1. *CFOPEN,fname, ext, –, Loc
This opens the specified file for writing with *cfwrite and *vwrite commands.  This is the preferred method.
2. /output,fname, ext, –, Loc
By default *VWRITE output to standard output – the jobname.out (batch) file or the command window (interactive). So if you use /output you can redirect to a file instead of the *.out or screen.  We don’t recommend this because other stuff might get written as well to the file.

Now you have a place to write to, next you need to use *VWRITE to write. *VWRITE is a unique command because it actually uses two lines.  The first contains *VWRITE and a list of parameters and/or arrays to write and the second contains a format statement.  We will cover the first line first, and the format second.

Parameter Arguments for *VWRITE

As you can see from the command, you can have up to 19 parameters listed on a *VWRITE command.  PAR1 through PAR19 can be array, scalar, or character parameters. They can also be a constant. This is where the real flexibility comes in.  You can do something like (just look at the *VWRITE line, we will talk about the rest further on):

`   1: adiv = ' | '`

`   2: *dim,nds, ,10`

`   3: *dim,temps,,10`

`   4: *vfill,nds(1),ramp,1,1`

`   5: *vfill,temps(1),rand,70,1500`

`   6: *cfopen,vw1.out`

`   7: *VWRITE,'Temp: ',nds(1),temps(1),adiv, 'TREF: ',70`

`   8: (A6,F8.0,g16.8,A3,A6,F10.4)`

`   9: *cfclose`

This mixes characters, arrays, and constants in one command.  As output you get:

`Temp:       1.   429.56308     | TREF:    70.0000Temp:       2.   263.55403     | TREF:    70.0000Temp:       3.   1482.8411     | TREF:    70.0000Temp:       4.   605.95819     | TREF:    70.0000Temp:       5.   782.33391     | TREF:    70.0000Temp:       6.   1301.1332     | TREF:    70.0000Temp:       7.   1119.4253     | TREF:    70.0000Temp:       8.   202.87298     | TREF:    70.0000Temp:       9.   1053.4121     | TREF:    70.0000Temp:      10.   805.71033     | TREF:    70.0000`

Array Parameters

The first thing you will notice is no *do loop.  If you supply an array parameter, *vwrite loops on the parameter from the given index (1 in this case) to the end of the array.  But if you don’t want the whole array written, you can control by placing *VLEN and/or *VMASK in front of the *VWRITE command:

• *VLEN,nrow,ninc
This will only write out nrow times, skipping based on ninc (defaults to 1)
• As an example, if you want to write just the fourth value in array A() you would do:
*VLEN,1
*VWRITE,A(4)
(G16.8)
You make a mask array of 0’s and 1 that is the same size as your array, and supply it to *VMASK.  *VWRITE will only write out values for your array if the mask array is not zero for the same index.

You can have a multiple dimensions on your array. *VWRITE only increments the first index. Say you specify X, Y, and Z coordinates in an array call xyz.  It would look like:

*VWRITE,xyz(1,1),XYZ(1,3),XYZ(1,3)
(3G16.8)

String Parameters

Being an older program, you are limited in what you can do with character parameters.  You are limited to 8 characters. So you just use a long string parameter several times and increment the index by 8:

`   1: *dim,mystring,string,80`

`   2: mystring(1) = 'This is a very long sentance'`

`   3: *cfopen,vw2.out`

`   4: *VWRITE,mystring(1), mystring(9), mystring(17), mystring(25), mystring(33)`

`   5: (5A) `

`   6: *cfclose`

Kind of hokey, but it works.

Integers

Sigh.  This is the one thing that I’m not fond of in *VWRITE. The original command did not support outputting integer values.  That is because the FORTRAN I descriptor was not supported, and ANSYS stores everything as an integer anyhow. But people needed to output integer values so they took the ‘C’ format routines for *MSG and made them work with *VWRITE. So you can do a %I.  See the section on ‘C’ formatting below for more information on this.

Close the File

Before you can do anything with the file you create you need to close it.  Not to hard: *CFCLOSE does the trick.

Other Stuff you Need to Know

Don’t put a blank in there.  If you do, *vwrite stops looking at parameters.  So if you need a blank in your file, put in a space ‘ ‘ or use the X FORTRAN descriptor.

Be aware of *CFWRITE as well.  It is a way to write APDL commands to a file. If what you want to do is have your macro write a macro, *CFWRITE is better because you don’t have to do format statements. And those can be a pain when you need commas.

If your arrays are of different lengths, *VWRITE will loop for the length of the longest array. Any shorter arrays will be replaced with zeros for number arrays and blanks for character/string arrays.

You can not use *VWRITE by pasting or typing into the command line.  You have to read it from a file.

Formatting

The key, and the difficult part of using *VWRITE is the format statement. We recommend that you use the FORTRAN formatting when you are writing out large amounts of columnar data, and use the ‘C’ format if you are writing out text rich information for a report or to inform the user of something.

Many users today may not even know what a FORTRAN statement looks like.  A good place to look is:
http://www.stanford.edu/class/me200c/tutorial_77/17_format.html

Just remember that you can’t use the Integer (I) format.  The list directed format (*) also does not work.  If you are new to it also remember everything goes in parenthesis and it has to fit on one line.  It does not have to start in column 8 (if you think that is funny, you are old)

As to ‘C’ formatting, you have a lot more options but we have found that the behavior is not as consistent between Linux and windows as the FORTRAN.  But if you are more comfortable with ‘C’, do that. Not all of ‘C’ formatting works in APDL, but what does is actually documented under the *MSG command.

Making it Work

We always recommend you work out your *VWRITE issues in a small macro that you can run over and over again as you work out the formatting.  If you are writing a snippet for ANSYS Mechanical. Go ahead and save your model, bring it up in MAPDL, then work on your *VWRITE statement till you get it right.

Some other useful suggestions are:

• Keep things simple. Don’t try and format a novel or an HTML page with *VWRITE.  You probably could, but there are better tools for that.
• Make sure you understand arrays, how they work in APDL, and how you have yours dimensioned.
• Get familiar with *VMASK and *VLEN. They are useful

Turning on Beta Features in Workbench

It is a busy week and I’m in between meetings for about 30 minutes, just enough time for a very short Focus posting for the week.  So, I thought I would share something I had to remember for the first time in a long time: How to access beta features in ANSYS Workbench.

First off a word of warning:  Beta features are beta features. They are capabilities in the software that have either not finished testing, are not fully documented, or that have a known issue.  They therefore must be used AT YOUR OWN RISK!!!!   If you find a bug or a problem, report it to your technical support provider, they need that feedback. But don’t call up indignant because it is not working the way you want it, or because the documentation is non-existent.  It is a beta feature.

Set the Option

Not too difficult.  From the Project Schematic page go to the tools menu and select Options.

Now in the Options dialog click on Appearance in the tree on the left.  You will not see Beta Options.  Scroll down and near the bottom there are a bunch of check boxes. Check “Beta Options”

Now, in your project toolbar you should see (Beta) next to the exposed beta functions:

This will also impact any beta features, if any, in the workbench native applications:  Parameter manager, Engineering Data, or DesignXplorer.

That is it. I promised short.  Off to another meeting.

Remember, USE AT YOUR OWN RISK.

Webinar Information: Constraint Equation Primer

Here are the files from the webinar held on Friday, April 27, 2012: A Constraint Equation Primer: How to Tie Degrees of Freedom Together

PowerPoint:

No models or anything on this one.

Executive Summary

Without losing you quickly due to fabulous marketing and product information. I thought it prudent if I get to the answer of my question as quickly as possible. What was the question? Should I invest in a companion processor and an ANSYS HPC Pack? Yes, now is the time.

• If your current workstation is beginning to show its age and you are unable to purchase new hardware.
• Get your critical results fast. It is painful waiting upwards of 50 hours for your solution to solve.
• Assign a dollar value to your current frustration level.
• Current pricing for Nvidia TESLA C2075 is around \$2500. Current pricing for the NVidia Quadro 6000 is around \$5,000.

“Keeping It Real”

Matt Sutton our Lead Software Development Engineer at PADT, Inc. was solving a very large ANSYS MAPDL acoustic ultrasound wave propagation model. The model labored over 12 hours to solve on Matt’s older Dell Precision 690 8 core workstation. We loaded the model onto our CUBE HVPC w8i with the NVidia Tesla C2075 and it solved the model in 50 minutes. That is a 15x speedup for Matt’s particular model!

Benchmarks

I started off with our benchmark assault using an industry standard ANSYS benchmark for HPC. This is a benchmark that NVidia requested we use for testing our TESLA C2075. The next benchmark that I used was an internal PADT, Inc. modal coupler benchmark.

Nvidia V14sp-5 ANSYS R14 Benchmark Matrix (Sparse solver, 2100k)

CUBE HVPC – w12a-GPU – 64GB

CONFIG A

64GB

#

Cores

Config A: Win v14 SMP In-Core

Config A: Win v14 SMP In-Core & GPU

Config A: Win v14 DIST In-Core

Config A: Win v14 DIST & GPU

Config A: Linux v14 SMP In-Core

Config A: Linux v14 SMP In-Core & GPU

Config A: Linux v14 DIST In-Core

Config A: Linux v14 DIST & GPU

speedup

2 1388.20 463.40 1425.50 294.40 1324.40 466.30 1312.30 292.10
4 870.00 447.20 855.00 232.40 817.00 422.00 975.40 221.80
6 670.60 440.20 588.00 245.10 625.50 394.00 601.60 208.10
8 594.30 439.10 605.00 222.00 480.10 383.90 542.30 181.10
10 539.50 426.90 435.00 283.40 491.30 392.80 396.90 220.80
12 538.40 437.10 402.20 211.10 480.10 394.80 343.80 183.10     2x

CUBE HVPC – w8i-GPU – 64GB CONFIG B
64GB # Cores Config B: Win v14 SMP In-Core Config B: Win v14 SMP In-Core & GPU Config B: Win v14 DIST In-Core Config B: Win v14 DIST & GPU Config B: Linux v14 SMP In-Core Config B: Linux v14 SMP In-Core & GPU Config B: Linux v14 DIST In-Core Config B: Linux v14 DIST & GPU
2 1078.50 361.00 1111.90 235.70 1036.00 350.20 1072.90 250.60
4 645.50 330.70 652.30 185.50 608.10 312.00 790.90 193.20
6 494.30 322.70 458.30 233.70 464.90 303.20 502.70 178.70
8 438.50 328.30 462.20 230.40 406.10 304.80 451.20 166.00     2.5x

 CUBE HVPC – w16i-GPU – 128GB

128 GB # Cores Config C: Linux v14 SMP In-Core & GPU Config C: Linux v14 DIST & GPU
2 296.6 208.10
4 254.4 160.70
6 254.2 164.20
8 239.9 138.20
10 238.6 159.70
12 246.3 129.60
14 237.6 129.1
16 248.9 130.5

CUBE HVPC – PADT, Inc. – Coupling Modal Benchmark (PCG solver, ~1 Million DOF, 50 modes)

 CUBE HVPC w12i w/GPU ANSYS 13.0 Shared Memory Parallel INTEL XEON 2×6 @3.47GHz /144GB of RAM w12i-GPU Processors Time Spent Computing Solution (secs) Date/Individuals Initials: 2 5416.4 11/7/2011 – DRJM 10+GPU 1914.2 (incore) 11/8/2011 – DRJM 12+GPU 1946 (incore) 11/8/2011 – DRJM CUBE HVPC w8i w/GPU ANSYS R14 Shared Memory Parallel INTEL XEON 2×4 @2.8GHz /64GB of RAM w8i-GPU 6+GPU 3659.4 (out of core) 4/11/12 – DRJM 8+GPU 3686.7 (out of core) 4/11/12 – DRJM CUBE HVPC w16i w/GPU ANSYS R14 Shared Memory Parallel INTEL XEON e5-2690 2×8 @2.9GHz /128GB of RAM W16i-GPU 14+GPU 2113 (incore) 4/18/12 – DRJM 16+GPU 1533.9 (incore) 4/18/12 – DRJM

Summary, Debates, Controversy & Conclusions

One question that I hear often is this: “What operating systems is faster. Linux or Windows?”. My typical response will begin with “Well, it depends…” However, what the data illustrates with the two independent CPU workstations as well as Operating Systems. The ANSYS benchmarks were all performed in the same relative time frame. The only exception was the ANSYS modal analysis benchmark.

Are you ready for the answer? Here it is…yes Linux is faster than windows! Even if you come from the AMD CPU or INTEL CPU side of the tracks. Linux is faster! No big discovery right? I know we all knew this already but we were maybe afraid to ask by Operating system as well as CPU manufacture how much? Well with our 2.1 million degree of freedom Nvidia benchmark. Ummm, not very much as the data clearly indicates. However once again, the Linux based OS does give you a system performance advantage! If you use AMD or INTEL it is still faster. The next question that I hear next is: “What processor is faster AMD or INTEL?”. My typical response will begin with “Well, it depends…”

You did buy the NVidia TESLA C2075 GPU, okay this is great news! However, you figured it best that you keep it safe, stay the same, and continue to use Shared Memory Parallel solve method. As you ponder further on the speed up values. You will see once again that the best results are going to be found on the Linux Operating System and when you choose to solve in Distributed Memory Parallel mode. The AMD based CPU had the best speedup values when comparing the workstation against the two solve modes with a 2.2x’s speed up.

Lets begin to unpack this data even more and see if you can come up with your own judgments. This is where things might start to get controversial…so hang on.

Windows, Linux, AMD or INTEL

Time Spent Computing The Solution

• ANSYS R14 Distributed Memory Parallel Results: (Incore)
• LINUX 64-bit:
• One minute forty-seven seconds faster on 2x AMD CPU (343.80 seconds vs. 451.20 seconds)
• Windows 7 Professional 64-bit:
• One minute faster on 2x AMD CPU (402.20 seconds vs. 462.20 seconds).
• ANSYS R14 Shared Memory Parallel Results: (Incore)
• LINUX 64-bit
• One minute fourteen seconds faster on 2x INTEL XEON CPU (406.10 seconds vs. 480.10 seconds)
• Windows 7 Professional 64-bit is sixty seconds faster solve time on AMD based CPU
• One minute forty seconds faster on 2x INTEL XEON CPU (438.50 vs. 538.40 seconds)

The Nvidia TESLA C2075 GPU and ANSYS R14 – GPU to the rescue!

• ANSYS R14 Distributed Memory Parallel with Nvidia TESLA C2075 GPU Assist Results
• LINUX 64-bit:
• Seventeen seconds faster on 2x INTEL XEON CPU (166 seconds vs. 183.10 seconds)
• 129.1 seconds was the fastest overall solve time achieved for any operating system on this benchmark!
• 2 x INTEL XEON E5-2690 – fourteen cores with ANSYS R14 DMP w/GPU assist
• Windows 7 Professional 64-bit:
• Nineteen seconds faster on 2x AMD CPU (211.10 seconds vs. 230.40 seconds).
• 211 seconds was the fastest solve time achieved using Windows for this benchmark!
• 2 x INTEL XEON X5560 – eight cores with ANSYS R14 DMP w/GPU assist
• ANSYS R14 Shared Memory Parallel with Nvidia TESLA C2075 GPU Assist Results
• LINUX 64-bit:
• Seventeen seconds faster on 2x INTEL XEON CPU (166 seconds vs. 183.10 seconds)
• Config C: 237.6 seconds was the fastest overall solve time achieved for any operating system on this benchmark!
• 2 x INTEL XEON E5-2690 – fourteen cores with ANSYS R14 SMP w/GPU assist
• Windows 7 Professional 64-bit:
• Nineteen seconds faster on 2x AMD CPU (211.10 seconds vs. 230.40 seconds).
• 211 seconds was the fastest solve time achieved using Windows for this benchmark!
• 2 x INTEL XEON X5560 – eight cores with ANSYS R14 SMP w/GPU assist

ANSYS R14 Distributed Memory Parallel with GPU vs. ANSYS R14 Shared Memory Parallel with GPU speed up results:

Distributed Memory Parallel (DMP) or Shared Memory Parallel (SMP)

• 2 x AMD Opteron 4184 based workstation vs. “DMP or SMP”
• 2.2x’s speedup on Linux Operating System (183.10 seconds vs. 394.80 seconds)
• 2x’s speedup on Windows Operating System (211.10 seconds vs. 437.10 seconds)
• 2 x INTEL XEON E5-2690 based CPU vs. “DMP or SMP”
• 1.9x’s speedup on Linux Operating System (129.41 seconds vs. 237.6 seconds)

With or Without GPU? ANSYS R14 Distributed Memory Parallel speed up results

Distributed Memory Parallel (DMP) with GPU vs. Distributed Memory Parallel without GPU

• AMD Opteron 4184 based workstation
• 2x’s speedup Linux Operating System (183.10 seconds vs. 343.80 seconds)
• 2x’s speedup on Windows Operating System (211.10 seconds vs. 402.2 seconds)
• INTEL XEON x5560 based workstation
• 2.5x’s speedup on Linux Operating System (166 seconds vs. 451.20 seconds)
• 1.4x’s speedup on Windows Operating System (230.40 seconds vs. 462.20 seconds)

Some Notes:

• ANSYS Base License – unlocks up to 2 CPU Cores
• ANSYS HPC Pack – unlocks up to 8 CPU Cores and GPU
• The total amount of system RAM you have affects your Distributed solve times. A minimum of 48GB of RAM is recommended.
• The processing speed of your CPU affects your Shared Memory Parallel solve times.
• Model limits for direct will depend on largest front sizes
• 6M DOF for 6GB Tesla C2075 and Quadro 6000
• Model limits for iterative PCG and JCG
• 3M DOF for 6GB Tesla C2075 and Quadro 6000

Hardware Specs of Workstations:

CONFIG A – CUBE HVPC w12a-GPU

The AMD® based CUBE HVPC w12a FEA Simulation Workstation:

• CPU: 2x AMD Opteron 4184 (2.8GHz Ghz 6 core)
• GPU: NVIDIA TESLA C2075 Companion Processor
• RAM: 64GB DDR3 1333Mhz ECC
• HDD: (os and apps): 450GB WD Velociraptor 10k
• HDD: (working directory): 6 x 1TB WD RE4 drives using LSI 2008 RAID 0
• OS: Windows 7 Professional 64-bit, Linux 64-bit
• OTHER: ANSYS R14, latest NVIDIA TESLA Drivers

CONFIG B – CUBE HVPC w8i-GPU

The INTEL® based CUBE HVPC w8i FEA Simulation Workstation:

• CPU: 2x INTEL XEON x5560 (2.8GHz 4 core)
• GPU: NVIDIA TESLA C2075 Companion Processor
• RAM: 64GB DDR3 1333Mhz ECC
• HDD: (os and apps): 146GB SAS 15k
• HDD: (working directory): 3 x 73GB SAS 15k – LSI RAID 0
• OS: Windows 7 Professional 64-bit, Linux 64-bit
• OTHER: ANSYS R14, latest NVIDIA TESLA Drivers

CONFIG C – CUBE HVPC c16i-GPU

The INTEL® based CUBE HVPC w16i-GPU FEA Server:

• CPU: 2x INTEL XEON e5-2690 (2.9GHz 8 core)
• RAM: 128GB DDR3 1600 MHz ECC
• HDD: (os and apps): 300GB SATA III 10k WD Velociraptor
• HDD: (working directory): 3 x 600GB SATA III 10k WD
• OS: Linux 64-bit
• OTHER: ANSYS R14, latest NVIDIA TESLA Drivers

CUBE HVPC w12i-GPU

The INTEL® based CUBE HVPC w12i-GPU FEA Simulation Workstation:

• CPU: 2x INTEL XEON x5690 (3.47GHz 6 core)
• RAM: 144GB DDR3 1333Mhz ECC
• HDD: (os and apps): 256GB SSD SATA III
• HDD: (working directory): 4 x 600GB SAS2 15k – LSI RAID 0
• OS: Windows 7 Professional 64-bit
• OTHER: ANSYS R13

References

Happy 10th Birthday: The Focus

Don’t you hate it when you miss someone’s birthday?  I was looking up an old article in the Focus and noticed that the first issue was published on January, 13th, 2002.

Happy Belated Birthday!

It was sometime in 2001 that Rod Scholl pointed out that there was no good ANSYS newsletter out there.  People would do one or two issues then get busy and never put out any more, or only publish once in a while.  So we decided that we would not only do a newsletter, but that we would keep it up and publish  on a regular basis. The first issue came out as an HTML email on January 13th of 2002.

The First Issue of The Focus

And Rod was instrumental in keeping us on track and making sure that with it. Since then we published 74 actual newsletters before switching to a blog format in 2010.  Just before this one goes up on the blog, we will have published 59 The Focus articles.

Thank you to everyone who subscribes to The Focus and reads our postings, rates us, and sends us such great comments and questions.

Here is to 10 more years!

Files for Webinar: A POST26 Primer

We just finished our webinar for 4/12/2012 on the basics of the POST26 Time History Post Processor.  As promised, here are the files used for examples in the webinar, as well as the PowerPoint:

Tower-of-Test (ToT) test model, APDL:

Tower-of-Test (ToT) test model, ANSYS Mechanical with APDL Snippets:

tower-of-test-transient-2013.wbpz (note this is an R15 file.  See why here.)

PowerPoint Presentation:

A recording of this webinar will be available on the following site after 4/13/2012:

Click on  PADT ANSYS Webinar Series to see all recordings.

Some Revolutionary HPC Talk: 208 Cores+896GB < \$60k, GPU’s, & ANSYS Distributed

The last couple of weeks a bunch of stuff has gone on in the area of High Performance computing, or HPC, so we thought we would throw it all into one Focus posting and share some things we have learned, along with some advice, with the greater ANSYS user community.

There is a little bit of a revolution going on in both the FEA and CFD simulation side of things amongst users of ANSYS, Inc. products.  For a while now customers with large numbers of users and big nasty problems to solve have been buying lots of parallel licenses and big monster clusters. The size of problems that these firms, mostly turbomachinery and aerospace, have been growing and growing. And even more so for CFD jobs.   But also FEA for HFSS and ANSYS Mechanical/Mechanical APDL.  That is where the revolution started.

But where it is gaining momentum, where the greater impact is being seen on how simulation is used around the world, is with the smaller customers.  People with one to five seats of ANSYS products.  In the past they were happy with their two “included” Mechanical shared memory parallel tasks.  Or they might spring for 3 or 4 CFD parallel licenses.  But as 4, 6, and 8 core CPU chips become mainstream, even on the desktop, and as ANSYS delivers greater value from parallel, we are seeing more and more people invest in high performance computing. And they are getting a quick return on that investment.

Affordable High Value Hardware

208 Cores + 869 GB + 25 TB + Infiniband + Mobile Rack = \$58k = HOT DAMN!

Yes, this is a commercial for PADT’s CUBE machines.  (www.CUBE-HVPC.com) Even if you would rather be an ALGOR user than purchase hardware from a lowly ANSYS Chanel Partner, keep reading. Even if you would rather go to an ANSYS meeting at HQ in the winter than brave asking your IT department if you can buy a machine not made by a major computer manufacturer, keep reading.

Because what we do with CUBE hardware is something you can do yourself, or that you can pressure your name brand hardware provider into doing.

We just got a very large CFD benchmark job for a customer.  They want multiple runs on a piece of “rotating machinery” to generate performance curves.  Lots of runs, and each run can take up to 4 or 5 days on one of our 32 core machines.  So we put together a 208 core cluster.  And we maxed out the RAM on each one to get to just under 900 GB. Here are the details:

Cores: 208 Total
3 servers x48 2.3 GHz cores each server
2 servers x32 3.0 GHz cores each server
RAM: 896 GB
3 servers  x128GB DDR3 1333MHz ECC RAM each server
2 servers  x256GB DDR3 1600MHz ECC RAM each server
DATA Array:  ~25TB
Interconnect: 5 x Infiniband 40Gbps QDR
Infiniband Switch: 8 port 40Gbps QDR
Mobile Departmental Cluster Rack – 12U

All of this cost us around \$58,000 if you add up what we spent on various parts over the past year or so.  That much horsepower for \$58,000.  If you look at your hourly burden rate and the impact of schedule slip on project cost, spending \$60k on hardware has a quick payback.

You do need to purchase parallel licenses. But if you go with this type of hardware configuration what you save over a high-priced solution will go a long way towards paying for those licenses.  Even if you do spend \$150k on hardware, your payback with the hardware and the license is still pretty quick.

Now this is old hardware (six months to a year –  dinosaur bones).  How much would a mini-cluster departmental server cost now and what would it have inside:

Cores: 320 Total
5 servers x64 2.3 GHz cores each server
RAM: 2.56 TB
5 servers x512 GB DDR3 RAM each server
DATA Array: ~50TB
Interconnect: 5 x Infiniband 40Gbps QDR
Infiniband Switch: 8 port 40Gbps QDR
Mobile Departmental Cluster Rack – 12U

The cost?  around \$80,000.  That is \$250/core.  Now you need big problems to take advantage of that many cores.  If your problems are not that big, just drop the number of servers in the mini-cluster.  And drop the price proportionally.

Same if you are a Mechanical user.  The matrices in FEA just don’t scale in parallel like they do for CFD, so a 300+ core machine won’t be 300 times faster. It might even be slower than say 32 cores.  But the cost drop is the same.  See below for some numbers.

Bottom line, hardware cost is now in a range where you will see payback quickly in increased productivity.

GPU’s

NVIDIA’s Tesla GPU
We think they should have some 80’s era super model
draped over the front like those Ferrari posters we

For you Mechanical/Mechanical APDL users, let’s talk GPU’s.

We just got an NVIDIA TESLA C2075 GPU.  We are not done testing, but our basic results show that no matter how we couple it with cores and solvers, it speeds things up.  Anywhere from 3 times faster to 50% faster depending on the problem, shared vs. distributed memory, and how many cores we throw in with the GPU.

This is a fundamental problem with answering the question “How much faster?” because it depends a lot on the problem and the hardware. You need to try it on your problems on your hardware. But we feel comfortable in saying if you buy an HPC pack and run on 8 cores with a GPU, the GPU should double your speed relative to just running on 8 cores.  It could even run faster on some problems.

For some, that is a disappointment.  “The GPU has hundreds of processors, why isn’t it 10 or 50 times faster?”  Well, getting the problem broken up and running on all of those processors takes time.  But still, twice as fast for between \$2,000 to \$3,000 is a bargain. I don’t know what your burden rate is but it doesn’t take very many hours of saved run time to recover that cost.  And there is no additional license cost because you get a GPU license with an HPC pack.

Plus, at R14 the solver supports a GPU with distributed ANSYS, so even more improvements.  Add to that support for the unsymmetrical or damped Eigensolvers and general GPU performance increases at R14.

PADT’s advice? If you are running ANSYS Mechanical or Mechanical APDL get the HPC Pack and a GPU along with a 12 core machine with gobs of RAM (PADT’s 12 core AMD system with 64GB or RAM and 3TB of disk costs only \$6,300 without the GPU, \$8,500 with).  You can solve on 8 cores and play Angry Birds on the remaining 4.

Distributed ANSYS

For a long time many of users out there have avoided distributed ANSYS. It was hard to install, and unless you had the right problem you didn’t see much of a benefit for many of the early releases. Shared Memory Parallel, or SMP, was dirt easy – get an HPC license and tell it to run on 8 cores and go.

Well at R14 of ANSYS Mechanical APDL it is time to go distributed.  First off they make the install much easier.  To be honest, we found that this was the biggest deterrent for many small company users.

Second, at R14 a lot more things are supported in distributed ANSYS. This has been going on for some time so most of what people use is supported. At this release they added subspace eigensolving, TRANS, INFINI and PLANE121/230 elements (electrostatics), and SURF251/252.

Some “issues” have been fixed like restart robustness and you now have control on when and how multiple restart files are combined after the solve.

All and all, if you have R14, you are solving mechanical problems, and you have an HPC pack, you should be using distributed most of the time.

Conclusions

We get a ton of questions from people about what they should buy and how much.  And every situation is different. But for small and medium sized companies, the HPC revolution is here and everyone should be budgeting for taking advantage of HPC:

• At least one HPC pack
• Some new faster/cheaper multicore hardware (CUBE anyone?)
• A GPU.

STOP!  I know you were scrolling down to the comments section to write some tirade about how ANSYS, Inc overcharges for parallel, how it is on a moral equivalence with drowning puppies, and how much more reasonable competitor A, B, or C is with parallel costs.  Let me save you the time.

HPC delivers huge value.  Big productivity improvements.  And it does not write itself. It is not an enhancement to existing software.  Scores of developers are going into solver code and implementing complex logic that allows efficiency with older hardware, shared memory, distributed memory, and GPU’s. It has to be written, tested, fixed, tested again, and back and forth every release.  That value is real, and there is a cost associated with it.

And the competitors pricing model? The only thing they can do to differentiate themselves is charge nothing or very little. They have not put the effort or don’t have the expertise to deliver the kind of parallel performance that the ANSYS, Inc. solvers do.  So they give it away to compete.  Trust me, they are not being nice because they like you. They have the same business drivers as ANSYS, Inc.  They price the way they do because they did not incur as much cost and they know if they charged for it you would have no reason to use their solvers.

ANSYS users of the world unit!  Load your multicore hardware with HPC packs, feed it with a GPU, and join the revolution!

Changing Results Values in ANSYS Mechanical APDL–DNSOL and *VPUT

So it is Friday afternoon and that big, involved, deep-dive into some arcane ANSYS capability is still not written.  So, time for plan B and the list of not so involved but still somewhat arcane capabilities that we like to expose in The Focus.  At the top of that list right now is how to change the results in an ANSYS Mechanical APDL (MAPDL) solve.

One might ask why would you want to do this?  Well the most common usage is that you want to use APDL to calculate some result value and then display it in a plot.  Similarly, you may want to do some sort of calculation or post processing on MAPDL results but using and external piece of code, but still show the results in ANSYS.  Another common usage is to use MAPDL as a post processor for some external solver.

And, it turns out, it is pretty easy.  And, as you probably have learned by now if you use MAPDL a lot, there is more than one way to do it.

The “Database”

Before we get into how to do this, we need to talk about the “database” in MAPDL. If you read through the documentation on the commands we will use, it will talk about the database.  This is not the jobname.db file.  That is the db file.  The database refers to the representation of your model, including results and the currently active loads, in memory when you are running MAPDL.

When you do a RESUME command, MAPDL reads the DB file and stores the model, including geometry, mesh, loads, and run parameters, in memory.  When you do a SET command, it then adds the results and related information into memory.

So when we use the commands we will talk about next, you are changing what is in the database, not what is in the DB file on your disk.  And you are storing it temporarily.  Many different commands cause the database to go back to its original values.  So you need to be very careful in how you use these tools, and don’t assume that once you have used them, the changes are permanent.

DNSOL

The simplest way to do it is with the DNSOL command: DNSOL, NODE, Item, Comp, V1, V2, V3, V4, V5, V6

So, if you do a dnsol,32,u,x,3.145  the value in memory for deflection in the X direction will be changed to 3.145.  dnsol,32,s,x,1.1,2.2,3.3 will change the stress on node 32 in the X direction to 1.1, in the Y direction to 2.2, and in the Z direction to 3.3.

The second argument can also be a component, so you can assign a uniform result value to many nodes at the same time.

Here is an example, very simple, of a block where we set the deflection on the top nodes to 1 in.

`   1: finish`

`   2: /clear`

`   3: /prep7`

`   4: blc4,-2,-2,4,4,20`

`   5: et,1,185`

`   6: mptemp,1,70`

`   7: mpdata,ex,1,1,20e6`

`   8: mpdata,nuxy,1,1,.23`

`   9: mpdata,dens,1,1,.001`

`  10:  `

`  11: vmesh,all`

`  12: /view,1,1,1,1`

`  13: /vup,1,z`

`  14: /RGB,INDEX,100,100,100, 0`

`  15: /RGB,INDEX, 80, 80, 80,13`

`  16: /RGB,INDEX, 60, 60, 60,14`

`  17: /RGB,INDEX, 0, 0, 0,15`

`  18: /show,png`

`  19: eplot`

`  20:  `

`  21: nsel,s,loc,z,0`

`  22: cm,nbt,node`

`  23: d,all,all`

`  24: nsel,s,loc,z,20`

`  25: cm,ntp,node`

`  26: f,all,fx,10`

`  27: f,all,fy,12`

`  28: allsel`

`  29: save`

`  30:  `

`  31: /solu`

`  32: solve`

`  33: finish`

`  34: /post1`

`  35: plnsol,u,sum`

`  36:  `

`  37: dnsol,ntp,u,y,1`

`  38: plnsol,u,sum`

`  39:  `

`  40: /show,close`

The Fancy Mesh

The Normal Solution

Solution with 1” deflection DNSOL’d onto the top nodes

Pretty simple.

NOTE: One key thing to remember is you can not use this with Powergraphics. You must have /graph,full turned on.

*VPUT

The DNSOL is great for a few nodes here and there, or a constant value. But it makes for a big nasty *do loop if you want to do a lot of nodes. So ANSYS, Inc. give us the *VPUT command:

*VPUT, ParR, Entity, ENTNUM, Item1, IT1NUM, Item2, IT2NUM, KLOOP

As you can see, this command has a lot of options, so read the Help before you use it. Most of the time you have an array that stores the value you want stuck on your model in an array (ParR).  Then you specify what MAPDL result you want to overwrite and it takes care of it for you.

*vput,nws1(1),node,1,s,1 will place new values for maximum principal stress on all the nodes covered in the array , starting at 1.

Here is an example of the code, after the same solve as above, to do a *vput instead of a DNSOL:

`   1: finish`

`   2: /post1`

`   3: plnsol,u,sum`

`   4:  `

`   5: *get,nmnd,node,,num,max`

`   6: *dim,nwux,,nmnd`

`   7: *do,i,1,nmnd`

`   8:     nwux(i) =  i`

`   9: *enddo`

`  10:     `

`  11: *vput,nwux(1),node,1,u,x`

`  12: plnsol,u,sum`

The code places a displacement in the X direction equal to its node number.  So node 80 has a UX of 80.

A key thing to note with *vput is that it is much more transient. Pretty much any other command that involves anything other than plotting or listing blows away the values you assigned with *VPUT.  So we recommend that you do a *vput before every plot or listing you do.

Thoughts, Questions, and Conclusion

Of course you should never use this to fudge results.

You can get very fancy with this. When I use it I often turn off all the plot controls and then create my own legend. That way I can put hours of life on as temperature or SX and then make my own legend that says LIFE or something of the sort.

Another thing to note is that if you DNSOL or *VPUT a displacement, then ANSYS will distort your plot by that much.  That is OK if you are changing deflection, but not so good if you are plotting life or some esoteric stress value.

A common question when you play with these commands is if you can store the modified results in the RST file.  You can for degree of freedom results, but not for stresses.  You use LCDEF and RAPPND.

And what about ANSYS Mechanical?  Well it works totally different, and better. You can do all of this using Design Assessment, which was covered in a webinar and a Focus article you can find here.

The key to using these two commands is pretty much the same as any APDL command: Start on a small test model, write a macro to do it, and keep things simple. And, read the manual.  Everything you need to know is in there.

Utilizing Element Birth and Death for Contact Elements in Workbench Mechanical

In case you missed it, Doug Oatis wrote a Focus blog entry on general use of element birth and death in ANSYS Workbench back in November, 2011, which you can access here:

This new article narrows the focus to contact elements specifically.  We recently had a tech support question about how to utilize element birth and death for contact elements in ANSYS Workbench.  So, a simple example was put together and is explained below.

The main idea is that we need multiple load steps (labeled Steps in Workbench) in order for elements to change status from alive to dead or vice versa.  We also need a way to select the elements so that we can identify which ones will be killed or made alive.

Keep in mind that ANSYS Workbench Mechanical is a newer pre- and post-processor for good old ANSYS Mechanical APDL.  That means we can insert ANSYS commands into the object tree in Workbench Mechanical and those commands will be executed when the solver reads the batch input file that is created when we click the solve button.

So, we need at least one set of Mechanical APDL commands to identify which contact/target pairs or contact regions we need to kill or make alive.  In our example we’ll focus on killing elements but the same principal applies to making killed elements come alive.  Note that killing elements does not remove them from the model.  Rather, it reduces their stiffness by a default value of six orders of magnitude so that effectively they do not participate.  The Mechanical APDL commands needed are for the contact/target pair identification are scalar parameter commands.

ANSYS Workbench employs some ‘magic’ parameter names that automatically plug in the integer pointers used behind the scenes for identification of element types and material properties.  In the case of contact and target elements, these parameter names are ‘cid’ for the contact elements and ‘tid’ for the target elements.  Thus, for each contact region we want to be able to kill, we need to create unique scalar parameter names, such as:

mycont=cid
mytarg=tid

If we had more than one pair, we might use

mycont1=cid
mytarg1=tid

and increment the ‘1’ in the parameter names on the left side for each contact pair so that we end up with mycont1, mycont2, etc.

These commands need to be inserted directly under each desired contact region so that they will be located in the appropriate place in the solver batch input file at solution time.

The next command snippet needed is the one that selects the desired contact and target elements and then employs the ANSYS Mechanical APDL command to kill them.  Finally we need to re-select all the elements in the model so that they are all active when the solution takes place.  An example of this command object is:

esel,s,type,,mycont
esel,a,type,,mytarg

!kill selected elements (contact and target)
ekill,all

!select everything
allsel

Note that anything that occurs to the left of a “!” is considered a comment.  This second command object needs to be inserted under the analysis type branch.

Next, we need to tell ANSYS to perform at least 2 steps (load steps).  This is accomplished in the Details view for Analysis Settings.  For Step Controls, number of steps needs to be 2 (or more than 2).  Once 2 load steps are specified, we can tell ANSYS to only activate the EKILL command snippet for load step 2.  This is done in the Details view for the command snippet.  Step Selection Mode can be set to By Number and the Step Number set to 2, meaning that the command object will only be active for load step 2.  This will result in the contact elements that have been selected by the above commands being killed in load step 2.

In our example, we have two semicircular rings, connected by contact elements where they touch.  One side of the interface uses bonded contact, active for both loads steps.  The other side of the interface uses frictionless contact, active in load step 1 and killed in load step 2.  We would expect that under a compressive load, the frictionless contacts will prevent penetration in load step 1 but allow penetration in load step 2 since they have been killed.

That is exactly what the results show.  The contact status for the frictionless contact region goes from 2 (sliding) at the end of load step 1 to zero (far or not touching) at the end of load step 2.

Deformation plots indicate that penetration is prevented in load step 1.

In load step 2, penetration is allowed because the contact elements at this location have been killed.

So, here is a fairly simple Workbench Mechanical example of utilizing command objects to select contact and target elements, and to kill those elements using the Mechanical APDL EKILL command.  You can read up on element birth and death in the Mechanical APDL Help for more details on element birth and death.  We hope this is useful information to you.

Workshops for “Intro to Workbench Framework Scripting”

At noon Phoenix time today we will be presenting the Webinar “Intro to Workbench Scripting: Controlling Projects, Materials, and Solution Execution with Python”  This is a very high level, and probably short, introduction to the basics of using the python scripting in Workbench.

To support the talking, I’ve put together four workshops, mostly based on ANSYS material or examples we have presented before here on the Focus.  They should be enough for anyone that is a good programmer or better to customize the Workbench Framework.  We also present it here as a tool for those who don’t attend the webinar (and as this weeks Focus posting, cause we didn’t have time to write one…)

Warning: These were done in a hurry between meetings and some travel… so they are not great from a grammar or typo standpoint. Regardless, we hope you find them useful.

The files you need to run them are in this zip file:

The Workshop Document is here:

We hope to add to this document over the next year or so and provide it as a more or less complete tutorial for those who want to automate their analysis.

Starting ANSYS Products From the Command Line

Sometimes you just get tired of clicking on icons.  Sometimes you just need to feel the control and power of launching your applications from the command line.  You type it in, you hit the enter key, and sometimes you can actually hear the disk spin up or the fan run faster to cool the processor as the program you asked for, the program you took time to type out, leaps to life in front of you. Satisfaction.

OK, maybe not.  More often you are scripting some batch solves.  Or maybe you are using the graphical user interface in Workbench but you need to set options for the solvers you are running from within workbench.  Because most of the solvers in the ANSYS family of products predate such new-fangled concepts as GUI’s, and because they are often run remotely on other machines, they have command line interfaces. And that gives the knowledgeable user more power and control.

General Concepts for Launching from the Command Line

Although the number of options available changes from application to applications, there a few common things you should know.

Paths

The first and most important concept is to be aware of the path.  This is where most errors happen. One of the big changes over the years is that as software gets more complicated, the executable program or script that you use to launch a solver is now buried deep down inside a directory structure.  Since we never run in that actual directory we need to tell the operating system where the executable is. You can do this by including the full directory path in your command line argument, or by adding it to your path.

On Linux follow the directions in the help:

// Installation and Licensing Documentation // Linux Installation Guide // 5. Post-Installation Instructions

for each of the products you want to run. Generally, you need to set the PATH environmental variable in your .cshrc, .login, or .profile.

On Windows it is not documented, the assumption being that you will be clicking on icons and not typing into a command window.  So a little detective work is needed. Use a file explorer and the Linux documentation on launching to locate the executable for solvers you want to use:

// Installation and Licensing Documentation // Linux Installation Guide // 5. Post-Installation Instructions // 5.3. Launching ANSYS, Inc. Products

The /ansys_inc part is usually replaced with c:\Program Files\ANSYS Inc.  The rest of the path is pretty much the same, swapping forward slashes with backward slashes.  Use these paths in your command line or add to your path by:

• From the desktop, right-click My Computer and click Properties.
• In the System Properties window, click on the Advanced tab.
• In the Advanced section, click the Environment Variables button.
• Finally, in the Environment Variables window (as shown below), highlight the Path variable in the Systems Variable section and click the Edit button. Add or modify the path lines with the paths you wish the computer to access. Each different directory is separated with a semicolon as shown below.

Important note for Windows:  If you are typing the path in on the command line, you need to put it in double quotes.  The convention on Windows is to specify directories with spaces in the name.  But the convention is not to have a command line parser that recognizes this.  So you will get an error if you type:

C:\Program Files\ANSYS Inc\v140\ansys\bin\winx64\ansys140.exe

But if you put it in quotes, it works fine:

"C:\Program Files\ANSYS Inc\v140\ansys\bin\winx64\ansys140.exe"

Versions Numbering

If you look at the example for launching MAPDL above you will notice that 140 is used in the directory path and in the name of the executable.  This will change with version:  v130, v145, etc…  Just be aware of that if you are reading this blog posting in 3 years and we are on release 16.5.  you would use:

"C:\Program Files\ANSYS Inc\v165\ansys\bin\winx64\ansys165.exe"

Where do you Launch From?

You of course need a command line to launch a solver.  This is usually a window that lets you type operating system commands: called a Command Prompt in Windows or a shell on Linux.  On Linux it can be an xterm window, a console window, or some other terminal window you have opened.

But you can also launch from a script, and that script can be launched from a command prompt or shell, or it can be launched by an icon.  All that needs to happen is that the script needs to be executed with the environmental variables required for the command prompt/script or the GUI.  If you don’t know how to make that happen, contact your IT support or someone who understands your operating system and how it runs processes.

ANSYS Mechanical APDL

The solver with the most options and capabilities from the command line is Mechanical APDL. So we will start there.  It is important to know these even if you use Mechanical most of the time.  That is because you can set these, and better control your solves, under Tools->Options->Mechanical APDL.  Here is what that dialog looks like:

The most common settings have their own widgets,  and the others can all be accessed by using “- string” command line style arguments in the first text widget aptly named: Command Line Options.

Here are the options, grouped for your studying pleasure:

The ones that everyone should know about are: –p, –m, –db.  We find that not using these to define what license to use (-p) or to control how memory is pre-allocated (-m, –db) generate the most tech support questions.    The next most important is the –np command. Use this to define more processors if you have HPC licenses.

Cheating – Use the Launcher

Sometimes the options can get long and confusing.  So what I do is I use the “ANSYS Mechanical APDL Launcher” and fill in all the forms. then go to tools->View Display Command Line and I can see all the options.

Here is a a fancy command line that got generated that way:

"C:\Program Files\ANSYS Inc\v140\ANSYS\bin\winx64\ansys140.exe"  -g -p ane3flds -np 2 -acc nvidia -dir "c:\temp" -j "grgewrt1" -s read -m 5000 -db 1000 -l en-us -lstp1 32 -t -d win32 -custom "/temp/myansys.exe"

It runs interactive (-g), uses a multiphysics license (-p ane3flds), grabs two processors (-np 2), uses an NVIDEA GPU (-acc nvidia (I don’t have one…)), runs in my temp directory (-dir), uses a job name of grgewrt1 (-j), reads the start.ans file (-s), grabs 5000mb and 1000mb for memory (-m, –db), uses English, passes in a parameter called lstp1 and sets it to 32 (-), uses the win32 graphics driver (-d) and runs my custom ANSYS executable (-custom).

I have no idea what –t is.  Some undocumented option I guess…

ANSYS Workbench

• ANSYS Workbench also has some command line arguments. not as rich as what is available in MAPDL, but still powerful. It allows you to run Mechanical in Batch or Interactive mode, supply python commands as needed.  The key thing to remember is that the workbench interface is not Mechanical or FLUENT. It is the infrastructure that other programs run on. Scripting in workbench allows you to control material properties, parameters, and how systems are created and executed.

Here are the options:

 Argument Operation -B Run ANSYS Workbench in batch mode. In this mode, the user interface is not displayed and a console window is opened. The functionality of the console window is the same as the ANSYS Workbench Command Window. -R Replay the specified ANSYS Workbench script file on start-up. If specified in conjunction with –B, ANSYS Workbench will start in batch mode, execute the specified script, and shut down at the completion of script execution. -I Run ANSYS Workbench in interactive mode. This is typically the default, but if specified in conjunction with –B, both the user interface and console window are opened. -X Run ANSYS Workbench interactively and then exit upon completion of script execution. Typically used in conjunction with –R. -F Load the specified ANSYS Workbench project file on start-up. -E Execute the specified ANSYS Workbench scripting command on start-up. You can issue multiple commands, separated with a semicolon (;), or specify this argument multiple times and the commands will be executed in order.

The big deal in this list is the –B argument.  This allows you to run workbench, and applications controlled by the project page, in batch mode.  You will usually use the –R argument to specify the Iron Python script you want to run.  In most cases you will often want to throw in a –X to make sure it exists when the script is done.

Other ANSYS Products

Here is where it gets boring.  The other products just don’t have all those options. At least not documented.  So you simply find the executable and run it.  Here is the list for Linux. Use it to find the location on Windows.

 Product Command Mechanical APDL /ansys_inc/v140/ansys/bin/ansys140 ANSYS Workbench /ansys_inc/v140/Framework/bin//runwb2 ANSYS CFX /ansys_inc/v140/CFX/bin/cfx5 ANSYS FLUENT /ansys_inc/v140/fluent/bin/fluent ANSYS ICEM CFD /ansys_inc/v140/icemcfd//bin/icemcfd ANSYS POLYFLOW /ansys_inc/v140/polyflow/bin/polyman ANSYS CFD-Post /ansys_inc/v140/CFD-Post/bin/cfdpost ANSYS Icepak /ansys_inc/v140/Icepak/bin/icepak ANSYS TurboGrid /ansys_inc/v140/TurboGrid/bin/cfxtg ANSYS AUTODYN /ansys_inc/v140/autodyn/bin/autodyn140

APDL Math.  It is one of the most powerful, uber-user, deep down under the hood, wrapping your hands around the neck of what FEA is, capabilities in the ANSYS Mechanical APDL (MAPDL) solver.  And most users don’t even know it is there.  It kind of snuck in over time, with the developers adding more and more capability each release.  Now it gives you access that you needed custom FORTRAN code to get to in the past… or NASTRAN DMAP, which is really something none of us ever want to do.

There is a lot of capability in this tool. This posting is just going to cover the basics so that you know what the tool can be used for in case you need it in the future, and hopefully motivate some of you to take a long look at the help.

What is APDL Math?

It is an extension to the APDL command language that drives MAPDL.   Although it runs in a different workspace (chunk of memory in the ANSYS database) it talks to standard APDL by importing and exporting APLD arrays (vectors or matrices).  It consists, at R14, of 18 commands that can be executed at the /SOLU level at any time.  All of the commands start with a * character and look and act like standard APDL commands.

APDL Math is a tool for users to do two things: 1) get access to view, export or modify matrices and vectors created by the solver, and 2) to control import or modify matrices and vectors then solve them.  The most common uses we have seen is the exporting of a matrix from ANSYS for use in some other program, usually Matlab.  The other is working with sub-structure matrices.

The entire tool is documented in the Mechanical APDL section of the help under  // ANSYS Parametric Design Language Guide // 4. APDL Math.

The Commands

Below is a list of the APDL math commands.  As usual, you really need to read the manual entries to get the full functionality explained.  Just like parameters and arrays in APDL, the matrices and vectors in APDL Math use names to identify them.  Note that you have a set of commands to create the matrix/vector you want, which includes reading from a file or importing an APDL array.  then you have commands to do basic matrix/vector math like multiply, find dot products, and do Fast Fourier Transformations.  Then there are solver commands.

 Commands to create and delete matrices and vectors *DMAT, Matrix, Type, Method, Val1, Val2, Val3, Val4, Val5 Creates a dense matrix that is complex, double or integer. You can allocate it, resize an existing matrix, copy a matrix, or link to a portion of a matrix. You can also import from a file or an APDL variable. *SMAT, Matrix, Type, Method, Val1, Val2, Val3 Creates a sparse matrix. Double or Complex, copied or imported. *VEC, Vector, Type, Method, Val1, Val2, Val3, Val4 Creates a vector. Double, complex or integer. Similar arguments to *DMAT. *FREE, Name, Deletes a matrix or a solver object and frees its memory allocation. Important to remember to do. Commands to manipulate matrices *AXPY, vr, vi, M1, wr, wi, M2 Performs the matrix operation M2= v*M1 + w*M2. *DOT, Vector1, Vector2, Par_Real, Par_Imag Computes the dot (or inner) product of two vectors. *FFT, Type, InputData, OutputData, DIM1, DIM2, ResultFormat Computes the fast Fourier transformation of the specified matrix or vector. *INIT, Name, Method, Val1, Val2, Val3 Initializes a vector or dense matrix. Used to fill vectors or matrices with zero’s, constant values, random values, or values on the diaganol. *MULT, M1, T1, M2, T2, M3 Performs the matrix multiplication M3 = M1(T1)*M2(T2). *NRM, Name, NormType, ParR, Normalize Computes the norm of the specified vector or matrix. *COMP, Matrix, Algorithm, Threshold Compresses the columns of a matrix using a specified Singular value decomposition algorithm (default) or Modified Gram-Schmidt algorithm Commands to perform solutions *LSENGINE, Type, EngineName, Matrix, Option Creates a linear solver engine and assignes a name to be used when you want to execute the solve. Does Boeing sparse, MKL sparse, LAPACK or Distributed Sparse. *LSFACTOR, EngineName, Option Performs the numerical factorization of a linear solver system. *LSBAC, EngineName, RhsVector, SolVector Performs the solve (forward/backward substitution) of a factorized linear system. *ITENGINE, Type, EngineName, PrecondName, Matrix, RhsVector, SolVector, MaxIter, Toler Performs a solution using an iterative solver. *EIGEN, Kmatrix, Mmatrix, Cmatrix, Evals, Evects Performs a modal solution with unsymmetric or damping matrices. Commands to output matrices *EXPORT, Matrix, Format, Fname, Val1, Val2, Val3 Exports a matrix to a file in the specified format. Supports Matrix Market, ANSYS SUB, DMIG, Harwell-Boeing and ANSYS EMAT. Also used to put values in an APDL array or print in a formated way to a postscript file. *PRINT, Matrix, Fname Prints the non-zero matrix values to a text file. Useful APDL Commands /CLEAR, Read Wipes all APDL and APDL Math values from memory WRFULL, Ldstep Stops solution after assembling global matrices. Use this to make matrices you need when you don’t want a full solve /SYS, String Executes an operating system command. This can be used in APDL Math to do some sort of an external operation on a matrix you wrote out to a file, like running a matlab script. After execution the matrix can be read back in and used.

How You Use APDL Math

When using APDL math you should follow some basic steps.  I’m always a big proponent of the crawl-walk-run approach to anything, so I also recommend that you start with small, simple models to figure stuff out.

First Step: Know the Math

The first step, or maybe the zero’th, is to understand your math.  If you charge in and start grabbing the stiffness matrix from the .FULL file and changing values, who knows what you will end up with. Chart out the math you want to do on paper or in a tool like Mathematica.

Then makes sure that you understand the math in ANSYS, and that includes the files being used by ANSYS.  A good place to look is in the Programmer’s Manual, Chapter 1 lists the various files and what is in them.  It might also not be a bad idea for you to crack open the theory manual  We all know that ANSYS solve Kx = F, but how, and what matrices and vectors are used.  Section 17.1 explains the static analysis approach used, with lots of links to more detailed explanations.

Since the whole point of using APDL math is to do stuff with Matrices/Vectors, you need to start by creating them.  Note that we are not doing anything with APDL Math yet.  We are using APDL, ANSYS, or an external program to get our matrix/vector so that we can then get it into APDL Math.. There are three types of sources you can get matrices/vectors from:

1. Use APDL to create an array.  *DIM, *SET, *VREAD, *MOPER, etc…
2. Use ANSYS to make them as part of a solve, or as part of an “almost solve” using WRFULL.  You can read the .FULL, .EMAT, .SUB, .MODE or .RST
3. Get a file from some other source and put it into a format that APDL Math can read. It supports Harwell-Boeing, Matrix Market, or NASTRAN DMIG format as well as APDL Math’s own format.

Third Step: Get the Matrices/Vectors into APDL Math

Using *DMAT, *SMAT, and *VEC you convert the APDL array, ANSYS file, or external format file into a matrix or a vector.  You can also use *INIT to make one from scratch, filling it with constants, zeros, random numbers or by setting the diagonal or anti-diagonal  values.

Fourth Step: Manipulate the Matrices/Vectors

In this step you can modify matrices in a lot of different ways.  The simplest is to use *MULT or *AXPY to do math with several matrices/vectors to get a new matrix/vector.

Another simple way to change things is to simply refer to the entries in the matrix using APDL.  As an example, to change the damping at I=4 and J =5 in a damping matrix called dmpmtrx just use dmpmtrx(4,5) – 124.321e4.

You can take that one step further and use the APDL operators that work on arrays like *SET, *MOPER, *VGUN and whatever *DO loops you need.

If you can’t do the modification you need in APDL or APDL Math, then you can use *EXPORT to write the matrices out and use an external tool like matlab or Mathematica to do your manipulation. Of course you then use *DMAT with an IMPORT argument to read the modified matrix/vector back inside.

Fifth Step: Use the Matrix

Now it is finally time to use your matrices/vectors.  The most common use is to bundle it up in a substructure matrix (*EXPORT,,SUB) and use it in a solve.  What is great about this is that you can also (I know, we don’t want to use the N-word) save the file as a NASTRAN substructure and give it to that annoying DMAP guru who insists that NASTRAN is the only structural analysis code in the world.  He gets his file, and you get to use ANSYS.

You can also solve using APDL Math.  This can require multiple steps depending on what you want to do. A typical solve involves using the *LSENGINE command to define how you want to solve, then factor your matrix with *LSFACTOR, then solve with *LSBAK.

There are also ways to continue a solve in ANSYS after changing the EMAT file.  Unfortunately my plans to do an example of this have been thwarted and ANSYS did not provide one, so I’m not 100% sure on the steps required.  But a LSSOLVE that does not force ANSYS to recreate the matrices should work.  Maybe a topic for a future posting.

There is a lot more too this, and the help is where you can learn more.  But a few things everyone needs to be ware of are listed here.

DOF Order

A key area where people have problems is understanding how the DOF’s in your matrix or vector are ordered. This is because ANSYS reorders things for efficiency of memory and solve.  The nice thing is that ANSYS stores a map in the full file that you can use to convert back and forth using the *MULT command.

Please read section 4.4 ( // ANSYS Parametric Design Language Guide // 4. APDL Math // 4.4. Degree of Freedom Ordering) in the manual. They have a great description and some examples.

Just remember to remember to deal with DOF ordering.

Limitations

This is still a new tool set and as users apply it to real world problems, they are adding functionality.  Right now there are some limitations.

• The biggest is that that all of this works on linear matrices.  So you have to be working on a linear problem. Material or geometric non-linarites just don’t work.  This makes perfect sense but may be one of those things some users might not figure out till they have invested some serious time.
• You can not modify a sparse matrix in APDL Math.  You have to write it out using *EXPORT, modify it with something like matlab, then read it back in with *SMAT.
• *MULT can not be used to multiply two sparse matrices. One or both must be dense.  the result is always dense.
• The BCS, DSS and DSP solves only work with sparse matrices. Only the LAPACK solver works for a dense matrix.

Real and Imaginary

Most of the features in APDL math work with matrices that have imaginary terms in them. Be careful and read the help if you are using complex math, it can get tricky.  Especially read 4.3 on how to specify a position in a matrix for real or imaginary numbers.

Examples

There are not a lot out there.  The manual has some in section 4.7.  Take a look at these before you do anything. If you can not find a specific example, contact your technical support provider and see if they have one, or if they can ask development for one.

Give it a Shot, and Share

This is a new feature, and a power user feature.  So what is happening is some very smart people are doing some very cool things with it, and not sharing.  It is very important that you share your effort with your Channel Partner or ANSYS, Inc so that they can not only see how people use this tool, but also modify the tool to do even more cool things.

It would also be cool if you posted what you do on XANSYS for the masses to see.  Very cool.

Dean Kamen Visits Phoenix

Inventor of medical devices, the man behind the Segway, and FIRST backer Dean Kamenvisited the Phoenix area yesterday and today and we were lucky enough to have people from PADT invited to two different events at which he spoke.  For engineers involved in product development, this is like a visit from an NFL quarterback for most people.  He turned out to be open, engaging, and a very good speaker.

We could go on in adoration and explore the guilt and envy we feel after seeing all that he has done.  Instead we thought we would highlight two things we learned from his visit:

1. The FIRST program that he started and still heads is making a huge difference in this country and around the world.  PADT has been peripherally involved, focusing instead more on the underwater robot scholastic competitions that are very popular here in Phoenix.  But FIRST is now huge, and is still growing.  But what we learned is the positive impact it is having: Students who participate in FIRST are 3 times more likely to become engineers, 30% more likely to attend college, and twice as likely to volunteer in their communities. Those are some positive numbers.  Those of us in the engineering world should take advantage of that and support FIRST.

2. Second, he offered a unique perspective on how engineers see the world.  When he was young he heard the story of David and Goliath.   Most people see a religious message in this story, there are various interpretations. But as a child, Dean Kamen did not see those messages.  What he saw was that David won because he had better technology. He had a sling shot.  That is how he beat the giant.  I found that a very interesting point of view. If you don’t get it, ask an engineer.

If you ever have the chance to explore what his company, DEKA, is currently doing with a revolutionary power generation and water purification solution for areas of the globe without power or clean water, do so.  It is very leading edge stirling engine and distillation technology

ANSYS Training Face to Face

This weeks Focus posting is not going to be very technical. In fact, it is a bit of an editorial.

Over the past five years or so we have seen a lot of companies who use ANSYS, Inc products move away from traditional face-to-face training with instructors in a classroom.  There are a lot of reasons for this.  The two most common are that 1) the company does not have a travel budget and 2) that training labor hours are considered overhead and managers all have very strict overhead restrictions.  On top of these two, many companies are just plain trying to save money on the cost of training or limiting their overall training budgets.

What we have seen is a larger number of users either trying to train themselves from manuals or downloaded training material, or people trying to do web-based training.  One can certainly learn how to use an FEA or CFD tool this way, but through our tech support we are starting to see the negative side of this shift: users only understand some of the aspects of the tools and do not have a depth of knowledge that goes beyond the basics.  So when they run into a problem that requires a move beyond those basics, or that might require a more nuanced approach, they struggle or they call tech support for on-the-spot training.

Interaction

Even though we engineers are not the most social sub-species of humans, we can still heavily benefit from face-to-face interaction during training.  When PADT teaches a training class we find that a small portion of the time is spent lecturing on and doing workshops for the basics.  Most of the time is spent answering questions that occur to students while they take these basics in.  Some are industry or user specific, some delve deeper into the tool than the training material does.  But they all provide an education to the whole class that never occurs otherwise.

We have taught, and been students in, web based training classes. The interaction is just not the same.  There are not as many questions and the instructor is not able to use body language clues to see if the class is really getting what they are saying.  In fact, we feel this is the biggest issue. When you are on the phone and sharing a screen you can not even tell if the students are listening.  So the instructor pushes on, the students drift further away, and the true benefits of the class are lost.

Make a Case for Classroom Training

The point of all of this is that we feel users out there need to make a case for real classroom training.  When your boss says that there is no travel budget, not enough overhead allocation, or just not enough money, argue strongly that the cost differences of online or self training are not that significant when compared to the productivity problem of not having deep, interactive training.  If you are a boss, admit it, you know we are right.  You should fight a bit harder for the budget because in the long run you will save money.

Another way to look at it is the relative cost of classroom training versus how much you will use the ANSYS tools you are trained on.  Even if we assume that the company you work for kind of sucks and most engineers move out of there in five years, one to three week of training is nothing when compared to five years as a user.  If your productivity is just 5% higher during that time, the savings are significant.

Do the full classroom training.  You will not regret it.

As a full disclosure:
We are partly motivated to express this opinion by the fact that we make money doing such training classes, but in reality very few of you reading this will do training with us (although you could use us if you wanted to… hint, hint).  Most of you do your training through other Channel Partners or ANSYS, Inc.  So this posting is not entirely self serving.