Caps and Limits on Hardware Resources in Microsoft Windows and Red Hat Enterprise Linux

windows-caps(Revised and updated February 10, 2014 to include pertinent, relevant Windows Server 2012 information as it relates to the world of numerical simulation)

Hi – One of our more popular blog articles from January 14, 2011. It has been over three years now and the blog article needs a refresh. It seems that as operating system provider’s release a new OS iteration, for Windows Operating System or Linux, that this may contribute to confusion when selecting the proper licensing for the numerical simulation computers physical hardware.

Hopefully this updated blog article will assist you in making sure your numerical simulation machines are licensed properly.

Sometime around 3am in October 2010. I found myself beating my head up against a server rack. I was frustrated with trying to figure out what was limiting my server hardware. I was aware of a couple limits that Microsoft had placed into its OS software. However, I had no idea how far reaching the limits were. I researched into two manufactures of two of the most used Operating Systems on the planet. I figured it would be best if I had a better understanding of these hardware limits. The physical socket and memory limit caps that are placed on the hardware by two of the most popular Operating Systems on the planet: Microsoft Windows 7, Windows Server 2008 R2, Windows Server 2012 and Red Hat Enterprise Linux.

So now let us fast-forward over three years, not much has changed because change is constant. The new Windows Server 2012 changes up the naming convention on us IT geeks. So pay attention because the Windows Server Standard or Enterprise edition you may have been used to has changed.

Limits on Cores, RAM, and USERS by Operating System

  • Microsoft Windows Operating Systems
    • Windows 7
      • Professional / Enterprise / Ultimate
        • Processor: 2 Socket limit (many cores)
        • Core limits:
          • 64-bit: 256 max quantity of cores in 1 physical processor
          • 32-bit: 32 cores max quantity of cores in 1 physical processor
        • RAM: 192 GB limit to amount of accessible
      • Home Premium
        • RAM: 16GB
      • Home Basic
        • RAM: 8GB
      • Starter Edition
        • RAM: 2 GB
    • Windows Server 2008
      • Standard & R2
        • Processor: 4 socket limit – (many cores)
          • (4 – Parts x 12core) = 48 cores
        • RAM: 32 GB
      • Windows Server 2008 R2 Foundation  (R2 releases are 64-bit only)
        • RAM: 128 GB
      • HPC Edition 2008 R2 (R2 releases are 64-bit only)
        • RAM: 128 GB
      • Windows Server 2008 R2 Datacenter (R2 releases are 64-bit only)
        • Processor: 8 socket limit
        • RAM: 2TB
      • Windows Server 2008 R2 Enterprise (R2 releases are 64-bit only)
        • Processor: 8 socket limit
        • RAM: 2TB
    • Windows Server 2012
      • Foundation
        • Processor: 1 socket licensed – (many cores)
        • RAM: 32 GB
        • User Limit: 15 users
      • Essentials
        • Processor: 2 socket licensed – (many cores)
        • RAM: 64 GB
        • User Limit: 25 users
      • Standard
        • Processor:  4 socket licensed* – (many cores)
        • RAM: 4TB
        • User Limit: unlimited
      • Datacenter
        • Processor: 4 socket licensed* – (many cores)
        • RAM: 4TB
        • User Limit: unlimited
      • R2
        • Processor: 4 socket licensed* – (many cores)
        • RAM: 4TB
        • User Limit: unlimited
  • Red Hat Enterprise Linux – 64-bit
    • Red Hat defines a logical CPU as any schedulable entity. So every core/thread in a multi-core/thread processor is a logical CPU
    • This information is by Product default.  Not the maximums of a fully licensed/subscribed REHL product.
    • Desktop
      • Processor: 1-2 CPU
      • RAM: 64 GB
    • Basic
      • Processor: 1-2 CPU
      • RAM: 16 GB
    • Enterprise
      • Processor: 1-8 CPU
      • RAM: 64 GB
    • NOTE: Red Hat would be happy to create custom subscriptions with yearly fees for other configurations to fit your specific environment. Please contact Red Hat to check on costs.

Okay great but what operating system platforms can I use with ANSYS R15?

ANSYS 15.0 Supported Platforms

ANSYS 15.0 is the currently released version. The specific operating system versions supported by ANSYS 15.0 products and License Manager are documented and posted at: 
   www.ansys.com/Support/Platform+Support.

ANSYS 15.0 includes support for the following:

  • Windows XP and Windows 7 (32-bit and 64-bit Professional and Enterprise versions)
  • Windows 8 (64-bit Professional and Enterprise versions)
  • Windows Server 2008 R2 Enterprise
  • Windows HPC Server 2008 R2 (64-bit)
  • Windows Server 2012 Standard version
  • Red Hat Enterprise Linux (RHEL) 5.7-5.9 and 6.2-6.4 (64-bit)
  • SUSE Enterprise Linux Server and Desktop (SLES / SLED) 11 SP1-SP2 (64-bit)

Not all applications are supported on all of these platforms. See detailed information, by product, at the URL noted above.

Final Thoughts

Approximate additional licensing cost to License Windows Server 2012 for a Quad Socket CPU motherboard:

  • Windows Server 2012 Foundation: Please call your OEM partner
  • Windows Server 2012 Essentials: $429 + User Client Access Licensing $$$
  • Windows Server 2012 Standard:  $ 1,500  + User Client Access Licensing $$$
  • Windows Server 2012 Datacenter: $ 10,500 + User Client Access Licensing $$$

References

 

Help! My New HPC System is not High Performance!

It is an all too common feeling, that sinking feeling that leads to the phrase “Oh Crap” being muttered under your breath. You just spent almost a year getting management to pay for a new compute workstation, server or cluster. You did the ROI and showed an eight-month payback because of how much faster your team’s runs will be. But now you have the benchmark data on real models, and they are not good. “Oh Crap”

Although a frequent problem, and the root causes are often the same, the solutions can very. In this posting I will try and share with you what our IT and ANSYS technical support staff here at PADT have learned.

Hopefully this article can help you learn what to do to avoid or circumvent any future or current pitfalls if you order an HPC system. PADT loves numerical simulation, we have been doing this for twenty years now. We enjoy helping, and if you are stuck in this situation let us know.

Wall Clock Time

It is very easy to get excited about clock speeds, bus bandwidth, and disk access latency. But if you are solving large FEA or CFD models you really only care about one thing. Wall Clock Time. We cannot tell you how many times we have worked with customers, hardware vendors, and sometimes developers, who get all wrapped up in the optimization of one little aspect of the solving process. The problem with this is that high performance computing is about working in a system, and the system is only as good as its weakest link.

We see people spend thousands on disk drives and high speed disk controllers but come to discover that their solves are CPU bound, adding better disk drives makes no difference. We also see people blow their budget on the very best CPU’s but don’t invest in enough memory to solve their problems in-core. This often happens because when they look at benchmark data they look at one small portion and maximize that measurement, when that measurement often doesn’t really matter.

The fundamental thing that you need to keep in mind while ordering or fixing an HPC system for numerical simulation is this: all that matters is how long it takes in the real world from when you click “Solve” till your job is finished. I bring this up first because it is so fundamental, and so often ignored.

The Causes

As mentioned above, an HPC server or cluster is a system made up of hardware, software, and people who support it. And it is only as good as its weakest link. The key to designing or fixing your HPC system is to look at it as a system, find the weakest links, and improve that links performance. (OK, who remembers the “Weakest Link” lady? You know you kind of miss her…)

In our experience we have found that the cause for most poorly performing systems can be grouped into one of these categories:

  • Unbalanced System for the Problems Being Solved:

    One of the components in the system cannot keep up with the others. This can be hardware or software. More often than not it is the hardware being used. Let’s take a quick look at several gotchas in a misconfigured numerical simulation machine.

  • I/O is a Bottleneck
    Number crunching, memory, and storage are only as fast as the devices that transfer data between them.
  • Configured Wrong

    Out of simple lack of experience the wrong hardware is used, the OS settings are wrong, or drivers are not configured properly.

  • Unnecessary Stuff Added out of Fear

    People tend to overcompensate out of fear that something bad might happen, so they burden a system with software and redundant hardware to avoid a one in a hundred chance of failure, and slow down the other ninety-nine runs in the process.

Avoiding an Expensive Medium Performance Computing (MPC) System

The key to avoiding these situations is to work with an expert who knows the hardware AND the software, or become that expert yourself. That starts with reading the ANSYS documentation, which is fairly complete and detailed.

Often times your hardware provider will present themselves as the expert, and their heart may be in the right place. But only a handful of hardware providers really understand HPC for simulation. Most simply try and sell you the “best” configuration you can afford and don’t understand the causes of poor performance listed above. More often than we like, they sell a system that is great for databases, web serving, or virtual machines. That is not what you need.

A true numerical simulation hardware or software expert should ask you questions about the following, if they don’t, you should move on:

  • What solver will you use the most?
  • What is more important, cost or performance? Or better: Where do you want to be on the cost vs. performance curve?
  • How much scratch space do you need during a solve? How much storage do you need for the files you keep from a run?
  • How will you be accessing the systems, sending data back and forth, and managing your runs?

Another good test of an expert is if you have both FEA and CFD needs, they should not recommend a single system for you. You may be constrained by budget, but an expert should know the difference between the two solvers vis-à-vis HPC and design separate solutions for each.

If they push virtual machines on you, show them the door.

The next thing you should do is step back and take the advice of writing instructors. Start cutting stuff. (I know, if you have read my blog posts for a while, you know I’m not practicing what I preach. But you should see the first drafts…) You really don’t need huge costly UPS’, the expensive archival backup system, or some arctic chill bubbling liquid nitrogen cooling system. Think of it as a race car, if it doesn’t make the car go faster or keep the driver safe, you don’t need it.

A hard but important step in cutting things down to the basics is to try and let go of the emotional aspect. It is in many ways like picking out a car and the truth is, the red paint job doesn’t make it go any faster, and the fancy tail pipes will look good, but also don’t help. Don’t design for the worst-case model either. If 90% of your models run in 32GB or RAM, don’t do a 128GB system for that one run you need to do a year that is that big. Suffer a slow solve on that one and use the money to get a faster CPU, a better disk array, or maybe a second box.

Pull back, be an engineer, and just get what you need. Tape robots look cool, blinky lights and flashy plastic case covers even cooler. Do you really need that? Most of time the numerical simulation cruncher is locked up in a cold dark room. Having an intern move data to USB drives once a month may be a more practical solution.

Another aspect of cutting back is dealing with that fear thing. The most common mistake we see is people using RAID configurations for storing redundant data, not read/write speed. Turn off that redundant writing and dump across as many drives as you can in parallel, RAID 0. Yes you may lose a drive. Yes that means you lose a run. But if that happens once every six months, which is very unlikely, the lost productivity from those lost runs is small compared to the lost productivity of solving all those other runs on a slow disk array.

Intel-AMD-Flunet-Part2-Chart2Lastly, benchmark. This is obvious but often hard to do right. The key is to find real problems that represent a spectrum of the runs you plan on doing. Often different runs, even within the same solver, have different HPC needs. It is a good idea to understand which are more common and bias your design to those. Do not benchmark with standard benchmarks, use industry accepted benchmarks for numerical simulation. Yes it’s an amazing feeling knowing that your new cluster is number 500 on the Top 500 list. However if it is number 5000 on the ANSYS Numerical simulation benchmark list nobody wins.

Fixing the System You Have

As of late we have started tearing down clusters in numerous companies around the US. Of course we would love to sell you new hardware however at PADT, as mentioned before, we love numerical simulation. Fixing your current system may allow you to stretch that investment another year or more. As a co-owner of a twenty year old company, this makes me feel good about that initial investment. When we sick our IT team on extending the life of one of our systems, I start thinking about and planning for that next $150k investment we will need to do in a year or more.

Breathing new life into your existing hardware basically requires almost the same steps as avoiding a bad system in the first place. PADT has sent our team around the country helping companies breath new life into their existing infrastructure. The steps they use are the same but instead of designing stuff, we change things. Work with an expert, start cutting stuff out, breath new life into the growing old hardware, avoid fear and “cool factor” based choices, and verify everything.

Take a look and understand the output from your solvers, there is a lot of data in there. As an example, here is an article we wrote describing some of those hidden gems within your numerical simulation outputs. http://www.padtinc.com/blog/the-focus/ansys-mechanical-io-bound-cpu-bound

Play with things, see what helps and what hurts. It may be time to bring in an outside expert to look at things with fresh eyes.

Do not be afraid to push back against what IT is suggesting, unless you are very fortunate, they probably don’t have the same understanding as you do when it comes to numerical simulation computing. They care about security and minimizing the cost of maintaining systems. They may not be risk takers and they don’t like non-standard solutions. All of these can often result in a system that is configured for IT, and not fast numerical simulation solves. You may have to bring in senior management to solve this issue.

PADT is Here to Help

Cube_Logo_Target1The easiest way to avoid all of this is to simply purchase your HPC hardware from PADT.  We know simulation, we know HPC, and we can translate between engineers and IT.  This is simply because simulation is what we do, and have done since 1994.   We can configure the right system to meet your needs, at that point on the price performance curve you want.  Our CUBE systems also come preloaded and tested with your simulation software, so you don’t have to worry about getting things to work once the hardware shows up.

If you already have a system or are locked in to a provider, we are still here to help.  Our system architects can consult over the phone or in person, bringing their expertise to the table on fixing existing systems or spec’ing new ones.  In fact, the idea for this article came when our IT manager was reconfiguring a customer’s “name brand” cluster here in Phoenix, and he got a call from a user in the Midwest that had the exact same problem.  Lots of expensive hardware, and disappointing performance. They both had the wrong hardware for their problems, system bottlenecks, and configuration issues.

Learn more on our HPC Server and Cluster Performance Tuning page, or by contacting us. We would love to help out. It is what we like to do and we are good at it.

I’m All Bound Up! : A Brief Discussion on What It Means To Be Compute Bound or I/O Bound

CUBE-HVPC-512-core-closeup2-1000hWe often get questions from our customers, both ANSYS product and CUBE HVPC users, on how to get their jobs to run faster. Should they get better disk drives or focus on CPU performance. We have found that disk drive performance often gets the blame when it is undeserving. To help figure this out, the first thing we do t is look at the output from an ANSYS Mechanical/Mechanical APDL run. Here is an email, slightly modified to leave the user anonymous, that shows our most recent case of this:

From: David Mastel – PADT, Inc.
To: John Engineering

Subject: Re: Re: Re: Relatively no difference between SSD vs. SAS2 15k RPM solve times?

Hi John, so I took a look at your ANSYS Mechanical output files – Based on the problem you are running the machine is Compute Bound. Here is the process on how I came to that conclusion. Additionally, at the end of this email I have included a few recommendations.

All the best,
David

Example 1:

The bottom section of an ANSYS out file for a 2 x 240GB
Samsung 843 SSD RAID0 array:

Total CPU time for main thread                    :      105.9 seconds
Total CPU time summed for all threads             :      119.1 seconds

Elapsed time spent pre-processing model (/PREP7)  :        0.0 seconds
Elapsed time spent solution – preprocessing       :       10.3 seconds
Elapsed time spent computing solution             :       83.5 seconds
Elapsed time spent solution – postprocessing      :        3.9 seconds
Elapsed time spent post-processing model (/POST1) :        0.0 seconds

Equation solver computational rate                :   319444.9 Mflops
Equation solver effective I/O rate                :    26540.1 MB/sec

Maximum total memory used                         :    48999.0 MB
Maximum total memory allocated                    :    54896.0 MB
Maximum total memory available                    :        128 GB

+—— E N D   D I S T R I B U T E D   A N S Y S   S T A T I S T I C S ——-+

*—————————————————————————*
|                                                                           |
|                       DISTRIBUTED ANSYS RUN COMPLETED                     |
|                                                                           |
|—————————————————————————|
|                                                                           |
|            Release 14.5.7         UP20130316         WINDOWS x64          |
|                                                                           |
|—————————————————————————|
|                                                                           |
| Database Requested(-db)   512 MB    Scratch Memory Requested       512 MB |
| Maximum Database Used     447 MB    Maximum Scratch Memory Used   4523 MB |
|                                                                           |
|—————————————————————————|
|                                                                           |
|        CP Time      (sec) =        119.01       Time  =  15:41:54         |
|        Elapsed Time (sec) =        117.000       Date  =  10/21/2013      |
|                                                                           |
*—————————————————————————*

For a quick refresher on what it means to be compute bound or I/O bound, let’s review what ANSYS Mechanical APDL tells you.

When looking at your ANSYS Mechanical APDL (this file is created during the solve in ANSYS Mechanical, since ANSYS Mechanical is just running ANSYS Mechanical APDL behind the scenes) out files; I/O bound and Compute bound are essentially the following:

I/O Bound:

  1. When Elapsed Time greater than Main Thread CPU time

Compute Bound:

  1. When Elapsed time is equals (approx) to Main Thread CPU time

Example 2:
CUBE HVPC – Samsung 843 – 240GB SATA III SSD 6Gbps – RAID 0
Total CPU time for main thread :  105.9  seconds
Elapsed Time (sec) :   117.000        seconds

CUBE HVPC – Hitachi 600GB SAS2 15k RPM – RAID 0
Total CPU time for main thread  :  109.0 seconds
Elapsed Time (sec) :   120.000       seconds

Recommendations for a CPU compute bound ANSYS server or workstation:

When computers are compute bound I normally recommend the following.

  1. Add faster processors – check!
  2. Use more cores for solve – I think you are in the process of doing this now?
  3. Instead of running SMP go DMP – unable to use DMP with your solve
  4. Add an Accelerator card (NVidia Tesla K20x). Which unfortunately does not help in your particular solving situation.

Please let me know if you need any further information:

David

David Mastel
Manager, Information Technology

Phoenix Analysis & Design Technologies
7755 S. Research Dr, Suite 110
Tempe, AZ  85284
David.Mastel@PADTINC.com

The hardware we have available changes every couple of months or so, but right now, for a user who is running this type of ANSYS Mechanical/Mechanical APDL model, we are recommending the following configuration:

CUBE HVPC Recommended Workstation for ANSYS Mechanical: CUBE HVPC w16-KGPU

CUBE HVPC
MODEL

COST

CHASIS

PROCESSOR

CORES

MEMORY

CUBE HVPC
w16i-kgpu

$ 16,164.00

Mid-Tower
(black quiet edition)

Dual Socket INTEL XEON
e5-2687 v2,
16 cores
@ 3.4GHz

16 = 2 x 8

128GB DDR3-1866 ECC REG

STORAGE

RAID
CONTROLLER

GRAPHICS

ACCELERATOR

OS

OTHER

4 x 240GB
SATA II SSD 6 Gbps
2 x 600GB
SAS2 6Gbps
(2.1 TB)

SMC LSI 2208
6Gbps

NVIDIA
QUADRO K5000

NVIDIA
TESLA K20

Microsoft
Windows 7
Professional 64-bit

ANSYS R14.5.7
ANSYS R15

Cube_logo_Trg_Wide_150w

Here are some references to the some basic information and the systems we recommend:
http://www.intel.com
http://en.wikipedia.org/wiki/CPU_bound
http://en.wikipedia.org/wiki/I/O_bound
http://www.supermicro.com
http://www.cube-hvpc.com
http://www.ansys.com
http://www.nvidia.com

ANSYS & 3D Printing: Converting your ANSYS Mechanical or MAPDL Model into an STL File

image3D printing is all the rage these days.  PADT has been involved in what should be called Additive Manufacturing since our founding twenty years ago.  So people in the ANSYS world often come to us for advice on things 3D Printer’ish.  And last week we got an email asking if we had a way to convert a deformed mesh into a STL file that can be used to print that deformed geometry.  This email caused neurons to fire that had not fired in some time. I remembered writing something but it was a long time ago.

Fortunately I have Google Desktop on my computer so I searched for ans2stl, knowing that I always called my translators ans2nnn of some kind. There it was.  Last updated in 2001, written in maybe 1995. C.  I guess I shouldn’t complain, it could have been FORTRAN. The notes say that the program has been successfully tested on Windows NT. That was a long time ago.

So I dusted it off and present it here as a way to get results from your ANSYS Mechanical or ANSYS Mechanical APDL model as a deformed STL file.

UPDATE – 7/8/2014

Since this article was written, we have done some more work with STL files. This Macro works fine on a tetrahedral mesh, but if you have hex elements, it won’t work – it assumes triangles on the face.  It also requires a macro and some ‘C’ code, which is an extra pain. So we wrote a more generic macro that works with Hex or Tet meshes, and writes the file directly. It can be a bit slow but no annoyingly slow.  We recommend you use this method instead of the ones outlined below.

Here is the macro:  writstl.zip

The Process

An STL file is basically a faceted representation of geometry. Triangles on the surface of your model. So to get an STL file of an FEA model, you simply need to generate triangles on your mesh face, write them out to a file, and convert them to an STL format.  If you want deformed geometry, simply use the UPGEOM command to move your nodes to the deformed position.

The Program

Here is the source code for the windows version of the program:

/*
---------------------------------------------------------------------------

 PADT--------------------------------------------------- Phoenix Analysis &
                                                        Design Technologies

---------------------------------------------------------------------------
                             www.padtinc.com
---------------------------------------------------------------------------

       Package: ans2stl

          File: ans2stl.c
          Args: rootname
        Author: Eric Miller, PADT
		(480) 813-4884 
		eric.miller@padtinc.com

	Simple program that takes the nodes and elements from the
	surface of an ANSYS FE model and converts it to a binary
	STL file.

	USAGE:
		Create and ANSYS surface mesh one of two ways:
			1: amesh the surface with triangles
			2: esurf an existing mesh with triangles
         	Write the triangle surface mesh out with nwrite/ewrite
		Run ans2stl with the rootname of the *.node and *.elem files
		   as the only argument
		This should create a binary STL file

	ASSUMPTIONS:
		The ANSYS elements are 4 noded shells (MESH200 is suggested)
		in triangular format (nodes 3 and 4 the same)

		This code has been succesfully compiled and tested
		on WindowsNT

		NOTE: There is a known issue on UNIX with byte order
				Please contact me if you need a UNIX version

	COMPILE:
		gcc -o ans2stl_win ans2stl_win.c

       10/31/01:       Cleaned up for release to XANSYS and such
       1/13/2014:	Yikes, its been 12+ years. A little update 
       			and publish on The Focus blog
			Checked it to see if it works with Windows 7. 
			It still compiles with GCC just fine.

---------------------------------------------------------------------------
PADT, Inc. provides this software to the general public as a curtesy.
Neither the company or its employees are responsible for the use or
accuracy of this software.  In short, it is free, and you get what
you pay for.
---------------------------------------------------------------------------
*/
/*======================================================

   SAMPLE ANSYS INPUT DECK THAT SHOWS USAGE

finish
/clear
/file,a2stest
/PREP7  
!----------
! Build silly geometry
BLC4,-0.6,0.35,1,-0.75,0.55 
SPH4,-0.8,-0.4,0.45 
CON4,-0.15,-0.55,0.05,0.35,0.55 
VADD,all
!------------------------
! Mesh surface with non-solved (MESH200) triangles
et,1,200,4
MSHAPE,1,2D   ! Use triangles for Areas
MSHKEY,0      ! Free mesh
SMRTSIZE,,,,,5
AMESH,all
!----------------------
! Write out nodes and elements
nwrite,a2stest,node
ewrite,a2stest,elem
!--------------------
! Execute the ans2stl program
/sys,ans2stl_win.exe a2stest

======================================================= */

#include 
#include 
#include 

typedef struct vertStruct *vert;
typedef struct facetStruct *facets;
typedef struct facetListStruct *facetList;

        int     ie[8][999999];
        float   coord[3][999999];
        int	np[999999];

struct vertStruct {
  float	x,y,z;
  float	nx,ny,nz;
  int  ivrt;
  facetList	firstFacet;
};

struct facetListStruct {
  facets	facet;
  facetList	next;
};

struct facetStruct {
  float	xn,yn,zn;
  vert	v1,v2,v3;
};

facets	theFacets;
vert	theVerts;

char	stlInpFile[80];
float	xmin,xmax,ymin,ymax,zmin,zmax;
float   ftrAngle;
int	nf,nv;  

void swapit();
void readBin();
void getnorm();
long readnodes();
long readelems();

/*--------------------------------*/
main(argc,argv)
     int argc;
     char *argv[];
{
  char nfname[255];
  char efname[255];
  char sfname[255];
  char s4[4];
  FILE	*sfile;
  int	nnode,nelem,i,i1,i2,i3;
  float	xn,yn,zn;

  if(argc <= 1){
        puts("Usage:  ans2stl file_root");
        exit(1);
  }
  sprintf(nfname,"%s.node",argv[1]);
  sprintf(efname,"%s.elem",argv[1]);
  sprintf(sfname,"%s.stl",argv[1]);

  nnode = readnodes(nfname);
  nelem = readelems(efname);
  nf = nelem;

  sfile = fopen(sfname,"wb");
  fwrite("PADT STL File, Solid Binary",80,1,sfile);
  swapit(&nelem,s4);    fwrite(s4,4,1,sfile);

  for(i=0;i<nelem;i++){ 
      i1 = np[ie[0][i]];
      i2 = np[ie[1][i]];
      i3 = np[ie[2][i]];
      getnorm(&xn,&yn,&zn,i1,i2,i3);

      swapit(&xn,s4);	fwrite(s4,4,1,sfile);
      swapit(&yn,s4);	fwrite(s4,4,1,sfile);
      swapit(&zn,s4);	fwrite(s4,4,1,sfile);

      swapit(&coord[0][i1],s4);	fwrite(s4,4,1,sfile);
      swapit(&coord[1][i1],s4);	fwrite(s4,4,1,sfile);
      swapit(&coord[2][i1],s4);	fwrite(s4,4,1,sfile);

      swapit(&coord[0][i2],s4);	fwrite(s4,4,1,sfile);
      swapit(&coord[1][i2],s4);	fwrite(s4,4,1,sfile);
      swapit(&coord[2][i2],s4);	fwrite(s4,4,1,sfile);

      swapit(&coord[0][i3],s4);	fwrite(s4,4,1,sfile);
      swapit(&coord[1][i3],s4);	fwrite(s4,4,1,sfile);
      swapit(&coord[2][i3],s4);	fwrite(s4,4,1,sfile);
      fwrite(s4,2,1,sfile);
  }
  fclose(sfile);
    puts(" ");
  printf("  STL Data Written to %s.stl \n",argv[1]);
    puts("  Done!!!!!!!!!");
  exit(0);
}

void  getnorm(xn,yn,zn,i1,i2,i3)
	float	*xn,*yn,*zn;
	int	i1,i2,i3;
{
	float	v1[3],v2[3];
	int	i;

        for(i=0;i<3;i++){
	  v1[i] = coord[i][i3] - coord[i][i2];
	  v2[i] = coord[i][i1] - coord[i][i2];
	}

	*xn = (v1[1]*v2[2]) - (v1[2]*v2[1]);
	*yn = (v1[2]*v2[0]) - (v1[0]*v2[2]);
	*zn = (v1[0]*v2[1]) - (v1[1]*v2[0]);
}
long readelems(fname)
        char    *fname;
{
        long num,i;
        FILE *nfile;
        char    string[256],s1[7];

        num = 0;
        nfile = fopen(fname,"r");
		if(!nfile){
			puts(" error on element file open, bye!");
			exit(1);
		}
        while(fgets(string,86,nfile)){
          for(i=0;i<8;i++){
            strncpy(s1,&string[6*i],6);
            s1[6] = '\0';
            sscanf(s1,"%d",&ie[i][num]);
          }
          num++;
        }

        printf("Number of element read: %d\n",num);
        return(num);
}

long readnodes(fname)
        char	*fname;
{
        FILE    *nfile;
        long     num,typeflag,nval,ifoo;
        char    string[256];

        num = 0;
        nfile = fopen(fname,"r");
		if(!nfile){
			puts(" error on node file open, bye!");
			exit(1);

		}
        while(fgets(string,100,nfile)){
          sscanf(string,"%d ",&nval);
          switch(nval){
            case(-888):
                typeflag = 1;
            break;
            case(-999):
                typeflag = 0;
            break;
            default:
                np[nval] = num;
                if(typeflag){
                        sscanf(string,"%d %g %g %g",
                           &ifoo,&coord[0][num],&coord[1][num],&coord[2][num]);
                }else{
                        sscanf(string,"%d %g %g %g",
                           &ifoo,&coord[0][num],&coord[1][num],&coord[2][num]);
                        fgets(string,81,nfile);
                }
num++;
            break;
        }

        }
        printf("Number of nodes read %d\n",num);
        return(num);

}

/* A Little ditty to swap the byte order, STL files are for DOS */
void swapit(s1,s2)
     char s1[4],s2[4];
{
  s2[0] = s1[0];
  s2[1] = s1[1];
  s2[2] = s1[2];
  s2[3] = s1[3];
}

ans2stl_win_2014_01_28.zip

Creating the Nodes and Elements

I’ve created a little example macro that can be used to make an STL of deformed geometry.  If you do not want the deformed geometry, simply remove or comment out the UPGEOM command.  This macro is good for MAPDL or ANSYS Mechanical, just comment out the last line  to use it with MAPDL:

et,999,200,4

type,999

esurf,all

finish ! exit whatever preprocessor your in

! move the RST file to a temp file for the UPCOORD. Comment out if you want

! the original geometry

/copy,file,rst,,stl_temp,rst

/prep7 ! Go in to PREP7

et,999,200,4 ! Create a dummy triangle element type, non-solved (200)

type,999 ! Make it the active type

esurf,all ! Surface mesh your model

!

! Update the geometry to the deformed shape

! The first argument is the scale factor, adjust to the appropriate level

! Comment this line out if you don’t want deformed geometry

upgeom,1000,,,stl_temp,rst

!

esel,type,999 ! Select those new elements

nelem ! Select the nodes associated with them

nwrite,stl_temp,node ! write the node file

ewrite,stl_temp,elem ! Write the element file

! Run the program to convert

! This assumes your executable in in c:\temp. If not, change to the proper

! location

/sys,c:\temp\ans2stl_win.exe stl_temp

! If this is a ANSYS Mechanical code snippet, then copy the resulting STL file up to

! the root directory for the project

! For MAPDL, Comment this line out.

/copy,stl_temp,stl,,stl_temp,stl,..\..

An Example

To prove this out using modern computing technology (remember, last time I used this was in 2001) I brought up my trusty valve body model and slammed 5000 lbs on one end, holding it on the top flange.  I then inserted the Commands object into the post processing branch:

image

When the model is solved, that command object will get executed after ANSYS is done doing all of its post processing, creating an STL of the deformed geometry. Here is what it looks like in the output file. You can see what it looks like when APDL executes the various commands:

/COPY FILE FROM FILE= file.rst

TO FILE= stl_temp.rst

FILE file.rst COPIED TO stl_temp.rst

1

***** ANSYS – ENGINEERING ANALYSIS SYSTEM RELEASE 15.0 *****

ANSYS Multiphysics

65420042 VERSION=WINDOWS x64 08:39:44 JAN 14, 2014 CP= 22.074

valve_stl–Static Structural (A5)

Note – This ANSYS version was linked by Licensee

***** ANSYS ANALYSIS DEFINITION (PREP7) *****

ELEMENT TYPE 999 IS MESH200 3-NODE TRIA MESHING FACET

KEYOPT( 1- 6)= 4 0 0 0 0 0

KEYOPT( 7-12)= 0 0 0 0 0 0

KEYOPT(13-18)= 0 0 0 0 0 0

CURRENT NODAL DOF SET IS UX UY UZ

THREE-DIMENSIONAL MODEL

ELEMENT TYPE SET TO 999

GENERATE ELEMENTS ON SURFACE DEFINED BY SELECTED NODES

TYPE= 999 REAL= 1 MATERIAL= 1 ESYS= 0

NUMBER OF ELEMENTS GENERATED= 13648

USING FILE stl_temp.rst

THE SCALE FACTOR HAS BEEN SET TO 1000.0

USING FILE stl_temp.rst

ESEL FOR LABEL= TYPE FROM 999 TO 999 BY 1

13648 ELEMENTS (OF 43707 DEFINED) SELECTED BY ESEL COMMAND.

SELECT ALL NODES HAVING ANY ELEMENT IN ELEMENT SET.

6814 NODES (OF 53895 DEFINED) SELECTED FROM

13648 SELECTED ELEMENTS BY NELE COMMAND.

WRITE ALL SELECTED NODES TO THE NODES FILE.

START WRITING AT THE BEGINNING OF FILE stl_temp.node

6814 NODES WERE WRITTEN TO FILE= stl_temp.node

WRITE ALL SELECTED ELEMENTS TO THE ELEMENT FILE.

START WRITTING AT THE BEGINNING OF FILE stl_temp.elem

Using Format = 14(I6)

13648 ELEMENTS WERE WRITTEN TO FILE= stl_temp.elem

SYSTEM=

c:\temp\ans2stl_win.exe stl_temp

Number of nodes read 6814

Number of element read: 13648

STL Data Written to stl_temp.stl

Done!!!!!!!!!

/COPY FILE FROM FILE= stl_temp.stl

TO FILE= ..\..\stl_temp.stl

FILE stl_temp.stl COPIED TO ..\..\stl_temp.stl

image

The resulting STL file looks great:

image

I use MeshLab to view my STL files because… well it is free.  Do note that the mesh looks coarser.  This is because the ANSYS mesh uses TETS with midside nodes.  When those faces get converted to triangles those midside nodes are removed, so you do get a coarser looking model.

And after getting bumped from the queue a couple of times by “paying” jobs, our RP group printed up a nice FDM version for me on one of our Stratasys uPrint Plus machines:

image

It’s kind of hard to see, so I went out to the parking lot and recorded a short video of the part, twisting it around a bit:

Here is the ANSYS Mechanical project archive if you want to play with it yourself.

Other Things to Consider

Using FE Modeler

You can use FE Modeler in a couple of different ways with STL files. First off, you can read an STL file made using the method above. If you don’t have an STL preview tool, it is an easy way to check your distorted mesh.  Just chose STL as the input file format:

image

You get this:

image

If you look back up at the open dialog you will notice that it reads a bunch of mesh formats. So one thing you could do instead of using my little program, is use FE Modeler to make your STL.  Instead of executing the program with a /SYS command, simply use a CDWRITE,DB command and then read the resulting *.CDB file into FE Modeler.  To write out the STL, just set the “Target System” to STL and then click “Write Solver File”

image

You may know, or may have noticed in the image above, that FE Modeler can read other FEA meshes.  So if you are using some other FEA package, which you should not, then you can make an STL file in FE Modeler as well.

Color Contours

The next obvious question is how do I get my color contours on the plot. Right now we don’t have that type of printer here at PADT, but I believe that the dominant 3D Color printer out, the former Z-Corp and now 3D Systems machines, will read ANSYS results files. Stratasys JUST announced a new color 3D Printer that makes usable parts. Right now they don’t have a way to do contours, but as soon as they do we will publish something.

Another option is to use a /SHOW,vrml option and then convert that to STL with the color information.

Scaling

Scaling is something you should think about. Not only the scaling on your deformed geometry, but the scaling on your model for printing.  Units can be tricky with STL files so make sure you check your model size before you print.

Smoother STL Surfaces

Your FEA mesh may be kind of coarse and the resulting STL file is even coarser because of the whole midside node thing.  Most of the smoothing tools out there will also get rid of sharp edges, so you don’t want those. Your best best is to refine your mesh or using a tool like Geomagic.

Making a CAD Model from my Deformed Mesh

Perhaps you stumbled on this posting not wanting to print your model. Maybe you want a CAD model of your deformed geometry.  You would use the same process, and then use Geomagic Studio.  It actually works very well and give you a usable CAD model when you are done.

Efficient Engineering Data, Part 2: Setting Default Materials and Assignments aka No, You’re Not Stuck with Structural Steel for the Rest of Your Life

Longer ago than I care to admit, I wrote an article about creating and using your own material libraries in Workbench. This is the long awaited follow-up, which concerns setting the default Engineering Data materials and default material assignments in Mechanical and other analysis editors.

imageNote:
Part of the reason it’s taken me this long is that I moved to New Mexico to help staff PADT’s new office there, and to shadow Walter White. It has been a hectic, exhausting endeavor but I’m here and I’m finally settled in. If you’re in New Mexico and are interested in ANSYS, engineering services, product development, or rapid prototyping (e.g. 3D printing), please feel free to contact me.

In order to make the best use of the procedures here, you will probably want to know how to create your own material libraries. Part 1 describes how to do this. This will also work with the material libraries that come with the ANSYS installation, though.

Pick Favorites

The first step is to get into Engineering Data and expose the material libraries by clicking on the book stack button ( image ). Then, drag the materials of your choice from the appropriate library(ies) to the Favorites Data Source. These can include materials you want to have available in Mechanical by default as well as materials that you would like to consolidate into a single location for quick access. At this point, the default material availability and assignments have not been altered. These will be handled in the next couple of steps.

image

Drag and Drop Materials to Favorites

Set Default Material Availability

To specify which materials will be immediately available for assignment in future analyses, go to the Favorites Data Source and check all applicable materials in column D. Though not assigned to the immediate set of engineering data, these will be on the default list of available materials in subsequent analyses, i.e. when you create a new analysis in the same project schematic or when you exit and reopen Workbench.

image

Check to Add to Default List of Available Materials

image

Materials Immediately Available Inside Mechanical

Set Default Material Assignment

Now our most commonly used materials are immediately available in our analysis editor. But Structural Steel still lingers. In many, if not most, cases, we would prefer our default assignment to be something else.

The fix is easy. Once again, go to the Favorites Data Source, right click the material you wish to have as your default material, and select Default Solid Material (and if you’re doing Emag or CFD, you can set your default fluid or field material with the right-click menu too). Your default solid material will now replace Structural Steel in subsequent analyses.

image

Example: Aluminum 6061-T651 Set as Default Material Assignment

image

Becomes Default Material Assignment in Analysis

Note that you can stop at any step in this process. If you want to consolidate favorite materials, but don’t want to have them immediately in your analysis editor, you can do that. If you want a default list of materials to select from without specifying a default material assignment, you can do that too. More than likely, though, you’ll want to do all three.

Press Release: PADT and M-Tech Industries to Highlight Fluid-Thermal System Modeling for Mining with Flownex at 2014 SME Annual Meeting and Exhibit

987786-Flownex-SME-2014_Mine-Simulation-3We are very excited about the upcoming 2014 SME Annual Meeting and Exhibit in Salt Lake City, Utah.  Not only is this in our very own back yard (or is it our front or side yard?) it is a great place for us to show off Flownex Simulation Environment and how useful it is for simulation mining systems. Besides promoting Flownex, we will hae a booth in the exhibit area and we will be presenting a paper on some work we did with ANSYS software for mining.  Last years show in Denver was a great experience and we know this years will be as well.

To promote the event and Flownex usage in the industry, we just published the following press release:

image

The release is accompanied by two great videos that Stephen did showing the usage of Flownex on some real mining problems.

Part 1

Part 2

Also, don’t forget that we still have room in our free Denver, Colorado Introduction to Flownex Class.

As always with Flownex, contact Roy Haynie (roy.haynie@padtinc.com) to learn more.

Video Tips: Multiphysics Simulation with ANSYS Maxwell and ANSYS Mechanical – Part 2

This is Part 2 of our 2 part video series showing you a multiphysics simulation with ANSYS Maxwell and ANSYS Mechanical. In this video we take the results from ANSYS Maxwell and use it to compute the temperature distribution and finally the structural deformation due to the current through the parts.

The Part 1 video can be found here

Video Tips: Multiphysics Simulation with ANSYS Maxwell and ANSYS Mechanical – Part 1

This Part 1 of 2 video shows you the first half of a multiphysics simulation using the low-frequency electromagnetics tool ANSYS Maxwell to do an eddy current analysis. Part 2 will involve taking the results of this analysis and transferring it to perform a thermal-structural analysis using ANSYS Mechanical.

Video Tips: Parallel Part by Part Meshing in ANSYS v15.0

This video shows you a new capability in ANSYS v15.0 that allows multiple parts to be simultaneously meshed on multiple CPU cores…with no additional licenses required!

A Word on Files and Evil Missing Files in ANSYS Workbench Projects

image_232A while back I did a webinar on POST26 in ANSYS Mechanical APDL and using it with ANSYS Mechanical.  You know it was a while ago because… well… it was a webinar and I have not had time to do one of those for a long chunk of time now. Anyway, as usual the files used were placed on the blog in a posting.

Last night I got an email from a student in Australia who tried to use the file and found a problem with it.  Now this was cool for a couple of reasons: 1) anytime someone from the opposite side of the earth reaches out to communicate, that is just makes my day, and 2) someone not only read the posting, but they tried to use it. Sometimes the only way we know people are using the content we create is when the find a problem.  I’ll take it.

In figuring out what was wrong I figured it might be a good time to point out some things about the file manager in ANSYS Workbench, and how using it, you can fix the problem that this project had.

Projects – A Big Directory Tree of Folders and Files

ansys-workbench-project-filesIf you ever looked at the directory your project is stored in you will see a big old tree of folders and files.  All of the info needed or created for your project are stored in this directory tree. Why? Because the briliant thing about the project page is that it is designed to take all these different programs like ANSYS Mechanical, ICEM CFD, FLUENT, CFX, etc… and allow you to interact between them in a single tool set.  In that each program was written by a different group of developers, and most of them when those developers worked at different companies, each one has its own unique file structure, files that it needs, and way of organization them.   By giving each tool its own directory in the project, you can have the legacy data structure you need, but still keep all your files in one place where the project page actually knows where to find the information it needs.

Looking at your Files

This is done with the Files View in the Workbench Project page. By default it is hidden. Just go to View on the menu and click Files so it has a check next to it:

turn-on-files-ansys-workbench

We did a post in the past explaining all the things this view does for you.  Read it here.

Fixing a Missing File

If you should get the type of error message that our Australian friend got, you can easily remedy it with the files view.   If it is missing it will show up with a big red X next to it instead of its normal icon. The whole line will in fact be red.

If you Right Mouse Button on it you will see a couple of options:

  1. Repair “filename”:  This allows you to hunt and find the missing file. It should say “Identify Missing File” or some such, but repair works too.  Click on that, find your missing file, and you are good.
  2. Remove “filename” is what you use if you don’t need the file, it is gone, and you want to get on with your life.
  3. Open Containing Folder is nice because it will take you to the folder that the file is supposed to be in. You might poke around in there and figure out what is going on.

fix-file-ansys-workbench

 

That is it.  Hopefully I made a mistake somewhere and someone from Argentina will email letting us know. I don’t believe we have been contacted by a user in Argentina.

Exercising Parallel Meshing in ANSYS Mechanical R15

[The following is an email that Manoj sent the tech support staff at PADT. I thought is was perfect for a The Focus posting, so here it is – Eric]

First of all I found out a way to get Mesh Generation time (if no one knew about this).  In ANSYS Mechanical go to Tools->Options->Miscellaneous and turn “Report Performance Diagnostics in Messages” to Yes.  It will give you “Elapsed Time for Last Mesh Generation” in the Messages window.

clip_image001

clip_image002

Next I did a benchmark on the Parallel Part by Part meshing of a Helicopter Rotor Hub with 502 bodies.  The mesh settings were getting a mesh of about 560,026 elements and 1.23 million nodes.

clip_image004

I did Parallel Part by Part Meshing on this model with 1,2,4,6 and 8 cores and here are the results.

Can I say “I LIKE IT!”

1 core: 172 seconds (1.0)
2 core:  89 seconds (1.9)
4 core:  52 seconds (3.3)
6 core:  38 seconds (4.5)
8 core:  33 seconds (5.2)

image

Of course this is a small mesh so as the number of cores goes up, the benefits go down.   I will be doing some testing on some models that take a lot longer to mesh but wanted to start simple. I’ll make a video summarizing that study showing how to set up the whole process and the results.

If you are curious, Manoj is running on a PADT CUBE server. As configured it would cost around $19k. You could drop a few thousand of the price if you changed up cards or went with CPU’s that were not so leading edge.

Here are the SPECs:

CUBE HVPC w8i-KGPU
CUBE Mid-Tower Chassis – 26db quiet edition
Two XEON e5-2637 v2 (4 cores per, 3.5GHz each)
128 GB of DDR3-1600 ECC Reg RAM
NVIDIA QUADRO K5000
NVIDIA TESLA K20x
7.1 HD Audio (to really rock your webinars…)
SMC LSI 2208 RAID Card – 6Gbps
OS Drive: 2 x 256GB SSD 6gbps
Solver Array: 3 x 600GB SAS2 15k RPM 6Gbps

CUBE Systems are Now Part of the ANSYS, Inc. HPC Partner Program

CUBE-HVPC-Logo-wide_thumb.png

The relationship between ANSYS, Inc. and PADT is a long one that runs deep. And that relationship just got stronger with PADT joining the HPC Partner Program with our line of CUBE compute systems specifically designed for simulation. The partner program was set up by ANSYS, Inc. to work:

CUBE-HVPC-512-core-closeup3-1000h_thumb.jpg“… with leaders in high-performance computing (HPC) to ensure that the engineering simulation software is optimized on the latest computing platforms. In addition, HPC partners work with ANSYS to develop specific guidelines and recommended hardware and system configurations. This helps customers to navigate the rapidly changing HPC landscape and acquire the optimum infrastructure for running ANSYS software. This mutual commitment means that ANSYS customers get outstanding value from their overall HPC investment.”

CUBE-HVPC-512-core-stairs-1000h_thumb.jpg

PADT is very excited to be part of this program and to contribute to the ANSYS/HPC community as much as we can.  Users know they can count on PADT’s strong technical expertise with ANSYS Mechanical, ANSYS Mechanical APDL, ANSYS FLUENT, ANSYS CFX, ANSYS Maxwell, ANSYS HFSS, and other ANSYS, Inc. products, a true differentiator when compared with other hardware providers.

Customers around the US have fallen in love with their CUBE workstations, servers, mini-clusters, and clusters finding them to be the right mix between price and performance. CUBE systems let users carry out larger simulations, with greater accuracy, in less time, at a lower cost than name-brand solutions. This leaves you more cash to buy more hardware or software.

Assembled by PADT’s IT staff, CUBE computing systems are delivered with the customer’s simulation software loaded and tested. We configure each system specifically for simulation, making choices based upon PADT’s extensive experience using similar systems for the same kind of work. We do not add things a simulation user does not need, and focus on the hardware and setup that delivers performance.

CUBE-HVPC-512-core-front1-1000h_thumb.jpg

Is it time for you to upgrade your systems?  Is it time for you to “step out of the box, and step in to a CUBE?”  Download a brochure of typical systems to see how much your money can actually buy, visit the website, or contact us.  Our experts will spend time with you to understand your needs, your budget, and what your true goals are for HPC. Then we will design your custom system to meet those needs.

 

The 10 Coolest New Features in R15 of ANSYS Mechanical APDL

On Tuesday we posted on what I thought were the 10 coolest features in ANSYS Mechanical for R15. Now it is time to take a good look at ANSYS Mechanical APDL, or MAPDL (classic ANSYS, black window ANSYS, or my favorite: ANSYS).  The developers have been very busy and added a lot of useful features, and there are a large number of “Oh Yes!” capabilities in this release that different groups of users will be very excited about.  For this posting though, we are going to stay focused on the things that impact larger groups of users and/or expand capability in the code.  As always, you can learn more by attending one of the many upcoming ANSYS webinars or reading the release notes in the help.

image1: Rezoning Enhancements and Additions

This is my favorite change in R15, a mix of some improvements and some new capabilities. The whole idea of rezoning is that when you have a part that sees a large amount deformation, the mesh often gets very distorted. It often gets so distorted that the elements are no longer accurate and crazy strains are calculated and the element literally blows up.  Or it turns inside out and generates an error in the solver.

Rezoning has been around for a while but at this release some holes are plugged and some big advances are made.  The first change was a hole plug, you can now rezone areas that contain surface effect  (SURF153/154) elements.  It is very common to have that type of load on highly distorted geometry, so this is welcomed.

The next change was adding mesh splitting for 3D  tetrahedral elements. This is used for manual rezoning with the REMESH command. It was available before with 2D elements.  The 2D example from the manual shows it best:

image

The advantage of this approach is that the subsequent stress field that is placed upon the new mesh is already accurate at the nodes that existed for the original mesh, and are fairly accurately interpolated for the new nodes.  When you read in a completely new mesh, you have to interpolate the stress field and then iterate till the stresses are accurate.  This approach can be much faster.

The third and best addition is Automatic Rezoning or Mesh Nonlinear Adaptivity.  This process is completely automatic and does not require the user interaction that rezoning does.  Both splitting and remeshing are used. You can turn remeshing on based upon position, energy levels, or contact conditions.

Here is an example from the user manual:

seal2

And here is an example that ANSYS, Inc. is showing on the new :

Seal_Squish1

image2: Bolt Thread Modeling

Modeling bolt threads.  Classic newby mistake right?  They model threads on 37 bolts and then try and set up contact on all of them.  Never goes well does it. ANSYS MAPDL has had a bolt modeling capability for some time that allows you to simulate a bolt as a cylinder with preload and everything. But what that approximation missed was the fact that the contact is at an angle that is not normal to the cylinder surface.

At R15 you can now specify your thread geometry and the contact algorithm will calculate the proper normal and contact pattern for the contact forces. Much more efficient. There is a great example in the Technology Demonstration Guide, Section 39, showing all three approaches: model the threads in the mesh, use the new contact threads, or just bond the threaded area.

image

Needless to say we will be doing an in-depth posting on this one in the future.

image3: Mode-Superposition for Harmonic Analysis of Cyclic Structures

I started my career in turbomachinary and from day one, one of the holy grails was to be able to do a harmonic analysis using blade pressure loads from a CFD run: getting the actual stresses in the blades caused by the varying aerodynamic load as they spun around and dealt with variations caused by passing frequencies and resonance in the flow itself.  It was always doable as a full 360 model on both the CFD and structural side. And you could have done it using the full method for a few releases.  But now we can use cyclic symmetry and mode-superposition.

The ANSYS MAPDL side of things is released in R15.  You can take your complex loading info from CFD and apply that as a load on your blades using the new /MAP pre-processor (see below), a bit of a pain to do in the past.  The other big change was making it all work with modal-superposition.  The grail is almost complete.

image4: Arc Length

This is one of those things buried in the code that the user really doesn’t have to do anything to benefit from. If you are not familiar with the method, it is an approach used on non-linear problems using the Newton-Raphson method. Most solves use other methods,  but for things like non-linear buckling it is a better method. Check out 14.12 in the Theory Reference for the math and all that.

The bottom line at R15 is that they changed the algorithm to use the Crisfield Method and to avoid Driftback.  What it means to you the user is those nasty non-linear buckling problems that always seem to have a hard time converging, or that require really small steps to converge, should converge now or converge faster.

image5: Mode Selection

This is another one of those advanced options that users of other solvers have been asking for in the past. When you do a modal analysis that produces a ton of modes, you often want to ignore the majority of them and focus on the few modes that are strong or that get excited.  In the past you could specify a range only, and only one range. At R15 you can now select which modes to use in modal-superposition analysis.  You can decide which modes to use based on the modal effective mass, the mode coefficient, or the DDAM Procedure. Or, if you have your own criteria, you can use APDL to create a table that specifies a 1 for keep, and a 0 for toss. Very handy.

image6: Acoustics Enhancements

This is really not one new feature, but an overall continuation of adding functionality to the acoustics capability in ANSYS MAPDL. For decades, this capability was not really focused on advanced acoustic modeling. But over the last couple of releases we have seen added functionality that put the functionality on a par with specialty acoustics codes.

The key enhancements at R15 are:

  • Frequency-dependent acoustic material properties
  • Surface impedance can be frequency-dependent
  • A new boundary layer impedance (BLI) model is available for visco-thermo fluids modeling
  • A wider range of units are now supported for acoustics, including support for user defined units (/UNITS)
  • Many enhancements for coupling acoustics with CFD for FSI
  • New postprocessing commands for calculating acoustics specific information like sound power level, A-weighted sound pressure level (dBA), and return loss, and transmission loss, amongst others.

image

image7: Shape Memory on Beam, Shell and Plane Strain

For whatever reason shape memory alloys have always fascinated me, and being able to simulate them accurately is very important for those that use the material in their products. Development has been adding more and more functionality in this area for many releases.  The material has two unique properties: it is super elastic and it has a memory effect.

With R15 development rounds out the capabilities with full support for beam and shell elements, adding the memory behavior.  This is important, and warrants top 10 status for me, because many of the geometries we have worked on that use Nitinol (the most common shape memory alloy) are made with wires.  In the past you had to model them in 3D, now we can use beams.  Faster, more accurate, etc…

image

image8: RSTMAC to Experimental Data

Something we should all be doing more is comparing our FEA results to experimental data. One excuse we often use is that it is too hard to compare the data from a vib test to our modal analysis results. Well, that is no longer to. The RSTMAC command has been modified to not only compare to ANSYS result files, but to also read the old Universal file format for results. Yipee!

Why that format? Because back in the days when SDRC was SDRC and IDEAS was their prep/post tool, they had some awesome result comparison tools. So a lot of test software out there writes to the file that IDEAS read, the Universal file.  If your software does not write to a Universal file, the key info you need is in the user manual: Basic Analysis Guide, section 7.3.8.5.  and here is a link to some documentation on it.

image

image9: Mapping Processor

It has been a long time since ANSYS, Inc. has added a new processor to ANSYS Mechanical APDL.  /PREP7, /SOLVE, /POST26.  So it was kind of cool to see that they are creating a new mapping processor called /MAP that will be a place for you to do load mapping. At this initial release, it maps surface pressures as a point cloud from a CFD model onto your mechanical model.

Under the hood it is actually just the algorithms used in the *MOPER APDL function.  But now it is exposed in through its own set of commands so that users don’t have to script their load mapping. And it supports imaginary loading for that fancy cyclic-symmetry stuff some of us need to do. As you can imagine, this needs its own article, but here are the high points:

  • Enter the processor with /MAP
  • Your model must have surface effect elements (SURF154) elements paved onto the outside of where you want the pressures.

 

  • Specify the nodes you want to map the pressures on to with the TARGET command
  • You can provide pressures in a variety of formats (specified with the FTYPE command):

 

  • CFX Transient Blade Row format is made by CFX and contains real and imaginary terms
  • The standard output file from ANSYS CFDPost
  • A fixed format file that has x, y, z, and pressure, and yes, you can specify the actual format in FORTRAN using the READ command
  • And of course the trusty comma delimited file format: x, y, z, pressure.

 

  • The READ command specifies some other parameters and reads in the point based pressure data.
  • They have given us a PLGEOM command to view the target nodes and the point cloud on top of each other so you can see if things are aligned
  • A whole slew of /PREP7 like commands to edit and move your point cloud data. Basically they are treated as nodes and you manipulate them like nodes. They are just nodes with a pressure assigned to them.
  • When everything is good, use MAP to actually do the interpolation.
  • View the results with PLMAP
  • When you are happy, save the pressures as SFE commands using WRITEMAP

 

There is no GUI interface for this yet.  It was put into place to support some advanced FSI modeling of turbomachinery, but it benefits all users.  We hope to see more in this new module in future releases.  Here is an example we were playing with at PADT:

image

image

Image10: Performance Enhancements, Including GPU Stuff

Last but certainly not least are the enhancements to solver performance that we have come to expect. New compilers, optimized code, and new hardware all come together to deliver better bang for your ANSYS buck.  There is a ton in there, all documented in section 2.3.1 of the ANSYS, Inc. Release Notes part of the help.  The highlights are:

  • The sparse solver now has some sophisticated error detection for handling singular or near singular matrices.  This should keep you from solving poorly constrained models, or models with really messed up elements. Do note, some models that ran in the past, maybe with a warning, will now not solve. This is a good thing since the matrices are not good
  • Better domain decomposition for distributed ANSYS, especially for larger core counts.
  • The subspace method has been added for solving modal analysis.  It is well suited for larger problems and runs well distributed.
  • Switching to the new Intel compiler has resulted in a 30% faster solve time on some problems when using Sandy Bridge Intel processors.
  • Harmonic analyses solves using the full method have been improved, resulting into up to 40% improvements in solve time.
  • The Intel Xeon Phi coprocessor is now supported – we have no data yet on the performance but will try and get some as soon as we can
  • The latest NVIDIA Kepler GPU’s are now supported and the sparse solver has been improved again to take advanged of the Kepler GPU’s.

CUBE HVPC 512 Core System

Thoughts

The hard part for me in writing this posting was picking the top 10. There are a lot of significant enhancements but few are world changing. Most improve existing technologies, provide functionality for a subset of users, or fill a hole in capability.  Taken as a whole though, they show ANSYS, Inc.’s strong commitment to core technology: new elements, new material models, faster solves, expanded advanced capability, etc…

The end result is giving greater power to the user through greater depth and breadth.  And in the end, isn’t this why we use ANSYS Mechanical APDL in the first place – the incredible breadth and depth of capability it offers?  Scrolling back up through the images you have to admit, this is some pretty cool stuff.

The 10 Coolest New Features in R15 of ANSYS Mechanical

It’s that time of year again, time for a new release of ANSYS, Inc’s products.  R15 is being released in stages to the user community this week so we thought we would take some time to point out ten features in R15 of ANSYS Mechanical that we find useful, important, or just plain cool.  There are a ton of new features and we will try and cover most of them in the coming months, but these are the ones we felt every user should know about.

This posting will focus on features in ANSYS Mechanical that are unique to ANSYS Mechanical.  Later this week or next week we will do the same for ANSYS Mechanical APDL, and we will cover solver changes that impact ANSYS Mechanical there as well.

image1: Mesh Based Geometry

This is by far the most far-reaching enhancement in R15.  A fundamental limitation of ANSYS Mechanical from the beginning was the requirement that you had to have a valid BREP geometry that can be correctly meshed in ANSYS Mechanical. For most problems this is fine, you have a CAD model, you mesh it, and you move on.  But there are often situations where you have a legacy model or a mesh from another source that you want to use. And in such cases you were just stuck. Most things in ANSYS Mechanical work on geometry and if you just have a mesh, and no geometry, there area not a lot of options.

No longer.  Yipidee yapidee dooooooo daaaa!!!!  That gets a leap for joy gif:

It works through an External Model system in the Workbench project page.  There are a slew of options to copy, translate, change units, etc.. for the model. However, the most important option is the tolerance angle. The way the mesh geometry import works is that it takes the external faces of your mesh and treats them as facets. Any facets that have an angle less than the tolerance angle are considered to be on the same face. Any angle greater than the tolerance treats the edge between facets as an edge on a face.  This is key to understanding how using an mesh in Mechanical works.

I’ll resist the temptation to get into the details and save that for a future posting where we can dig deep. 

To prove it out I searched my hard drive for old *.DB files and found one from 2001. A generic turbine blade ad disk model I made for some cyclic-symmetry testing ANSYS was doing.  Here is what is looks like in ANSYS Mechanical APDL:

image

And here it what it looks like in ANSYS Mechanical:

image

That is a sight for sore eyes.  It’s not perfect, the trailing edge muffs things up a big because the mesh is a bit coarse there. But a little work playing with the tolerance angle and/or named selections in MAPDL and that can be cleaned up. More on that in that promised post on this capability.  Here is a wireframe where you can see the internal cooling passages, and the funky elements.

image

Changing the tolerance from the default 45deg to 55deg cleans up most of the problems:

image

image2: Assembly Mechanical Models

A related capability to the mesh geometry mentioned above is the ability to create an assembly of multiple external models (mesh based) and other Mechanical Models.

You can take as many External Model, Mechanical Model, or various Analysis systems as you want, and feed them all into a new mechanical model or any analysis system you want to use them in. ANSYS, Inc. provided this really cool example of a model of a pad lock:

image

As much information as is possible is transferred over, depending on what makes sense.  Also note that you can apply transformations and units conversions to each model, so if you build your parts in different coordinate systems, you can move them around and get them set up when you build the assembled model.

image3: Parallel Meshing by Part

The first step in making meshing fully parallel in ANSYS Mechanical has been released in R15: parallel part meshing.  Basically, if you have more than one part, the program will mesh each part on its own CPU.  So if you have 8 cores and 6 parts, it will mesh on 6 cores at one time.  The default setting is 0, which tells the program to run on as many cores as it can.  Most users will want to keep it there, but if you do a lot of models with lots of large parts, you may want to set it at N-1 where N is the number of cores on your machine.  Leave one open to watch YouTube on while you are meshing.

The settings are in Tools->Options->Meshing->Meshing->Meshing->Number of CPUs for Meshing Methods

image4: Mechanical Shortcut Keys

You may love this one, or it may generate a “Meh.”  Shortcut Keys are almost a religious thing.  If you are on the “true believer” side then you now can use function and control keys to do the following actions:

Tree Outline Actions

F1: opens the Mechanical User’s Guide.
F2: rename a selected tree object.
Ctrl + S: save the project.

Graphics Actions

F6: toggles between the Shaded Exterior and Edges, Shaded Exterior, and Wireframe views
F7: executes Zoom to Fit option
F8: hide selected faces.
F9: hide selected bodies.
Ctrl + A: selects all entities based on the active selection filter (bodies, faces, edges, vertices, nodes).

Selection Filters

Ctrl + B: activate Body selection.
Ctrl + E: activate Edge selection.
Ctrl + F: activate Face selection.
Ctrl + P: activate Vertex selection.

Even if you are not a Shortcut Key acolyte, the selection filters and the F7 options look pretty useful.

image5: Element Selection and Grouping

In the last release the developers at ANSYS, Inc. gave users full access to nodes in ANSYS Mechanical. At R15 they have opened up access to elements.  Using the term “opened up access” is a bit misleading, they did not just change a parameter from FALSE to TRUE and boom, you have elements.  It was a major change to how data is stored and accessed in the program. 

Selecting works as you would expect, just like nodes. You choose “Select Mesh” from the Select Type icon:

image

Then you choose the Select “Body/Element” from the type choices (the green cube).  Here is where you can use those shortcuts: Ctrl + B selects it for you. Now you can pick elements or use box select to get what you want.

Names selections for elements work just like nodes.  Very useful indeed. And they do get converted to components in Mechanical APDL, avoiding that annoying snippet where you had to convert a nodal component into an element component.

image6: Mapping Enhancements

One feature set in ANSYS Mechanical that makes users of other ANSYS products jealous are the mapping tools. And at this release they got even better, adding more functionality, feedback, and making some beta features released features. If you are not familiar with mapping tools, they allow you to take a load specified on points in space, and interpolate that load on to your model. Again, this is a topic worthy of its own posting, but here are the highlights:

  • Support for Velocity
  • Support for Initial Stress or Strain
  • Support for Body Force Density (forces from an electromagnetic solution)
  • Pressure can now be applied to nodes as well as to elements
  • Acoustic loads from a Harmonic Response Analysis can be mapped as velocities
  • UV Mapping is now available for surfaces that don’t really line up.

That last one, UV Mapping, may be one of the more powerful. It is not that uncommon for you to get pressures on a surface that does not really sit on your model surface.

image7: Better Control of Hydrostatic Pressure

This is pretty specialized, but if you work on parts that see hydrostatic loading you always had to play around with APDL snippets to get the control you needed.  At R15 they have added those controls into the program for us.

image

The first addition is that you can turn the pressure on and off for a given load step.  This is not necessarily done in an intuitive way, but it works.  Select the step in the graph or in the table when you have the load selected in the tree. Then Right Mouse Button to get the context menu and activate or deactivate the load. It doesn’t show anything in the table, but it does show on the graph that the load is turned off.  Note, you can’t change the acceleration, you can only turn it on or off.

The second addition is simply that the values for fluid density and magnitude are parameters.

image

image8: Multiple GPU Support, and Intel Phi Support

GPU usage for ANSYS Mechanical solves is growing.  We are seeing good enhancements in performance at every release, and R15 is no exception.  But that is solver stuff and I said we would not talk about solver stuff…  What is important in this area for ANSYS Mechanical users is that you can now specify more than one GPU for a solve, and at R15 the new Intel Phi coprocessor, which is really not a GPU, is supported. You access the control, and all settings for HPC, under Tools-> Solve Process Settings-> Advanced.  Note that there are different settings for solving interactive and in the background.

image

image9: Follower Load for Rigid Body Dynamics

Because Rigid Body Dynamics are, well rigid body dynamics, they are generally inherently large deflection.  When you apply a load to an object you usually want that load to move with the objects motion, to follow it.  In the past, you had to create a dummy rigid part and apply a pressure to get this.  Now at R15 you can set “Follower Load” to yes in the details view for a Remote Force and it will go along for the ride.  If you do RBD, this is critical.

image

image10: Bearing Enhancements

The last item on our top ten list for this release are improvements to modeling bearings in ANSYS Mechanical. The ANSYS Mechanical APDL solver supports a wide range of bearing capabilities, and with this release most of them are now exposed in ANSYS Mechanical. 

The big change is that all of the solvers that support bearings are now supported in ANSYS Mechanical. In the past, it was only modal analysis. Now you can simplify your model and get the proper stiffness and damping of your bearing for transient, static, and any other type of run you want to do.

As you would expect with the support of the bearing joint on the pre-processing side, they have added a probe that allows you to get key information out of your bearing after the solve. Since a bearing joint is basically two spring-dampers, you can get spring type info for each spring: elastic force, damping force, elongation, and velocity (for transient runs).

Thoughts

If you look at this list you should notice one common thread, that most of these changes are not general, they are for specific analysis types.  As time has gone by ANSYS Mechanical has grown and matured, and there are less and less basic or general features that need to be added. So now we are in to the fun stuff, digging down into the nitty gritty and exposing more and more of the most powerful solver available (ANSYS Mechanical APDL), through what is the most powerful user interface for structural mechanics, ANSYS Mechanical.

This May Be the Fastest ANSYS Mechanical Workstation we Have Built So Far

The Build Up

Its 6:30am and a dark shadow looms in Eric’s doorway. I wait until Eric finishes his Monday morning company updates. “Eric check this out, the CUBE HVPC w16i-k20x we built for our latest customer ANSYS Mechanical scaled to 16 cores on our test run.” The left eyebrow of Eric’s slightly rises up. I know I have him now I have his full and complete attention.

Why is this huge news?

This is why; Eric knows and probably many of you reading this also know that solving differential equations, distributed, parallel along with using graphic processing unit makes our hearts skip a beat. The finite element method used for solving these equations is CPU intensive and I/O intensive. This is headline news type stuff to us geek types. We love scratching our way along the compute processing power grids to utilize every bit of performance out of our hardware!

Oh and yes a lower time to solve is better! No GPU’s were harmed in this tests. Only one NVIDIA TESLA k20X GPU was used during the test.

Take a Deep Breath and Start from the Beginning:

I have been gathering and hording years’ worth of ANSYS mechanical benchmark data. Why? Not sure really after all I am wanna-be ANSYS Analysts. However, it wasn’t until a couple weeks ago that I woke up to the why again. MY CUBE HVPC team sold a dual socket INTEL Ivy bridge based workstation to a customer out of Washington state. Once we got the order, our Supermicro reseller‘s phone has been bouncing of the desk. After some back and forth, this is how the parts arrive directly from Supermicro, California. Yes, designed in the U.S.A.  And they show up in one big box:

clip_image002[4]

Normal is as Normal Does

As per normal is as normal does, I ran the series of ANSYS benchmarks. You know the type of benchmarks that perform coupled-physics simulations and solving really huge matrix numbers. So I ran ANSYS v14sp-5, ANSYS FLUENT benchmarks and some benchmarks for this customer, the types of runs they want to use the new machine for. So I was talking these benchmark results over with Eric. He thought that now is a perfect time to release the flood of benchmark data. Well some/a smidge of the benchmark data. I do admit the data does get overwhelming so I have tried to trim down the charts and graphs to the bare minimum. So what makes this workstation recipe for the fastest ANSYS Mechanical workstation so special? What is truly exciting enough to tip me over in my overstuffed black leather chair?

The Fastest Ever? Yup we have been Changed Forever

Not only is it the fastest ANSYS Mechanical workstation running on CUBE HVPC hardware.  It uses two INTEL CPU’s at 22 nanometers. Additionally, this is the first time that we have had an INTEL dual socket based workstation continue to gain faster times on and up to its maximum core count when solving in ANSYS Mechanical APDL.

Previously the fastest time was on the CUBE HVPC w16i-GPU workstation listed below. And it peaked at 14 cores. 

Unfortunately we only had time before we shipped the system off to gather two runs: 14 and 16 cores on the new machine. But you can see how fast that was in this table.  It was close to the previous system at 14 cores, but blew past it at 16 whereas the older system actually got clogged up and slowed down:

  Run Time (Sec)
Cores Used Config B Config C Config D
14 129.1 95.1 91.7
16 130.5 99 83.5

And here are the results as a bar graph for all the runs with this benchmark:

CUBE-Benchmark-ANSYS-2013_11_01

  We can’t wait to build one of these with more than one motherboard, maybe a 32 core system with infinband connecting the two. That should allow some very fast run times on some very, very large problems.

ANSYS V14sp-5 ANSYS R14 Benchmark Details

  • Elements : SOLID187, CONTA174, TARGE170
  • Nodes : 715,008
  • Materials : linear elastic
  • Nonlinearities : standard contact
  • Loading : rotational velocity
  • Other : coupling, symentric, matrix, sparse solver
  • Total DOF : 2.123 million
  • ANSYS 14.5.7

Here are the details and the data of the March 8, 2013 workstation:

Configuration C = CUBE HVPC w16i-GPU

  • CPU: 2x INTEL XEON e5-2690 (2.9GHz 8 core)
  • GPU: NVIDIA TESLA K20 Companion Processor
  • GRAPHICS: NVIDIA QUADRO K5000
  • RAM: 128GB DDR3 1600Mhz ECC
  • HD RAID Controller: SMC LSI 2208 6Gbps
  • HDD: (os and apps): 160GB SATA III SSD
  • HDD: (working directory):6x 600GB SAS2 15k RPM 6Gbps
  • OS: Windows 7 Professional 64-bit, Linux 64-bit
  • Other: ANSYS R14.0.8 / ANSYS R14.5

Here are the details from the new, November 1, 2013 workstation:

Configuration D = CUBE HVPC w16i-k20x

  • CPU: 2x INTEL XEON e5-2687W V2 (3.4GHz)
  • GPU: NVIDIA TESLA K20X Companion Processor
  • GRAPHICS: NVIDIA QUADRO K4000
  • RAM: 128GB DDR3 1600Mhz ECC
  • HDD: (os and apps): 4 x 240GB Enterprise Class Samsung SSD 6Gbps
  • HD RAID CONTROLLER: SMC LSI 2208 6Gbps
  • OS: Windows 7 Professional 64-bit, Linux 64-bit
  • Other: ANSYS 14.5.7

You can view the output from the run on the newer box (Configuration D) here:

Here is a picture of the Configuration D machine with the info on its guts:

clip_image006[4]clip_image008[4]

What is Inside that Chip:

The one (or two) CPU that rules them all: http://ark.intel.com/products/76161/

Intel® Xeon® Processor E5-2687W v2

  • Status: Launched
  • Launch Date: Q3’13
  • Processor Number: E5-2687WV2
  • # of Cores: 8
  • # of Thread: 16
  • Clock Speed: 3.4 GHz
  • Max Turbo Frequency: 4 GHz
  • Cache:  25 MB
  • Intel® QPI Speed:  8 GT/s
  • # of QPI Link:  2
  • Instruction Se:  64-bit
  • Instruction Set Extension:  Intel® AVX
  • Embedded Options Available:  No
  • Lithography:  22 nm
  • Scalability:  2S Only
  • Max TDP:  150 W
  • VID Voltage Range:  0.65–1.30V
  • Recommended Customer Price:  BOX : $2112.00, TRAY: $2108.00

The GPU’s that just keep getting better and better:

Features

TESLA C2075

TESLA K20X

TESLA K20

Number and Type of GPU

FERMI

Kepler GK110

Kepler GK110

Peak double precision floating point performance

515 Gflops

1.31 Tflops

1.17 Tflops

Peak single precision floating point performance

1.03 Tflops

3.95 Tflops

3.52 Tflops

Memory Bandwidth (ECC off)

144 GB/sec

250 GB/sec

208 GB/sec

Memory Size (GDDR5)

6GB

6GB

5GB

CUDA Cores

448

2688

2496

clip_image012[4]

Ready to Try one Out?

If you are as impressed as we are, then it is time for you to try out this next iteration of the Intel chip, configured for simulation by PADT, on your problems.  There is no reason for you to be using a CAD box or a bloated web server as your HPC workstation for running ANSYS Mechanical and solving in ANSYS Mechanical APDL.  Give us a call, our team will take the time to understand the types of problems you run, the IT environment you run in, and custom configure the right system for you:

http://www.padtinc.com/products/hardware/cube-hvpc,
email: garrett.smith@padtinc.com,
or call 480.813.4884