TACC and Local outputs not equivalent for circular NorSand model

MPM
#1

Issue

Running circular NorSand model with cb-geo:develop on local machine and in TACC are resulting in axisymmetric and non-axisymmetric pdstrain outputs, respectively. Identical input files are used.

Local (Expected Behavior)

Build

  • upstream is cb-geo:develop
  • Local branch is up-to-date

image

Compiling

mkdir build && cd build && cmake -DCMAKE_CXX_COMPILER=g++ -DNO_KAHIP=True..
make clean && make -j2

Run & Output

  • Running on -p 1
  • Output is axisymmetric (as expected after @ezrayst bug fix)

image
image1187×684 856 KB

Number of Attempts

  • Run 2x locally with identical axisymmetric outputs

TACC (Unexpected Behavior)

Build

  • TACC is up-to-date with origin
  • origin is personal fork of cb-geo:develop

image

  • Personal fork is even with cb-geo:develop

image
image1248×337 18.2 KB

Compiling

mkdir build && cd build && cmake -DBOOST_ROOT=$TACC_BOOST_DIR -DBOOST_INCLUDE_DIRS=$TACC_BOOST_INC -DCMAKE_BUILD_TYPE=Release -DEIGEN3_INCLUDE_DIR=$HOME/eigen -DKAHIP_ROOT=$HOME/KaHIP ..
make -j2

Run & Output

  • Running on -p 1
  • Output is not axisymmetric

image
image1187×684 862 KB

Number of Attempts

  • Run 3x on TACC with non-axisymmetric outputs for all attempts
  • Third attempt included removing entire mpm directory and git clone from scratch

Other Notes

  • TACC build definitely includes the axisymmetric bug fix (dF_dsigma_deviatoric) in norsand.tcc

image
image854×189 125 KB

#2

Update

  • Instead of personal fork, built mpm in TACC directly from CB-Geo with:
git clone https://github.com/cb-geo/mpm.git
  • Continued to produce non-axisymmetric results

image
image1189×684 941 KB

#3

Thanks @jgiven100 for reporting the issue, could you please share the input files so I can try it?

#4

@kks32
Here is a link to all input files.
Thank you for looking into this!

#5

@kks32
Updated link with smaller input files.

  • Axisymmetric outputs observed for develop run locally
  • Unsure about the axisymmetric behavior for develop due to /work directory down for TACC
#6

Thanks @jgiven100, locally I get this:

image
image1704×1294 698 KB

However, when I zoom-in some points break symmetry. Is this expected?

image
image1950×878 739 KB

#7

@kks32 thank you for running locally.

I had intially been focused on pdstrain and not been looking at stresses Magnitude and missed this behavior. For this smaller model, I decreased the number of material points but did not think to decrease the magnitude of external load - I think this is resulting in more element crossings.

Although I am also getting some points which break symetry for stresses Magnitude, the values appear closer aligned than in your output:

image
image1179×675 287 KB

Even with the non-axisymmetric behavior of the stresses Magnitude, it appears my pdstrain output continues to be approximately axisymmetric at the same output step:

image
image1189×682 293 KB

Possible Solution?

I think a possible solution to the original question of this thread might be related to this discussion.

Previous TACC Compiling

Previously compiling on TACC would result in 100s of warnings such as:

image
image852×850 635 KB

Based on the recommendation of others, I had been ignoring these warnings since mpm was being successfully built.

Current TACC Compiling

When compiling using the updated procedures frrom CB-Geo

export CC=icc
export CXX=icpc
mkdir build && cd build && cmake -DBOOST_ROOT=$TACC_BOOST_DIR -DBOOST_INCLUDE_DIRS=$TACC_BOOST_INC -DCMAKE_BUILD_TYPE=Release -DEIGEN3_INCLUDE_DIR=$HOME/eigen -DKAHIP_ROOT=$HOME/KaHIP ..

make -j

I now get 0 warnings during compiling.

Current TACC Output for CB-Geo:Develop

When compiling with the updated procedures (above), my most recent TACC output is identical to my most recent local output (i.e. problem solved). :+1:

image
image1180×677 571 KB

Current TACC Output for Old Branches

If I’m using an outdated personal branch to reporduce older results, using updated compiling procedure (above) will not successfully compile. Therefore, I had been using just:

mkdir build && cd build && cmake -DBOOST_ROOT=$TACC_BOOST_DIR -DBOOST_INCLUDE_DIRS=$TACC_BOOST_INC -DCMAKE_BUILD_TYPE=Release -DEIGEN3_INCLUDE_DIR=$HOME/eigen -DKAHIP_ROOT=$HOME/KaHIP ..

make -j

The output when compiling with this old procedure is consistent with the non-axisymmetric results of the original post for this thread:

image
image1176×675 565 KB

Conclusion

  • Using updated TACC compiling (as currently outlined on CB-Geo website) matches local outputs for CB-Geo:Develop
  • If using an outdated branch, I can only run locally
2 Likes
#8

Thanks @jgiven100 for the detailed summary. We’ll update all the outdated branches, it’s possible they don’t have the NorSand fix in them.

#9

Thanks @jgiven100 for this! Wonderful!