May, 2025
Client
Dr. Terrence Tricco, Associate Professor, Computer Science, Memorial University
Research Consultant
Dr. Chris Geroux
Objective
The project’s goal was to optimize the Message Passing Interface (MPI) parallelization to enable scaling on distributed-memory architectures of the general purpose high-performance Phantom astrophysical simulation code used by dozens of research groups globally to model a wide range of astrophysics objects and systems.
Challenges
1) Phantom’s MPI parallelization has not scaled well on multi-node systems. As high performance computing (HPC) architectures expand their computational resources primarily through increasing node counts, less so by increasing the number of cores per node, this presents a limitation to the long-term health of the Phantom code and collaboration. Larger simulations were needed in order to resolve small-scale dynamics.
2) The group needed access to expertise in MPI parallel programming, performance profiling and optimization on distributed HPC systems, experience in computational fluid dynamics or astrophysical simulations, and access to an HPC system.
Results
ACENET research consultant, Chris Geroux, attempted to reproduce the poor scaling observed by the researcher between two MPI calculations. He conducted a scaling study of the code, running different combinations of number of threads (OpenMP) and number processes (MPI) for different problem types and sizes. This resulted in better understanding the problem.
Chris’ work suggested that both the threaded parallelism and the process based parallelism scale similarly with increasing core count, and that perhaps significantly larger, perhaps several orders of magnitude larger, problem sizes should be explored to see if scaling further improves.
There were also non-reproducable issues with memory access violations uncovered in the MPI version of the code.
The work highlighted several issues with the MPI performance beyond what was already suspected, specifically that the code does not scale well past certain core counts for certain problem sizes. Through this work, the group was able to exclude various potential reasons for this lack of scaling.
Future
The research group will work with the Phantom leadership council to develop a plan on how to build upon the exploration done in this project to create a targeted plan of action. This may include future requests for embedded technical support from ACENET.