PETSc

Publications:  Unstructured Mesh


Multi-Model Multi-Domain Computational Methods in Aerodynamics and Acoustics

PIs - David Keyes, Alex Pothen (Old Dominion University); Hafiz Atassi (University of Notre Dame); Xiao-Chuan Cai (University of Colorado-Boulder); William Gropp, Lois Curfman McInnes, Barry Smith (Argonne National Laboratory); Olof Widlund (Courant Institute of Mathematical Sciences, NYU); David P. Young (Boeing Computer Services).

This NSF funded multidisciplinary research project focuses on the development of effective multi-domain parallel solutions to multi-model coupled field problems. Here PETSc provides the foundation for designing advanced domain decomposition software for use within the context of a numerical laboratory for engineering analysis.

Performance Results

The following sample results were generated by an incompressible Euler flow code developed by Dinesh Kaushik and David Keyes at ODU and ICASE; this parallel code is based on a legacy sequential Fortran77 code written by W. Kyle Anderson of  NASA Langley. The code uses a three-dimensional, unstructured tetrahedral grid with an edge-based finite volume discretization. This code computes steady-state solutions using pseudo-transient continuation with a truncated Newton method that employs GMRES(30) with block Jacobi preconditioning using point-block ILU(0) on each processor. For this simulation PETSc provides not only the linear and nonlinear solvers, but also the ghost point update utilities. These calculations were performed on a fixed (nonadaptive) grid. The problem has 2,761,774 vertices (for a total of 11,047,096 unknowns). Results on a Cray T3E-900 (runs at 450MHz).

Fixed-Size Scalability

Processors Iterations Time Efficiency Mflops/Processor Total Gigaflops
128 164 6,048

---

67 8.5
256 166 3,242 93% 65 16.6
512 171 1,811

83%

63 32.1

The floating point rates are sustained over the entire calculation.

Gustafson-Style Scalability (Scaled Speedup)

Vertices Processors Vertices/Processor Iterations Time Time/Iteration
357,900 80 4474 78 560 7.18
53,961 12 4497 36 266 7.38
9,428 2 4714 19 131 6.89