PETSc

Publications: Structured Mesh


Research in New Generation Framework for Petroleum Reservoir Simulation

PIs - Gary Pope, Todd Arbogast, Clint Dawson, Bill Gropp, Larry Lake, Daene McKinney, Tom Morgan, Kamy Spehrnoori, Barry Smith, and Mary Wheeler.

Joint with the Center for Subsurface Modeling and the Center for Petroleum and Geosystems Engineering at the University of Texas at Austin.This DOE funded research project focuses on the development of scalable approaches to petroleum reservoir simulations. Here PETSc provides some of the scalable linear solver technology.

Performance Results

We present sample results for an EOS compositional oil reservoir simulator developed at UT Austin by Peng Wang and John Wheeler that uses the PETSc linear solvers. This problem uses a three-dimensional structured grid, with cell-centered finite differences. There are eight degrees of freedom per cell, and the Jacobian is stored using the PETSc MPIBAIJ matrix format with a block size of four. The Backward Euler method with a truncated Newton solver is used for time evolution of the nonlinear system. The resulting linear systems are solved approximately using GMRES(30) with block Jacobi preconditioning using point-block ILU(0) on each processor. These calculations were performed on a fixed (nonadaptive) grid. Results are given for experiments using an IBM SP (with Power 2 Super processors running at 160MHz and 2 memory cards; 4 memory cards gives about 30 percent better performance).  In a representative run, the linear solver sustained 83 megaflops per processor.

Number of Unknowns Processors Time (Minutes) Unknowns/Node/Minute
512,000 4 10.6 12,075
  8 5.8 11,034
  16 2.9 11,034
1,024,000 8 14.2 9,014
  16 7.5 8,556
8,064,000 32 54.8 4,598
  64 21.0 6,000
16,128,000 64 37.7 6,684
26,880,000 128 32.7 6,422

The following table compares performance on an IBM SP (with Power 2 Super chips running at 160 MHz and two memory cards) to a cluster of 300 MHz Pentium 2 processors connected using faster ethernet and MPICH. This is for a fixed size problem with 179,200 unknowns. (PETSc vector scatter options -vecscatter_merge -vecscatter_ssend).

Processors Matrix Vector Product
Flops
Linear Solver Time Speedup
  SP Cluster SP Cluster SP Cluster
1 106 47 155 364 --- ---
2 198 73 65 168 2.4 2.2
4 382 137 33 84 4.6 4.4
8 691 251 18 46 8.7 8.0
16 1398 443 10 26 15 14