Large model modifications to the exodusII library:

The changes are to support the storage of larger models. There are two
pieces of this. The first is the setting of the type of netcdf file
that will be created; either one with 32-bit offsets or one with
64-bit offsets. This can be specified in a couple ways:

1. Pass the EX_LARGE_MODEL flag in the mode argument to ex_create.
2. Set the environment variable EXODUS_LARGE_MODEL.

If either of these are set, then the library will pass the
NC_64BIT_OFFSET flag to the netcdf library. See the commit log for
netcdf library for more details.

The other change is to reduce the size of some of the datasets in an
exodusII library. Even with the netcdf changes, the maximum dataset
size is still 2GB. To reduce the size of the datasets, the nodal
coordinates and the nodal variables have been split to store by
component. The old behavior stored all x, y, and z coordinates in a
single dataset; in the new behavior, each component is stored
separately -- there is a coordx, coordy, and coordz dataset.

The nodal variables used to be stored in a single blob of dimension
(#times,#nodes,#variables). This has now been split into #variable
datasets of size (#times,#nodes).

These two changes should increase the maximum model sizes
significantly. Prior to the change, the maximum number of nodes that
could be stored in the coordinate dataset was about 90 Million nodes;
the new storage permits 270 Million nodes in double precision. The old
model was more restrictive if there were multiple nodal variables, but
the new storage should not depend on the number of nodal variables.

These changes were made such that the new library would create
old-style files by default and would read either old or new style
files. The version has been changed to 3.01 for the file version and
4.01 for the API version.

An additional attribute is now written to the file. It is called
"file_size" or ATT_FILESIZE. If it is 0 or not present, then the old
format is assumed; if it is 1, then the new format is assumed.

There is also a new internal function called ex_large_model(int exoid)
which will return 1 if new version; 0 if old version.

If the function is passed a negative exoid, then it will check the
environment variable "EXODUS_LARGE_MODEL" and return 1 if it is
defined. It also currently prints a warning message saying that the
large model size was selected via the environment variable.

If you are using the exodusII api, then the only change to the client
application is the passing of the EX_LARGE_MODEL flag to ex_create or
the setting of the EXODUS_LARGE_MODEL environment variable. If your
client application is reading the database, no changes are needed.

========================================================================
If your client application bypasses some or all of the exodusII API
and makes direct netcdf calls, you will need to modify the calls.  The
changes that were made are shown below along with the name of the
exodusII API function in which the changes were made. 

Alternatively, you can look at the changes that were made to the API at 
http://jal.sandia.gov/cgi-bin/cvsweb/ACCESS/libraries/exodusii/cbind/src.
The files that were changed are:

   exgnvt.c 	  1.12 
   expcor.c 	  1.12 
   expini.c 	  1.23 
   expnv.c 	  1.13 
   expvp.c 	  1.16 
   expvpc.c 	  1.4 
   ex_utils.c 	  1.33 	
   excopy.c 	  1.13 	
   excre.c 	  1.23 	
   exgcor.c 	  1.13 	
   exgnv.c 	  1.11 	


ex_create:
-- Check whether the EX_LARGE_MODEL mode was set.  If so, then the
mode passed to nccreate must have the NC_64BIT_OFFSET bit set.  For
example, "mode |= NC_64BIT_OFFSET;"

  NOTE: NC_64BIT_OFFSET is defined in the Sandia's netcdf version 
        "3.4-snl10X".  It should also be in netcdf-3.6.0 once it is released.

-- Write the exodus file size "ATT_FILESIZE" attribute (1=large, 0=normal):

   filesiz = (nclong)(((cmode & EX_LARGE_MODEL) != 0) || (ex_large_model(-1) == 1));
   if (ncattput (exoid, NC_GLOBAL, ATT_FILESIZE, NC_LONG, 1, &filesiz) == -1)
    ... handle errors...

ex_put_init:
-- If writing a "large model" capable database, then the coordinates
are defined as components instead of an array.  The variables are
VAR_COORD_X, VAR_COORD_Y (if 2D or 3D), VAR_COORD_Z (if 3D). If not,
define the VAR_COORD variable as is currently done.

     if (ex_large_model(exoid) == 1) {
       /* node coordinate arrays -- separate storage... */
       dim[0] = numnoddim;
       if (ncvardef (exoid, VAR_COORD_X, nc_flt_code(exoid), 1, dim) == -1)
	 { ... handle error }
       
       if (num_dim > 1) {
	 if (ncvardef (exoid, VAR_COORD_Y, nc_flt_code(exoid), 1, dim) == -1)
	   { ... handle error }
       }
       
       if (num_dim > 2) {
	 if (ncvardef (exoid, VAR_COORD_Z, nc_flt_code(exoid), 1, dim) == -1)
	   { ... handle error }
       }
     } else {
       /* node coordinate arrays: -- all stored together (old method) */
       .... define the old way...
     }       

ex_put_coord:
-- If writing a "large model" capable database, then the coordinates
are written a component at a time, otherwise write the old way as a single blob.
  
  if (ex_large_model(exoid) == 0) {
    ... write coordinates old way...
  } else {
    if ((coordidx = ncvarid (exoid, VAR_COORD_X)) == -1)
      { ... handle error }
    
    if (num_dim > 1) {
      if ((coordidy = ncvarid (exoid, VAR_COORD_Y)) == -1)
	{ ... handle error }
    } else {
      coordidy = 0;
    }
    if (num_dim > 2) {
      if ((coordidz = ncvarid (exoid, VAR_COORD_Z)) == -1)
	{ ... handle error }
    } else {
      coordidz = 0;
    }
    /* write out the coordinates  */
    for (i=0; i<= num_vars; i++) {
      dims[0] = time_dim;
      dims[1] = num_nod_dim;
      if ((ncvardef (exoid, VAR_NOD_VAR_NEW(i),
		     nc_flt_code(exoid), 2, dims)) == -1)
	{  ... handle error ...	}
    }
  }

ex_put_nodal_var:
 -- If the large model method, write the nodal variable data to the correct variable;
    if the old method, determine the location within the blob

   if (ex_large_model(exoid) == 0) {
     /* write values of the nodal variable */
     if ((varid = ncvarid (exoid, VAR_NOD_VAR)) == -1) {
       ... handle error...
     }
     start[0] = --time_step;
     start[1] = --nodal_var_index;
     start[2] = 0;
     
     count[0] = 1;
     count[1] = 1;
     count[2] = num_nodes;
   } else {
     /* nodal variables stored separately, find variable for this variable
	index */
     if ((varid = ncvarid (exoid, VAR_NOD_VAR_NEW(nodal_var_index))) == -1) {
       ... handle error ...
     }
       
     start[0] = --time_step;
     start[1] = 0;

     count[0] = 1;
     count[1] = num_nodes;
   }
 
   if (ncvarput (exoid, varid, start, count,
		 ex_conv_array(exoid,WRITE_CONVERT,nodal_var_vals,num_nodes)) == -1) {
     ...handle error ...
   }

If there are any questions, contact:
  Greg Sjaardema
  [email protected]
  505-844-2701