The ESMF infrastructure data classes are part of the framework's hierarchy of structures for handling Earth system model data and metadata on parallel platforms. The hierarchy is in complexity; the simplest data class in the infrastructure represents a distributed array and the most complex data class represents a bundle of physical fields that are discretized on the same grid. Data class methods are called both from user-written code and from other classes internal to the framework.
Data classes are distributed over DEs, or Decomposition Elements. A DE represents a piece of a decomposition. A DELayout is a collection of DEs with some associated connectivity that describes a specific distribution. For example, the distribution of a grid divided into four segments in the x-dimension would be expressed in ESMF as a DELayout with four DEs lying along an x-axis. This abstract concept enables a data decomposition to be defined in terms of threads, MPI processes, virtual decomposition elements, or combinations of these without changes to user code. This is a primary strategy for ensuring optimal performance and portability for codes using the ESMF for communications.
ESMF data classes are useful because they provide a standard, convenient way for developers to collect together information related to model or observational data. The information assembled in a data class includes a data pointer, a set of attributes (e.g. units, although attributes can also be user-defined), and a description of an associated grid. The same set of information within an ESMF data object can be used by the framework to arrange intercomponent data transfers, to perform I/O, for communications such as gathers and scatters, for simplification of interfaces within user code, for debugging, and for other functions. This unifies and organizes codes overall so that the user need not define different representations of metadata for the same field for I/O and for component coupling.
Since it is critical that users be able to introduce ESMF into their codes easily and incrementally, ESMF data classes can be created based on native Fortran pointers. Likewise, there are methods for retrieving native Fortran pointers from within ESMF data objects. This allows the user to perform allocations using ESMF, and to retrieve Fortran arrays later for optimized model calculations. The ESMF data classes do not have associated differential operators or other mathematical methods.
For flexibility, it is not necessary to build an ESMF data object all at once. For example, it's possible to create a field but to defer allocation of the associated field data until a later time.
|
Key Features |
| Hierarchy of data structures designed specifically for the Earth system domain and high performance, parallel computing. |
| Multi-use ESMF structures simplify user code overall. |
| Data objects support incremental construction and deferred allocation. |
| Native Fortran arrays can be associated with or retrieved from ESMF data objects, for ease of adoption, convenience, and performance. |
The main classes that are used for model and observational data manipulation are as follows:
Data elements in Arrays are partitioned into categories defined by the role the data element plays in distributed halo operations. Haloing - sometimes called ghosting - is the practice of copying portions of array data to multiple memory locations to ensure that data dependencies can be satisfied quickly when performing a calculation. ESMF Arrays contain an exclusive domain, which contains data elements updated exclusively and definitively by a given DE; a computational domain, which contains all data elements with values that are updated by the DE in computations; and a total domain, which includes both the computational domain and data elements from other DEs which may be read but are not updated in computations.
Bundle objects contain methods for setting and retrieving constituent fields, regridding, data I/O, and reordering of data in memory.
The following is a simplified UML diagram showing the relationships among ESMF Field, Grid and Bundle classes. See Appendix A, A Brief Introduction to UML, for a translation table that lists the symbols in the diagram and their meaning.
The Bundle class represents ``bundles'' of Fields that are discretized on the same Grid and distributed in the same manner. Fields within a Bundle may be located at different locations relative to the vertices of their common Grid. The Fields in a Bundle may be of different dimensions, as long as the Grid dimensions that are distributed are the same. For example, a surface Field on a distributed lat/lon Grid and a 3D Field with an added vertical dimension on the same distributed lat/lon Grid can be included in the same Bundle.
Bundles currently function mainly as convenient containers for storing Fields. Bundles can be created and destroyed, can have attributes added or retrieved, and can have Fields added or retrieved. Methods include a variety of queries that return information about the attributes and the Fields that a Bundle contains. The Fortran data pointer of a Field within a Bundle can be obtained by passing the Bundle a Field name.
Memory layout information is stored in a BundleDataMap object which is attached to the Bundle. It can be accessed by querying the Bundle. Although we have made the BundleDataMap public, many of the memory layout options have not been implemented.
Bundles are one of the data objects that can be added to States, which are used for sending to or receiving data from other components.
In the future Bundles will serve as a mechanism for performance optimization. ESMF will take advantage of the similarities of the Fields within a Bundle in order to implement collective communication, IO, and regridding. See Section 17.4 for a description of features that are being planned.
DESCRIPTION:
Specifies whether a Bundle is packed or not. A packed
Bundle contains an array in which all the data in its
constituent Fields is packed contiguously. Bundles that
are not packed are not guaranteed to carry a contiguous
array of their data. This flag is not yet implemented;
the value is always set to ESMF_NO_PACKED_DATA.
Valid values are:
Examples of creating, destroying and accessing Bundles and their constituent Fields are provided in this section, along with some notes on Bundle methods.
After creating multiple Fields, a Bundle can be created by passing a list of the Fields into the method ESMF_BundleCreate(). The Bundle will contain references to the Fields. An empty Bundle can also be created and Fields added singularly or in groups.
The feature which requests a packed Array be created from the combined Field data arrays is not implemented in this version of the framework.
To access data in a Bundle the user can provide a Field name and retrieve the Field's Fortran data pointer. Alternatively, the user can retrieve the data in the form of an ESMF Field and use the Field-level interfaces.
The packed Array feature of Bundles is not implemented in this version of the Framework.
The user must call ESMF_BundleDestroy() before deleting any of the Fields it contains. Because Fields can be shared by multiple Bundles and States, they are not deleted by this call.
See the following code fragments for examples of how to create new Bundles.
! Example program showing various ways to create a Bundle object.
program ESMF_BundleCreateEx
! ESMF Framework module
use ESMF_Mod
implicit none
! Local variables
integer :: i, rc, fieldcount
type(ESMF_Grid) :: grid
type(ESMF_ArraySpec) :: arrayspec
!type(ESMF_FieldDataMap) :: datamap
type(ESMF_DELayout) :: delayout
type(ESMF_VM) :: vm
character (len = ESMF_MAXSTR) :: bname1, fname1, fname2
type(ESMF_Field) :: field(10), returnedfield1, returnedfield2, simplefield
type(ESMF_Bundle) :: bundle1, bundle2, bundle3
!real (selected_real_kind(6,45)), dimension(:,:), pointer :: f90ptr1, f90ptr2
integer :: counts(2)
real(ESMF_KIND_R8) :: min_coord(2), max_coord(2)
! ! Create several Fields and add them to a new Bundle.
! counts = (/ 100, 200 /)
! min_coord = (/ 0.0, 0.0 /)
! max_coord = (/ 50.0, 60.0 /)
delayout = ESMF_DELayoutCreate(vm, rc=rc)
grid = ESMF_GridCreateShapeTile(minIndex=(/1,1/), maxIndex=(/100,200/), &
regDecomp=(/2,2/), name="atmgrid", rc=rc)
call ESMF_ArraySpecSet(arrayspec, 2, ESMF_TYPEKIND_R8, rc=rc)
if (rc.NE.ESMF_SUCCESS) finalrc = ESMF_FAILURE
field(1) = ESMF_FieldCreate(grid, arrayspec, &
staggerloc=ESMF_STAGGERLOC_CENTER, &
name="pressure", rc=rc)
field(2) = ESMF_FieldCreate(grid, arrayspec, &
staggerloc=ESMF_STAGGERLOC_CENTER, &
name="temperature", rc=rc)
field(3) = ESMF_FieldCreate(grid, arrayspec, &
staggerloc=ESMF_STAGGERLOC_CENTER, &
name="heat flux", rc=rc)
bundle1 = ESMF_BundleCreate(3, field, name="atmosphere data", rc=rc)
print *, "Bundle example 1 returned"
!-------------------------------------------------------------------------
! ! Create an empty Bundle and then add a single field to it.
simplefield = ESMF_FieldCreate(grid, arrayspec, &
staggerloc=ESMF_STAGGERLOC_CENTER, name="rh", rc=rc)
bundle2 = ESMF_BundleCreate(name="time step 1", rc=rc)
call ESMF_BundleAddField(bundle2, simplefield, rc)
call ESMF_BundleGet(bundle2, fieldCount=fieldcount, rc=rc)
print *, "Bundle example 2 returned, fieldcount =", fieldcount
!-------------------------------------------------------------------------
! ! Create an empty Bundle and then add multiple fields to it.
bundle3 = ESMF_BundleCreate(name="southern hemisphere", rc=rc)
call ESMF_BundleAddField(bundle3, 3, field, rc)
call ESMF_BundleGet(bundle3, fieldCount=fieldcount, rc=rc)
print *, "Bundle example 3 returned, fieldcount =", fieldcount
!-------------------------------------------------------------------------
! ! Get a Field back from a Bundle, first by name and then by index.
! ! Also get the Bundle name.
call ESMF_BundleGetField(bundle1, "pressure", returnedfield1, rc)
call ESMF_FieldGet(returnedfield1, name=fname1, rc=rc)
call ESMF_BundleGetField(bundle1, 2, returnedfield2, rc)
call ESMF_FieldGet(returnedfield2, name=fname2, rc=rc)
call ESMF_BundleGet(bundle1, name=bname1, rc=rc)
print *, "Bundle example 4 returned, field names = ", &
trim(fname1), ", ", trim(fname2)
print *, "Bundle name = ", trim(bname1)
!-------------------------------------------------------------------------
call ESMF_BundleDestroy(bundle1, rc=rc)
call ESMF_BundleDestroy(bundle2, rc=rc)
call ESMF_BundleDestroy(bundle3, rc=rc)
do i=1, 3
call ESMF_FieldDestroy(field(i),rc=rc)
enddo
call ESMF_FieldDestroy(simplefield, rc=rc)
end program ESMF_BundleCreateEx
INTERFACE:
! Private name; call using ESMF_BundleAddField()
subroutine ESMF_BundleAddOneField(bundle, field, rc)
ARGUMENTS:
type(ESMF_Bundle), intent(inout) :: bundle
type(ESMF_Field), intent(inout) :: field
integer, intent(out), optional :: rc
DESCRIPTION:
Adds a single field to an existing bundle. The field must be associated with the same ESMF_Grid as the other ESMF_Fields in the bundle. The field is referenced by the bundle, not copied.
The arguments are:
INTERFACE:
! Private name; call using ESMF_BundleAddField()
subroutine ESMF_BundleAddFieldList(bundle, fieldCount, fieldList, rc)
ARGUMENTS:
type(ESMF_Bundle), intent(inout) :: bundle
integer, intent(in) :: fieldCount
type(ESMF_Field), dimension(:), intent(inout) :: fieldList
integer, intent(out), optional :: rc
DESCRIPTION:
Adds a fieldList to an existing ESMF_Bundle. The items added from the ESMF_fieldList must be associated with the same ESMF_Grid as the other ESMF_Fields in the bundle. The items in the fieldList are referenced by the bundle, not copied.
The arguments are:
INTERFACE:
! Private name; call using ESMF_BundleCreate()
function ESMF_BundleCreateNew(fieldCount, fieldList, &
packflag, bundleinterleave, name, iospec, rc)
RETURN VALUE:
type(ESMF_Bundle) :: ESMF_BundleCreateNewARGUMENTS:
integer, intent(in) :: fieldCount
type(ESMF_Field), dimension (:) :: fieldList
type(ESMF_PackFlag), intent(in), optional :: packflag
type(ESMF_InterleaveFlag), intent(in), optional :: bundleinterleave
character (len = *), intent(in), optional :: name
type(ESMF_IOSpec), intent(in), optional :: iospec
integer, intent(out), optional :: rc
DESCRIPTION:
Creates an ESMF_Bundle from a list of existing ESMF_Fields stored in a fieldList. All items in the fieldList must be associated with the same ESMF_Grid. Returns a new ESMF_Bundle.
The arguments are:
![[*]](crossref.png)
for anticipated values. The flag is not applicable
to the current implementation,
since it applies only to packed data (see the packflag argument).
INTERFACE:
! Private name; call using ESMF_BundleCreate()
function ESMF_BundleCreateNoFields(grid, name, iospec, rc)
RETURN VALUE:
type(ESMF_Bundle) :: ESMF_BundleCreateNoFieldsARGUMENTS:
type(ESMF_Grid), intent(in), optional :: grid
character (len = *), intent(in), optional :: name
type(ESMF_IOSpec), intent(in), optional :: iospec
integer, intent(out), optional :: rc
DESCRIPTION:
Creates an ESMF_Bundle with no associated ESMF_Fields.
The arguments are:
INTERFACE:
subroutine ESMF_BundleDestroy(bundle, rc)ARGUMENTS:
type(ESMF_Bundle) :: bundle
integer, intent(out), optional :: rc
DESCRIPTION:
Releases resources associated with the bundle. This method does not destroy the ESMF_Fields that the bundle contains. The bundle should be destroyed before the ESMF_Fields within it are.
INTERFACE:
subroutine ESMF_BundleGet(bundle, grid, fieldCount, name, rc)ARGUMENTS:
type(ESMF_Bundle), intent(inout) :: bundle
type(ESMF_Grid), intent(out), optional :: grid
integer, intent(out), optional :: fieldCount
character (len = *), intent(out), optional :: name
integer, intent(out), optional :: rc
DESCRIPTION:
Returns information about the bundle. If the ESMF_Bundle was originally created without specifying a name, a unique name will have been generated by the framework.
The arguments are:
INTERFACE:
subroutine ESMF_BundleGetAttribute(bundle, name, <value argument>, rc)ARGUMENTS:
type(ESMF_Bundle), intent(inout) :: bundle
character (len = *), intent(in) :: name
<value argument>, see below for supported values
integer, intent(out), optional :: rc
DESCRIPTION:
Returns an attribute from the bundle. Supported values for <value argument> are:
The arguments are:
INTERFACE:
subroutine ESMF_BundleGetAttributeCount(bundle, count, rc)ARGUMENTS:
type(ESMF_Bundle), intent(inout) :: bundle
integer, intent(out) :: count
integer, intent(out), optional :: rc
DESCRIPTION:
Returns the number of attributes associated with the given bundle in the argument count.
The arguments are:
INTERFACE:
! Private name; call using ESMF_BundleGetAttributeInfo()
subroutine ESMF_BundleGetAttrInfoByName(bundle, name, typekind, count, rc)
ARGUMENTS:
type(ESMF_Bundle), intent(in) :: bundle
character(len=*), intent(in) :: name
type(ESMF_TypeKind), intent(out), optional :: typekind
integer, intent(out), optional :: count
integer, intent(out), optional :: rc
DESCRIPTION:
Returns information associated with the named attribute, including typekind and item count.
The arguments are:
INTERFACE:
! Private name; call using ESMF_BundleGetAttributeInfo()
subroutine ESMF_BundleGetAttrInfoByNum(bundle, attributeIndex, name, &
typekind, count, rc)
ARGUMENTS:
type(ESMF_Bundle), intent(inout) :: bundle
integer, intent(in) :: attributeIndex
character(len=*), intent(out), optional :: name
type(ESMF_TypeKind), intent(out), optional :: typekind
integer, intent(out), optional :: count
integer, intent(out), optional :: rc
DESCRIPTION:
Returns information associated with the indexed attribute, including typekind and item count.
The arguments are:
INTERFACE:
! Private name; call using ESMF_BundleGetField()
subroutine ESMF_BundleGetFieldByName(bundle, name, field, rc)
ARGUMENTS:
type(ESMF_Bundle), intent(inout) :: bundle
character (len = *), intent(in) :: name
type(ESMF_Field), intent(out) :: field
integer, intent(out), optional :: rc
DESCRIPTION:
Returns a field from a bundle using the field's name.
The arguments are:
INTERFACE:
! Private name; call using ESMF_BundleGetField()
subroutine ESMF_BundleGetFieldByNum(bundle, fieldIndex, field, rc)
ARGUMENTS:
type(ESMF_Bundle), intent(inout) :: bundle
integer, intent(in) :: fieldIndex
type(ESMF_Field), intent(out) :: field
integer, intent(out), optional :: rc
DESCRIPTION:
Returns a field from a bundle by index number.
The arguments are:
INTERFACE:
subroutine ESMF_BundleGetFieldNames(bundle, nameList, nameCount, rc)ARGUMENTS:
type(ESMF_Bundle), intent(inout) :: bundle
character (len = *), intent(out) :: nameList(:)
integer, intent(out), optional :: nameCount
integer, intent(out), optional :: rc
DESCRIPTION:
Returns an array of ESMF_Field names in an ESMF_Bundle.
The arguments are:
INTERFACE:
subroutine ESMF_BundlePrint(bundle, options, rc)ARGUMENTS:
type(ESMF_Bundle), intent(inout) :: bundle
character (len=*), intent(in), optional :: options
integer, intent(out), optional :: rc
DESCRIPTION:
Prints diagnostic information about the bundle to stdout.
The arguments are:
INTERFACE:
subroutine ESMF_BundleSetAttribute(bundle, name, <value argument>, rc)ARGUMENTS:
type(ESMF_Bundle), intent(inout) :: bundle
character (len = *), intent(in) :: name
<value argument>, see below for supported values
integer, intent(out), optional :: rc
DESCRIPTION:
Attaches an attribute to the bundle. The attribute has a name and either a value or a valueList. Supported values for the <value argument> are:
The arguments are:
INTERFACE:
subroutine ESMF_BundleSetGrid(bundle, grid, rc)ARGUMENTS:
type(ESMF_Bundle), intent(inout) :: bundle
type(ESMF_Grid), intent(in) :: grid
integer, intent(out), optional :: rc
DESCRIPTION:
Sets the grid for a bundle that contains no ESMF_Fields. All ESMF_Fields added to this bundle must be associated with the same ESMF_Grid. Returns an error if there is already an ESMF_Grid associated with the bundle.
The arguments are:
INTERFACE:
subroutine ESMF_BundleValidate(bundle, options, rc)ARGUMENTS:
type(ESMF_Bundle), intent(in) :: bundle
character (len=*), intent(in), optional :: options
integer, intent(out), optional :: rc
DESCRIPTION:
Validates that the bundle is internally consistent. Currently this method determines if the bundle is uninitialized or already destroyed. The method returns an error code if problems are found.
The arguments are:
An ESMF Field represents a physical field, such as temperature. The motivation for including Fields in ESMF is that bundles of Fields are the entities that are normally exchanged when coupling Components.
The ESMF Field class contains discretized field data, a reference to its associated grid, and metadata. The Field class maintains the relationship of how a data array maps onto a grid (e.g. one item per cell located at the cell center, one item per cell located at the NW corner, one item per cell vertex). This means that different Fields which are on the same underlying ESMF Grid but have different staggerings can share the same Grid object without needing to replicate it multiple times.
ESMF does not currently support vector fields, so the components of a vector field must be stored as separate Field objects.
The Field class provides methods for initialization, setting and retrieving data values, general data redistribution and regridding, standard communication methods such as gather and scatter, and manipulation of attributes. Field methods are called from objects internal to the framework and can also be called from user code.
A Field serves as an annotator of data, since it carries a description of the grid it is associated with and metadata such as name and units. Fields can be used in this capacity alone, as convenient, descriptive containers into which arrays can be placed and retrieved. However, for most codes the primary use of Fields is in the context of import and export States, which are the objects that carry coupling information between Components. Fields enable data to be self-describing, and a State holding ESMF Fields contains data in a standard format that is easy to query and manipulate.
The information that is necessary for describing a Field to another Component is similar to the information needed to write history files. Another use of Fields is as a mechanism for writing out data to files for history and restart.
The sections below go into more detail about Field usage.
Fields can be created and destroyed at any time during application execution. However, these Field methods require some time to complete. We do not recommend that the user create or destroy Fields inside performance-critical computational loops.
All versions of the ESMF_FieldCreate() routines require a Grid object as input, or require a Grid be added before most operations involving Fields can be performed. The Grid contains the information needed to know which Decomposition Elements (DEs) are participating in the processing of this Field, and which subsets of the data are local to a particular DE.
The details of how the create process happens depends on which of the variants of the ESMF_FieldCreate() call is used. Some of the variants are discussed below.
There are versions of the ESMF_FieldCreate() interface which create the Field based on the input Grid. The ESMF can allocate the proper amount of space but not assign initial values. The user code can then get the pointer to the uninitialized buffer and set the initial data values.
Other versions of the ESMF_FieldCreate() interface allow user code to attach arrays that have already been allocated by the user. Empty Fields can also be created in which case the data can be added at some later time.
For versions of Create which do not specify data values, user code can create an ArraySpec object, which contains information about the Type, TypeKind, and Rank of the data values in the array. Then at Field create time, the appropriate amount of memory is allocated to contain the data which is local to each DE.
There is a ESMF_FieldDestroy() method which releases any data buffers which were allocated or internally referenced by this Field, and deletes the Field object. Since a single Grid reference can be shared by multiple Fields, the internal Grid is not deleted by this call. Field provides copy behavior through creation and set interface, the internally referenced ESMF_Array object will be deleted upon Field destruction.
grid = ESMF_GridCreateShapeTile(minIndex=(/1,1/), maxIndex=(/10,20/), &
regDecomp=(/2,2/), name="atmgrid", rc=rc)
if (rc.NE.ESMF_SUCCESS) finalrc = ESMF_FAILURE
call ESMF_GridGet(grid, distgrid=distgrid, rc=rc)
if (rc.NE.ESMF_SUCCESS) finalrc = ESMF_FAILURE
call ESMF_ArraySpecSet(arrayspec, 2, ESMF_TYPEKIND_R4, rc)
if (rc.NE.ESMF_SUCCESS) finalrc = ESMF_FAILURE
field1 = ESMF_FieldCreate(grid, arrayspec, &
staggerloc=ESMF_STAGGERLOC_CENTER, name="pressure", rc=rc)
if (rc.NE.ESMF_SUCCESS) finalrc = ESMF_FAILURE
call ESMF_GridGet(grid, staggerloc=ESMF_STAGGERLOC_CENTER, &
computationalEdgeLWidth=compEdgeLWdith, &
computationalEdgeUWidth=compEdgeUWdith, rc=rc)
if (rc.NE.ESMF_SUCCESS) finalrc = ESMF_FAILURE
array2 = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, staggerLoc=0, &
computationalEdgeLWidth=compEdgeLWdith, &
computationalEdgeUWidth=compEdgeUWdith, rc=rc)
if(rc .ne. ESMF_SUCCESS) finalrc = ESMF_FAILURE
call ESMF_FieldSetArray(field1, array2, rc=rc)
if (rc.NE.ESMF_SUCCESS) finalrc = ESMF_FAILURE
When finished with an ESMF_Field, the destroy method removes it. However, the objects inside the ESMF_Field that has external reference should be deleted separately, since objects can be added to more than one ESMF_Field, for example the same ESMF_Grid can be used in multiple ESMF_Fields.
call ESMF_FieldDestroy(field1, rc=rc)
Through the ESMF_FieldSetDataPtr interface user can reset the intrinsic Fortran data pointer contained in the internal ESMF_Array object of a ESMF_Field. ESMF_FieldSetDataPtr is an overloaded interface based on the type, kind, and rank of the input fortran pointer argument. In this example, a rank 3 ESMF_KIND_R8 fortran data pointer is used. This method creates an internally referenced ESMF_Array inside the field. The previous ESMF_Array will be deleted if it was internally referenced by the field, otherwise it's replaced by the newly created array.
allocate(farray(xdim,ydim,zdim))
call ESMF_FieldSetDataPtr(f8, farray, rc=rc)
User can reset the internal ESMF_Grid object of a ESMF_Field through ESMF_FieldSetGrid interface. Invalid Grid will be rejected, check return code for status.
grid = ESMF_GridCreateShapeTile(minIndex=(/1,1,1/), maxIndex=(/4*xdim-1,ydim-1,zdim-1/), &
regDecomp=(/4,1,1/), name="grid", rc=rc)
if(rc .ne. ESMF_SUCCESS) finalrc = ESMF_FAILURE
call ESMF_FieldSetGrid(f8, grid=grid, rc=rc)
INTERFACE:
subroutine ESMF_FieldDestroy(field, rc)ARGUMENTS:
type(ESMF_Field) :: field
integer, intent(out), optional :: rc
DESCRIPTION:
Releases all resources associated with the ESMF_Field.
The arguments are:
INTERFACE:
subroutine ESMF_FieldGet(field, grid, array, &
typekind, rank, staggerloc, &
gridToFieldMap, ungriddedLBound, &
ungriddedUBound, maxHaloLWidth, &
maxHaloUWidth, name, iospec, rc)
ARGUMENTS:
type(ESMF_Field), intent(inout) :: field
type(ESMF_Grid), intent(out), optional :: grid
type(ESMF_Array), intent(out), optional :: array
type(ESMF_TypeKind), intent(out), optional :: typekind
integer, intent(out), optional :: rank
type(ESMF_StaggerLoc), intent(out), optional :: staggerloc
integer, intent(out), optional :: gridToFieldMap(:)
integer, intent(out), optional :: ungriddedLBound(:)
integer, intent(out), optional :: ungriddedUBound(:)
integer, intent(out), optional :: maxHaloLWidth(:)
integer, intent(out), optional :: maxHaloUWidth(:)
character(len=*), intent(out), optional :: name
type(ESMF_IOSpec), intent(out), optional :: iospec ! NOT IMPLEMENTED
integer, intent(out), optional :: rc
DESCRIPTION:
Query an ESMF_Field for various things. All arguments after the Field are optional. To select individual items use the named_argument=value syntax.
The arguments are:
INTERFACE:
subroutine ESMF_FieldGetArray(field, array, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field
type(ESMF_Array), intent(out) :: array
integer, intent(out), optional :: rc
DESCRIPTION:
Get data in ESMF_Array form.
The arguments are:
INTERFACE:
subroutine ESMF_FieldGetAttribute(field, name, <value argument>, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field
character (len = *), intent(in) :: name
<value argument>, see below for supported values
integer, intent(out), optional :: rc
DESCRIPTION:
Returns an attribute from the field. Supported values for <value argument> are:
The arguments are:
INTERFACE:
subroutine ESMF_FieldGetAttributeCount(field, count, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field
integer, intent(out) :: count
integer, intent(out), optional :: rc
DESCRIPTION:
Returns the number of attributes associated with the given field in the argument count.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldGetAttributeInfo()
subroutine ESMF_FieldGetAttrInfoByName(field, name, typekind, count, rc)
ARGUMENTS:
type(ESMF_Field), intent(inout) :: field
character(len=*), intent(in) :: name
type(ESMF_TypeKind), intent(out), optional :: typekind
integer, intent(out), optional :: count
integer, intent(out), optional :: rc
DESCRIPTION:
Returns information associated with the named attribute, including typekind and count.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldGetAttributeInfo()
subroutine ESMF_FieldGetAttrInfoByNum(field, attributeIndex, name, &
typekind, count, rc)
ARGUMENTS:
type(ESMF_Field), intent(inout) :: field
integer, intent(in) :: attributeIndex
character(len=*), intent(out), optional :: name
type(ESMF_TypeKind), intent(out), optional :: typekind
integer, intent(out), optional :: count
integer, intent(out), optional :: rc
DESCRIPTION:
Returns information associated with the indexed attribute, including name, typekind and count.
The arguments are:
INTERFACE:
subroutine ESMF_FieldPrint(field, options, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field
character (len = *), intent(in), optional :: options
integer, intent(out), optional :: rc
DESCRIPTION:
Prints information about the field to stdout. This subroutine goes through the internal data members of a field data type and prints information of each data member.
The arguments are:
INTERFACE:
subroutine ESMF_FieldSetArray(field, array, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field
type(ESMF_Array), intent(in) :: array
integer, intent(out), optional :: rc
DESCRIPTION:
Set data in ESMF_Array form.
The arguments are:
INTERFACE:
subroutine ESMF_FieldSetAttribute(field, name, <value argument>, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field
character (len = *), intent(in) :: name
<value argument>, see below for supported values
integer, intent(out), optional :: rc
DESCRIPTION:
Attaches an attribute to the field. The attribute has a name and either a value or a valueList. Supported values for the <value argument> are:
The arguments are:
INTERFACE:
subroutine ESMF_FieldValidate(field, options, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field
character (len = *), intent(in), optional :: options
integer, intent(out), optional :: rc
DESCRIPTION:
Validates that the field is internally consistent. Currently this method determines if the field is uninitialized or already destroyed. It validates the contained array and grid objects. The code also checks if the array and grid sizes agree. This check compares the distgrid contained in array and grid; then it proceeds to compare the computational bounds contained in array and grid.
The method returns an error code if problems are found.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate()
function ESMF_FieldCreateNew(grid, arrayspec, allocflag, staggerloc, &
gridToFieldMap, ungriddedLBound, &
ungriddedUBound, maxHaloLWidth, &
maxHaloUWidth, name, iospec, rc)
RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateNewARGUMENTS:
type(ESMF_Grid) :: grid
type(ESMF_ArraySpec), intent(inout) :: arrayspec
type(ESMF_AllocFlag), intent(in), optional :: allocflag
type(ESMF_StaggerLoc), intent(in), optional :: staggerloc
integer, intent(in), optional :: gridToFieldMap(:)
integer, intent(in), optional :: ungriddedLBound(:)
integer, intent(in), optional :: ungriddedUBound(:)
integer, intent(in), optional :: maxHaloLWidth(:)
integer, intent(in), optional :: maxHaloUWidth(:)
character (len=*), intent(in), optional :: name
type(ESMF_IOSpec), intent(in), optional :: iospec
integer, intent(out), optional :: rc
DESCRIPTION:
An interface function to ESMF_FieldCreate(). Create an ESMF_Field and allocate space internally for a gridded ESMF_Array. Return a new ESMF_Field.
The arguments are:
The ESMF_Array class is an index space based, distributed data storage class. It provides DE-local memory allocations within DE-centric index regions and defines the relationship to the index space described by DistGrid. The Array class offers common communication patterns within the index space formalism. As part of the ESMF index space layer Array has close relationship to the DistGrid and DELayout classes.
An ESMF_Array is a distributed object that must exist on all PETs of the current context. Each PET-local instance of an Array object contains memory allocations for all PET-local DEs. There may be 0, 1, or more DEs per PET and the number of DEs per PET can differ between PETs for the same Array object. Memory allocations may be provided for each PET by the user during Array creation or can be allocated as part of the Array create call. Many of the concepts of the proposed ESMF_Array class are illustrated by the following examples.
The create call of the ESMF_Array class has been overloaded extensively to facilitate the need for generality while keeping simple cases simple. The following program demonstrates one of the simpler cases, where existing local Fortran90 arrays are to be used to provide the PET-local memory allocations for the Array object.
program ESMF_ArrayFarrayEx use ESMF_Mod implicit none
The Fortran90 language provides a variety of ways to define and allocate an array. Actual Fortran90 array objects must either be explicit-shape or deferred-shape. In the first case the memory allocation and deallocation is automatic from the user's perspective and the details of the allocation (static or dynamic, heap or stack) are left to the compiler. (Compiler flags may be used to control some of the details). In the second case, i.e. for deferred-shape actual objects, the array definition must include the pointer or allocatable attribute and it is the user's responsibility to allocate memory. While it is also the user's responsibility to deallocate memory for arrays with pointer attribute the compiler will automatically deallocate allocatable arrays under certain circumstances defined by the Fortran standard.
The ESMF_ArrayCreate() interface has been written to accept native Fortran90 arrays of any flavor as a means to allow user-contolled memory management. The Array create call will check on each PET if sufficient memory has been provided by the specified Fortran90 arrays and will indicate an error if a problem is detected. However, the Array create call cannot validate the lifetime of the provided memory allocations. If, for instance, an Array object was created in a subroutine from an automatic explicit-shape array or an allocatable array, the memory allocations referenced by the Array object will be automatically deallocated on return from the subroutine unless provissions are made by the application writer to prevent such behavior. The Array object cannot contol when memory that has been provided by the user during Array creation becomes deallocated, however, the Array will indicate an error if it's memory references have been invalidated.
The easiest, portable way to provide safe native Fortran90 memory allocations to Array create is to use arrays with the pointer attribute. Memory allocated for an array pointer will not be deallocated automatically. However, in this case the possibility of memory leaks becomes an issue of concern. The deallocation of memory provided to an Array in form of a native Fortan90 allocation will remain the users responsibility.
None of the concerns discussed above are an issue in this example where the
native Fortran90 array farray is defined in the main program. All
different types of array memory allocation are demonstrated in this example.
First farrayE is defined as a 2D explicit-shape array on each PET which
will automatically provide memory for 
elements.
! local variables real(ESMF_KIND_R8) :: farrayE(10,10) ! explicit shape F90 array
Then an allocatable array farrayA is declared which will be used to show user-controlled dynamic memory allocation.
real(ESMF_KIND_R8), allocatable :: farrayA(:,:) ! allocatable F90 array
Finally an array with pointer attribute farrayP is declared, also used for user-controlled dynamic memory allocation.
real(ESMF_KIND_R8), pointer :: farrayP(:,:) ! F90 array pointer
A matching array pointer must also be available to gain access to the arrays held by an Array object.
real(ESMF_KIND_R8), pointer :: farrayPtr(:,:) ! matching F90 array pointer type(ESMF_DistGrid) :: distgrid ! DistGrid object type(ESMF_Array) :: array ! Array object integer :: rc
call ESMF_Initialize(rc=rc) if (rc /= ESMF_SUCCESS) call ESMF_Finalize(terminationflag=ESMF_ABORT)
On each PET farrayE can be accessed directly to initialize the entire PET-local array.
farrayE = 12.45d0 ! initialize to some value
In order to create an Array object a DistGrid must first be created that describes the total index space and how it is decomposed and distributed. In the simplest case only the minIndex and maxIndex of the total space must be provided.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/40,10/), rc=rc)
This example is assumed to run on 4 PETs. The default 2D decomposition will then be into 4 x 1 DEs as to ensure 1 DE per PET.
Now the Array object can be created using the farrayE and the DistGrid just created.
array = ESMF_ArrayCreate(farray=farrayE, distgrid=distgrid, rc=rc)
The 40 x 10 index space defined by the minIndex and maxIndex arguments paired with the default decomposition will result in the following distributed Array.
+---------------------------> 2nd dimension
| (1,1)-------+
| | |
| | DE 0 | <--- farray on PET 0
| | |
| +------(10,10)
| (11,1)-------+
| | |
| | DE 1 | <--- farray on PET 1
| | |
| +------(20,10)
| (21,1)-------+
| | |
| | DE 2 | <--- farray on PET 2
| | |
| +------(30,10)
| (31,1)-------+
| | |
| | DE 3 | <--- farray on PET 3
| | |
| +------(40,10)
v
1st dimension
Providing farrayE during Array creation does not change anything about the actual farrayE object. This means that each PET can use its local farrayE directly to access the memory referenced by the Array object.
print *, farrayE
Another way of accessing the memory associated with an Array object is to use ArrayGet() to obtain an Fortran90 pointer that references the PET-local array.
call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc)
print *, farrayPtr
Finally the Array object must be destroyed. The PET-local memory of the farrayEs will remain in user control and will not be altered by ArrayDestroy().
call ESMF_ArrayDestroy(array, rc=rc)
Since the memory allocation for each farrayE is automatic there is nothing more to do.
The interaction between farrayE and the Array class is representative also for the two other cases farrayA and farrayP. The only difference is in the handling of memory allocations.
allocate(farrayA(10,10)) ! user controlled allocation farrayA = 23.67d0 ! initialize to some value array = ESMF_ArrayCreate(farray=farrayA, distgrid=distgrid, rc=rc)
print *, farrayA ! print PET-local farrayA directly call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc) ! obtain array pointer print *, farrayPtr ! print PET-local piece of Array through pointer call ESMF_ArrayDestroy(array, rc=rc) ! destroy the Array deallocate(farrayA) ! user controlled de-allocation
The farrayP case is identical.
allocate(farrayP(10,10)) ! user controlled allocation farrayP = 56.81d0 ! initialize to some value array = ESMF_ArrayCreate(farray=farrayP, distgrid=distgrid, rc=rc)
print *, farrayP ! print PET-local farrayA directly call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc) ! obtain array pointer print *, farrayPtr ! print PET-local piece of Array through pointer call ESMF_ArrayDestroy(array, rc=rc) ! destroy the Array deallocate(farrayP) ! user controlled de-allocation
To wrap things up the DistGrid object is destroyed and ESMF can be finalized.
call ESMF_DistGridDestroy(distgrid, rc=rc) ! destroy the DistGrid
call ESMF_Finalize(rc=rc)
end program
The example of the previous section showed how easy it is to create an Array object from existing PET-local Fortran90 arrays. The example did, however, not define any halos around the DE-local regions. The following code demonstrates how an Array object with space for a halo can be set up.
program ESMF_ArrayFarrayHaloEx use ESMF_Mod implicit none
The allocatable array farrayA will be used to provide the PET-local Fortran90 array for this example.
! local variables real(ESMF_KIND_R8), allocatable :: farrayA(:,:) ! allocatable F90 array real(ESMF_KIND_R8), pointer :: farrayPtr(:,:) ! matching F90 array pointer type(ESMF_DistGrid) :: distgrid ! DistGrid object type(ESMF_Array) :: array ! Array object integer :: rc, i, j real :: localSum
call ESMF_Initialize(rc=rc) if (rc /= ESMF_SUCCESS) call ESMF_Finalize(terminationflag=ESMF_ABORT)
The Array is to cover the exact same index space as in the previous example. Furthermore decomposition and distribution are also kept the same. Hence the same DistGrid object will be created and it is expected to execute this example with 4 PETs.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/40,10/), rc=rc)
This DistGrid describes a 40 x 10 index space that will be decomposed into 4 DEs when executed on 4 PETs, associating 1 DE per PET. Each DE-local exclusive region contains 10 x 10 elements. The DistGrid also stores and provides information about the relationship between DEs in index space, however, DistGrid does not contain information about halos. Arrays contain halo information and it is possible to create multiple Arrays covering the same index space with identical decomposition and distribution using the same DistGrid object, while defining different, Array-specific halo regions.
The extra memory required to cover the halo in the Array object must be taken into account when allocating the PET-local farrayA arrays. For a halo of 2 elements in each direction the following allocation will suffice.
allocate(farrayA(14,14)) ! Fortran90 array with halo: 14 = 10 + 2 * 2
The farrayA can now be used to create an Array object with enough space for a two element halo in each direction. The Array creation method checks for each PET that the local Fortran90 array can accomodate the requested regions.
The default behavior of ArrayCreate() is to center the exclusive region within the total region. Consequently the following call will provide the 2 extra elements on each side of the exclusive 10 x 10 region without having to specify any additional arguments.
array = ESMF_ArrayCreate(farray=farrayA, distgrid=distgrid, rc=rc)
The exclusive Array region on each PET can be accessed through a suitable Fortran90 array pointer. See section 19.2.6 for more details on Array regions.
call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc)
Following Array bounds convention, which by default puts the beginning of the exclusive region at (1, 1, ...), the following loop will add up the values of the local exclusive region for each DE, regardless of how the bounds were chosen for the original PET-local farrayA arrays.
localSum = 0.
do j=1, 10
do i=1, 10
localSum = localSum + farrayPtr(i, j)
enddo
enddo
Elements with 
or 
in the [-1,0] or [11,12] ranges are located outside the
exclusive region and may be used to define extra computational points or
halo operations.
Cleanup and shut down ESMF.
call ESMF_ArrayDestroy(array, rc=rc) deallocate(farrayA) call ESMF_DistGridDestroy(distgrid, rc=rc)
call ESMF_Finalize(rc=rc)
end program
Alternative to the direct usage of Fortran90 arrays during Array creation it is also possible to first create an ESMF_LocalArray and create the Array from it. While this may seem more burdensome for the 1 DE per PET cases discussed in the previous sections it allows a straight forward generallization to the multiple DE per PET case. The following example first recaptures the previous example using an ESMF_LocalArray and then expands to the multiple DE per PET case.
program ESMF_ArrayLarrayEx use ESMF_Mod implicit none
The current ESMF_LocalArray interface requires Fortran90 arrays to be defined with pointer attribute.
! local variables real(ESMF_KIND_R8), pointer :: farrayP(:,:) ! F90 array pointer real(ESMF_KIND_R8), pointer :: farrayPtr(:,:) ! matching F90 array pointer type(ESMF_LocalArray) :: larray ! ESMF_LocalArray object type(ESMF_LocalArray) :: larrayRef ! ESMF_LocalArray object type(ESMF_DistGrid) :: distgrid ! DistGrid object type(ESMF_Array) :: array ! Array object integer :: rc, i, j, de real :: localSum type(ESMF_LocalArray), allocatable :: larrayList(:) ! ESMF_LocalArray object list type(ESMF_LocalArray), allocatable :: larrayRefList(:) ! ESMF_LocalArray object list type(ESMF_VM):: vm integer:: localPet, petCount
call ESMF_Initialize(vm=vm, rc=rc) if (rc /= ESMF_SUCCESS) call ESMF_Finalize(terminationflag=ESMF_ABORT) call ESMF_VMGet(vm, localPet=localPet, petCount=petCount, rc=rc) if (rc /= ESMF_SUCCESS) call ESMF_Finalize(terminationflag=ESMF_ABORT) if (petCount /= 4) goto 10 ! TODO: use EXAMPLES_MULTI_ONLY once available
DistGrid and array allocation remains unchanged.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/40,10/), rc=rc)
allocate(farrayP(14,14)) ! allocate Fortran90 array on each PET with halo
Now instead of directly creating an Array object using the PET-local farrayPs an ESMF_LocalArray object will be created on each PET.
larray = ESMF_LocalArrayCreate(farrayP, ESMF_DATA_REF, rc=rc)
The Array object can now be created from larray. The Array creation method checks for each PET that the LocalArray can accomodate the requested regions.
array = ESMF_ArrayCreate(larrayList=(/larray/), distgrid=distgrid, rc=rc)
Once created there is no difference in how the Array object can be used. The exclusive Array region on each PET can be accessed through a suitable Fortran90 array pointer as before.
call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc)
Alternatively it is also possible (independent of how the Array object was created) to obtain the reference to the array allocation held by Array in form of an ESMF_LocalArry object. The farrayPtr can then be extracted using LocalArray methods.
call ESMF_ArrayGet(array, larray=larrayRef, rc=rc)
call ESMF_LocalArrayGet(larrayRef, farrayPtr, rc=rc)
Either way the farrayPtr reference can be used now to add up the values of the local exclusive region for each DE. The following loop works regardless of how the bounds were chosen for the original PET-local farrayP arrays and consequently the PET-local larray objects.
localSum = 0.
do j=1, 10
do i=1, 10
localSum = localSum + farrayPtr(i, j)
enddo
enddo
print *, "localSum=", localSum
Cleanup.
call ESMF_ArrayDestroy(array, rc=rc) call ESMF_LocalArrayDestroy(larray, rc=rc) deallocate(farrayP) ! use the pointer that was used in allocate statement call ESMF_DistGridDestroy(distgrid, rc=rc)
While the usage of LocalArrays is unnecessarily cumbersome for 1 DE per PET Arrays, it provides a straight forward path for extenting the interfaces to multiple DEs per PET.
In the following example a 8 x 8 index space will be decomposed into 2 x 4 = 8 DEs. The situation is captured by the following DistGrid object.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/8,8/), &
regDecomp=(/2,4/), rc=rc)
The distgrid object created in this manner will contain 8 DEs no matter how many PETs are available during execution. Assuming an execution on 4 PETs will result in the following distribution of the decomposition.
+---------------------------------------> 2nd dimension
| (1,1)
| +-----------+-----------+-----------+-----------+
| | DE0, PET0 | DE2, PET1 | DE4, PET2 | DE6, PET3 |
| | * * | * * | * * | * * |
| | | | | |
| | * * | * * | * * | * * |
| | | | | |
| | * * | * * | * * | * * |
| | | | | |
| | * * | * * | * * | * * |
| +-----------+-----------+-----------+-----------+
| | DE1, PET0 | DE3, PET1 | DE5, PET2 | DE7, PET3 |
| | * * | * * | * * | * * |
| | | | | |
| | * * | * * | * * | * * |
| | | | | |
| | * * | * * | * * | * * |
| | | | | |
| | * * | * * | * * | * * |
| +-----------+-----------+-----------+-----------+
| (8,8)
v
1st dimension
Obviously each PET is associated with 2 DEs. Each PET must allocate enough space for all its DEs. This is done by allocating as many DE-local arrays as there are DEs on the PET. The reference to these array allocations is passed into ArrayCreate via a LocalArray list argument that holds as many elements as there are DEs on the PET. Here each PET must allocate for two DEs.
allocate(larrayList(2)) ! 2 DEs per PET allocate(farrayP(4, 2)) ! without halo each DE is of size 4 x 2 farrayP = 123.456d0 larrayList(1) = ESMF_LocalArrayCreate(farrayP, ESMF_DATA_REF, rc=rc) ! 1st DE allocate(farrayP(4, 2)) ! without halo each DE is of size 4 x 2 farrayP = 456.789d0 larrayList(2) = ESMF_LocalArrayCreate(farrayP, ESMF_DATA_REF, rc=rc) ! 2nd DE
Notice that it is perfectly fine to re-use farrayP for all allocations of DE-local Fortran90 arrays. The allocated memory can be deallocated at the end using the array pointer contained in the larrayList.
With this information an Array object can be created. The distgrid object indicates 2 DEs for each PET and ArrayCreate() expects to find two LocalArray elements in larrayList.
array = ESMF_ArrayCreate(larrayList=larrayList, distgrid=distgrid, rc=rc)
Usage of a LocalArray list is the only way to provide a list of variable length of Fortran90 array allocations to ArrayCreate() for each PET. The array object created by the above call is an ESMF distributed object. As such it must follow the ESMF convention that requires that the call to ESMF_ArrayCreate() must be issued in unison by all PETs of the current context. Each PET only calls ArrayCreate() once, even if there are multiple DEs per PET.
The ArrayGet() method provides access to the list of LocalArrays on each PET.
allocate(larrayRefList(2)) call ESMF_ArrayGet(array, larrayList=larrayRefList, rc=rc)
Finally, access to the actual Fortran90 pointers is done on a per DE basis. Generally each PET will loop over its DEs.
do de=1, 2
call ESMF_LocalArrayGet(larrayRefList(de), farrayPtr, rc=rc)
localSum = 0.
do j=1, 2
do i=1, 4
localSum = localSum + farrayPtr(i, j)
enddo
enddo
print *, "localSum=", localSum
enddo
Note: If the VM associates multiple PEs with a PET the application writter may decide to use OpenMP loop parallelization on the de loop.
Cleanup requires that the PET-local deallocations are done before the pointers to the actual Fortran90 arrays are lost. Notice that larrayList is used to obtain the pointers used in the deallocate statement. Pointers obtained from the larrayRefList, while pointing to the same data, cannot be used to deallocated the array allocations!
do de=1, 2
call ESMF_LocalArrayGet(larrayList(de), farrayPtr, rc=rc)
deallocate(farrayPtr)
call ESMF_LocalArrayDestroy(larrayList(de), rc=rc)
enddo
deallocate(larrayList)
deallocate(larrayRefList)
call ESMF_ArrayDestroy(array, rc=rc)
call ESMF_DistGridDestroy(distgrid, rc=rc)
With that ESMF can be shut down cleanly.
call ESMF_Finalize(rc=rc)
end program
The examples of the previous sections made the user responsible for providing memory allocations for the PET-local regions of the Array object. The user was able to use any of the Fortran90 array methods or go through the ESMF_LocalArray interfaces to obtain memory allocations before passing them into ArrayCreate(). Alternatively, users may wish for ESMF to handle memory allocation of an Array object directly. The following example shows the interfaces that are available to the user to do just this.
To create an ESMF_Array object without providing an existing Fortran90 array or ESMF_LocalArray the type, kind and rank (tkr) of the Array must be specified in form of an ESMF_ArraySpec argument. Here a 2D Array of double precision real numbers is to be created:
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
Further an ESMF_DistGrid argument must be constructed that holds information about the entire domain (patchwork) and the decomposition into DE-local exclusive regions. The following line creates a DistGrid for a 5x5 global LR domain that is decomposed into 2 x 3 = 6 DEs.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
regDecomp=(/2,3/), rc=rc)
This is enough information to create a Array object with default settings.
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, rc=rc)
The array object created by the above call is an ESMF distributed object. As such it must follow the ESMF convention that requires that the call to ESMF_ArrayCreate() must be issued in unison by all PETs of the current context.
The index space covered by the Array object and the decomposition into DE-local exclusive regions, as it is described by the DistGrid object, is illustrated in the following diagram. Each asterix (*) represents a single element.
+---------------------------------------> 2nd dimension
| (1,1)
| +-----------+-----------+------+
| | DE 0 | DE 2 | DE 4 |
| | | | |
| | * * | * * | * |
| | | | |
| | * * | * * | * |
| | | | |
| | * * | * * | * |
| +-----------+-----------+------+
| | | | |
| | DE 1 | DE 3 | DE 5 |
| | | | |
| | * * | * * | * |
| | | | |
| | * * | * * | * |
| +-----------+-----------+------+
| (5,5)
v
1st dimension
The exact decomposition of the index space covered by the array object into DEs is contained in the distgrid object. Further, the layout of the DEs across the PETs of the component is stored in the delayout contained within the distgrid object. In the above example a default DELayout was created during the ESMF_DistGridCreate() call (see the refDoc / proposal for ESMF_DELayout and ESMF_DistGrid for details).
In order to use the array object it is necessary to know the local DEs located on each calling PET.
call ESMF_ArrayGet(array, localDeCount=localDeCount, rc=rc) allocate(localDeList(localDeCount)) call ESMF_ArrayGet(array, localDeList=localDeList, rc=rc)
In general it must be assumed that there may be multiple DEs associated with the calling PET, i.e. localDeCount >= 1. The situation where there is exactly one DE for each PET, i.e. localDeCount = 1 on every PET, is merely a special case of the more general formulation.
Consequently, in order to gain access to the DE-local memory segments that have been allocated on each PET by the ArrayCreate() call the Array must be queried for a list of LocalArray objects, each element corresponding to one PET-local DE.
allocate(larrayList(localDeCount)) call ESMF_ArrayGet(array, larrayList=larrayList, rc=rc)
Now each PET can loop through its local list of DEs and access the associated
memory through a suitable Fortran90 pointer. In the current example the native
pointer myF90Array must be declared as
real(ESMF_KIND_R8), pointer:: myF90Array(:,:)
in order to match the arrayspec that was used to create the
array object. The following loop uses the native language access to
initialize the entire memory chunks of all PET-local DEs to 0 using
Fortran90 array syntax.
do de=1, localDeCount
call ESMF_LocalArrayGet(larrayList(de), myF90Array, ESMF_DATA_REF, rc=rc)
myF90Array = 0.
enddo
Each ESMF_Array object is decomposed into DEs as specified by the associated ESMF_DistGrid object. Each piece of this decomposition, i.e. each DE, holds a chunk of the Array data in its own local piece of memory. The details of the Array decomposition are described in the following paragraphs.
At the center of the Array decomposition is the ESMF_DistGrid class. The DistGrid object specified during Array creation contains three essential pieces of information:
Each element of an Array is associated with a single DE. The union of elements associated with a DE, as defined by the DistGrid above, corresponds to a LR chunk of index space, called the exclusive region of the DE.
There is a hierarchy of four regions that can be identified for each DE in an Array object. Their definition and relationship to each other is as follows:
+-totalLBound(:)----------------------------------+
|\ |
| \ <--- totalLWidth(:) |
| \ |
| +-computationalLBound(:)------------------+ |
| |\ | |
| | \ <--- computationalLWidth(:) | |
| | \ | |
| | +-exclusiveLBound(:)-------------+ | |
| | | | | |
| | | +------+ +-----+ | | |
| | | | | | | | | |
| | | | +------+ | | | |
| | | | "Interior Region" | | | |
| | | +-----+ | | | |
| | | | | | | |
| | | +-------------+ | | |
| | | | | |
| | | "Exclusive Region" | | |
| | +-------------exclusiveUBound(:)-+ | |
| | \ | |
| | computationalUWidth(:) --> \ | |
| | \ | |
| | "Computational Region" \| |
| +------------------computationalUBound(:)-+ |
| \ |
| totalUWidth(:) -> \ |
| "Total Region" \|
+--------------------------------- totalUBound(:)-+
With the following definitions:
computationalLWidth(:) = exclusiveLBound(:) - computationalLBound(:) computationalUWidth(:) = computationalUBound(:) - exclusiveLBound(:)and
totalLWidth(:) = computationalLBound(:) - totalLBound(:) totalUWidth(:) = totalUBound(:) - computationalUBound(:)
The exclusive region is determined during Array creation by the DistGrid argument. Optional arguments may be used to specify the computational region when the Array is created, by default it will be set equal to the exclusive region. The total region, i.e. the actual memory allocation for each DE, is also determined during Array creation. When creating the Array object from existing Fortran90 arrays the total region is set equal to the memory provided by the Fortran90 arrays. Otherwise the default is to allocate as much memory as is needed to accomodate the union of the DE-local exclusive and computational region. Finally it is also possible to use optional arguments to the ArrayCreate() call to specify the total region of the object explicitly.
The ESMF_ArrayCreate() call checks that the input parameters are consistent and will result in an Array that fulfills all of the above mentioned requirements for its DE-local regions.
Once an Array object has been created the exclusive and total regions are fixed. The computational region, however, may be adjusted within the limits of the total region using the ArraySet() call.
The interior region is very different from the other regions in that
it cannot be specified. The interior region for each DE is a consequence of the choices made for the other regions collectively across
all DEs into which an Array object is decomposed. An Array object can be
queried for its DE-local interior regions as to offer additional
information to the user necessary to write more efficient code. See section
(not yet implemented) for more details.
By default the bounds of each DE-local total region are defined as to put the start of the DE-local exclusive region at the "origin" of the local index space, i.e. at (1, 1, ..., 1). With that definition the following loop will access each element of the DE-local memory segment for each PET-local DE of the Array object used in the previous sections and print its content.
do de=1, localDeCount
call ESMF_LocalArrayGet(larrayList(de), myF90Array, ESMF_DATA_REF, rc=rc)
do i=1, size(myF90Array, 1)
do j=1, size(myF90Array, 2)
print *, "PET-local DE=", de, ": array(",i,",",j,")=", myF90Array(i,j)
enddo
enddo
enddo
The loop over Array elements at the end of the last section only works correctly because of the default definition of the computational and total regions used in the example. In general, without such specific knowledge about an Array object, it is necessary to use a more formal approach to access its regions with DE-local indices.
The DE-local exclusive region takes a central role in the definition of Array bounds. Even as the computational region may adjust during the course of execution the exclusive region remains unchanged. Furthermore the exclusive region is identical for all stagger locations (discussed in a later section) and as such provides a unique reference frame for the index space of all Arrays associated with the same DistGrid.
There is a choice between two indexing options that needs to be made during Array creation. By default each DE-local exclusive region starts at (1, 1, ..., 1). However, for some computational kernels it may be more convenient to choose the index bounds of the DE-local exclusive regions to match the index space coordinates as they are defined in the corresponding DistGrid object. The second option is only available if the DistGrid object does not contain any non-contiguous decompositions (such as cyclically decomposed dimensions).
The following example code demonstrates the safe way of dereferencing the DE-local exclusive regions of the previously created array object.
allocate(exclusiveUBound(2, localDeCount)) ! dimCount=2
allocate(exclusiveLBound(2, localDeCount)) ! dimCount=2
call ESMF_ArrayGet(array, indexflag=indexflag, &
exclusiveLBound=exclusiveLBound, exclusiveUBound=exclusiveUBound, rc=rc)
if (indexflag == ESMF_INDEX_DELOCAL) then
! this is the default
! print *, "DE-local exclusive regions start at (1,1)"
do de=1, localDeCount
call ESMF_LocalArrayGet(larrayList(de), myF90Array, ESMF_DATA_REF, rc=rc)
do i=1, exclusiveUBound(1, de)
do j=1, exclusiveUBound(2, de)
! print *, "DE-local exclusive region for PET-local DE=", de, &
! ": array(",i,",",j,")=", myF90Array(i,j)
enddo
enddo
enddo
else if (indexflag == ESMF_INDEX_GLOBAL) then
! only if set during ESMF_ArrayCreate()
! print *, "DE-local exclusive regions of this Array have global bounds"
do de=1, localDeCount
call ESMF_LocalArrayGet(larrayList(de), myF90Array, ESMF_DATA_REF, rc=rc)
do i=exclusiveLBound(1, de), exclusiveUBound(1, de)
do j=exclusiveLBound(2, de), exclusiveUBound(2, de)
! print *, "DE-local exclusive region for PET-local DE=", de, &
! ": array(",i,",",j,")=", myF90Array(i,j)
enddo
enddo
enddo
endif
call ESMF_ArrayDestroy(array, rc=rc) ! destroy the array object
Obviously the second branch of this simple code will work for either case, however, if a complex computational kernel was written assuming ESMF_INDEX_DELOCAL type bounds the second branch would simply be used to indicate the problem and bail out.
The advantage of the ESMF_INDEX_GLOBAL index option is that the Array bounds directly contain information on where the DE-local Array piece is located in a global index space sense. When the ESMF_INDEX_DELOCAL option is used the correspondence between local and global index space must be made by querying the associated DistGrid for the DE-local indexList arguments.
In the previous examples the computational region of array was chosen by default to be identical to the exclusive region defined by the DistGrid argument during Array creation. In the following the same arrayspec and distgrid objects as before will be used to create an Array but now a larger computational region shall be defined around each DE-local exclusive region. Furthermore, extra space will be defined around the computational region of each DE to accommodate a halo and/or serve as memory padding.
In this example the indexflag argument is set to ESMF_INDEX_GLOBAL indicating that the bounds of the exclusive region correspond to the index space coordinates as they are defined by the DistGrid object.
The same arrayspec and distgrid objects as before are used which also allows the reuse of the already allocated larrayList variable.
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
computationalLWidth=(/0,3/), computationalUWidth=(/1,1/), &
totalLWidth=(/1,4/), totalUWidth=(/3,1/), &
indexflag=ESMF_INDEX_GLOBAL, rc=rc)
Obtain the larrayList on every PET.
call ESMF_ArrayGet(array, larrayList=larrayList, rc=rc)
The bounds of DE 1 for array are shown in the following diagram to illustrate the situation. Notice that the totalLWidth and totalUWidth arguments in the ArrayCreate() call define the total region with respect to the exclusive region given for each DE by the distgrid argument.
+-(3,-3)---------------------------------+
|\ |
| +-(4,-2)-+-(4,1)--------------------+--+
| | | | |
| | | | |
| | | DE 1 | |
| | | | |
| | | | |
| | | Exclusive Region | |
| | +--------------------(5,2)-+ |
| | Computational Region |
| +-------------------------------(6,3)--+
| |
| Total Region |
+---------------------------------(8,3)--+
When working with this array it is possible for the computational kernel to overstep the exclusive region for both read/write access (computational region) and potentially read-only access into the total region outside of the computational region, if a halo operation provides valid entries for these elements.
The Array object can be queried for absolute bounds
allocate(computationalLBound(2, localDeCount)) ! dimCount=2
allocate(computationalUBound(2, localDeCount)) ! dimCount=2
allocate(totalLBound(2, localDeCount)) ! dimCount=2
allocate(totalUBound(2, localDeCount)) ! dimCount=2
call ESMF_ArrayGet(array, exclusiveLBound=exclusiveLBound, &
exclusiveUBound=exclusiveUBound, computationalLBound=computationalLBound, &
computationalUBound=computationalUBound, totalLBound=totalLBound, &
totalUBound=totalUBound, rc=rc)
or for the relative widths.
allocate(computationalLWidth(2, localDeCount)) ! dimCount=2
allocate(computationalUWidth(2, localDeCount)) ! dimCount=2
allocate(totalLWidth(2, localDeCount)) ! dimCount=2
allocate(totalUWidth(2, localDeCount)) ! dimCount=2
call ESMF_ArrayGet(array, computationalLWidth=computationalLWidth, &
computationalUWidth=computationalUWidth, totalLWidth=totalLWidth, &
totalUWidth=totalUWidth, rc=rc)
Either way the dereferencing of Array data is centered around the DE-local exclusive region:
do de=1, localDeCount
call ESMF_LocalArrayGet(larrayList(de), myF90Array, ESMF_DATA_REF, rc=rc)
! initialize the DE-local array
myF90Array = 0.1d0 * localDeList(de)
! first time through the total region of array
! print *, "myF90Array bounds for DE=", localDeList(de), lbound(myF90Array), &
! ubound(myF90Array)
do j=exclusiveLBound(2, de), exclusiveUBound(2, de)
do i=exclusiveLBound(1, de), exclusiveUBound(1, de)
! print *, "Excl region DE=", localDeList(de), ": array(",i,",",j,")=", &
! myF90Array(i,j)
enddo
enddo
do j=computationalLBound(2, de), computationalUBound(2, de)
do i=computationalLBound(1, de), computationalUBound(1, de)
! print *, "Excl region DE=", localDeList(de), ": array(",i,",",j,")=", &
! myF90Array(i,j)
enddo
enddo
do j=totalLBound(2, de), totalUBound(2, de)
do i=totalLBound(1, de), totalUBound(1, de)
! print *, "Total region DE=", localDeList(de), ": array(",i,",",j,")=", &
! myF90Array(i,j)
enddo
enddo
! second time through the total region of array
do j=exclusiveLBound(2, de)-totalLWidth(2, de), &
exclusiveUBound(2, de)+totalUWidth(2, de)
do i=exclusiveLBound(1, de)-totalLWidth(1, de), &
exclusiveUBound(1, de)+totalUWidth(1, de)
! print *, "Excl region DE=", localDeList(de), ": array(",i,",",j,")=", &
! myF90Array(i,j)
enddo
enddo
enddo
All previous examples were written for the 2D case. There is, however, no restriction within the Array or DistGrid class that limits the dimensionality of Array objects beyond the language specific limitations (7D for Fortran).
In order to create an n-dimensional Array the rank indicated by both the arrayspec and the distgrid arguments specified during Array create must be equal to n. A 1D Array of double precision real data hence requires the following arrayspec.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=1, rc=rc)
The index space covered by the Array and the decomposition description is
provided to the Array create method by the distgrid argument. The index
space in this example has 16 elements and covers the interval ![$[-10, 5]$](img17.png){kind=small}
. It is
decomposed into as many DEs as there are PETs in the current context.
distgrid1D = ESMF_DistGridCreate(minIndex=(/-10/), maxIndex=(/5/), &
regDecomp=(/petCount/), rc=rc)
A 1D Array object with default regions can now be created.
array1D = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid1D, rc=rc)
The creation of a 3D Array proceeds analogous to the 1D case. The rank of the arrayspec must be changed to 3
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=3, rc=rc)
and an appropriate 3D DistGrid object must be created
distgrid3D = ESMF_DistGridCreate(minIndex=(/1,1,1/), maxIndex=(/16,16,16/), &
regDecomp=(/4,4,4/), rc=rc)
before an Array object can be created.
array3D = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid3D, rc=rc)
The distgrid3D object decomposes the 3-dimensional index space into
DEs. These DEs are laid out across the computational
resources (PETs) of the current component according to a default DELayout that
is created during the DistGrid create call. Notice that in the index space
proposal a DELayout does not have a sense of dimensionality. The DELayout
function is simply to map DEs to PETs. The DistGrid maps chunks of index space
against DEs and thus its rank is equal to the number of index space
dimensions.
The previously defined DistGrid and the derived Array object decompose the index space along all three dimension. It is, however, not a requirement that the decomposition be along all dimensions. An Array with the same 3D index space could as well be decomposed along just one or along two of the dimensions. The following example shows how for the same index space only the last two dimensions are decomposed while the first Array dimension has full extent on all DEs.
call ESMF_ArrayDestroy(array3D, rc=rc)
call ESMF_DistGridDestroy(distgrid3D, rc=rc)
distgrid3D = ESMF_DistGridCreate(minIndex=(/1,1,1/), maxIndex=(/16,16,16/), &
regDecomp=(/1,4,4/), rc=rc)
array3D = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid3D, rc=rc)
Finally, the definition and usage of the stagger location index as it was
described in sections and
for the 2D case applies without change to
1D, 3D or any other dimensionality. Connections defined in the DistGrid object
may utilize the stagger location index in order to express characteristics of
the index space topology. The concept is completely rank independent.
call ESMF_DistGridGet(distgrid3D, delayout=delayout, rc=rc) ! get DELayout
distgrid2D = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/16,16/), &
regDecomp=(/4,4/), delayout=delayout, rc=rc)
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
array2D = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid2D, rc=rc)
Now the following kernel is sure to work with array3D and array2D.
call ESMF_DELayoutGet(delayout, localDeCount=localDeCount, rc=rc)
allocate(larrayList1(localDeCount))
call ESMF_ArrayGet(array3D, larrayList=larrayList1, rc=rc)
allocate(larrayList2(localDeCount))
call ESMF_ArrayGet(array2D, larrayList=larrayList2, rc=rc)
do de=1, localDeCount
call ESMF_LocalArrayGet(larrayList1(de), myF90Array3D, ESMF_DATA_REF, &
rc=rc)
myF90Array3D = 0.1d0 * de ! initialize
call ESMF_LocalArrayGet(larrayList2(de), myF90Array2D, ESMF_DATA_REF, &
rc=rc)
myF90Array2D = 0.5d0 * de ! initialize
do k=1, 4
do j=1, 4
dummySum = 0.d0
do i=1, 16
dummySum = dummySum + myF90Array3D(i,j,k) ! sum up the (j,k) column
enddo
dummySum = dummySum * myF90Array2D(j,k) ! multiply with local 2D element
! print *, "dummySum(",j,k,")=",dummySum
enddo
enddo
enddo
All of the Array create interfaces require the specification of at least the arrayspec and the distgrid arguments. Both arguments contain a sense of dimensionality. The relationship between these two arguments deserves extra attention.
The arrayspec argument is of type ESMF_ArraySpec and determines, among other things, the rank of the Array, i.e. the dimensionality of the actual data storage. This means, for example, that the rank of a native language array extracted from an Array object is equal to the rank specified by the arrayspec argument. It is also equal to the rank that is returned as by the ESMF_ArrayGet() call. The arrayspec argument does not determine, however, how the Array dimensions are decomposed and distributed.
The rank specification contained in the distgrid argument, which is of type ESMF_DistGrid, on the other hand has no affect on the rank of the Array. The dimCount specified by the DistGrid object, which may be equal, greater or less than the Array rank, determines the dimensionality of the decomposition.
While there is no constraint between DistGrid dimCount and Array rank, there is an important relationship between the two, resulting in the concept of index space dimensionality. Array dimensions can be arbitrarily mapped against DistGrid dimension, rendering them decomposed dimensions. The index space dimensionality is equal to the number of decomposed Array dimensions.
Array dimensions that are not mapped to DistGrid dimensions are considered extra or tensor dimensions of the Array. They are not part of the index space. The mapping is specified during ESMF_ArrayCreate() via the the distgridToArrayMap argument. DistGrid dimensions that have not been associated with Array dimensions are replicating dimensions. The Array will be replicated across the DEs that lie along replication DistGrid dimensions.
Array tensor dimensions can be used to store multi-dimensional data for each
Array index space element. A special purpose of tensor dimensions is to store
multiple data arrays in the same Array object. It is, for example,
possible to store array1 and array2 of section
in a single Array object using one
tensor dimension of size 2. The same distgrid object as
before can be used to create the Array.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
regDecomp=(/2,3/), rc=rc)
The rank in the arrayspec argument, however, must change from 2 to 3 in order to provide for the extra Array dimension.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=3, rc=rc)
During Array creation with extra dimension(s) it is necessary to specify the bounds of these tensor dimension(s). This requires two additional arguments, undistLBound and undistUBound, which are vectors in order to accommodate higher order tensor dimensions. The other arguments remain unchanged and apply across all tensor components.
The optional arguments used in the following call are identical to those
used to create array1 of section . This
will set the total region and the stagger location of both tensor components
to be those of array1.
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
totalLWidth=(/0,1/), totalUWidth=(/0,1/), staggerLoc=1, &
undistLBound=(/1/), undistUBound=(/2/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(terminationflag=ESMF_ABORT)
This will create array with 2+1 dimensions. The 2D DistGrid is used to describe decomposition into DEs with 2 Array dimensions mapped to the DistGrid dimensions resulting in a 2D index space. The extra Array dimension provides storeage for multiple 2D user data arrays that are kept in a single Array object. By default the distgrid dimensions are associated with the first Array dimensions in sequence. For the example above this means that the first 2 Array dimensions are decomposed according to the provided 2D DistGrid. The 3rd Array dimension does not have an associated DistGrid dimension, rendering it a tensor dimension.
The optional arguments that were used to create array ensure that the total region is large enough to accommodate the arrays for tensor component 1 and 2. The Array class provides a special Set() method that allows to individually address tensor elements in an Array and set staggerLoc and vectorDim arguments.
call ESMF_ArraySet(array, tensorIndex=(/2/), staggerLoc=2, rc=rc)
Native language access to an Array with tensor dimensions is in principle the same as without extra dimensions.
call ESMF_ArrayGet(array, localDeCount=localDeCount, rc=rc) allocate(larrayList(localDeCount)) call ESMF_ArrayGet(array, larrayList=larrayList, rc=rc)
The following loop shows how a Fortran pointer to the DE-local data chunks can be obtained and used to set data values in the exclusive regions. The myF90Array3D variable must be of rank 3 to match the Array rank of array. However, variables such as exclusiveUBound that store the information about the decomposition, remain to be allocated for a 2D decomposition.
call ESMF_ArrayGet(array, exclusiveLBound=exclusiveLBound, &
exclusiveUBound=exclusiveUBound, rc=rc)
do de=1, localDeCount
call ESMF_LocalArrayGet(larrayList(de), myF90Array3D, ESMF_DATA_REF, rc=rc)
myF90Array3D = 0.0 ! initialize
myF90Array3D(exclusiveLBound(1,de):exclusiveUBound(1,de), &
exclusiveLBound(2,de):exclusiveUBound(2,de), 1) = 5.1 ! dummy assignment
myF90Array3D(exclusiveLBound(1,de):exclusiveUBound(1,de), &
exclusiveLBound(2,de):exclusiveUBound(2,de), 2) = 2.5 ! dummy assignment
enddo
deallocate(larrayList)
For some applications the default association rules between DistGrid and Array dimensions may not satisfy the user's needs. The optional distgridToArrayMap argument may be used during Array creation to explicitly specify the mapping between Array and DistGrid dimensions. To demonstrate this the following lines of code reproduce the above example but with rearranged dimensions. Here the distgridToArrayMap argument is a list with two elements corresponding to the DistGrid dimCount of 2. The first element indicates which Array dimension the first DistGrid dimension is mapped against. Here the 1st DistGrid dimension maps against the 3rd Array and the 2nd DistGrid dimension maps against the 1st Array dimension. This leaves the 2nd Array dimension to be the extra or tensor dimension of the created Array object.
call ESMF_ArrayDestroy(array, rc=rc)
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
distgridToArrayMap=(/3, 1/), totalLWidth=(/0,1/), totalUWidth=(/0,1/), &
undistLBound=(/1/), undistUBound=(/2/), rc=rc)
call ESMF_ArraySet(array, tensorIndex=(/1/), staggerLoc=1, rc=rc)
call ESMF_ArraySet(array, tensorIndex=(/2/), staggerLoc=2, rc=rc)
Operations on the Array object as a whole are unchanged by the different mapping of dimensions.
When working with Arrays that contain explicitly mapped Array and DistGrid dimensions it is critical to understand that width and bound arguments are always defined in terms of the DistGrid dimension order. The Array dimensions indicate how the data is actually stored in the Array object, and that can be different for each Array, even if the same DistGrid is used. The decomposition defined by the DistGrid, however, does not change and is the same for each Array that uses it, regardless of the dimension order in the Array. The DistGrid dimension order thus becomes a common reference frame for all Arrays that use the same DistGrid.
The distgridToArrrayMap argument optionally provided during Array create indicates the DistGrid to Array dimension mapping. Depending on the formulation of the computational kernel, the inverse mapping, i.e. Array to DistGrid dimension mapping, is just as important. The ESMF_ArrayGet() call offers both mappings as distgridToArrrayMap and arrayToDistGridMap, respectively. The number of elements in arrayToDistGridMap is equal to the rank of the Array. Each element corresponds to an Array dimension and indicates the associated DistGrid dimension by an integer number. An entry of "0" indicates an extra Array dimension.
The association between Array and DistGrid dimensions becomes critical for correct native language access to the Array. In the following example the inverse mapping information is used to determine the correct bounds of the Array dimensions and to verify that the kernel's assumption about which Array dimension is a tensor dimension is correct.
allocate(arrayToDistGridMap(3)) ! arrayRank = 3
call ESMF_ArrayGet(array, arrayToDistGridMap=arrayToDistGridMap, &
exclusiveLBound=exclusiveLBound, exclusiveUBound=exclusiveUBound, &
localDeCount=localDeCount, rc=rc)
if (arrayToDistGridMap(2) /= 0) then ! check if extra dimension at expected index
! indicate problem and bail out
endif
! obtain larrayList for local DEs
allocate(larrayList(localDeCount))
call ESMF_ArrayGet(array, larrayList=larrayList, rc=rc)
! prepare inverse distgridToArrayMap variables for kernel loop
idm1=arrayToDistGridMap(1)
idm3=arrayToDistGridMap(3)
do de=1, localDeCount
call ESMF_LocalArrayGet(larrayList(de), myF90Array3D, ESMF_DATA_REF, rc=rc)
myF90Array3D(exclusiveLBound(idm1,de):exclusiveUBound(idm1,de), &
1, exclusiveLBound(idm3,de):exclusiveUBound(idm3,de)) = 10.5 ! dummy assignment
myF90Array3D(exclusiveLBound(idm1,de):exclusiveUBound(idm1,de), &
2, exclusiveLBound(idm3,de):exclusiveUBound(idm3,de)) = 23.3 ! dummy assignment
enddo
deallocate(exclusiveLBound, exclusiveUBound)
deallocate(arrayToDistGridMap)
deallocate(larrayList)
call ESMF_ArrayDestroy(array, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(terminationflag=ESMF_ABORT)
Thus far most examples demonstrated cases where the DistGrid dimCount was equal to the Array rank. The previous section introduced the concept of Array tensor dimensions when dimCount < rank. In this section dimCount and rank are assumed completely unconstrained and the relationship to distgridToArrayMap and arrayToDistGridMap will be discussed.
The Array class allows completely arbitrary mapping between Array and DistGrid dimensions. Most cases considered in the previous sections used the default mapping which assigns the DistGrid dimensions in sequence to the lower Array dimensions. Extra Array dimensions, if present, are considered non-distributed tensor dimensions for which the optional undistLBound and undistUBound arguments must be specified.
The optional distgridToArrayMap argument provides the option to override the default DistGrid to Array dimension mapping. The entries of the distgridToArrayMap array correspond to the DistGrid dimensions in sequence and assign a unique Array dimension to each DistGrid dimension. DistGrid and Array dimensions are indexed starting at 1 for the lowest dimension. A value of "0" in the distgridToArrayMap array indicates that the respective DistGrid dimension is not mapped against any Array dimension. What this means is that the Array will be replicated along this DistGrid dimension.
As a first example consider the case where a 1D Array
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=1, rc=rc)
is created on the 2D DistGrid used during the previous section.
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, rc=rc)
Here the default DistGrid to Array dimension mapping is used which assigns the Array dimensions in sequence to the DistGrid dimensions starting with dimension "1". Extra DistGrid dimensions are considerd replicator dimensions because the Array will be replicated along those dimensions. In the above example the 2nd DistGrid dimension will cause 1D Array pieces to be replicated along the DEs of the 2nd DistGrid dimension. Replication in the context of ESMF_ArrayCreate() does not mean that data values are communicated and replicated between different DEs, but it means that different DEs provide memory allocations for identical exclusive elements.
Access to the data storage of an Array that has been replicated along DistGrid dimensions is the same as for Arrays without replication.
call ESMF_ArrayGet(array, localDeCount=localDeCount, rc=rc)
allocate(larrayList(localDeCount))
allocate(localDeList(localDeCount))
call ESMF_ArrayGet(array, larrayList=larrayList, localDeList=localDeList, &
rc=rc)
The array object was created without additional padding which means that the bounds of the Fortran array pointer correspond to the bounds of the exclusive region. The following loop will cycle through all local DEs, print the DE number as well as the Fortran array pointer bounds. The bounds should be:
lbound ubound
DE 0: 1 3 --+
DE 2: 1 3 --| 1st replication set
DE 4: 1 3 --+
DE 1: 1 2 --+
DE 3: 1 2 --| 2nd replication set
DE 5: 1 2 --+
do de=1, localDeCount
call ESMF_LocalArrayGet(larrayList(de), myF90Array1D, ESMF_DATA_REF, &
rc=rc)
print *, "DE ",localDeList(de)," [", lbound(myF90Array1D), &
ubound(myF90Array1D),"]"
enddo
deallocate(larrayList)
deallocate(localDeList)
call ESMF_ArrayDestroy(array, rc=rc)
The Fortran array pointer in the above loop was of rank 1 because the Array object was of rank 1. However, the distgrid object associated with array is 2-dimensional! Consequently DistGrid based information queried from array will be 2D. The distgridToArrayMap and arrayToDistGridMap arrays provide the necessary mapping to correctly associate DistGrid based information with Array dimensions.
The next example creates a 2D Array
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
on the previously used 2D DistGrid. By default, i.e. without the distgridToArrayMap argument, both DistGrid dimensions would be associated with the two Array dimensions. However, the distgridToArrayMap specified in the following call will only associate the second DistGrid dimension with the first Array dimension. This will render the first DistGrid dimension a replicator dimension and the second Array dimension a tensor dimension for which 1D undistLBound and undistUBound arguments must be supplied.
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
distgridToArrayMap=(/0,1/), undistLBound=(/11/), undistUBound=(/14/), rc=rc)
call ESMF_ArrayDestroy(array, rc=rc)
Finally, the same arrayspec and distgrid arguments are used to create a 2D Array that is fully replicated in both dimensions of the DistGrid. Both Array dimensions are now tensor dimensions and both DistGrid dimensions are replicator dimensions.
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
distgridToArrayMap=(/0,0/), undistLBound=(/11,21/), undistUBound=(/14,22/), &
rc=rc)
The result will be an Array with local lower bound (/11,21/) and upper bound (/14,22/) on all 6 DEs of the DistGrid.
call ESMF_ArrayDestroy(array, rc=rc)
call ESMF_DistGridDestroy(distgrid, rc=rc)
Replicated Arrays can also be created from existing local Fortran arrays. The following Fortran array allocation will provide a 3 x 10 array on each PET.
allocate(myF90Array2D(3,10))
Assuming a petCount of 4 the following DistGrid defines a 2D index space that is distributed across the PETs along the first dimension.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/40,10/), rc=rc)
The following call creates an Array object on the above distgrid using the locally existing myF90Array2D Fortran arrays. The difference compared to the case with automatic memory allocation is that instead of arrayspec the Fortran array is provided as argument. Futhermore, the undistLBound and undistUBound arguments can be omitted, defaulting into Array tensor dimension lower bound of 1 and an upper bound equal to the size of the respective Fortran array dimension.
array = ESMF_ArrayCreate(farray=myF90Array2D, distgrid=distgrid, &
distgridToArrayMap=(/0,2/), rc=rc)
The array object associates the 2nd DistGrid dimension with the 2nd Array dimension. The first DistGrid dimension is not associated with any Array dimension and will lead to replication of the Array along the DEs of this direction.
call ESMF_ArrayDestroy(array, rc=rc)
call ESMF_DistGridDestroy(distgrid, rc=rc)
It is a common situation, particularily in legacy code, that an ESMF Array object must be filled with data originating from a large Fortran array stored on a single PET.
if (localPet == 0) then
allocate(farray(10,20,30))
do k=1, 30
do j=1, 20
do i=1, 10
farray(i, j, k) = k*1000 + j*100 + i
enddo
enddo
enddo
endif
distgrid = ESMF_DistGridCreate(minIndex=(/1,1,1/), maxIndex=(/10,20,30/), &
rc=rc)
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_I4, rank=3, rc=rc)
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, rc=rc)
The ESMF_ArrayScatter() method provides a convenient way of scattering array data from a single root PET across the DEs of an ESMF Array object.
call ESMF_ArrayScatter(array, farray=farray, rootPet=0, rc=rc)
if (localPet == 0) then
deallocate(farray)
endif
The destination of the ArrayScatter() operation are all the DEs of a single patch. For multi-patch Arrays the destination patch can be specified. The shape of the scattered Fortran array must match the shape of the destination patch in the ESMF Array.
Gathering data decomposed and distributed across the DEs of an ESMF Array object into a single Fortran array on root PET is accomplished by calling ESMF_ArrayGather().
if (localPet == 3) then
allocate(farray(10,20,30))
endif
call ESMF_ArrayGather(array, farray=farray, rootPet=3, rc=rc)
if (localPet == 3) then
deallocate(farray)
endif
The source of the ArrayGather() operation are all the DEs of a single patch. For multi-patch Arrays the source patch can be specified. The shape of the gathered Fortran array must match the shape of the source patch in the ESMF Array.
The ESMF_ArrayScatter() operation allows to fill entire replicated Array objects with data coming from a single root PET.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
regDecomp=(/2,3/), rc=rc)
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
distgridToArrayMap=(/0,0/), undistLBound=(/11,21/), undistUBound=(/14,22/), &
rc=rc)
The shape of the Fortran source array used in the Scatter() call must be that of the contracted Array, i.e. contracted DistGrid dimensions do not count. For the array just created this means that the source array on rootPet must be of shape 4 x 2.
if (localPet == 0) then
allocate(myF90Array2D(4,2))
do j=1,2
do i=1,4
myF90Array2D(i,j) = i * 100.d0 + j * 1.2345d0 ! initialize
enddo
enddo
endif
call ESMF_ArrayScatter(array, farray=myF90Array2D, rootPet=0, rc=rc)
if (localPet == 0) then
deallocate(myF90Array2D)
endif
This will have filled each local 4 x 2 Array piece with the replicated data of myF90Array2D.
call ESMF_ArrayDestroy(array, rc=rc)
call ESMF_DistGridDestroy(distgrid, rc=rc)
As a second example for the use of Scatter() and Gather() consider the following replicated Array created from existing local Fortran arrays.
allocate(myF90Array2D(3,10)) distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/40,10/), rc=rc)
array = ESMF_ArrayCreate(farray=myF90Array2D, distgrid=distgrid, &
distgridToArrayMap=(/0,2/), rc=rc)
The array object associates the 2nd DistGrid dimension with the 2nd Array dimension. The first DistGrid dimension is not associated with any Array dimension and will lead to replication of the Array along the DEs of this direction. Still, the local arrays that comprise the array object refer to independent pieces of memory and can be initialized indpendently.
myF90Array2D = localPet ! initialize
However, the notion of replication becomes visible when an array of shape 3 x 10 on root PET 0 is scattered across the Array object.
if (localPet == 0) then
allocate(myF90Array2D2(5:7,11:20))
do j=11,20
do i=5,7
myF90Array2D2(i,j) = i * 100.d0 + j * 1.2345d0 ! initialize
enddo
enddo
endif
call ESMF_ArrayScatter(array, farray=myF90Array2D2, rootPet=0, rc=rc)
if (localPet == 0) then
deallocate(myF90Array2D2)
endif
The Array pieces on every DE will receive the same source data, resulting in a replication of data along DistGrid dimension 1.
When the inverse operation, i.e. ESMF_ArrayGather(), is applied to a replicated Array an intrinsic ambiguity needs to be considered. ESMF defines the gathering of data of a replicated Array as the collection of data originating from the numerically higher DEs. This means that data in replicated elements associated with numerically lower DEs will be ignored during ESMF_ArrayGather(). For the current example this means that changing the Array contents on PET 1, which here corresponds to DE 1,
if (localPet == 1) then
myF90Array2D = real(1.2345, ESMF_KIND_R8)
endif
will not affect the result of
allocate(myF90Array2D2(3,10)) myF90Array2D2 = 0.d0 ! initialize to a known value call ESMF_ArrayGather(array, farray=myF90Array2D2, rootPet=0, rc=rc)
The result remains completely defined by the unmodified values of Array in DE 3, the numerically highest DE. However, overriding the DE-local Array piece on DE 3
if (localPet==3) then
myF90Array2D = real(5.4321, ESMF_KIND_R8)
endif
will change the outcome of
call ESMF_ArrayGather(array, farray=myF90Array2D2, rootPet=0, rc=rc)
as expected.
deallocate(myF90Array2D2) call ESMF_ArrayDestroy(array, rc=rc)
call ESMF_DistGridDestroy(distgrid, rc=rc)
Arrays used in different models often cover the same index space region, however, the distribution of the Arrays may be different, e.g. the models run on exclusive sets of PETs. Even if the Arrays are defined on the same list of PETs the decomposition may be different.
srcDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/10,20/), &
regDecomp=(/4,1/), rc=rc)
dstDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/10,20/), &
regDecomp=(/1,4/), rc=rc)
The number of elements covered by srcDistgrid is identical to the number of elements covered by dstDistgrid - in fact the index space regions covered by both DistGrid objects are congruent.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
srcArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=srcDistgrid, rc=rc)
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, rc=rc)
By construction srcArray and dstArray are of identical type and kind. Further the number of exclusive elements matches between both Arrays. These are the prerequesites for the application of an Array redistribution in default mode. In order to increase performance of the actual redistribution the communication patter must be precomputed and stored.
call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
The redistHandle can now be used repeatedly on the srcArray, dstArray pair to redistributed data from source to destination Array.
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
The use of the redistHandle is not restricted to srcArray and dstArray. The redistHandle can be applied to redistribute data between any Array pairs that are congruent to the Array pair used during precomputation. Arrays are congruent if they are defined on matching DistGrids and the shape of local array allocations match for all DEs.
The resources held by redistHandle need to be deallocated by the user code before the handle becomes inaccessible.
call ESMF_RouteHandleRelease(routehandle=redistHandle, rc=rc)
In default mode, i.e. without providing the optional srcToDstTransposeMap argument, ESMF_ArrayRedistStore() does not require equal number of dimensions in source and destination Array. Only the total number of elements must match.
Specifying srcToDstTransposeMap switches ESMF_ArrayRedistStore() into transpose mode. In this mode each dimension of srcArray is uniquely associated with a dimension in dstArray. The sizes of associated dimensions must match for each pair.
dstDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/20,10/), rc=rc)
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, rc=rc)
This dstArray object covers a 20 x 10 index space while the srcArray, defined further up, covers a 10 x 20 index space. Setting srcToDstTransposeMap = (/2,1/) will associate the first and second dimension of srcArray with the second and first dimension of dstArray, respectively. This corresponds to a transpose of dimensions. Since the decomposition and distribution of dimensions may be different for source and destination redistribution may occur at the same time.
call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, srcToDstTransposeMap=(/2,1/), rc=rc)
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
The transpose mode of ESMF_ArrayRedist() is not limited to distributed dimensions of Arrays. The srcToDstTransposeMap argument can be used to transpose undistributed dimensions in the same manner. Furthermore transposing distributed and undistributed dimensions between Arrays is also supported.
The srcArray used in the following examples is of rank 4 with 2 distributed and 2 undistributed dimensions. The distributed dimensions are the two first dimensions of the Array and are distributed according to the srcDistgrid which describes a total index space region of 100 x 200 elements. The last two Array dimensions are undistributed dimensions of size 2 and 3, respectively.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=4, rc=rc)
srcDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/100,200/), &
rc=rc)
srcArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=srcDistgrid, &
undistLBound=(/1,1/), undistUBound=(/2,3/), rc=rc)
The first dstArray to consider is defined on a DistGrid that also describes a 100 x 200 index space region. The distribution indicated by dstDistgrid may be different from the source distribution. Again the first two Array dimensions are associated with the DistGrid dimensions in sequence. Furthermore, the last two Array dimensions are undistributed dimensions, however, the sizes are 3 and 2, respectively.
dstDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/100,200/), &
rc=rc)
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
undistLBound=(/1,1/), undistUBound=(/3,2/), rc=rc)
The desired mapping between srcArray and dstArray dimensions is expressed by srcToDstTransposeMap = (/1,2,4,3/), transposing only the two undistributed dimensions.
call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, srcToDstTransposeMap=(/1,2,4,3/), rc=rc)
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
Next consider a dstArray that is defined on the same dstDistgrid, but with a different order of Array dimensions. The desired order is specified during Array creation using the argument distgridToArrayMap = (/2,3/). This map associates the first and second DistGrid dimensions with the second and third Array dimensions, respectively, leaving Array dimensions one and four undistributed.
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
distgridToArrayMap=(/2,3/), undistLBound=(/1,1/), undistUBound=(/3,2/), &
rc=rc)
Again the sizes of the undistributed dimensions are chosen in reverse order compared to srcArray. The desired transpose mapping in this case will be srcToDstTransposeMap = (/2,3,4,1/).
call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, srcToDstTransposeMap=(/2,3,4,1/), rc=rc)
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
Finally consider the case where dstArray is constructed on a 200 x 3 index space and where the undistributed dimensions are of size 100 and 2.
dstDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/200,3/), &
rc=rc)
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
undistLBound=(/1,1/), undistUBound=(/100,2/), rc=rc)
By construction srcArray and dstArray hold the same number of elements, albeit in a very different layout. Nevertheless, with a srcToDstTransposeMap that maps matching dimensions from source to destination an Array redistribution becomes a well defined operation between srcArray and dstArray.
call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, srcToDstTransposeMap=(/3,1,4,2/), rc=rc)
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
The default mode of Array redistribution, i.e. without providing a srcToDstTransposeMap to ESMF_ArrayRedistStore(), also supports undistributed Array dimensions. The requirement in this case is that the total undistributed element count, i.e. the product of the sizes of all undistributed dimensions, be the same for source and destination Array. In this mode the number of undistributed dimensions need not match between source and destination.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=4, rc=rc)
srcDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/10,20/), &
regDecomp=(/4,1/), rc=rc)
srcArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=srcDistgrid, &
undistLBound=(/1,1/), undistUBound=(/2,4/), rc=rc)
dstDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/10,20/), &
regDecomp=(/1,4/), rc=rc)
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
distgridToArrayMap=(/2,3/), undistLBound=(/1,1/), undistUBound=(/2,4/), &
rc=rc)
Both srcArray and dstArray have two undistributed dimensions and
a total count of undistributed elements of
.
The Array redistribution operation is defined in terms of sequentialized undistributed dimensions. In the above case this means that a unique sequence index will be assigned to each of the 8 undistributed elements. The sequence indices will be 1, 2, ..., 8, where sequence index 1 is assigned to the first element in the first (i.e. fastest varying in memory) undistributed dimension. The following undistributed elements are labeled in consecutive order as they are stored in memory.
call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
The redistribution operation by default applies the identity operation between the elements of undistributed dimensions. This means that source element with sequence index 1 will be mapped against destination element with sequence index 1 and so forth. Because of the way source and destination Arrays in the current example were constructed this corresponds to a mapping of dimensions 3 and 4 on srcArray to dimensions 1 and 4 on dstArray, respectively.
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
Array redistribution does not require the same number of undistributed dimensions in source and destination Array, merely the total number of undistributed elements must match.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=3, rc=rc)
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
distgridToArrayMap=(/1,3/), undistLBound=(/11/), undistUBound=(/18/), &
rc=rc)
This dstArray object only has a single undistributed dimension, while the srcArray, defined further back, has two undistributed dimensions. However, the total undistributed element count for both Arrays is 8.
call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
In this case the default identity operation between the elements of undistributed dimensions corresponds to a merging of dimensions 3 and 4 on srcArray into dimension 2 on dstArray.
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
Sparse matrix multiplication is a fundamental Array communication method. One frequently used application of this method is the interpolation between pairs of Arrays. The principle is this: the value of each element in the exclusive region of the destination Array is expressed as a linear combination of potentially all the exclusive elements of the source Array. Naturally most of the coefficients of these linear combinations will be zero and it is more efficient to store explicit information about the non-zero elements than to keep track of all the coefficients.
There is a choice to be made with respect to the format in which to store the information about the non-zero elements. One option is to store the value of each coefficient together with the corresponding destination element index and source element index. Destination and source indices could be expressed in terms of the corresponding DistGrid patch index together with the coordinate tuple within the patch. While this format may be the most natural way to express elements in the source and destination Array, it has two major drawbacks. First the coordinate tuple is dimCount specific and second the format is extremly bulky. For 2D source and destination Arrays it would require 6 integers to store the source and destination element information for each non-zero coefficient and matters get worse for higher dimensions.
Both problems can be circumvented by interpreting source and destination Arrays as sequentialized strings or vectors of elements. This is done by assigning a unique sequence index to each exclusive element in both Arrays. With that the operation of updating the elements in the destination Array as linear combinations of source Array elements takes the form of a sparse matrix multiplication.
The default sequence index rule assigns index 
to the minIndex corner
element of the first patch of the DistGrid on which the Array is defined. It then
increments the sequence index by for each element running through the
DistGrid dimensions by order. The index space position of the DistGrid patches
does not affect the sequence labeling of elements. The default sequence indices
for
srcDistgrid = ESMF_DistGridCreate(minIndex=(/-1,0/), maxIndex=(/1,3/), rc=rc)
for each element are:
-------------------------------------> 2nd dim
|
| +------+------+------+------+
| |(-1,0)| | |(-1,3)|
| | | | | |
| | 1 | 4 | 7 | 10 |
| +------+------+------+------+
| | | | | |
| | | | | |
| | 2 | 5 | 8 | 11 |
| +------+------+------+------+
| | (1,0)| | | (1,3)|
| | | | | |
| | 3 | 6 | 9 | 12 |
| +------+------+------+------+
|
v
1st dim
The assigned sequence indices are decomposition and distribution invariant by construction. Furthermore, when an Array is created with extra elements per DE on a DistGrid the sequence indices (which only cover the exclusive elements) remain unchanged.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
srcArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=srcDistgrid, &
totalLWidth=(/1,1/), totalUWidth=(/1,1/), indexflag=ESMF_INDEX_GLOBAL, &
rc=rc)
The extra padding of 1 element in each direction around the exclusive elements on each DE are "invisible" to the Array spare matrix multiplication method. These extra elements are either updated by the computational kernel or by Array halo operations (not yet implemented!).
An alternative way to assign sequence indices to all the elements in the patches covered by a DistGrid object is to use a special ESMF_DistGridCreate() call. This call has been specifically designed for 1D cases with arbitrary, user-supplied sequence indices.
seqIndexList(1) = localPet*10 seqIndexList(2) = localPet*10 + 1 dstDistgrid = ESMF_DistGridCreate(arbSeqIndexList=seqIndexList, rc=rc)
This call to ESMF_DistGridCreate() is collective across the current VM. The arbSeqIndexList argument specifies the PET-local arbitrary sequence indices that need to be covered by the local DE. The resulting DistGrid has one local DE per PET which covers the entire PET-local index range. The user supplied sequence indices must be unique, but the sequence may be interrupted. The four DEs of dstDistgrid have the following local 1D index space coordinates (given between "()") and sequence indices:
covered by DE 0 covered by DE 1 covered by DE 2 covered by DE 3
on PET 0 on PET 1 on PET 2 on PET 3
----------------------------------------------------------------------
(1) : 0 (1) : 10 (1) : 20 (1) : 30
(2) : 1 (2) : 11 (2) : 21 (2) : 31
Again the DistGrid object provides the sequence index labeling for the exclusive elements of an Array created on the DistGrid regardless of extra, non-exclusive elements.
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, rc=rc)
With the definition of sequence indices, either by the default rule or as user provided arbitrary sequence indices, it is now possible to uniquely identify each exclusive element in the source and destination Array by a single integer number. Specifying a pair of source and destination elements takes two integer number regardless of the number of dimensions.
The information required to carry out a sparse matrix multiplication are the pair of source and destination sequence indices and the associated multiplication factor for each pair. ESMF requires this information in form of two Fortran arrays. The factors are stored in a 1D array of the appropriate type and kind, e.g. real(ESMF_KIND_R8)::factorList(:). Array sparse matrix multiplications are only supported between Arrays of the same type and kind using factors of identical type and kind. The sequence index pairs associated with the factors provided by factorList are stored in a 2D Fortran array of default integer kind of the shape integer::factorIndexList(2,:). The sequence indices of the source Array elements are stored in the first row of factorIndexList while the sequence indices of the destination Array elements are stored in the second row.
Each PET in the current VM must call into ESMF_ArraySparseMatMulStore() to precompute and store the communication pattern for the sparse matrix multiplication. The multiplication factors may be provided in parallel, i.e. multiple PETs may specify factorList and factorIndexList arguments when calling into ESMF_ArraySparseMatMulStore(). PETs that do not provide factors either call with factorList and factorIndexList arguments containing zero elements or issue the call omitting both arguments.
if (localPet == 0) then
allocate(factorList(1)) ! PET 0 specifies 1 factor
allocate(factorIndexList(2,1))
factorList = (/0.2/) ! factors
factorIndexList(1,:) = (/5/) ! seq indices into srcArray
factorIndexList(2,:) = (/30/) ! seq indices into dstArray
call ESMF_ArraySparseMatMulStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, factorList=factorList, &
factorIndexList=factorIndexList, rc=rc)
deallocate(factorList)
deallocate(factorIndexList)
else if (localPet == 1) then
allocate(factorList(3)) ! PET 1 specifies 3 factor
allocate(factorIndexList(2,3))
factorList = (/0.5, 0.5, 0.8/) ! factors
factorIndexList(1,:) = (/8, 2, 12/) ! seq indices into srcArray
factorIndexList(2,:) = (/11, 11, 30/) ! seq indices into dstArray
call ESMF_ArraySparseMatMulStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, factorList=factorList, &
factorIndexList=factorIndexList, rc=rc)
deallocate(factorList)
deallocate(factorIndexList)
else
! PETs 2 and 3 do not provide factors
call ESMF_ArraySparseMatMulStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, rc=rc)
endif
The RouteHandle object sparseMatMulHandle produced by ESMF_ArraySparseMatMulStore() can now be used to call ESMF_ArraySparseMatMul() collectively across all PETs of the current VM to perform
dstArray = 0.0
do n=1, size(combinedFactorList)
dstArray(combinedFactorIndexList(2, n)) +=
combinedFactorList(n) * srcArray(combinedFactorIndexList(1, n))
enddo
in parallel. Here combinedFactorList and combinedFactorIndexList
are the combined lists defined by the respective local lists provided by
PETs 0 and 1 in parallel. For this example
call ESMF_ArraySparseMatMul(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, rc=rc)
will initialize the entire dstArray to 0.0 and then update two elements:
on DE 1: dstArray(2) = 0.5 * srcArray(0,0) + 0.5 * srcArray(0,2)
and
on DE 3: dstArray(1) = 0.2 * srcArray(0,1) + 0.8 * srcArray(1,3).
The call to ESMF_ArraySparseMatMul() does provide the option to turn the default dstArray initialization off. If argument zeroflag is set to ESMF_FALSE
call ESMF_ArraySparseMatMul(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, zeroflag=ESMF_FALSE, rc=rc)
skips the initialization and elements in dstArray are updated according to:
do n=1, size(combinedFactorList)
dstArray(combinedFactorIndexList(2, n)) +=
combinedFactorList(n) * srcArray(combinedFactorIndexList(1, n)).
enddo
The resources held by sparseMatMulHandle need to be deallocated by the user code before the handle becomes inaccessible.
call ESMF_RouteHandleRelease(routehandle=sparseMatMulHandle, rc=rc)
The Array sparse matrix multiplication also applies to Arrays with undistributed dimensions. The undistributed dimensions are interpreted in a sequentialized manner, much like the distributed dimensions, introducing a second sequence index for source and destination elements. Sequence index 1 is assigned to the first element in the first (i.e. fastest varying in memory) undistributed dimension. The following undistributed elements are labeled in consecutive order as they are stored in memory.
In the simplest case the Array sparse matrix multiplication will apply an identity matrix to the vector of sequentialized undistributed Array elements for every non-zero element in the sparse matrix. The requirement in this case is that the total undistributed element count, i.e. the product of the sizes of all undistributed dimensions, be the same for source and destination Array.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=3, rc=rc)
srcArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=srcDistgrid, &
totalLWidth=(/1,1/), totalUWidth=(/1,1/), indexflag=ESMF_INDEX_GLOBAL, &
distgridToArrayMap=(/1,2/), undistLBound=(/1/), undistUBound=(/2/), rc=rc)
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
distgridToArrayMap=(/2/), undistLBound=(/1/), undistUBound=(/2/), rc=rc)
Setting up factorList and factorIndexList is identical to the case for Arrays without undistributed dimensions. Also the call to ESMF_ArraySparseMatMulStore() remains unchanged. Internally, however, the source and destination Arrays are checked to make sure the total undistributed element count matches.
if (localPet == 0) then
allocate(factorList(1)) ! PET 0 specifies 1 factor
allocate(factorIndexList(2,1))
factorList = (/0.2/) ! factors
factorIndexList(1,:) = (/5/) ! seq indices into srcArray
factorIndexList(2,:) = (/30/) ! seq indices into dstArray
call ESMF_ArraySparseMatMulStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, factorList=factorList, &
factorIndexList=factorIndexList, rc=rc)
deallocate(factorList)
deallocate(factorIndexList)
else if (localPet == 1) then
allocate(factorList(3)) ! PET 1 specifies 3 factor
allocate(factorIndexList(2,3))
factorList = (/0.5, 0.5, 0.8/) ! factors
factorIndexList(1,:) = (/8, 2, 12/) ! seq indices into srcArray
factorIndexList(2,:) = (/11, 11, 30/) ! seq indices into dstArray
call ESMF_ArraySparseMatMulStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, factorList=factorList, &
factorIndexList=factorIndexList, rc=rc)
deallocate(factorList)
deallocate(factorIndexList)
else
! PETs 2 and 3 do not provide factors
call ESMF_ArraySparseMatMulStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, rc=rc)
endif
The call into the ESMF_ArraySparseMatMul() operation is completely transparent with respect to whether source and/or destination Arrays contain undistributed dimensions.
call ESMF_ArraySparseMatMul(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, rc=rc)
This operation will initialize the entire dstArray to 0.0 and then update four elements:
on DE 1: dstArray[1](2) = 0.5 * srcArray(0,0)[1] + 0.5 * srcArray(0,2)[1], dstArray[2](2) = 0.5 * srcArray(0,0)[2] + 0.5 * srcArray(0,2)[2]
and
on DE 3: dstArray[1](1) = 0.2 * srcArray(0,1)[1] + 0.8 * srcArray(1,3)[1], dstArray[2](1) = 0.2 * srcArray(0,1)[2] + 0.8 * srcArray(1,3)[2].
Here indices between "()" refer to distributed dimensions while indices between "[]" correspond to undistributed dimensions.
In a more general version of the Array sparse matrix multiplication the total undistributed element count, i.e. the product of the sizes of all undistributed dimensions, need not be the same for source and destination Array. In this formulation each non-zero element of the sparse matrix is identified with a unique element in the source and destination Array. This requires a generalization of the factorIndexList argument which now must contain four integer numbers for each element. These numbers in sequence are the sequence index of the distributed dimensions and the sequence index of the undistributed dimensions of the element in the source Array, followed by the sequence index of the distributed dimensions and the sequence index of the undistributed dimensions of the element in the destination Array.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=3, rc=rc)
srcArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=srcDistgrid, &
totalLWidth=(/1,1/), totalUWidth=(/1,1/), indexflag=ESMF_INDEX_GLOBAL, &
distgridToArrayMap=(/1,2/), undistLBound=(/1/), undistUBound=(/2/), rc=rc)
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
distgridToArrayMap=(/2/), undistLBound=(/1/), undistUBound=(/4/), rc=rc)
Setting up factorList is identical to the previous cases since there is still only one value associated with each non-zero matrix element. However, each entry in factorIndexList now has 4 instead of just 2 components.
if (localPet == 0) then
allocate(factorList(1)) ! PET 0 specifies 1 factor
allocate(factorIndexList(4,1))
factorList = (/0.2/) ! factors
factorIndexList(1,:) = (/5/) ! seq indices into srcArray
factorIndexList(2,:) = (/1/) ! undistr. seq indices into srcArray
factorIndexList(3,:) = (/30/) ! seq indices into dstArray
factorIndexList(4,:) = (/2/) ! undistr. seq indices into dstArray
call ESMF_ArraySparseMatMulStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, factorList=factorList, &
factorIndexList=factorIndexList, rc=rc)
deallocate(factorList)
deallocate(factorIndexList)
else if (localPet == 1) then
allocate(factorList(3)) ! PET 1 specifies 3 factor
allocate(factorIndexList(4,3))
factorList = (/0.5, 0.5, 0.8/) ! factors
factorIndexList(1,:) = (/8, 2, 12/) ! seq indices into srcArray
factorIndexList(2,:) = (/2, 1, 1/) ! undistr. seq indices into srcArray
factorIndexList(3,:) = (/11, 11, 30/) ! seq indices into dstArray
factorIndexList(4,:) = (/4, 4, 2/) ! undistr. seq indices into dstArray
call ESMF_ArraySparseMatMulStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, factorList=factorList, &
factorIndexList=factorIndexList, rc=rc)
deallocate(factorList)
deallocate(factorIndexList)
else
! PETs 2 and 3 do not provide factors
call ESMF_ArraySparseMatMulStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, rc=rc)
endif
The call into the ESMF_ArraySparseMatMul() operation remains unchanged.
call ESMF_ArraySparseMatMul(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, rc=rc)
This operation will initialize the entire dstArray to 0.0 and then update two elements:
on DE 1: dstArray[4](2) = 0.5 * srcArray(0,0)[1] + 0.5 * srcArray(0,2)[2],
and
on DE 3: dstArray[2](1) = 0.2 * srcArray(0,1)[1] + 0.8 * srcArray(1,3)[1],
Here indices in 
refer to distributed dimensions while indices in ![$[]$](img22.png){kind=small}
correspond to undistributed dimensions.
The Array class is part of the ESMF index space layer and is built ontop of the DistGrid and DELayout classes. The DELayout class introduces the notion of decomposition elements (DEs) and their layout across the available PETs. The DistGrid describes how index space is decomposed by assigning logically rectangular index space pieces or DE-local tiles to the DEs. The Array finally associates a local memory allocation with each local DE.
The following is a list of implementation specific details about the current ESMF Array.
INTERFACE:
! Private name; call using ESMF_ArrayCreate() function ESMF_ArrayCreateAssmdShape<rank><type><kind>(farray, & distgrid, distgridToArrayMap, computationalEdgeLWidth, & computationalEdgeUWidth, computationalLWidth, & computationalUWidth, totalLWidth, & totalUWidth, indexflag, staggerLoc, vectorDim, & undistLBound, undistUBound, name, rc)ARGUMENTS:
<type> (ESMF_KIND_<kind>),dimension(<rank>),intent(in),target :: farray type(ESMF_DistGrid), intent(in) :: distgrid integer, intent(in), optional :: distgridToArrayMap(:) integer, intent(in), optional :: computationalEdgeLWidth(:) integer, intent(in), optional :: computationalEdgeUWidth(:) integer, intent(in), optional :: computationalLWidth(:) integer, intent(in), optional :: computationalUWidth(:) integer, intent(in), optional :: totalLWidth(:) integer, intent(in), optional :: totalUWidth(:) type(ESMF_IndexFlag), intent(in), optional :: indexflag integer, intent(in), optional :: staggerLoc integer, intent(in), optional :: vectorDim integer, intent(in), optional :: undistLBound(:) integer, intent(in), optional :: undistUBound(:) character (len=*), intent(in), optional :: name integer, intent(out), optional :: rcRETURN VALUE:
type(ESMF_Array) :: ESMF_ArrayCreateAssmdShapemrankDmtypekindDESCRIPTION:
Create an ESMF_Array object from an existing local native Fortran90 array according to distgrid. Besides farray each PET must issue this call with identical arguments in order to create a consistent Array object. The local arrays provided must be dimensioned according to the DE-local total region. Bounds of the exclusive regions are set as specified in the distgrid argument. Bounds for array dimensions that are not distributed can be chosen freely using the undistLBound and undistUBound arguments.
This interface requires a 1 DE per PET decomposition. The Array object will not be created and an error will be returned if this condition is not met.
The not distributed Array dimensions form a tensor of rank = array.rank - distgrid.dimCount. By default all tensor elements are associated with stagger location 0. The widths of the computational region are set to the provided value, or zero by default, for all tensor elements. Use ESMF_ArraySet() to change these default settings after the Array object has been created.
The return value is the newly created ESMF_Array object.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayCreate()
function ESMF_ArrayCreateLocalArray(larrayList, distgrid, distgridToArrayMap, &
computationalEdgeLWidth, computationalEdgeUWidth, computationalLWidth, &
computationalUWidth, totalLWidth, totalUWidth, &
indexflag, staggerLoc, vectorDim, undistLBound, undistUBound, name, rc)
ARGUMENTS:
type(ESMF_LocalArray), intent(in) :: larrayList(:)
type(ESMF_DistGrid), intent(in) :: distgrid
integer, intent(in), optional :: distgridToArrayMap(:)
integer, intent(in), optional :: computationalEdgeLWidth(:)
integer, intent(in), optional :: computationalEdgeUWidth(:)
integer, intent(in), optional :: computationalLWidth(:)
integer, intent(in), optional :: computationalUWidth(:)
integer, intent(in), optional :: totalLWidth(:)
integer, intent(in), optional :: totalUWidth(:)
type(ESMF_IndexFlag), intent(in), optional :: indexflag
integer, intent(in), optional :: staggerLoc
integer, intent(in), optional :: vectorDim
integer, intent(in), optional :: undistLBound(:)
integer, intent(in), optional :: undistUBound(:)
character (len=*), intent(in), optional :: name
integer, intent(out), optional :: rc
RETURN VALUE:
type(ESMF_Array) :: ESMF_ArrayCreateLocalArrayDESCRIPTION:
Create an ESMF_Array object from existing ESMF_LocalArray objects according to distgrid. Besides larrayList each PET must issue this call with identical arguments in order to create a consistent Array object. The local arrays provided must be dimensioned according to the DE-local total region. Bounds of the exclusive regions are set as specified in the distgrid argument. Bounds for array dimensions that are not distributed can be chosen freely using the undistLBound and undistUBound arguments.
This interface is able to handle multiple DEs per PET.
The not distributed Array dimensions form a tensor of rank = array.rank - distgrid.dimCount. By default all tensor elements are associated with stagger location 0. The widths of the computational region are set to the provided value, or zero by default, for all tensor elements. Use ESMF_ArraySet() to change these default settings after the Array object has been created.
The return value is the newly created ESMF_Array object.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayCreate()
function ESMF_ArrayCreateAllocate(arrayspec, distgrid, distgridToArrayMap, &
computationalEdgeLWidth, computationalEdgeUWidth, &
computationalLWidth, computationalUWidth, totalLWidth, totalUWidth, &
indexflag, staggerLoc, vectorDim, undistLBound, undistUBound, name, rc)
ARGUMENTS:
type(ESMF_ArraySpec), intent(inout) :: arrayspec
type(ESMF_DistGrid), intent(in) :: distgrid
integer, intent(in), optional :: distgridToArrayMap(:)
integer, intent(in), optional :: computationalEdgeLWidth(:)
integer, intent(in), optional :: computationalEdgeUWidth(:)
integer, intent(in), optional :: computationalLWidth(:)
integer, intent(in), optional :: computationalUWidth(:)
integer, intent(in), optional :: totalLWidth(:)
integer, intent(in), optional :: totalUWidth(:)
type(ESMF_IndexFlag), intent(in), optional :: indexflag
integer, intent(in), optional :: staggerLoc
integer, intent(in), optional :: vectorDim
integer, intent(in), optional :: undistLBound(:)
integer, intent(in), optional :: undistUBound(:)
character (len=*), intent(in), optional :: name
integer, intent(out), optional :: rc
RETURN VALUE:
type(ESMF_Array) :: ESMF_ArrayCreateAllocateDESCRIPTION:
Create an ESMF_Array object and allocate uninitialized data space according to arrayspec and distgrid. Each PET must issue this call with identical arguments in order to create a consistent Array object. DE-local allocations are made according to the total region defined by the arguments to this call: distgrid and the optional Width arguments.
The return value is the newly created ESMF_Array object.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayCreate() function ESMF_ArrayCreateCopy(array, rc)ARGUMENTS:
type(ESMF_Array), intent(in) :: array
integer, intent(out), optional :: rc
RETURN VALUE:
type(ESMF_Array) :: ESMF_ArrayCreateCopyDESCRIPTION:
Create an ESMF_Array object as the copy of an existing Array.
The return value is the newly created ESMF_Array object.
The arguments are:
INTERFACE:
subroutine ESMF_ArrayDestroy(array, rc)ARGUMENTS:
type(ESMF_Array), intent(inout) :: array
integer, intent(out), optional :: rc
DESCRIPTION:
Destroy an ESMF_Array object.
The arguments are:
INTERFACE:
subroutine ESMF_ArrayGather<rank><type><kind>(array, farray, patch, & rootPet, vm, rc)ARGUMENTS:
type(ESMF_Array), intent(inout) :: array mtype (ESMF_KIND_mtypekind),dimension(mdim),intent(in),target :: farray integer, intent(in), optional :: patch integer, intent(in) :: rootPet type(ESMF_VM), intent(in), optional :: vm integer, intent(out), optional :: rcDESCRIPTION:
Gather the data of an ESMF_Array object into the farray located on rootPET. A single DistGrid patch of array must be gathered into farray. The optional patch argument allows selection of the patch. For Arrays defined on a single patch DistGrid the default selection (patch 1) will be correct. The shape of farray must match the shape of the patch in Array.
If the Array contains replicating DistGrid dimensions data will be gathered from the numerically higher DEs. Replicated data elements in numericaly lower DEs will be ignored.
This version of the interface implements the PET-based blocking paradigm: Each PET of the VM must issue this call exactly once for all of its DEs. The call will block until all PET-local data objects are accessible.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayGet()
subroutine ESMF_ArrayGetDefault(array, typekind, rank, larrayList, &
indexflag, distgridToArrayMap, arrayToDistGridMap, undistLBound, &
undistUBound, exclusiveLBound, exclusiveUBound, &
computationalLBound, computationalUBound, totalLBound, totalUBound, &
computationalLWidth, computationalUWidth, totalLWidth, totalUWidth, &
name, distgrid, dimCount, patchCount, minIndexPDimPPatch, &
maxIndexPDimPPatch, patchListPDe, indexCountPDimPDe, delayout, deCount, &
localDeCount, localDeList, rc)
ARGUMENTS:
type(ESMF_Array), intent(in) :: array
type(ESMF_TypeKind), intent(out), optional :: typekind
integer, intent(out), optional :: rank
type(ESMF_LocalArray), target, intent(out), optional :: larrayList(:)
type(ESMF_IndexFlag), intent(out), optional :: indexflag
integer, intent(out), optional :: distgridToArrayMap(:)
integer, intent(out), optional :: arrayToDistGridMap(:)
integer, intent(out), optional :: undistLBound(:)
integer, intent(out), optional :: undistUBound(:)
integer, intent(out), optional :: exclusiveLBound(:,:)
integer, intent(out), optional :: exclusiveUBound(:,:)
integer, intent(out), optional :: computationalLBound(:,:)
integer, intent(out), optional :: computationalUBound(:,:)
integer, intent(out), optional :: totalLBound(:,:)
integer, intent(out), optional :: totalUBound(:,:)
integer, intent(out), optional :: computationalLWidth(:,:)
integer, intent(out), optional :: computationalUWidth(:,:)
integer, intent(out), optional :: totalLWidth(:,:)
integer, intent(out), optional :: totalUWidth(:,:)
character(len=*), intent(out), optional :: name
type(ESMF_DistGrid), intent(out), optional :: distgrid
integer, intent(out), optional :: dimCount
integer, intent(out), optional :: patchCount
integer, intent(out), optional :: minIndexPDimPPatch(:,:)
integer, intent(out), optional :: maxIndexPDimPPatch(:,:)
integer, intent(out), optional :: patchListPDe(:)
integer, intent(out), optional :: indexCountPDimPDe(:,:)
type(ESMF_DELayout), intent(out), optional :: delayout
integer, intent(out), optional :: deCount
integer, intent(out), optional :: localDeCount
integer, intent(out), optional :: localDeList(:)
integer, intent(out), optional :: rc
DESCRIPTION:
Get internal information.
This interface works for any number of DEs per PET.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayGet() subroutine ESMF_ArrayGetPLocalDePDim(array, localDe, dim, indexCount, indexList, rc)ARGUMENTS:
type(ESMF_Array), intent(in) :: array
integer, intent(in) :: localDe
integer, intent(in) :: dim
integer, intent(out), optional :: indexCount
integer, intent(out), optional :: indexList(:)
integer, intent(out), optional :: rc
DESCRIPTION:
Get internal information per local DE, per dim.
This interface works for any number of DEs per PET.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayGet() subroutine ESMF_ArrayGetFPtr<rank><type><kind>(array, farrayPtr, rc)ARGUMENTS:
type(ESMF_Array), intent(in) :: array <type> (ESMF_KIND_<kind>),dimension(<rank>),pointer :: farrayPtr integer, intent(out), optional :: rcDESCRIPTION:
Get Fortran90 pointer to DE-local memory regions in Array object.
This interface requires that exactly 1 DE is associated with the calling PET. An error will be returned if this condition is not met.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayGet() subroutine ESMF_ArrayGetLarray(array, larray, rc)ARGUMENTS:
type(ESMF_Array), intent(in) :: array
type(ESMF_LocalArray), intent(inout) :: larray
integer, intent(out), optional :: rc
DESCRIPTION:
Get internal information.
This interface requires that exactly 1 DE is associated with the calling PET. An error will be returned if this condition is not met.
The arguments are:
INTERFACE:
subroutine ESMF_ArrayScatter<rank><type><kind>(array, farray, patch, & rootPet, vm, rc)ARGUMENTS:
type(ESMF_Array), intent(inout) :: array mtype (ESMF_KIND_mtypekind),dimension(mdim),intent(in),target :: farray integer, intent(in), optional :: patch integer, intent(in) :: rootPet type(ESMF_VM), intent(in), optional :: vm integer, intent(out), optional :: rcDESCRIPTION:
Scatter the data of farray located on rootPET across an ESMF_Array object. A single farray must be scattered across a single DistGrid patch in Array. The optional patch argument allows selection of the patch. For Arrays defined on a single patch DistGrid the default selection (patch 1) will be correct. The shape of farray must match the shape of the patch in Array.
If the Array contains replicating DistGrid dimensions data will be scattered across all of the replicated pieces.
This version of the interface implements the PET-based blocking paradigm: Each PET of the VM must issue this call exactly once for all of its DEs. The call will block until all PET-local data objects are accessible.
The arguments are:
INTERFACE:
subroutine ESMF_ArrayPrint(array, options, rc)ARGUMENTS:
type(ESMF_Array), intent(in) :: array
character(len=*), intent(in), optional :: options
integer, intent(out), optional :: rc
DESCRIPTION:
Print internal information of the specified ESMF_Array object.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayRedistStore()
subroutine ESMF_ArrayRedistStore<type><kind>(srcArray, dstArray, routehandle, &
factor, srcToDstTransposeMap, rc)
ARGUMENTS:
type(ESMF_Array), intent(in) :: srcArray
type(ESMF_Array), intent(inout) :: dstArray
type(ESMF_RouteHandle), intent(inout) :: routehandle
<type>(ESMF_KIND_<kind>), intent(in) :: factor
integer, intent(in), optional :: srcToDstTransposeMap(:)
integer, intent(out), optional :: rc
DESCRIPTION:
Store an Array redistribution operation from srcArray to dstArray. PETs that specify a factor argument must use the <type><kind> overloaded interface. Other PETs call into the interface without factor argument. If multiple PETs specify the factor argument its type and kind as well as its value must match across all PETs. If none of the PETs specifies a factor argument the default will be a factor of 1.
Both srcArray and dstArray are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of patches within the DistGrid or by user-supplied arbitrary sequence indices. See section 19.2.15 for details on the definition of sequence indices. Redistribution corresponds to an identity mapping of the source Array vector to the destination Array vector.
Source and destination Arrays may be of different <type><kind>. Further source and destination Arrays may differ in shape, however, the number of elements must match.
It is erroneous to specify the identical Array object for srcArray and dstArray arguments.
The routine returns an ESMF_RouteHandle that can be used to call
ESMF_ArrayRedist() on any pair of Arrays that are congruent
and typekind conform with the srcArray, dstArray pair.
Congruent Arrays possess matching DistGrids and the shape of the local
array tiles matches between the Arrays for every DE.
This call is collective across the current VM.
INTERFACE:
! Private name; call using ESMF_ArrayRedistStore()
subroutine ESMF_ArrayRedistStoreNF(srcArray, dstArray, routehandle, &
srcToDstTransposeMap, rc)
ARGUMENTS:
type(ESMF_Array), intent(in) :: srcArray
type(ESMF_Array), intent(inout) :: dstArray
type(ESMF_RouteHandle), intent(inout) :: routehandle
integer, intent(in), optional :: srcToDstTransposeMap(:)
integer, intent(out), optional :: rc
DESCRIPTION:
Store an Array redistribution operation from srcArray to dstArray. PETs that specify a factor argument must use the <type><kind> overloaded interface. Other PETs call into the interface without factor argument. If multiple PETs specify the factor argument its type and kind as well as its value must match across all PETs. If none of the PETs specifies a factor argument the default will be a factor of 1.
Both srcArray and dstArray are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of patches within the DistGrid or by user-supplied arbitrary sequence indices. See section 19.2.15 for details on the definition of sequence indices. Redistribution corresponds to an identity mapping of the source Array vector to the destination Array vector.
Source and destination Arrays may be of different <type><kind>. Further source and destination Arrays may differ in shape, however, the number of elements must match.
It is erroneous to specify the identical Array object for srcArray and dstArray arguments.
The routine returns an ESMF_RouteHandle that can be used to call
ESMF_ArrayRedist() on any pair of Arrays that are congruent
and typekind conform with the srcArray, dstArray pair.
Congruent Arrays possess matching DistGrids and the shape of the local
array tiles matches between the Arrays for every DE.
This call is collective across the current VM.
INTERFACE:
subroutine ESMF_ArrayRedist(srcArray, dstArray, routehandle, checkflag, rc)ARGUMENTS:
type(ESMF_Array), intent(in), optional :: srcArray
type(ESMF_Array), intent(inout),optional :: dstArray
type(ESMF_RouteHandle), intent(inout) :: routehandle
type(ESMF_Logical), intent(in), optional :: checkflag
integer, intent(out), optional :: rc
DESCRIPTION:
Execute a precomputed Array redistribution from srcArray to dstArray. Both srcArray and dstArray must be congruent and typekind conform with the respective Arrays used during ESMF_ArrayRedistStore(). Congruent Arrays possess matching DistGrids and the shape of the local array tiles matches between the Arrays for every DE.
It is erroneous to specify the identical Array object for srcArray and dstArray arguments.
See ESMF_ArrayRedistStore() on how to precompute routehandle.
This call is collective across the current VM.
INTERFACE:
subroutine ESMF_ArraySet(array, name, staggerLoc, vectorDim, &
computationalLWidth, computationalUWidth, rc)
ARGUMENTS:
type(ESMF_Array), intent(inout) :: array
character(len = *), intent(in), optional :: name
integer, intent(in), optional :: staggerLoc
integer, intent(in), optional :: vectorDim
integer, intent(in), optional :: computationalLWidth(:,:)
integer, intent(in), optional :: computationalUWidth(:,:)
integer, intent(out), optional :: rc
DESCRIPTION:
Sets adjustable settings in an ESMF_Array object. Arrays with tensor dimensions will set values for all tensor components.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArraySet() subroutine ESMF_ArraySetTensor(array, tensorIndex, staggerLoc, vectorDim, rc)ARGUMENTS:
type(ESMF_Array), intent(in) :: array
integer, intent(in) :: tensorIndex(:)
integer, intent(in), optional :: staggerLoc
integer, intent(in), optional :: vectorDim
integer, intent(out), optional :: rc
DESCRIPTION:
Sets adjustable settings in an ESMF_Array object for a specific tensor component.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArraySparseMatMulStore()
subroutine ESMF_ArraySparseMatMulStore<type><kind>(srcArray, dstArray, &
routehandle, factorList, factorIndexList, rc)
ARGUMENTS:
type(ESMF_Array), intent(in) :: srcArray
type(ESMF_Array), intent(inout) :: dstArray
type(ESMF_RouteHandle), intent(inout) :: routehandle
<type>(ESMF_KIND_<kind>), target, intent(in) :: factorList(:)
integer, intent(in) :: factorIndexList(:,:)
integer, intent(out), optional :: rc
DESCRIPTION:
Store an Array sparse matrix multiplication operation from srcArray to dstArray. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.
Both srcArray and dstArray are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of patches within the DistGrid or by user-supplied arbitrary sequence indices. See section 19.2.15 for details on the definition of sequence indices.
Source and destination Arrays, as well as the supplied factorList argument, may be of different <type><kind>. Further source and destination Arrays may differ in shape and number of elements.
It is erroneous to specify the identical Array object for srcArray and dstArray arguments.
The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArraySparseMatMul() on any pair of Arrays that are congruent and typekind conform with the srcArray, dstArray pair. Congruent Arrays possess matching DistGrids and the shape of the local array tiles matches between the Arrays for every DE.
This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.
This call is collective across the current VM.
The second dimension of factorIndexList steps through the list of pairs, i.e. size(factorIndexList,2) == size(factorList). The first dimension of factorIndexList is either of size 2 or size 4.
In the size 2 format factorIndexList(1,:) specifies the sequence index of the source element in the srcArray while factorIndexList(2,:) specifies the sequence index of the destination element in dstArray. For this format to be a valid option source and destination Arrays must have matching number of tensor elements (the product of the sizes of all Array tensor dimensions). Under this condition an identiy matrix can be applied within the space of tensor elements for each sparse matrix factor.
The size 4 format is more general and does not require a matching tensor element count. Here the factorIndexList(1,:) specifies the sequence index while factorIndexList(2,:) specifies the tensor sequence index of the source element in the srcArray. Further factorIndexList(3,:) specifies the sequence index and factorIndexList(4,:) specifies the tensor sequence index of the destination element in the dstArray.
See section 19.2.15 for details on the definition of Array sequence indices and tensor sequence indices.
INTERFACE:
! Private name; call using ESMF_ArraySparseMatMulStore()
subroutine ESMF_ArraySparseMatMulStoreNF(srcArray, dstArray, routehandle, &
rc)
ARGUMENTS:
type(ESMF_Array), intent(in) :: srcArray
type(ESMF_Array), intent(inout) :: dstArray
type(ESMF_RouteHandle), intent(inout) :: routehandle
integer, intent(out), optional :: rc
DESCRIPTION:
Store an Array sparse matrix multiplication operation from srcArray to dstArray. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.
Both srcArray and dstArray are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of patches within the DistGrid or by user-supplied arbitrary sequence indices. See section 19.2.15 for details on the definition of sequence indices.
Source and destination Arrays, as well as the supplied factorList argument, may be of different <type><kind>. Further source and destination Arrays may differ in shape and number of elements.
It is erroneous to specify the identical Array object for srcArray and dstArray arguments.
The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArraySparseMatMul() on any pair of Arrays that are congruent and typekind conform with the srcArray, dstArray pair. Congruent Arrays possess matching DistGrids and the shape of the local array tiles matches between the Arrays for every DE.
This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.
This call is collective across the current VM.
INTERFACE:
subroutine ESMF_ArraySparseMatMul(srcArray, dstArray, routehandle, &
zeroflag, checkflag, rc)
ARGUMENTS:
type(ESMF_Array), intent(in), optional :: srcArray
type(ESMF_Array), intent(inout),optional :: dstArray
type(ESMF_RouteHandle), intent(inout) :: routehandle
type(ESMF_Logical), intent(in), optional :: zeroflag
type(ESMF_Logical), intent(in), optional :: checkflag
integer, intent(out), optional :: rc
DESCRIPTION:
Execute a precomputed Array sparse matrix multiplication from srcArray to dstArray. Both srcArray and dstArray must be congruent and typekind conform with the respective Arrays used during ESMF_ArraySparseMatMulStore(). Congruent Arrays possess matching DistGrids and the shape of the local array tiles matches between the Arrays for every DE.
It is erroneous to specify the identical Array object for srcArray and dstArray arguments.
See ESMF_ArraySparseMatMulStore() on how to precompute routehandle. See section 19.2.15 for details on the operation ESMF_ArraySparseMatMul() performs.
This call is collective across the current VM.
INTERFACE:
subroutine ESMF_ArrayValidate(array, rc)ARGUMENTS:
type(ESMF_Array), intent(in) :: array
integer, intent(out), optional :: rc
DESCRIPTION:
Validates that the Array is internally consistent. The method returns an error code if problems are found.
The arguments are:
The ESMF_LocalArray class provides a language independent representation of data in array format. One of the major functions of the LocalArray class is to bridge the Fortran/C/C++ language difference that exists with respect to array representation. All ESMF Field and Array data is internally stored in ESMF LocalArray objects allowing transparent access from Fortran and C/C++.
In the ESMF Fortran API the LocalArray becomes visible in those cases where a local PET may be associated with multiple pieces of an Array, e.g. if there are multiple DEs associated with a single PET. The Fortran language standard does not provide an array of arrays construct, however arrays of derived types holding arrays are possible. ESMF calls use arguments that are of type ESMF_LocalArray with dimension attribute where necessary.
INTERFACE:
! Private name; call using ESMF_LocalArrayCreate()
function ESMF_LocalArrayCreateByTKR(rank, typekind, counts, lbounds, &
ubounds, rc)
RETURN VALUE:
type(ESMF_LocalArray) :: ESMF_LocalArrayCreateByTKRARGUMENTS:
integer, intent(in) :: rank
type(ESMF_TypeKind), intent(in) :: typekind
integer, intent(in), optional :: counts(:)
integer, intent(in), optional :: lbounds(:)
integer, intent(in), optional :: ubounds(:)
integer, intent(out), optional :: rc
DESCRIPTION:
Create a new ESMF_LocalArray and allocate data space, which remains uninitialized. The return value is a new LocalArray.
The arguments are:
INTERFACE:
! Private name; call using ESMF_LocalArrayCreate() function ESMF_LocalArrayCreateBySpec(arrayspec, counts, lbounds, ubounds, rc)RETURN VALUE:
type(ESMF_LocalArray) :: ESMF_LocalArrayCreateBySpecARGUMENTS:
type(ESMF_ArraySpec), intent(inout) :: arrayspec
integer, intent(in), optional :: counts(:)
integer, intent(in), optional :: lbounds(:)
integer, intent(in), optional :: ubounds(:)
integer, intent(out), optional :: rc
DESCRIPTION:
Create a new ESMF_LocalArray and allocate data space, which remains uninitialized. The return value is a new LocalArray.
The arguments are:
INTERFACE:
! Private name; call using ESMF_LocalArrayCreate() function ESMF_LocalArrayCreateCopy(larray, rc)RETURN VALUE:
type(ESMF_LocalArray) :: ESMF_LocalArrayCreateCopyARGUMENTS:
type(ESMF_LocalArray), intent(in) :: larray
integer, intent(out), optional :: rc
DESCRIPTION:
Perform a deep copy of an existing ESMF_LocalArray object. The return value is a new LocalArray.
The arguments are:
INTERFACE:
! Private name; call using ESMF_LocalArrayCreate() function ESMF_LocalArrCreateByPtr<rank><type><kind>(fptr, docopy, counts, & lbounds, ubounds, rc)RETURN VALUE:
type(ESMF_LocalArray) :: ESMF_LocalArrCreateByPtr<rank><type><kind>ARGUMENTS:
<type> (ESMF_KIND_<kind>), dimension(<rank>), pointer :: fptr type(ESMF_CopyFlag), intent(in), optional :: docopy integer, intent(in), optional :: counts(:) integer, intent(in), optional :: lbounds(:) integer, intent(in), optional :: ubounds(:) integer, intent(out), optional :: rcDESCRIPTION:
Creates an ESMF_LocalArray based on a Fortran array pointer. Two cases must be distinguished.
First, if fptr is associated the optional docopy argument may be used to indicate whether the associated data is to be copied or referenced. For associated fptr the optional counts, lbounds and ubounds arguments need not be specified. However, all present arguments will be checked against fptr for consistency.
Second, if fptr is unassociated the optional argument docopy must not be specified. However, in this case a complete set of counts and bounds information must be provided. Any combination of present counts lbounds and ubounds arguments that provides a complete specification is valid. All input information will be checked for consistency.
The arguments are:
INTERFACE:
subroutine ESMF_LocalArrayDestroy(larray, rc)ARGUMENTS:
type(ESMF_LocalArray), intent(inout) :: larray
integer, intent(out), optional :: rc
DESCRIPTION:
Releases all resources associated with this ESMF_LocalArray object.
The arguments are:
INTERFACE:
! Private name; call using ESMF_LocalArrayGet()
subroutine ESMF_LocalArrayGetDefault(larray, rank, typekind, counts, lbounds, &
ubounds, base, name, rc)
ARGUMENTS:
type(ESMF_LocalArray), intent(in) :: larray
integer, intent(out), optional :: rank
type(ESMF_TypeKind), intent(out), optional :: typekind
integer, intent(out), optional :: counts(:)
integer, intent(out), optional :: lbounds(:)
integer, intent(out), optional :: ubounds(:)
type(ESMF_Pointer), intent(out), optional :: base
character(len=ESMF_MAXSTR), intent(out), optional :: name
integer, intent(out), optional :: rc
DESCRIPTION:
Returns information about the ESMF_LocalArray.
The arguments are:
INTERFACE:
! Private name; call using ESMF_LocalArrayGet() subroutine ESMF_LocalArrayGetData<rank><type><kind>(larray, fptr, docopy, rc)ARGUMENTS:
type(ESMF_LocalArray) :: larray <type> (ESMF_KIND_<kind>), dimension(<rank>), pointer :: fptr type(ESMF_CopyFlag), intent(in), optional :: docopy integer, intent(out), optional :: rcDESCRIPTION:
Return a Fortran pointer to the data buffer, or return a Fortran pointer to a new copy of the data.
The arguments are:
An ArraySpec is a very simple class that contains type, kind, and rank information about an array. This information is stored in two parameters. TypeKind describes the data type of the elements in the array and their precision. Rank is the number of dimensions in the array.
The only methods that are associated with the ArraySpec class are those that allow you to set and retrieve this information.
The ArraySpec is passed in as an argument at Field and Bundle creation in order to describe an Array that will be allocated or attached at a later time. There are any number of situations in which this approach is useful. One common example is a case in which the user wants to create a very flexible export State with many diagnostic variables predefined, but only a subset desired and consequently allocated for a particular run.
! !PROGRAM: ESMF_ArraySpecEx - ArraySpec manipulation examples
!
! !DESCRIPTION:
!
! This program shows examples of ArraySpec set and get usage
!-----------------------------------------------------------------------------
! ESMF Framework module
use ESMF_Mod
implicit none
! local variables
type(ESMF_ArraySpec) :: arrayDS
integer :: myrank
type(ESMF_TypeKind) :: mytypekind
! return code
integer:: rc
! initialize ESMF framework
call ESMF_Initialize(rc=rc)
This example shows how to set values in an ESMF_ArraySpec.
call ESMF_ArraySpecSet(arrayDS, rank=2, &
typekind=ESMF_TYPEKIND_R8, rc=rc)
This example shows how to query an ESMF_ArraySpec.
call ESMF_ArraySpecGet(arrayDS, myrank, mytypekind, rc)
print *, "Returned values from ArraySpec:"
print *, "rank =", myrank
! finalize ESMF framework
call ESMF_Finalize(rc=rc)
end program ESMF_ArraySpecEx
The information contained in an ESMF_ArraySpec is used to create ESMF_Array objects.
ESMF_ArraySpec is a shallow class, and only set and get methods are needed. They do not need to be created or destroyed.
INTERFACE:
subroutine ESMF_ArraySpecGet(arrayspec, rank, typekind, rc)ARGUMENTS:
type(ESMF_ArraySpec), intent(inout) :: arrayspec
integer, intent(out), optional :: rank
type(ESMF_TypeKind), intent(out), optional :: typekind
integer, intent(out), optional :: rc
DESCRIPTION:
Returns information about the contents of an ESMF_ArraySpec.
The arguments are:
INTERFACE:
subroutine ESMF_ArraySpecSet(arrayspec, rank, typekind, rc)ARGUMENTS:
type(ESMF_ArraySpec), intent(inout) :: arrayspec
integer, intent(in) :: rank
type(ESMF_TypeKind), intent(in) :: typekind
integer, intent(out), optional :: rc
DESCRIPTION:
Creates a description of the data - the typekind, the rank, and the dimensionality.
The arguments are:
INTERFACE:
subroutine ESMF_ArraySpecValidate(arrayspec, rc)ARGUMENTS:
type(ESMF_ArraySpec), intent(inout) :: arrayspec
integer, intent(out), optional :: rc
DESCRIPTION:
Validates that the arrayspec is internally consistent. The method returns an error code if problems are found.
The arguments are:
INTERFACE:
subroutine ESMF_ArraySpecPrint(arrayspec, rc)ARGUMENTS:
type(ESMF_ArraySpec), intent(in) :: arrayspec
integer, intent(out), optional :: rc
DESCRIPTION:
Print ArraySpec internals.
The arguments are:
The ESMF Grid class is used to describe the geometry and discretization of logically rectangular physical grids. It also contains the description of the grid's underlying topology and the decomposition of the physical grid across the available computational resources. The most frequent use of the Grid class is to describe physical grids in user code so that sufficient information is available to perform ESMF methods such as regridding.
In the current release (v3.1.0) the functionality in this class is partially implemented. Multi-tile grids are not supported, and edge connectivities are not implemented and default to aperiodic. Both regular and irregular distributions can be defined, but arbitrary distributions have not yet been implemented. Other constraints of the current implementation are noted in the usage section and in the API descriptions.
|
Key Features |
| Representation of grids formed by logically rectangular regions, including uniform and rectilinear grids (e.g. lat-lon grids), curvilinear grids (e.g. displaced pole grids), and grids formed by connected logically rectangular regions (e.g. cubed sphere grids) [CONNECTED REGIONS ARE NOT YET SUPPORTED]. |
| Support for 1D, 2D, 3D, and higher dimension grids. |
| Distribution of grids across computational resources for parallel operations - users set which grid dimensions are distributed. |
| Grids can be created already distributed, so that no single resource needs global information during the creation process. |
| Options to define periodicity and other edge connectivities either explicitly or implicitly via shape shortcuts [EDGE CONNECTIVITIES CURRENTLY DEFAULT TO APERIODIC BOUNDS]. |
| Options for users to define grid coordinates themselves or call prefabricated coordinate generation routines for standard grids [NO GENERATION ROUTINES YET]. |
| Options for incremental construction of grids. |
| Options for using a set of pre-defined stagger locations or for setting custom stagger locations. |
ESMF Grids are based on the concepts described in A Standard Description of Grids Used in Earth System Models [Balaji 2006]. In this document Balaji introduces the mosaic concept as a means of describing a wide variety of Earth system model grids. A mosaic is composed of grid tiles connected at their edges. Mosaic grids includes simple, single tile grids as a special case.
The ESMF Grid class is a representation of a mosaic grid. Each ESMF Grid is constructed of one or more logically rectangular Tiles. A Tile will usually have some physical significance (e.g. the region of the world covered by one face of a cubed sphere grid).
The piece of a Tile that resides on one DE (for simple cases, a DE can be thought of as a processor - see section on the DELayout) is called a LocalTile. For example, the six faces of a cubed sphere grid are each Tiles, and each Tile can be divided into many LocalTiles.
Every ESMF Grid contains a DistGrid object, which defines the Grid's index space, topology, distribution, and connectivities. It enables the user to define the complex edge relationships of tripole and other grids. The DistGrid can be created explicitly and passed into a Grid creation routine, or it can be created implicitly if the user takes a Grid creation shortcut. Options for grid creation are described in more detail in section 22.1.8.
The range of supported grids in ESMF can be defined by:
The table below shows the ESMF_GridConn settings used to create standard shapes in 2D using the ESMF_GridCreateShapeTile() call. Two values are specified for each dimension, one for the low end and one for the high end of the dimension's index values. Note that connectivities have not been implemented as of v3.1.0 and default to aperiodic bounds.
| 2D Shape | connDim1(1) | connDim1(2) | connDim2(1) | connDim2(2) |
| Rectangle | NONE | NONE | NONE | NONE |
| Bipole Sphere | POLE | POLE | PERIODIC | PERIODIC |
| Tripole Sphere | POLE | BIPOLE | PERIODIC | PERIODIC |
| Cylinder | NONE | NONE | PERIODIC | PERIODIC |
| Torus | PERIODIC | PERIODIC | PERIODIC | PERIODIC |
If the user's grid shape is too complex for an ESMF shortcut routine, or involves more than three dimensions, a DistGrid can be created to specify the shape in detail. This DistGrid is then passed into a Grid create call.
The main distribution options are regular, irregular, and arbitrary. A regular distribution is one in which the same number of contiguous grid points are assigned to each DE in the distributed dimension. A irregular distribution is one in which unequal numbers of contiguous gridpoints are assigned to each DE in the distributed dimension. An arbitrary distribution is one in which any gridpoint can be assigned to any DE. Any of these distribution options can be applied to any of the grid shapes (i.e., rectangle) or types (i.e., rectilinear). Arbitrary distributions have not been implemented as of v3.1.0.
Figure 11 illustrates options for distribution.
 |
A distribution can also be specified using the DistGrid, by passing object into a Grid create call.
Any of these logically rectangular grid types can be combined through edge connections to form a mosaic. Cubed sphere and yin-yang grids are examples of mosaic grids. Note that as of v3.1.0 multi-tile grids have not yet been implemented.
 |
Each of these coordinate types can be set for each of the standard grid shapes described in section 22.1.3.
The table below shows how examples of common single Tile grids fall into this shape and coordinate taxonomy. Note that any of the grids in the table can have a regular or arbitrary distribution.
| Uniform | Rectilinear | Curvilinear | |
| Sphere | Global uniform lat-lon grid | Gaussian grid | Displaced pole grid |
| Rectangle | Regional uniform lat-lon grid | Gaussian grid section | Polar stereographic grid section |
There are two ways of specifying coordinates in ESMF. The first way is for the user to set the coordinates. The second way is to take a shortcut and have the framework generate the coordinates.
No ESMF generation routines are currently available.
See Section 22.2.6 for more description and examples of setting coordinates.
Staggering is a finite difference technique in which the values of different physical quantities are placed at different locations within a grid cell.
The ESMF Grid class supports a variety of stagger locations, including cell centers, corners, and edge centers. Combinations of these are sufficient to specify any of the Arakawa staggers. ESMF also supports staggering in 3D and higher dimensions. There are shortcuts for standard staggers, and interfaces through which users can create custom staggers.
As a default the ESMF Grid class provides symmetric staggering, so that cell centers are enclosed by cell perimeter (e.g. corner) stagger locations. This means the coordinate arrays for stagger locations other than the center will have an additional element of padding in order to enclose the cell center locations. However, to achieve other types of staggering, the user may alter or eliminate this padding by using the appropriate options when adding coordinates to a Grid.
For examples and a full description of the stagger interface see Section 22.2.6.
ESMF Grid objects must represent a wide range of grid types and use cases, some of them quite complex. As a result, multiple ways to build Grid objects are required. This section describes the stages to building Grids, the options for each stage, and typical calling sequences.
In ESMF there are two main stages to building Grids. The ESMF_GridStatus value stored within the Grid object reflects the stage the Grid has attained (see Section 22.5.2). These stages are:
The options for specifying the Grid shape are:
When creating the Grid shape and specifying the Grid coordinates, the user can either specify all required information at once, or can provide information incrementally. The call ESMF_GridCreateEmpty() builds a Grid object container that can be filled in with subsequent calls to ESMF_GridSet() methods. A Grid that is empty has the ESMF_GridStatus value ESMF_GRIDSTATUS_NOT_READY.
The following table summarizes possible call sequences for building Grids.
| Create Shape |
| From shape shortcut |
| grid = ESMF_GridCreateShapeTile(...) |
| Using DistGrid with general create interface |
| distgrid = ESMF_DistGridCreate(...) |
| grid = ESMF_GridCreate(distgrid, ...) |
| Incremental |
| grid = ESMF_GridCreateEmpty(...) |
| call ESMF_GridSetCommitShapeTile(grid, ...) |
| Set Coordinates |
| Set coordinates by copy or reference |
| call ESMF_GridSetCoord(grid, ...) |
| Retrieve ESMF Array of coordinates from Grid and set values |
| call ESMF_GridGetCoord(grid, esmfArray, ...), set values |
| Retrieve local bounds and native array from Grid and set values |
| call ESMF_GridGetCoord(grid, lbound, ubound, array), set values |
This section describes the use of the ESMF Grid class. It first discusses the more user friendly shape specific interface to the Grid. During this discussion it covers creation and options, adding stagger locations, coordinate data access, and other grid functionality. After this initial phase the document discusses the more advanced options which the user can employ should they need more customized interaction with the Grid class.
The method ESMF_GridCreateShapeTile() is a shortcut for building single tile logically rectangular Grids up to three dimensions. It is partially implemented. The user can specify Grid size, rank and distribution, but cannot specify tile edge connectivities yet. The default is that Grid edges are not connected. Once completed, this method will enable users to create many common grid shapes, including rectangle, bipole sphere, and tripole sphere.
The ESMF_GridCreateShapeTile() method will eventually support the three types of distributions described in Section 22.1.4. It currently supports two of these types: regular and irregular.
To create a Grid with a regular distribution the user specifies the global maximum and minimum ranges of the Grid cell index space (maxIndex and minIndex), and the number of pieces in which to partition each dimension (via a regDecomp argument). ESMF then divides the index space as evenly as possible into the specified number of pieces. If there are cells left over then they are distributed one per DE starting from the first DE until they are gone.
If minIndex is not specified, then the bottom of the Grid cell index range is assumed to be (1,1,...,1). If regDecomp is not specified, then by default ESMF creates a distribution that partitions the grid cells in the first dimension (e.g. NPx1x1...1) as evenly as possible by the number of processors NP. The remaining dimensions are not partitioned. The rank of the Grid is the size of maxIndex. To create an undistributed dimension set that entry in regDecomp to 1, and set the corresponding entry in distDim to .false.. The following is an example of creating a 10x20x30 3D grid where the first dimensions is broken into 2 pieces, the second is broken into 4 pieces, and the third is undistributed.
grid3D=ESMF_GridCreateShapeTile(regDecomp=(/2,4,1/), maxIndex=(/10,20,30/), &
distDim=(/.true.,.true.,.false./), rc=rc)
Irregular distribution requires the user to specify the exact number of Grid cells per DE in each dimension. In the ESMF_GridCreateShapeTile() call the countsPerDEDim1, countsPerDim2, and countsPerDim3 arguments are used to specify a rectangular distribution containing size(countsPerDEDim1) by size(countsPerDEDim2) by size(countsPerDEDim3) DEs. The entries in each of these arrays specify the number of grid cells per DE in that dimension. The rank of the grid is determined by the presence of countsPerDEDim3. If it's present the Grid will be 3D. If just countsPerDEDim1 and countsPerDEDim2 are specified the Grid will be 2D.
The following call illustrates the creation of a 10x20 two dimensional rectangular Grid distributed across six DEs that are arranged 2x3. In the first dimension there are 3 grid cells on the first DE and 7 cells on the second DE. The second dimension has 3 DEs with 11,2, and 7 cells, respectively.
grid2D=ESMF_GridCreateShapeTile(countsPerDEDim1=(/3,7/), &
countsPerDEDim2=(/11,2,7/), rc=rc)
To add a distributed third dimension of size 30, broken up into two groups of 15, the above call would be altered as follows.
grid3d=ESMF_GridCreateShapeTile(countsPerDEDim1=(/3,7/), &
countsPerDEDim2=(/11,2,7/), countsPerDEDim3=(/15,15/), rc=rc)
Alternatively, if the third dimension were to be undistributed, then countsPerDEDim3 in the call would have only a single term.
grid3D=ESMF_GridCreateShapeTile(countsPerDEDim1=(/3,7/), &
countsPerDEDim2=(/11,2,7/), countsPerDEDim3=(/30/), rc=rc)
The petMap parameter may be used to specify on to which specific PETs the DEs in the Grid are assigned. Note that this parameter is only available for the regular and irregular distribution types. The petMap array is a 3D array, for a 3D Grid each of its dimensions correspond to a Grid dimension. If the Grid is 2D, then the first two dimensions correspond to Grid dimensions and the last dimension should be of size 1. The size of each petMap dimension is the number of DE's along that dimension in the Grid. For a regular Grid, the size is equal to the number in regDecomp (i.e. size(petMap,d)=regDecomp(d) for all dimensions d in the Grid). For an irregular Grid the size is equal to the number of items in the corresponding countsPerDEDim variable (i.e. size(petMap,d)=size(countsPerDEDimd) for all dimensions d in the Grid).
Each entry in petMap specifies to which PET the corresponding DE should be assigned. For example, petMap(3,2)=4 tells the Grid create call to put the DE located at column 3 row 2 on PET 4.
The following example demonstrates how to specify the PET to DE association for an ESMF_GridCreateShapeTile() call.
! allocate memory for petMap
allocate( petMap(2,2,1) )
! Set petMap
petMap(:,1,1) = (/3,2/) ! DE (1,1,1) on PET 3 and DE (2,1,1) on PET 2
petMap(:,2,1) = (/1,0/) ! DE (1,2,1) on PET 1 and DE (2,2,1) on PET 0
! Let the 3D grid be be distributed only in the first two dimensions.
grid2D=ESMF_GridCreateShapeTile(countsPerDEDim1=(/3,7/), &
countsPerDEDim2=(/7,6/), petMap=petMap, rc=rc)
Arbitrary distribution has not yet been implemented. The design is that the user specifies the global minimum and maximum ranges of the index space with the arguments minIndex and maxIndex, and through a localIndices argument specifies the set of index space locations residing on the local DE. Again, if minIndex is not specified, then the bottom of the index range is assumed to be (1,1,...). The rank of the Grid is equal to the size of maxIndex.
The following example creates a 2D Grid of dimensions 5x5, and places the diagonal elements (i.e. indices (i,i) where i goes from 1 to 5) on the local DE. The remaining DEs would individually declare the remainder of the Grid locations.
! allocate memory for local allocate( localIndices(2,5) ) ! Set local indices localIndices(:,1)=(/1,1/) localIndices(:,2)=(/2,2/) localIndices(:,3)=(/3,3/) localIndices(:,4)=(/4,4/) localIndices(:,5)=(/5,5/) ! Create Grid ! grid2D=ESMF_GridCreateShapeTile(maxIndex=(/5,5/), & ! localIndices=localIndices, rc=rc)
The following is an example of creating a simple rectilinear grid and loading in a set of coordinates. It illustrates a straightforward use of the ESMF_GridCreateShapeTile() call described in the previous section. This code creates a 10x20 2D grid with uniformly spaced coordinates varying from (10,10) to (100,200). The grid is partitioned using an irregular distribution. The first dimension it is divided into two pieces, the first with 3 grid cells per DE and the second with 7 grid cells per DE. In the second dimension, the Grid is divided into 3 pieces, with 5, 9, and 6 cells per DE respectively. The Grid is created with global indices. After grid creation the local bounds and native Fortran arrays are retrieved and the coordinates are set by the user.
!-------------------------------------------------------------------
! Create the Grid: Allocate space for the Grid object, define the
! topology and distribution of the grid, and specify that it
! will have global coordinates. Note that aperiodic bounds are
! specified by default - if periodic bounds were desired they
! would need to be specified using an additional gridConn argument
! (which isn't implemented yet).
!-------------------------------------------------------------------
grid2D=ESMF_GridCreateShapeTile( &
! Define an irregular distribution
countsPerDEDim1=(/3,7/), &
countsPerDEDim2=(/11,2,7/), &
! Specify mapping of coords dim to Grid dim
coordDep1=(/1/), &
coordDep2=(/2/), &
indexflag=ESMF_INDEX_GLOBAL, & ! Use global indices
rc=rc)
!-------------------------------------------------------------------
! Allocate coordinate storage and associate it with the center
! stagger location. Since no coordinate values are specified in
! this call no coordinate values are set yet.
!-------------------------------------------------------------------
call ESMF_GridAllocCoord(grid2D, &
staggerloc=ESMF_STAGGERLOC_CENTER, rc=rc)
!-------------------------------------------------------------------
! Get the pointer to the first coordinate array and the bounds
! of its global indices on the local DE.
!-------------------------------------------------------------------
call ESMF_GridGetCoord(grid2D, coordDim=1, &
staggerloc=ESMF_STAGGERLOC_CENTER, &
computationalLBound=lbnd, computationalUBound=ubnd, fptr=coordX, rc=rc)
!-------------------------------------------------------------------
! Calculate and set coordinates in the first dimension [10-100].
!-------------------------------------------------------------------
do i=lbnd(1),ubnd(1)
coordX(i) = i*10.0
enddo
!-------------------------------------------------------------------
! Get the pointer to the second coordinate array and the bounds of
! its global indices on the local DE.
!-------------------------------------------------------------------
call ESMF_GridGetCoord(grid2D, coordDim=2, &
staggerloc=ESMF_STAGGERLOC_CENTER, &
computationalLBound=lbnd, computationalUBound=ubnd, fptr=coordY, rc=rc)
!-------------------------------------------------------------------
! Calculate and set coordinates in the second dimension [10-200]
!-------------------------------------------------------------------
do j=lbnd(1),ubnd(1)
coordY(j) = j*10.0
enddo
The following is an example of creating a simple curvilinear grid and loading in a set of coordinates. It creates a 10x20 2D grid where the coordinates vary along every dimension. The grid is partitioned using an irregular distribution. The first dimension is divided into two pieces, the first with 3 grid cells per DE and the second with 7 grid cells per DE. In the second dimension, the Grid is divided into 3 pieces, with 5, 9, and 6 cells per DE respectively. The Grid is created with global indices. After grid creation the local bounds and native Fortran arrays are retrieved and the coordinates are set by the user.
!-------------------------------------------------------------------
! Create the Grid: Allocate space for the Grid object, define the
! distribution of the grid, and specify that it
! will have global coordinates. Note that aperiodic bounds are
! specified by default - if periodic bounds were desired they
! would need to be specified using an additional gridConn argument
! (which isn't implemented yet).
!-------------------------------------------------------------------
grid2D=ESMF_GridCreateShapeTile( &
! Define an irregular distribution
countsPerDEDim1=(/3,7/), &
countsPerDEDim2=(/11,2,7/), &
indexflag=ESMF_INDEX_GLOBAL, & ! Use global indices
rc=rc)
!-------------------------------------------------------------------
! Allocate coordinate storage and associate it with the center
! stagger location. Since no coordinate values are specified in
! this call no coordinate values are set yet.
!-------------------------------------------------------------------
call ESMF_GridAllocCoord(grid2D, &
staggerloc=ESMF_STAGGERLOC_CENTER, rc=rc)
!-------------------------------------------------------------------
! Get the pointer to the first coordinate array and the bounds
! of its global indices on the local DE.
!-------------------------------------------------------------------
call ESMF_GridGetCoord(grid2D, coordDim=1, &
staggerloc=ESMF_STAGGERLOC_CENTER, &
computationalLBound=lbnd, computationalUBound=ubnd, fptr=coordX2D, rc=rc)
!-------------------------------------------------------------------
! Calculate and set coordinates in the first dimension [10-100].
!-------------------------------------------------------------------
do j=lbnd(2),ubnd(2)
do i=lbnd(1),ubnd(1)
coordX2D(i,j) = i+j
enddo
enddo
!-------------------------------------------------------------------
! Get the pointer to the second coordinate array and the bounds of
! its global indices on the local DE.
!-------------------------------------------------------------------
call ESMF_GridGetCoord(grid2D, coordDim=2, &
staggerloc=ESMF_STAGGERLOC_CENTER, &
computationalLBound=lbnd, computationalUBound=ubnd, fptr=coordY2D, rc=rc)
!-------------------------------------------------------------------
! Calculate and set coordinates in the second dimension [10-200]
!-------------------------------------------------------------------
do j=lbnd(2),ubnd(2)
do i=lbnd(1),ubnd(1)
coordY2D(i,j) = j-i/100.0
enddo
enddo
This example demonstrates how a user can build a rectilinear horizontal grid with an undistributed vertical dimension. The Grid contains both the center and corner stagger locations (i.e. Arakawa B-Grid).
!-------------------------------------------------------------------
! Create the Grid: Allocate space for the Grid object. The
! grid is defined to be 180 elements in Dim1 (longitude), 90 in
! Dim2 (longitude), and 40 in Dim3 (depth). The first dimension is
! decomposed over 4 DEs, the second over 3 DEs, and the third is
! undistributed. The connectivities in each dimension default
! to aperiodic since they are not yet implemented. (Once connectivities
! are implemented, the longitude dimension will be periodic, the
! poles will be defined at each end of the latitude dimension,
! and the depth dimension will remain aperiodic.
!-------------------------------------------------------------------
grid3D=ESMF_GridCreateShapeTile( &
countsPerDEDim1=(/45,75,40,20/), &
countsPerDEDim2=(/30,40,20/), &
countsPerDEDim3=(/40/), &
coordDep1=(/1/), &
coordDep2=(/2/), &
coordDep3=(/3/), &
distDim=(/.true.,.true.,.false./), &
indexflag=ESMF_INDEX_GLOBAL, & ! Use global indices
rc=rc)
!-------------------------------------------------------------------
! Allocate coordinate storage for both center and corner stagger
! locations. Since no coordinate values are specified in this
! call no coordinate values are set yet.
!-------------------------------------------------------------------
call ESMF_GridAllocCoord(grid3D, &
staggerloc=ESMF_STAGGERLOC_CENTER_VCENTER, rc=rc)
call ESMF_GridAllocCoord(grid3D, &
staggerloc=ESMF_STAGGERLOC_CORNER_VCENTER, rc=rc)
!----------------------------------------------------------------------
! Fill in the coordinates for the corner stagger location first.
!----------------------------------------------------------------------
!-------------------------------------------------------------------
! Get the local bounds of the global indexing for the first
! coordinate array on the local DE. If the number of processors
! is less than the total number of DEs then the rest of this
! example would be in a loop over the local DEs. Also get the
! pointer to the first coordinate array.
!-------------------------------------------------------------------
call ESMF_GridGetCoord(grid3D, coordDim=1, &
staggerLoc=ESMF_STAGGERLOC_CORNER_VCENTER, &
computationalLBound=lbnd_corner, &
computationalUBound=ubnd_corner, &
fptr=cornerX, rc=rc)
!-------------------------------------------------------------------
! Calculate and set coordinates in the first dimension.
!-------------------------------------------------------------------
do i=lbnd_corner(1),ubnd_corner(1)
cornerX(i) = (i-1)*(360.0/180.0)
enddo
!-------------------------------------------------------------------
! Get the local bounds of the global indexing for the second
! coordinate array on the local DE. Also get the pointer to the
! second coordinate array.
!-------------------------------------------------------------------
call ESMF_GridGetCoord(grid3D, coordDim=2, &
staggerLoc=ESMF_STAGGERLOC_CORNER_VCENTER, &
computationalLBound=lbnd_corner, &
computationalUBound=ubnd_corner, &
fptr=cornerY, rc=rc)
!-------------------------------------------------------------------
! Calculate and set coordinates in the second dimension.
!-------------------------------------------------------------------
do j=lbnd_corner(1),ubnd_corner(1)
cornerY(j) = (j-1)*(180.0/90.0)
enddo
!----------------------------------------------------------------------
! Now fill the coordinates for the center stagger location with
! the average of the corner coordinate location values.
!----------------------------------------------------------------------
!-------------------------------------------------------------------
! Get the local bounds of the global indexing for the first
! coordinate array on the local DE, and the pointer to the array.
!-------------------------------------------------------------------
call ESMF_GridGetCoord(grid3D, coordDim=1, &
staggerloc=ESMF_STAGGERLOC_CENTER_VCENTER, &
computationalLBound=lbnd, computationalUBound=ubnd, fptr=centerX, rc=rc)
!-------------------------------------------------------------------
! Calculate and set coordinates in the first dimension.
!-------------------------------------------------------------------
do i=lbnd(1),ubnd(1)
centerX(i) = 0.5*(i-1 + i)*(360.0/180.0)
enddo
!-------------------------------------------------------------------
! Get the local bounds of the global indexing for the second
! coordinate array on the local DE, and the pointer to the array.
!-------------------------------------------------------------------
call ESMF_GridGetCoord(grid3D, coordDim=2, &
staggerloc=ESMF_STAGGERLOC_CENTER_VCENTER, &
computationalLBound=lbnd, computationalUBound=ubnd, fptr=centerY, rc=rc)
!-------------------------------------------------------------------
! Calculate and set coordinates in the second dimension.
!-------------------------------------------------------------------
do j=lbnd(1),ubnd(1)
centerY(j) = 0.5*(j-1 + j)*(180.0/90.0)
enddo
!-------------------------------------------------------------------
! Get the local bounds of the global indexing for the third
! coordinate array on the local DE, and the pointer to the array.
!-------------------------------------------------------------------
call ESMF_GridGetCoord(grid3D, coordDim=3, &
staggerloc=ESMF_STAGGERLOC_CENTER_VCENTER, &
computationalLBound=lbnd, computationalUBound=ubnd, fptr=centerZ, rc=rc)
!-------------------------------------------------------------------
! Calculate and set the vertical coordinates
! Note that the centerZ array needs to be 3D because currently
! all coordinate arrays must each have the same rank as the grid, but
! because the coordinate is rectilinear we only need to store a 1D array.
!-------------------------------------------------------------------
do k=lbnd(1),ubnd(1)
centerZ(k) = 4000.0*( (1./39.)*(k-1) )**2
enddo
ESMF Grids can be created using an incremental paradigm. To do this, the user first calls ESMF_GridCreateEmpty to allocate the shell of a Grid. Next, a series of ESMF_GridSet() calls are used to fill in the details of the grid. Here we use a convenient ESMF_GridSetCommitShapeTile() call that fills in the Grid via an interface much like the ESMF_GridCreateShapeTile() call. When using ESMF_GridSet the user must explicity call ESMF_GridCommit afterwards to make the Grid usable. ESMF_GridSetCommitShapeTile() contains this commit internally, so it doesn't need to done separately. For consistency's sake the initial ESMF_GridCreateEmpty call must occur on the same or a superset of the processors as the ESMF_GridSet() calls. The following example uses the incremental technique to create a rectangular 10x20 grid with coordinates at the center and corner stagger locations.
!---------------------------------------------------------------------------
! IN THE PARENT COMPONENT:
! Create an empty grid in the parent component for use in a child component.
! The parent may be defined on more PETs than the child component.
! The child's [vm or pet list] is passed into the create call so that
! the Grid is defined on the appropriate subset of the parent's PETs.
!---------------------------------------------------------------------------
grid2D=ESMF_GridCreateEmpty(rc=rc)
!---------------------------------------------------------------------------
! IN THE CHILD COMPONENT:
! Set the grid topology. Here we define an irregularly distributed
! rectangular Grid.
!---------------------------------------------------------------------------
call ESMF_GridSetCommitShapeTile(grid2D, &
countsPerDEDim1=(/6,4/), &
countsPerDEDim2=(/10,3,7/), rc=rc)
!---------------------------------------------------------------------------
! Set Grid coordinates at the cell center location.
!---------------------------------------------------------------------------
call ESMF_GridAllocCoord(grid2D, staggerLoc=ESMF_STAGGERLOC_CENTER, rc=rc)
!---------------------------------------------------------------------------
! Set Grid coordinates at the corner stagger location.
!---------------------------------------------------------------------------
call ESMF_GridAllocCoord(grid2D, staggerLoc=ESMF_STAGGERLOC_CORNER, rc=rc)
A useful finite difference technique is to place different physical quantities at different locations within a grid cell. This staggering of the physical variables on the mesh is introduced so that the difference of a field is naturally defined at the location of another variable. This method was first formalized by Mesinger and Arakawa (1976).
To support the staggering of variables, the Grid provides the idea of stagger locations. Stagger locations refer to the places in a Grid cell that contain coordinates and, once a Grid is associated with a Field object, field data. Typically data can be located at the cell center, at the cell corners, or at the cell faces, in 2D, 3D, and higher dimensions. (Note that any Arakawa stagger can be constructed of a set of Grid stagger locations.) Users can put coordinates, which are necessary for operations such as regrid, at multiple stagger locations in a Grid. In addition, the user can put Field data at any of the stagger locations in a Grid.
By default the coordinate array at the center stagger location starts at the bottom index of the Grid (default (1,1..,1)) and extends up to the maximum cell index in the Grid (e.g. given by the maxIndex argument). Other stagger locations also start at the bottom index of the Grid, however, they can extend to +1 element beyond the center in some dimensions to allow for the extra space to surround the center elements. See Section 22.2.15 for a description of this extra space and how to adjust if it necessary. The subroutine ESMF_GridGetCoord can be used to retrieve the stagger bounds for the piece of a coordinate array on a particular DE.
The user can allocate coordinate arrays without setting coordinates using the ESMF_AllocCoord() call, or allocate and set coordinates in a single call with ESMF_SetCoord(). When adding or accessing coordinate data, the stagger location is specified to tell the Grid method where in the cell to get the data. There are predefined stagger locations (see Section 22.5.3), or, should the user wish to specify their own, there is also a set of methods for generating custom locations (See Section 22.2.15).
The following example adds coordinate storage to the corner stagger location in a grid using one of the predefined stagger locations.
call ESMF_GridAllocCoord(grid2D, staggerLoc=ESMF_STAGGERLOC_CORNER, rc=rc)
To specify how the coordinate arrays are mapped to the index dimensions the arguments coordDep1, coordDep2, and coordDep3 are used, each of which is a Fortran array. The values of the elements in a coordDep array specify which index dimension the corresponding coordinate dimension maps to. For example, coordDep1=(/1,2/) means that the first dimension of coordinate 1 maps to index dimension 1 and the second maps to index dimension 2. If the coordDep arrays are not specified, then coordDep1 defaults to /1,2..,gridrank/. This default thus specifies a curvilinear grid.
The following call demonstrates the creation of a 10x20 2D rectilinear grid where the first coordinate component is mapped to the second index dimension (i.e. is of size 20) and the second coordinate component is mapped to the first index dimension (i.e. is of size 10).
grid2D=ESMF_GridCreateShapeTile(countsPerDEDim1=(/5,5/), &
countsPerDEDim2=(/7,7,6/), &
coordDep1=(/2/), &
coordDep2=(/1/), rc=rc)
The following call demonstrates the creation of a 10x20x30 2D plus 1 curvilinear grid where coordinate component 1 and 2 are still 10x20, but coordinate component 3 is mapped just to the third index dimension.
grid2D=ESMF_GridCreateShapeTile(countsPerDEDim1=(/6,4/), &
countsPerDEDim2=(/10,7,3/), countsPerDEDim3=(/30/), &
coordDep1=(/1,2/), coordDep2=(/1,2/), &
coordDep3=(/3/), rc=rc)
By default the local piece of the array on each processor starts at (1,1,..), however, the indexing for each grid coordinate array on each DE may be shifted to the global indices by using the indexflag. For example, the following call switches the grid to use global indices.
grid2D=ESMF_GridCreateShapeTile(countsPerDEDim1=(/6,4/), &
countsPerDEDim2=(/10,7,3/), indexflag=ESMF_INDEX_GLOBAL, rc=rc)
Once a Grid has been created, the user has several options to access the Grid coordinate data. The first of these, ESMF_GridSetCoord(), enables the user to set data for one stagger location across the whole Grid, using ESMF Arrays. For example, the following sets the coordinates in the first dimension (e.g. x) for the center stagger location to those in the ESMF Array arrayCoordX.
call ESMF_GridSetCoord(grid2D, &
staggerLoc=ESMF_STAGGERLOC_CORNER, &
coordDim=1, array=arrayCoordX, rc=rc)
The method ESMF_GridGetCoord() allows the user to access the Array, as a direct reference, which contains the coordinate data for a stagger location on a Grid. The user can then employ any of the standard ESMF_Array tools to operate on the data. The following copies the coordinates from the second component of the corner and puts it into the ESMF Array arrayCoordY.
call ESMF_GridGetCoord(grid2D, &
staggerLoc=ESMF_STAGGERLOC_CORNER, &
coordDim=2, &
array=arrayCoordY, rc=rc)
Alternatively, the call ESMF_GridGetCoord() gets a Fortran pointer to the coordinate data. The user can then operate on this array in the usual manner. The following call gets a reference to the Fortran array which holds the data for the second coordinate (e.g. y).
call ESMF_GridGetCoord(grid2D, coordDim=2, &
staggerloc=ESMF_STAGGERLOC_CORNER, fptr=coordY2D, rc=rc)
The index space of the Grid contains three regions with associated bounds. This section describes these regions and bounds in general. The next couple of sections describe getting the bound information for specific types of Grid information.
The exclusive region is the index space defined by the distgrid, undistLBound, and undistUBound of the Grid. These correspond to the maximum index space usable by any stagger location in the Grid. The size of the exclusive region is the index space for the Grid cells, plus the stagger padding.
The default stagger padding depends on the topology of the Grid. For an unconnected dimension the stagger padding is a width of 1 on the upper side (i.e. gridEdgeUWidth=(1,1,1,1...)). For a periodic dimension there is no stagger padding. By adjusting gridEdgeLWidth and gridEdgeUWidth, the user can set the stagger padding for the whole Grid and thus the exclusive region can be adjusted at will around the index space corresponding to the cells. The user can also use staggerEdgeLWidth and staggerEdgeUWidth to adjust individual stagger location padding within the Grid's padding (Please see Section 22.2.16 for further discussion of customizing the stagger padding).
The Grid computational region is a subset of the the Grid exclusive region. The computational region indicates which index locations in the Grid are active for a particular stagger. The computational region is typically the region that a user would compute over (e.g. would iterate over setting values for).
The total region is the outermost boundary of the memory allocated on each DE to hold the data for the Grid on that DE. This region can be as small as the computational region, but may be larger to include space for halos, memory padding, etc. The total region is what is enlarged to include space for halos, and the total region must be large enough to contain the maximum halo operation on the Grid. The total region is a property of an underlying Array and so information about it is only retrievable when an Array has been set or allocated.
The user can retrieve a set of bounds for each index space region described above: exclusive bounds, computational bounds, and total bounds. Note that although some of these are similar to bounds provided by ESMF_Array subroutines (see Section 19.2.6) the format here is different. The Array bounds are only for distributed dimensions and are ordered to correspond to the dimension order in the associated DistGrid. The bounds provided by the Grid are for both distributed and undistributed dimensions and are ordered according to the order of dimensions of the data in question. This means that the bounds provided should be usable "as is" to access the data.
Each of the three types of bounds refers to the maximum and minimum per dimension of the index ranges of a particular region. The paramters referring to the maximums contain a 'U' for upper. The parameters referring to the minimums contain an 'L' for lower. The bounds and associated quantities are almost always given on a per DE basis. The three types of bounds exclusiveBounds, computationalBounds, and totalBounds refer to the ranges of the exlusive region, the computational region, and the total region. Each of these bounds also has a corresponding count parameter which gives the number of items across that region (on a DE) in each dimension. (e.g. totalCount(d)=totallUBound(i)-totalLBound(i)+1). Width parameters give the spacing between two different types of region. The computationalWidth argument gives the spacing between the exclusive region and the computational region. The totalWidth argument gives the spacing between the total region and the computational region. Like the other bound information these are typically on a per DE basis, for example specifying totalLWidth=(1,1) makes the bottom of the total region one lower in each dimension than the computational region on each DE. The exceptions to the per DE rule are computationalEdgeWidth, staggerEdgeWidth, and gridEdgeWidth which give the spacing only on the DEs along the boundary of the Grid.
When operating on coordinates the user may often wish to retrieve the bounds of the piece of coordinate data on a particular local DE. This is useful for iterating through the data to set coordinates, retrieve coordinates, or do calculations. The method ESMF_GridGetCoord allows the user to retrieve bound information for a particular coordinate array.
As described in the previous section there are three types of bounds the user can get: exclusive bounds, computational bounds, and total bounds. The exclusive and computational bounds may be retrieved from an unallocated stagger location, but the total bounds may only be retrieved from a stagger location which contains a valid coordinate Array. The bounds provided by ESMF_GridGetCoord are for both distributed and undistributed dimensions and are ordered according to the order of dimensions in the coordinate. This means that the bounds provided should be usable "as is" to access data in the coordinate array. In the case of factorized coordinate Arrays where a coordinate may have a smaller dimension than its associated Grid, then the dimension of the coordinate's bounds are the dimension of the coordinate, not the Grid.
The following is an example of retrieving the bounds for the first coordinate array from the corner stagger location.
call ESMF_GridGetCoord(grid2D, localDE=0, coordDim=1, &
staggerLoc=ESMF_STAGGERLOC_CORNER, &
exclusiveLBound=elbnd, exclusiveUBound=eubnd, &
computationalLBound=clbnd, computationalUBound=cubnd, &
totalLBound=tlbnd, totalUBound=tubnd, rc=rc)
When operating on data stored at a particular stagger in a Grid the user may find it useful to be able to retrieve the bounds of the data on a particular local DE. This is useful for iterating through the data for computations or allocating arrays to hold the data. The method ESMF_GridGet allows the user to retrieve bound information for a particular stagger location.
As described in Section 22.2.9 there are three types of bounds the user can typically get, however, the Grid doesn't hold data at a stagger location (that is the job of the Field), and so no Array is contained there and so no total region exists, so the user may only retrieve exclusive and computational bounds from a stagger location. The bounds provided by ESMF_GridGet are for both distributed and undistributed dimensions and are ordered according to the order of dimensions in the Grid.
The following is an example of retrieving the bounds for localDE 0 from the corner stagger location.
call ESMF_GridGet(grid2D, localDE=0, &
staggerLoc=ESMF_STAGGERLOC_CORNER, &
exclusiveLBound=elbnd, exclusiveUBound=eubnd, &
computationalLBound=clbnd, computationalUBound=cubnd, rc=rc)
In addition to the per DE information that can be accessed about a stagger location there is a set of global information that can accessed by using ESMF_GridGet without specifying a localDE. One of the main uses of this information is to create an ESMF Array to hold data for a stagger location.
There are several types of information currently available from a stagger location. The first set is computationalEdgeLWidth and computationalEdgeUWidth these give the difference between the lower and upper boundary of the exclusive region and the computational region. This information is just for the distributed dimensions and so is the same size as the distgrid dimension and is arranged to correspond to the distgrid dimensions.
The second set is undistLBound and undistUBound these are the bounds of the undistributed dimensions of the stagger locations. These will be of the same size as the undistributed dimensions of the Grid. The values of these bounds are the undistributed bounds of the Grid modified by the padding for this stagger location.
The following is an example of retrieving information from the corner stagger location.
! Get info about staggerloc
call ESMF_GridGet(grid3D, staggerLoc=ESMF_STAGGERLOC_CORNER, &
computationalEdgeLWidth=celwdth, computationalEdgeUWidth=ceuwdth, &
undistLBound=lbnd, undistUBound=ubnd, rc=rc)
In order to create an Array to correspond to a Grid stagger location several pieces of information need to be obtained from both the Grid and the stagger location in the Grid.
The information that needs to be obtained from the Grid is the distgrid and distgridToGridMap to ensure that the new Array has the correct size and distribution and that its dimensions are mapped correctly to the Grid. These are obtained using the ESMF_GridGet method.
The information that needs to be obtained from the stagger location are the distributed and undistributed sizes. The distributed sizes are give as offsets from the edges of the exclusive region of the distgrid. These may be obtained from ESMF_GridGet by passing in the stagger location and the arguments computationalEdgeLWidth and computationalEdgeUWidth. The undistributed sizes may may be obtained from the same call using the undistLBound and undistUBound arguments. Note that if the Grid doesn't contain undistributed dimensions, then this second set of bounds shouldn't be used.
The following is an example of using information from the Grid to create an Array corresponding to a stagger location. The Grid is 3D and contains both distributed and undistributed dimensions.
! Get info from Grid
call ESMF_GridGet(grid3D, distgrid=distgrid, distgridToGridMap=distgridToGridMap, rc=rc)
! Get info about staggerloc
call ESMF_GridGet(grid3D, staggerLoc=ESMF_STAGGERLOC_CORNER, &
computationalEdgeLWidth=celwdth, computationalEdgeUWidth=ceuwdth, &
undistLBound=lbnd1D, undistUBound=ubnd1D, rc=rc)
! construct ArraySpec
call ESMF_ArraySpecSet(arrayspec, rank=3, typekind=ESMF_TYPEKIND_R8, rc=rc)
! Create an Array based on the presence of distributed dimensions
array=ESMF_ArrayCreate(arrayspec=arrayspec, &
distgrid=distgrid, distgridToArrayMap=distgridToGridMap, &
computationalEdgeLWidth=celwdth, &
computationalEdgeUWidth=ceuwdth, &
undistLBound=lbnd1D, undistUBound=ubnd1D, rc=rc)
Besides the shortcut methods for creating a Grid object such as ESMF_GridCreateShapeTile(), there is a set of methods which give the user more control over the specifics of the grid. The following describes the more general interface, using DistGrid. The basic idea is to first create an ESMF DistGrid object describing the distribution and shape of the Grid, and then to employ that to either directly create the Grid or first create Arrays and then create the Grid from those. This method gives the user maximum control over the topology and distribution of the Grid. See the DistGrid documentation in Section 23.1 for an in-depth description of its interface and use.
A DistGrid describes only the distributed dimensions of the index space, so when creating a Grid from a DistGrid some arguments are needed to describe the undistributed part, To add undistributed dimensions to the Grid, the arguments undistLBound and undistUBound are used. The undistLBound argument contains the lower bounds of the undistributed dimensions and undistUBound contains the upper bounds. As an example, the following call constructs a 10x20x30 Grid with a lower bound of (1,2,3), with the third dimension undistributed.
! Create DistGrid
distgrid2D = ESMF_DistGridCreate(minIndex=(/1,2/), maxIndex=(/11,22/), rc=rc)
! Create Grid
grid3D=ESMF_GridCreate(distGrid=distgrid2D, &
undistLBound=(/3/), undistUBound=(/33/), &
rc=rc)
To alter which dimensions are distributed, the distgridToGridMap argument can be used. The distgridToGridMap is used to set which dimensions of the Grid are mapped to the dimensions described by maxIndex. In other words, it describes how the dimensions of the underlying default DistGrid are mapped to the Grid. Each entry in distgridToGridMap contains the Grid dimension to which the cooresponding DistGrid dimension should be mapped. The following example illustrates the creation of a Grid where the undistributed dimension is first. To accomplish this the two distributed dimensions are mapped to the last two Grid dimensions (i.e. 2 and 3).
! Create DistGrid
distgrid2D = ESMF_DistGridCreate(minIndex=(/1,2/), maxIndex=(/11,22/), rc=rc)
! Create Grid
grid3D=ESMF_GridCreate(distGrid=distgrid2D, undistLBound=(/3/), undistUBound=(/33/), &
distgridToGridMap=(/2,3/), rc=rc)
Although ESMF provides a set of predefined stagger locations (See Section 22.5.3), the user may need one outside this set. This section describes the construction of custom stagger locations.
To completely specify stagger for an arbitrary number of dimensions, we define the stagger location in terms of a set of cartesian coordinates. The cell is represented by a n-dimensional cube with sides of length 2, and the coordinate origin located at the center of the cell. The geometry of the cell is for reference purposes only, and does not literally represent the actual shape of the cell. Think of this method instead as an easy way to specify a part (e.g. center, corner, face) of a higher dimensional cell which is extensible to any number of dimensions.
To illustrate this approach, consider a 2D cell. In 2 dimensions
the cell is represented by a square. An xy axis is placed at its center, with the
positive x-axis oriented East and the positive y-axis oriented North.
The resulting coordinate for the lower left corner is at 
, and upper right
corner at 
.
However, because our staggers are symmetric they don't need to distinguish between
the 
, and the , so we only need concern ourselves with the first quadrant of
this cell. We only need to use the , and the 
, and many of the cell locations
collapse together (e.g. we only need to represent one corner).
(0,1)-EDGE2 (1,1)-CORNER
1 *-------*-------* 1 *---------------*
| | |
| | |
| | | |
0 * +- * |
| | |
| | | | EDGE1
| | | | |
-1 *-------*-------* 0 *---- * (1,0)
-1 0 1 0 1
Full Cell Just The 1st Quadrant
The cell center is represented by the coordinate pair 
indicating the origin.
The cell corner is 
in each direction, giving a coordinate pair of .
The edges are each in one dimension and in the other indicating that
they're even with the center in one dimension and offset in the other.
For three dimensions, the vertical component of the stagger location can be added by
simply adding an additional coordinate. The three dimensional generalization of the
cell center becomes 
and the cell corner becomes 
. The rest of
the 3D stagger locations are combinations of offsets from the center.
To generalize this to 
dimensions, to represent a dimensional stagger
location. A set of and is used to specify for each dimension
whether a stagger location is aligned with the cell center in that dimension (),
or offset by in that dimension (). Using this scheme we can represent
any symmetric stagger location.
To construct a custom stagger location in ESMF the subroutine ESMF_StaggerLocSet() is used to specify, for each dimension, whether the stagger is located at the interior (0) or on the boundary (1) of the cell. This method allows users to construct stagger locations for which there is no predefined value. In this example, it's used to set the 4D center and 4D corner locations.
! Set Center call ESMF_StaggerLocSet(staggerLoc,loc=(/0,0,0,0/),rc=rc) call ESMF_GridAllocCoord(grid4D, staggerLoc=staggerLoc, rc=rc) ! Set Corner call ESMF_StaggerLocSet(staggerLoc,loc=(/1,1,1,1/),rc=rc) call ESMF_GridAllocCoord(grid4D, staggerLoc=staggerLoc, rc=rc)
There is an added complication with the data (e.g. coordinates) stored at stagger locations in that they can require different amounts of storage depending on the underlying Grid type.
*-------*-------*-------*
| | | |
| + | + | + |
| | | |
*-------*-------*-------*
| | | |
| + | + | + |
| | | |
*-------*-------*-------*
Example Grid
Consider the preceeding 2D grid, where the ``*'' represents the cell corners and the ``+'' represents the cell centers. For the corners to completely enclose the cell centers (symmetric stagger), the number of corners in each dimension needs to be one greater then the number of cell centers. In the above figure, there are two rows and three columns of cell centers. To enclose the cell centers, there must be three rows and four columns of cell corners. This is true in general for Grids without periodicity or other connections. In fact, for a symmetric stagger, given that the center location requires n x m storage, the corresponding corner location requires n+1 x m+1, and the edges, depending on the side, require n+1 x m or m+1 x n. In order to add the extra storage, but also to allow the the different stagger location arrays to remain on the same DistGrid, the capability of the ESMF Array class to have computational bounds different from exclusive bounds is used. By default, when the coordinate arrays are created, one extra layer of padding is added to the arrays to create symmetric staggers (i.e. the center location is surrounded). The default is to add this padding on the positive side, and to only add this padding where needed (e.g. no padding for the center, padding on both dimensions for the corner, in only one dimension for the edge in 2D.) There are two ways for the user to change these defaults.
One way is to use the GridEdgeWidth or GridAlign arguments when creating a Grid. These arguments can be used to change the padding around the Grid cell index space. This extra padding is the extra space used by all the stagger locations, and no stagger location can extend outside of it.
The gridEdgeLWidth and
gridEdgeUWidth arguments are both 1D arrays of the
same size as the Grid rank. The entries in the arrays
give the extra offset from the outer boundary of
the grid cell index space to the exclusive region of the
Grid. The following example shows the creation of
a Grid with all the extra space to hold stagger padding
on the negative side of a Grid. This is the reverse of
the default behavior. The resulting Grid will have
an exclusive region which extends from to
, however, the cell center stagger location
will still extend from to .
grid2D=ESMF_GridCreateShapeTile(minIndex=(/1,1/),maxIndex=(/10,10/), &
gridEdgeLWidth=(/1,1/), gridEdgeUWidth=(/0,0/), rc=rc)
To indicate how the data in a Grid's stagger locations are aligned with the cell centers, the optional gridAlign parameter may be used. This parameter indicates which stagger elements in a cell share the same index values as the cell center. For example, in a 2D cell, it would indicate which of the four corners has the same index value as the center. To set gridAlign, the values -1,+1 are used to indicate the alignment in each dimension. This parameter is mostly informational, however, if the gridEdgeWidth parameters are not set then its value determines where the default padding is placed. If not specified, then the default is to align all staggers to the most negative, so the padding is on the positive side. The following code illustrates creating a Grid aligned to the reverse of default (with everything to the positive side). This creates a Grid identical to that created in the previous example.
grid2D=ESMF_GridCreateShapeTile(minIndex=(/1,1/),maxIndex=(/10,10/), &
gridAlign=(/1,1/), rc=rc)
The gridEdgeWidth and gridAlign arguments both allow the user to set the extra padding available to be used by stagger locations in a Grid. By default, stagger locations allocated in a Grid set their stagger padding based on these values. A stagger location's padding in each dimension is equal to the value of gridEdgeWidth (or the value implied by gridAlign), unless the stagger location is centered in a dimension in which case the stagger padding is 0. For example, the cell center stagger location has 0 stagger padding in all dimensions, whereas the edge stagger location lower padding is equal to gridEdgeLWidth and the upper padding is equal to gridEdgeUWidth in one dimension, but both are 0 in the other, centered, dimension. If the user wishes to set the stagger padding individually for each stagger location they may use the staggerEdgeWidth and staggerAlign arguments, however, the padding set this way must be within that specified by the gridEdgeWidth and gridAlign used when the Grid was created (or the defaults if none were used).
The staggerEdgeLWidth and staggerEdgeUWidth arguments are both 1D arrays of the same size as the Grid rank. The entries in the arrays give the extra offset from the Grid cell index space for a stagger location. The following example shows the addition of two stagger locations. The corner location has no extra boundary and the center has a single layer of extra padding on the negative side and none on the positive. This is the reverse of the default behavior.
grid2D=ESMF_GridCreate(distgrid=distgrid2D, &
gridEdgeLWidth=(/1,1/), gridEdgeUWidth=(/0,0/), rc=rc)
call ESMF_GridAllocCoord(grid2D, &
staggerLoc=ESMF_STAGGERLOC_CORNER, &
staggerEdgeLWidth=(/0,0/), staggerEdgeUWidth=(/0,0/), rc=rc)
call ESMF_GridAllocCoord(grid2D, &
staggerLoc=ESMF_STAGGERLOC_CENTER, &
staggerEdgeLWidth=(/1,1/), staggerEdgeUWidth=(/0,0/), rc=rc)
To indicate how the data at a particular stagger location is aligned with the cell center, the optional staggerAlign parameter may be used. This parameter indicates which stagger elements in a cell share the same index values as the cell center. For example, in a 2D cell, it would indicate which of the four corners has the same index value as the center. To set staggerAlign, the values -1,+1 are used to indicate the alignment in each dimension. If a stagger location is centered in a dimension (e.g. an edge in 2D), then that dimension is ignored in the alignment. This parameter is mostly informational, however, if the staggerEdgeWidth parameters are not set then its value determines where the default padding is placed. If not specified, then the default is to align all staggers to the most negative, so the padding is on the positive side. The following code illustrates aligning the positive (northeast in 2D) corner with the center.
call ESMF_GridAllocCoord(grid2D, &
staggerLoc=ESMF_STAGGERLOC_CORNER, staggerAlign=(/1,1/), rc=rc)
The ESMF_Grid class depends upon the ESMF_DistGrid class for the specification of its topology. That is, when creating a Grid, first an ESMF_DistGrid is created to describe the appropriate index space topology. This decision was made because it seemed redundant to have a system for doing this in both classes. It also seems most appropriate for the machinary for topology creation to be located at the lowest level possible so that it can be used by other classes (e.g. the ESMF_Array class). Because of this, however, the authors recommend that as a natural part of the implementation of subroutines to generate standard grid shapes (e.g. ESMF_GridGenSphere) a set of standard topology generation subroutines be implemented (e.g. ESMF_DistGridGenSphere) for users who want to create a standard topology, but a custom geometry.
The primarily complication in the storage of multiple stagger locations in a Grid is that different variables in a symmetric stagger can require a different amount of storage depending on the underlying grid type. For example while h,u, and v on an A-grid all require n x m arrays, on a B-grid u and v require n+1 x m+1. On a C or D grid one vector component requires n+1 x m and the other n x m+1. To handle this complication the natural approach would be to define each stagger's storage to what is necessary for that grid type. This approach introduces a problem when the arrays are distributed, because they are different sizes. It is non-trivial to guarantee that the (i,j) element of all three of arrays ends up on the same processor. It is simpler to guarantee a consistent distribution of the arrays when using the same distGrid if they are the same size.
This may sound like a contradiction, but to be more precise we choose the exclusive region of each array to be the same size (say n x m), and pad the arrays that need it with additional memory from the computational region of the Array class. Recall that the exclusive region is defined as the cells for which the DE claims exclusive ownership. These are the cells updated by computations local to that DE. The exclusive region is a subset of the computation region. The computational region contains all the cells kept locally on the DE in addition to the exclusive cells. By using the computational region as padding we are able to guarantee a consistent distribution of the arrays and at the same time impose a symmetric stagger. This approach extends naturally to the connected/periodic cases because the padding can be used to hold the values across the branch cut.
The biased configuration (where each stagger location has the same number of elements) falls out trivially by setting an optional padding argument (staggerWidth) to zero. This argument can also be used to adjust where the stagger padding is located or to add extra for halos.
DESCRIPTION:
The ESMF Grid class can exist in three states. These states are
present so that the library code can detect if a Grid has been
appropriately setup for the task at hand. The following
are the valid values of ESMF_GRIDSTATUS.
DESCRIPTION:
In the ESMF Grid class, data can be located at different positions in a
Grid cell. When setting or retrieving coordinate data the stagger location is
specified to tell the Grid method from where in the cell to get the data.
Although the user may define their own custom stagger locations,
ESMF provides a set of predefined locations for ease of use. The
following are the valid predefined stagger locations.
2----4----2
| |
| |
3 1 3
| |
| |
2----4----2
Diagram Illustrating 2D Stagger Locations
(Numbers correspond to bracketed numbers in list.)
The 2D predefined stagger locations are:
5----7----5 2----4----2 5----7----5
| | | | | |
| | | | | |
6 8 6 3 1 3 6 8 6
| | | | | |
| | | | | |
5----7----5 2----4----2 5----7----5
Cell Top Cell Middle Cell Bottom
Diagram Illustrating 3D Stagger Locations
(Numbers correspond to bracketed numbers in list.)
The 3D predefined stagger locations are:
INTERFACE:
! Private name; call using ESMF_GridAllocCoord()
subroutine ESMF_GridAllocCoordNoValues(grid, staggerloc, &
staggerEdgeLWidth, staggerEdgeUWidth, staggerAlign, &
totalLWidth, totalUWidth,rc)
ARGUMENTS:
type(ESMF_Grid), intent(in) :: grid
type (ESMF_StaggerLoc), intent(in),optional :: staggerloc
integer, intent(in),optional :: staggerEdgeLWidth(:)
integer, intent(in),optional :: staggerEdgeUWidth(:)
integer, intent(in),optional :: staggerAlign(:)
integer, intent(out), optional :: totalLWidth(:) ! N. IMP
integer, intent(out), optional :: totalUWidth(:) ! N. IMP
integer, intent(out),optional :: rc
DESCRIPTION:
When a Grid is created all of its potential stagger locations can hold coordinate data, but none of them have storage allocated. This call allocates coordinate storage (creates internal ESMF_Arrays and associated memory) for a particular stagger location. Note that this call doesn't assign any values to the storage, it only allocates it. The remaining options staggerEdgeLWidth, etc. allow the user to adjust the padding on the coordinate arrays.
The arguments are:
INTERFACE:
subroutine ESMF_GridCommit(grid, status, defaultflag, rc)ARGUMENTS:
type(ESMF_Grid), intent(inout) :: grid
type(ESMF_GridStatus),optional :: status ! NOT IMPLEMENTED
type(ESMF_DefaultFlag), optional :: defaultflag ! NOT IMPLEMENTED
integer, intent(out), optional :: rc
DESCRIPTION:
This call is used to complete the grid so that it is usable at the level indicated by the status flag. For example, once committed with a status value of ESMF_GRIDSTATUS_SHAPE_READY, the grid will have sufficient size, rank, and distribution information to be used as the basis for allocating Field data. (The integration of Field and Grid classes has't yet happened, so you can't currently allocate Fields based on Grids no matter what the status.)
It is necessary to call the ESMF_GridCommit() method after creating a Grid object using the ESMF_GridCreateEmpty() method and incrementally filling it in with ESMF_GridSet() calls. The EMF_GridCommit() call is a signal to the Grid that it can combine the pieces of information that it's received and finish building any necessary internal structures. For example, an ESMF_GridCommit() call with the status flag set to ESMF_GRIDSTATUS_SHAPE_READY will trigger the grid to build an internal DistGrid object that contains topology and distribution information.
It's possible using the ESMF_GridCreateEmpty()/ESMF_GridSet() approach that not all information is present when the ESMF_GridCommit call is made. If this is the case and the defaultflag is set to ESMF_USE_DEFAULTS the Grid will attempt to build any internal objects necessary to get to the desired status by using reasonable defaults. If the defaultflag is set to ESMF_NO_DEFAULTS and any information is missing, the ESMF_GridCommit call will fail. If the defaultflag argument is not passed in, no defaults are used.
The arguments are:
INTERFACE:
! Private name; call using ESMF_GridCreate()
function ESMF_GridCreateFromDistGrid(name,coordTypeKind,distgrid, &
distgridToGridMap, undistLBound, undistUBound, coordRank, coordDimMap, &
gridEdgeLWidth, gridEdgeUWidth, gridAlign, indexflag, rc)
RETURN VALUE:
type(ESMF_Grid) :: ESMF_GridCreateFromDistGridARGUMENTS:
character (len=*), intent(in), optional :: name
type(ESMF_TypeKind), intent(in), optional :: coordTypeKind
type(ESMF_DistGrid), intent(in) :: distgrid
integer, intent(in), optional :: distgridToGridMap(:)
integer, intent(in), optional :: undistLBound(:)
integer, intent(in), optional :: undistUBound(:)
integer, intent(in), optional :: coordRank(:)
integer, intent(in), optional :: coordDimMap(:,:)
integer, intent(in), optional :: gridEdgeLWidth(:)
integer, intent(in), optional :: gridEdgeUWidth(:)
integer, intent(in), optional :: gridAlign(:)
type(ESMF_IndexFlag), intent(in), optional :: indexflag
integer, intent(out), optional :: rc
DESCRIPTION:
This is the most general form of creation for an ESMF_Grid object. It allows the user to fully specify the topology and index space (of the distributed dimensions) using the DistGrid methods and then build a grid out of the resulting distgrid. Optional lbound and ubound arguments can be used to specify extra undistributed dimensions. The distgridToGridMap argument specifies how the distributed (from distgrid) and undistributed (from bounds) dimensions are intermixed. The coordRank and coordDimMap arguments allow the user to specify how the coordinate arrays should map to the grid dimensions. (Note, though, that creating a grid does not allocate coordinate storage. A method such as ESMF_GridAllocCoord() must be called before adding coordinate values.)
The arguments are:
INTERFACE:
function ESMF_GridCreateEmpty(rc)RETURN VALUE:
type(ESMF_Grid) :: ESMF_GridCreateEmptyARGUMENTS:
integer, intent(out), optional :: rcDESCRIPTION:
Partially create an ESMF_Grid object. This function allocates an ESMF_Grid object, but doesn't allocate any coordinate storage or other internal structures. The ESMF_GridSet calls can be used to set the values in the grid object. Before using the grid, ESMF_GridCommit needs to be called to validate the internal state and construct internal structures.
The arguments are:
INTERFACE:
! Private name; call using ESMF_GridCreateShapeTile()
function ESMF_GridCreateShapeTileIrreg(name,coordTypeKind, minIndex, &
countsPerDEDim1,countsPerDeDim2, countsPerDEDim3, &
connDim1, connDim2, connDim3, &
poleStaggerLoc1, poleStaggerLoc2, poleStaggerLoc3, &
bipolePos1, bipolePos2, bipolePos3, &
coordDep1, coordDep2, coordDep3, &
gridEdgeLWidth, gridEdgeUWidth, gridAlign, &
indexflag, distDim, petMap, rc)
RETURN VALUE:
type(ESMF_Grid) :: ESMF_GridCreateShapeTileIrregARGUMENTS:
character (len=*), intent(in), optional :: name
type(ESMF_TypeKind), intent(in), optional :: coordTypeKind
integer, intent(in), optional :: minIndex(:)
integer, intent(in) :: countsPerDEDim1(:)
integer, intent(in) :: countsPerDEDim2(:)
integer, intent(in), optional :: countsPerDEDim3(:)
type(ESMF_GridConn), intent(in), optional :: connDim1(:) ! N. IMP.
type(ESMF_GridConn), intent(in), optional :: connDim2(:) ! N. IMP.
type(ESMF_GridConn), intent(in), optional :: connDim3(:) ! N. IMP.
type(ESMF_StaggerLoc), intent(in), optional :: poleStaggerLoc1(2) ! N. IMP.
type(ESMF_StaggerLoc), intent(in), optional :: poleStaggerLoc2(2) ! N. IMP.
type(ESMF_StaggerLoc), intent(in), optional :: poleStaggerLoc3(2) ! N. IMP.
integer, intent(in), optional :: bipolePos1(2) ! N. IMP.
integer, intent(in), optional :: bipolePos2(2) ! N. IMP.
integer, intent(in), optional :: bipolePos3(2) ! N. IMP.
integer, intent(in), optional :: coordDep1(:)
integer, intent(in), optional :: coordDep2(:)
integer, intent(in), optional :: coordDep3(:)
integer, intent(in), optional :: gridEdgeLWidth(:)
integer, intent(in), optional :: gridEdgeUWidth(:)
integer, intent(in), optional :: gridAlign(:)
type(ESMF_IndexFlag), intent(in), optional :: indexflag
logical, intent(in), optional :: distDim(:)
integer, intent(in), optional :: petMap(:,:,:)
integer, intent(out), optional :: rc
DESCRIPTION:
This method creates a single tile, irregularly distributed grid (see Figure 11). To specify the irregular distribution, the user passes in an array for each grid dimension, where the length of the array is the number of DEs in the dimension. Up to three dimensions can be specified, using the countsPerDEDim1, countsPerDEDim2, countsPerDEDim3 arguments. The index of each array element corresponds to a DE number. The array value at the index is the number of grid cells on the DE in that dimension. The rank of the grid is equal to the number of countsPerDEDim<> arrays that are specified.
To specify an undistributed dimension, the array in that dimension should have only one element and the corresponding entry in distDim should be false.
Section 22.2.2 shows an example of using this method to create a 2D Grid with uniformly spaced coordinates. This creation method can also be used as the basis for grids with rectilinear coordinates or curvilinear coordinates.
The arguments are:
INTERFACE:
! Private name; call using ESMF_GridCreateShapeTile()
function ESMF_GridCreateShapeTileReg(name, coordTypeKind, &
regDecomp, decompFlag, minIndex, maxIndex, &
connDim1, connDim2, connDim3, &
poleStaggerLoc1, poleStaggerLoc2, poleStaggerLoc3, &
bipolePos1, bipolePos2, bipolePos3, &
coordDep1, coordDep2, coordDep3, &
gridEdgeLWidth, gridEdgeUWidth, gridAlign, &
indexflag, distDim, petMap, rc)
RETURN VALUE:
type(ESMF_Grid) :: ESMF_GridCreateShapeTileRegARGUMENTS:
character (len=*), intent(in), optional :: name
type(ESMF_TypeKind), intent(in), optional :: coordTypeKind
integer, intent(in), optional :: regDecomp(:)
type(ESMF_DecompFlag), intent(in), optional :: decompflag(:)
integer, intent(in), optional :: minIndex(:)
integer, intent(in) :: maxIndex(:)
type(ESMF_GridConn), intent(in), optional :: connDim1(:) ! N. IMP.
type(ESMF_GridConn), intent(in), optional :: connDim2(:) ! N. IMP.
type(ESMF_GridConn), intent(in), optional :: connDim3(:) ! N. IMP.
type(ESMF_StaggerLoc), intent(in), optional :: poleStaggerLoc1(2) ! N. IMP.
type(ESMF_StaggerLoc), intent(in), optional :: poleStaggerLoc2(2) ! N. IMP.
type(ESMF_StaggerLoc), intent(in), optional :: poleStaggerLoc3(2) ! N. IMP.
integer, intent(in), optional :: bipolePos1(2) ! N. IMP.
integer, intent(in), optional :: bipolePos2(2) ! N. IMP.
integer, intent(in), optional :: bipolePos3(2) ! N. IMP.
integer, intent(in), optional :: coordDep1(:)
integer, intent(in), optional :: coordDep2(:)
integer, intent(in), optional :: coordDep3(:)
integer, intent(in), optional :: gridEdgeLWidth(:)
integer, intent(in), optional :: gridEdgeUWidth(:)
integer, intent(in), optional :: gridAlign(:)
type(ESMF_IndexFlag), intent(in), optional :: indexflag
logical, intent(in), optional :: distDim(:)
integer, intent(in), optional :: petMap(:,:,:)
integer, intent(out), optional :: rc
DESCRIPTION:
This method creates a single tile, regularly distributed grid (see Figure 11). To specify the distribution, the user passes in an array (regDecomp) specifying the number of DEs to divide each dimension into. If the number of DEs is 1 than the dimension is undistributed. The array decompFlag indicates how the division into DEs is to occur. The default is to divide the range as evenly as possible.
The arguments are:
INTERFACE:
subroutine ESMF_GridDestroy(grid, rc)ARGUMENTS:
type(ESMF_Grid) :: grid
integer, intent(out), optional :: rc
DESCRIPTION:
Destroys an ESMF_Grid object and related internal structures. This call does not destroy the internal coordinate Arrays, or the internally generated DistGrid.
The arguments are:
INTERFACE:
subroutine ESMF_GridGet(grid, name, coordTypeKind, &
rank, distRank, undistRank, &
tileCount, staggerlocsCount, localDECount, distgrid, &
distgridToGridMap, undistLBound, undistUBound, coordRank, coordDimMap, &
gridEdgeLWidth, gridEdgeUWidth, gridAlign, &
indexFlag, rc)
ARGUMENTS:
type(ESMF_Grid), intent(in) :: grid
character (len=*), intent(out), optional :: name
type(ESMF_TypeKind), intent(out), optional :: coordTypeKind
integer, intent(out), optional :: rank
integer, intent(out), optional :: distRank
integer, intent(out), optional :: undistRank
integer, intent(out), optional :: tileCount
integer, intent(out), optional :: staggerlocsCount
integer, intent(out), optional :: localDECount
type(ESMF_DistGrid), intent(out), optional :: distgrid
integer, intent(out), optional :: distgridToGridMap(:)
integer, intent(out), optional :: undistLBound(:)
integer, intent(out), optional :: undistUBound(:)
integer, intent(out), optional :: coordRank(:)
integer, intent(out), optional :: coordDimMap(:,:)
integer, intent(out), optional :: gridEdgeLWidth(:)
integer, intent(out), optional :: gridEdgeUWidth(:)
integer, intent(out), optional :: gridAlign(:)
type(ESMF_IndexFlag), intent(out), optional :: indexflag
integer, intent(out), optional :: rc
DESCRIPTION:
Gets various types of information about a grid.
The arguments are:
INTERFACE:
! Private name; call using ESMF_GridGet()
subroutine ESMF_GridGetPLocalDePSloc(grid, localDe, staggerloc, &
exclusiveLBound, exclusiveUBound, exclusiveCount, &
computationalLBound, computationalUBound, computationalCount, rc)
ARGUMENTS:
type(ESMF_Grid), intent(in) :: grid
integer, intent(in) :: localDe
type (ESMF_StaggerLoc), intent(in) :: staggerloc
integer, intent(out), optional :: exclusiveLBound(:)
integer, intent(out), optional :: exclusiveUBound(:)
integer, intent(out), optional :: exclusiveCount(:)
integer, intent(out), optional :: computationalLBound(:)
integer, intent(out), optional :: computationalUBound(:)
integer, intent(out), optional :: computationalCount(:)
integer, intent(out), optional :: rc
DESCRIPTION:
This method gets information about the range of index space which a particular stagger location occupies. This call differs from the coordinate bound calls (e.g. ESMF_GridGetCoord) in that a given coordinate array may only occupy a subset of the Grid's dimensions, and so these calls may not give all the bounds of the stagger location. The bounds from this call are the full bounds, and so for example, give the appropriate bounds for allocating a F90 array to hold data residing on the stagger location. Note that unlike the output from the Array, these values also include the undistributed dimensions and are ordered to reflect the order of the indices in the Grid. This call will still give correct values even if the stagger location does not contain coordinate arrays (e.g. if ESMF_GridAllocCoord hasn't yet been called on the stagger location).
The arguments are:
INTERFACE:
! Private name; call using ESMF_GridGet()
subroutine ESMF_GridGetPSloc(grid, staggerloc, &
computationalEdgeLWidth, computationalEdgeUWidth, &
undistLBound,undistUBound, rc)
ARGUMENTS:
type(ESMF_Grid), intent(in) :: grid
type (ESMF_StaggerLoc), intent(in) :: staggerloc
integer, intent(out), optional :: computationalEdgeLWidth(:)
integer, intent(out), optional :: computationalEdgeUWidth(:)
integer, intent(out), optional :: undistLBound(:)
integer, intent(out), optional :: undistUBound(:)
integer, intent(out), optional :: rc
DESCRIPTION:
This method gets information about a particular stagger location. This information is useful for creating an ESMF Array to hold the data at the stagger location.
The arguments are:
INTERFACE:
subroutine ESMF_GridGetCoord(grid, localDE, coordDim, staggerloc, &
exclusiveLBound, exclusiveUBound, exclusiveCount, &
computationalLBound, computationalUBound, computationalCount, &
totalLBound, totalUBound, totalCount, &
<pointer argument>, doCopy, rc)
ARGUMENTS:
type(ESMF_Grid), intent(in) :: grid
integer, intent(in), optional :: localDE
integer, intent(in) :: coordDim
type (ESMF_StaggerLoc), intent(in), optional :: staggerloc
integer, intent(out), optional :: exclusiveLBound(:)
integer, intent(out), optional :: exclusiveUBound(:)
integer, intent(out), optional :: exclusiveCount(:)
integer, intent(out), optional :: computationalLBound(:)
integer, intent(out), optional :: computationalUBound(:)
integer, intent(out), optional :: computationalCount(:)
integer, intent(out), optional :: totalLBound(:)
integer, intent(out), optional :: totalUBound(:)
integer, intent(out), optional :: totalCount(:)
<pointer argument>, see below for supported values
type(ESMF_CopyFlag), intent(in), optional :: docopy
integer, intent(out), optional :: rc
DESCRIPTION:
This method gets a Fortran pointer to the piece of memory which holds the coordinate data on the local DE for the given coordinate dimension and stagger locations. This is useful, for example, for setting the coordinate values in a Grid, or for reading the coordinate values. Currently this method supports up to three coordinate dimensions, of either R4 or R8 datatype. See below for specific supported values. If the coordinates that you are trying to retrieve are of higher dimension, use the ESMF_GetCoord() interface that returns coordinate values in an ESMF_Array instead. That interface supports the retrieval of coordinates up to 7D.
Supported values for the <pointer argument> are:
The arguments are:
INTERFACE:
! Private name; call using ESMF_GridGetCoord()
subroutine ESMF_GridGetCoordBounds(grid, localDE, coordDim, staggerloc, &
exclusiveLBound, exclusiveUBound, exclusiveCount, &
computationalLBound, computationalUBound, computationalCount, &
totalLBound, totalUBound, totalCount, rc)
ARGUMENTS:
type(ESMF_Grid), intent(in) :: grid
integer, intent(in), optional :: localDE
integer, intent(in) :: coordDim
type (ESMF_StaggerLoc), intent(in), optional :: staggerloc
integer, intent(out), optional :: exclusiveLBound(:)
integer, intent(out), optional :: exclusiveUBound(:)
integer, intent(out), optional :: exclusiveCount(:)
integer, intent(out), optional :: computationalLBound(:)
integer, intent(out), optional :: computationalUBound(:)
integer, intent(out), optional :: computationalCount(:)
integer, intent(out), optional :: totalLBound(:)
integer, intent(out), optional :: totalUBound(:)
integer, intent(out), optional :: totalCount(:)
integer, intent(out), optional :: rc
DESCRIPTION:
This method gets information about the range of index space which a particular piece of coordinate data occupies. In other words, this method returns the bounds of the coordinate arrays. Note that unlike the output from the Array, these values also include the undistributed dimensions and are ordered to reflect the order of the indices in the coordinate. So, for example, totalLBound and totalUBound should match the bounds of the Fortran array retrieved by ESMF_GridGetCoord.
The arguments are:
INTERFACE:
! Private name; call using ESMF_GridGetCoord()
subroutine ESMF_GridGetCoordIntoArray(grid, staggerloc,coordDim, array, &
docopy, rc)
ARGUMENTS:
type(ESMF_Grid), intent(in) :: grid
type (ESMF_StaggerLoc), intent(in),optional :: staggerloc
integer, intent(in) :: coordDim
type(ESMF_Array), intent(out) :: array
type(ESMF_CopyFlag), intent(in), optional :: docopy ! NOT IMPLEMENTED
integer, intent(out), optional :: rc
DESCRIPTION:
This method allows the user to get access to the ESMF Array holding coordinate data at a particular stagger location. This is useful, for example, to set the coordinate values. To have an Array to access, the coordinate Arrays must have already been allocated, for example by ESMF_GridAllocCoord or ESMF_GridSetCoord.
The arguments are:
INTERFACE:
! Private name; call using ESMF_GridSet()
subroutine ESMF_GridSetFromDistGrid(grid, name, coordTypeKind, distgrid, &
distgridToGridMap, undistLBound, undistUBound, coordRank, coordDimMap, &
gridEdgeLWidth, gridEdgeUWidth, gridAlign, &
indexflag, rc)
RETURN VALUE:
ARGUMENTS:
type(ESMF_Grid), intent(inout) :: grid
character (len=*), intent(in), optional :: name
type(ESMF_TypeKind), intent(in), optional :: coordTypeKind
type(ESMF_DistGrid), intent(in), optional :: distgrid
integer, intent(in), optional :: distgridToGridMap(:)
integer, intent(in), optional :: undistLBound(:)
integer, intent(in), optional :: undistUBound(:)
integer, intent(in), optional :: coordRank(:)
integer, intent(in), optional :: coordDimMap(:,:)
integer, intent(in), optional :: gridEdgeLWidth(:)
integer, intent(in), optional :: gridEdgeUWidth(:)
integer, intent(in), optional :: gridAlign(:)
type(ESMF_IndexFlag), intent(in), optional :: indexflag
integer, intent(out), optional :: rc
DESCRIPTION:
Set values in a grid in preparation for committing and creating a grid. This method is called between ESMF_GridCreateEmpty and ESMF_GridCommit. Note that once a grid is committed and created it's an error to try to set values in it. Note also that new values overwrite old values if previously set.
The arguments are:
INTERFACE:
subroutine ESMF_GridSetCoordFromArray(grid, staggerloc, coordDim, &
array, doCopy, rc)
ARGUMENTS:
type(ESMF_Grid), intent(in) :: grid
type (ESMF_StaggerLoc), intent(in), optional :: staggerloc
integer, intent(in) :: coordDim
type(ESMF_Array), intent(in) :: array
type(ESMF_CopyFlag), intent(in), optional :: docopy ! NOT IMPLEMENTED
integer, intent(out), optional :: rc
DESCRIPTION:
This method sets the passed in Array as the holder of the coordinate data for stagger location staggerloc and coordinate coord. If the location already contains an Array, then this one overwrites it.
The arguments are:
INTERFACE:
! Private name; call using ESMF_GridSetCommitShapeTile()
subroutine ESMF_GridSetCmmitShapeTileIrreg(grid, name,coordTypeKind, minIndex, &
countsPerDEDim1,countsPerDeDim2, countsPerDEDim3, &
connDim1, connDim2, connDim3, &
poleStaggerLoc1, poleStaggerLoc2, poleStaggerLoc3, &
bipolePos1, bipolePos2, bipolePos3, &
coordDep1, coordDep2, coordDep3, &
gridEdgeLWidth, gridEdgeUWidth, gridAlign, &
indexflag, distDim, petMap, rc)
ARGUMENTS:
type (ESMF_Grid) :: grid
character (len=*), intent(in), optional :: name
type(ESMF_TypeKind), intent(in), optional :: coordTypeKind
integer, intent(in), optional :: minIndex(:)
integer, intent(in) :: countsPerDEDim1(:)
integer, intent(in) :: countsPerDEDim2(:)
integer, intent(in), optional :: countsPerDEDim3(:)
type(ESMF_GridConn), intent(in), optional :: connDim1(:) ! N. IMP.
type(ESMF_GridConn), intent(in), optional :: connDim2(:) ! N. IMP.
type(ESMF_GridConn), intent(in), optional :: connDim3(:) ! N. IMP.
type(ESMF_StaggerLoc), intent(in), optional :: poleStaggerLoc1(2) ! N. IMP.
type(ESMF_StaggerLoc), intent(in), optional :: poleStaggerLoc2(2) ! N. IMP.
type(ESMF_StaggerLoc), intent(in), optional :: poleStaggerLoc3(2) ! N. IMP.
integer, intent(in), optional :: bipolePos1(2) ! N. IMP.
integer, intent(in), optional :: bipolePos2(2) ! N. IMP.
integer, intent(in), optional :: bipolePos3(2) ! N. IMP.
integer, intent(in), optional :: coordDep1(:)
integer, intent(in), optional :: coordDep2(:)
integer, intent(in), optional :: coordDep3(:)
integer, intent(in), optional :: gridEdgeLWidth(:)
integer, intent(in), optional :: gridEdgeUWidth(:)
integer, intent(in), optional :: gridAlign(:)
type(ESMF_IndexFlag), intent(in), optional :: indexflag
logical, intent(in), optional :: distDim(:)
integer, intent(in), optional :: petMap(:,:,:)
integer, intent(out), optional :: rc
DESCRIPTION:
This method sets information into an empty Grid and then commits it to create a single tile, irregularly distributed grid (see Figure 11). To specify the irregular distribution, the user passes in an array for each grid dimension, where the length of the array is the number of DEs in the dimension. Up to three dimensions can be specified, using the countsPerDEDim1, countsPerDEDim2, countsPerDEDim3 arguments. The index of each array element corresponds to a DE number. The array value at the index is the number of grid cells on the DE in that dimension. The rank of the grid is equal to the number of countsPerDEDim<> arrays that are specified.
To specify an undistributed dimension, the array in that dimension should have only one element and the corresponding entry in distDim should be false.
Section 22.2.2 shows an example of using this method to create a 2D Grid with uniformly spaced coordinates. This creation method can also be used as the basis for grids with rectilinear coordinates or curvilinear coordinates.
The arguments are:
INTERFACE:
! Private name; call using ESMF_GridSetCommitShapeTile()
subroutine ESMF_GridSetCmmitShapeTileReg(grid, name, coordTypeKind, &
regDecomp, decompFlag, minIndex, maxIndex, &
connDim1, connDim2, connDim3, &
poleStaggerLoc1, poleStaggerLoc2, poleStaggerLoc3, &
bipolePos1, bipolePos2, bipolePos3, &
coordDep1, coordDep2, coordDep3, &
gridEdgeLWidth, gridEdgeUWidth, gridAlign, &
indexflag, distDim, petMap, rc)
ARGUMENTS:
type(ESMF_Grid), intent(inout) :: grid
character (len=*), intent(in), optional :: name
type(ESMF_TypeKind), intent(in), optional :: coordTypeKind
integer, intent(in), optional :: regDecomp(:)
type(ESMF_DecompFlag), intent(in), optional :: decompflag(:)
integer, intent(in), optional :: minIndex(:)
integer, intent(in) :: maxIndex(:)
type(ESMF_GridConn), intent(in), optional :: connDim1(:) ! N. IMP.
type(ESMF_GridConn), intent(in), optional :: connDim2(:) ! N. IMP.
type(ESMF_GridConn), intent(in), optional :: connDim3(:) ! N. IMP.
type(ESMF_StaggerLoc), intent(in), optional :: poleStaggerLoc1(2) ! N. IMP.
type(ESMF_StaggerLoc), intent(in), optional :: poleStaggerLoc2(2) ! N. IMP.
type(ESMF_StaggerLoc), intent(in), optional :: poleStaggerLoc3(2) ! N. IMP.
integer, intent(in), optional :: bipolePos1(2) ! N. IMP.
integer, intent(in), optional :: bipolePos2(2) ! N. IMP.
integer, intent(in), optional :: bipolePos3(2) ! N. IMP.
integer, intent(in), optional :: coordDep1(:)
integer, intent(in), optional :: coordDep2(:)
integer, intent(in), optional :: coordDep3(:)
integer, intent(in), optional :: gridEdgeLWidth(:)
integer, intent(in), optional :: gridEdgeUWidth(:)
integer, intent(in), optional :: gridAlign(:)
type(ESMF_IndexFlag), intent(in), optional :: indexflag
logical, intent(in), optional :: distDim(:)
integer, intent(in), optional :: petMap(:,:,:)
integer, intent(out), optional :: rc
DESCRIPTION:
This method sets information into an empty Grid and then commits it to create a single tile, regularly distributed grid (see Figure 11). To specify the distribution, the user passes in an array (regDecomp) specifying the number of DEs to divide each dimension into. If the number of DEs is 1 than the dimension is undistributed. The array decompFlag indicates how the division into DEs is to occur. The default is to divide the range as evenly as possible.
The arguments are:
INTERFACE:
subroutine ESMF_GridValidate(grid, rc)ARGUMENTS:
type(ESMF_Grid), intent(in) :: grid
integer, intent(out), optional :: rc
DESCRIPTION:
Validates that the Grid is internally consistent. Note that one of the checks that the Grid validate does is the Grid status. Currently, the validate will return an error if the grid is not at least ESMF_GRIDSTATUS_SHAPE_READY. This means if a Grid was created with ESMF_GridCreateEmpty it must also have been commited with ESMF_GridCommit to be valid. If a Grid was created with another create call it should automatically have the correct status level to pass the status part of the validate. The Grid validate at this time doesn't check for the presence or consistency of the Grid coordinates. The method returns an error code if problems are found.
The arguments are:
INTERFACE:
! Private name; call using ESMF_StaggerLocSet()
subroutine ESMF_StaggerLocSetAllDim(staggerloc,loc,rc)
ARGUMENTS:
type (ESMF_StaggerLoc), intent(inout) :: staggerloc
integer, intent(in) :: loc(:)
integer, optional :: rc
DESCRIPTION:
Sets a custom staggerloc to a position in a cell by using the array loc. The values in the array should only be 0,1. If loc(i) is 0 it means the position should be in the center in that dimension. If loc(i) is 1 then for dimension i, the position should be on the side of the cell. Please see Section 22.2.15 for diagrams and further discussion of custom stagger locations.
The arguments are:
INTERFACE:
! Private name; call using ESMF_StaggerLocSet()
subroutine ESMF_StaggerLocSetDim(staggerloc,dim,loc,rc)
ARGUMENTS:
type (ESMF_StaggerLoc), intent(inout) :: staggerloc
integer, intent(in) :: dim,loc
integer, optional :: rc
DESCRIPTION:
Sets a particular dimension of a custom staggerloc to a position in a cell by using the variable loc. The variable loc should only be 0,1. If loc is 0 it means the position should be in the center in that dimension. If loc is +1 then for the dimension, the position should be on the positive side of the cell. Please see Section 22.2.15 for diagrams and further discussion of custom stagger locations.
The arguments are:
INTERFACE:
subroutine ESMF_StaggerLocString(staggerloc, string, rc)ARGUMENTS:
type(ESMF_StaggerLoc), intent(in) :: staggerloc
character (len = *), intent(out) :: string
integer, intent(out), optional :: rc
DESCRIPTION:
Return an ESMF_StaggerLoc as a printable string.
The arguments are:
INTERFACE:
subroutine ESMF_StaggerLocPrint(staggerloc, rc)ARGUMENTS:
type (ESMF_StaggerLoc), intent(in) :: staggerloc
integer, intent(out), optional :: rc
DESCRIPTION:
Print the internal data members of a ESMF_StaggerLoc object
The arguments are:
The ESMF_DistGrid class sits on top of the DELayout class and holds domain information in index space. A DistGrid object captures the index space topology and describes its decomposition in terms of DEs. Combined with DELayout and VM the DistGrid defines the data distribution of a domain decomposition across the computational resources of an ESMF component.
The global domain is defined as the union or ``patchwork'' of logically rectangular (LR) sub-domains or patches. The DistGrid create methods allow the specification of such a patchwork global domain and its decomposition into exclusive, DE-local LR regions according to various degrees of user specified constraints. Complex index space topologies can be constructed by specifying connection relationships between patches during creation.
The DistGrid class holds domain information for all DEs. Each DE is associated with a local LR region. No overlap of the regions is allowed. The DistGrid offers query methods that allow DE-local topology information to be extracted, e.g. for the construction of halos by higher classes.
A DistGrid object only contains decomposable dimensions. The minimum rank for a DistGrid object is 1. A maximum rank does not exist for DistGrid objects, however, ranks greater than 7 may lead to difficulties with respect to the Fortran API of higher classes based on DistGrid. The rank of a DELayout object contained within a DistGrid object must be equal to the DistGrid rank. Higher class objects that use the DistGrid, such as an Array object, may be of different rank than the associated DistGrid object. The higher class object will hold the mapping information between its dimensions and the DistGrid dimensions.
The following examples demonstrate how to create, use and destroy DistGrid objects. In order to produce complete and valid DistGrid objects all of the ESMF_DistGridCreate() calls require to be called in unison i.e. on all PETs of a component with a complete set of valid arguments.
The minimum information required to create an ESMF_DistGrid object for a single patch with default decomposition are the corners of the patch in index space. The following call will create a 1D DistGrid for a 1D index space patch with elements from 1 through 1000.
distgrid = ESMF_DistGridCreate(minIndex=(/1/), maxIndex=(/1000/), rc=rc)
A default DELayout with 1 DE per PET will be created during
ESMF_DistGridCreate(). The 1000 elements of the specified 1D patch will
then be block decomposed across the available DEs, i.e. across all PETs.
Hence, for 4 PETs the (min) 
(max) corners of the DE-local LR regions
will be:
DE 0 - (1) ~ (250)
DE 1 - (251) ~ (500)
DE 2 - (501) ~ (750)
DE 3 - (751) ~ (1000)
DistGrids with rank > 1 can also be created with default decompositions, specifying only the corners of the patch. The following will create a 2D DistGrid for a 5x5 patch with default decomposition.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), rc=rc)
The default decomposition for a DistGrid of rank 
will be
, where 
is the number of DEs in the DELayout
and there are 
factors of . For the 2D example above this means
a 
regular decomposition if executed on 4 PETs and will result
in the following DE-local LR regions:
DE 0 - (1,1) ~ (2,5)
DE 1 - (3,1) ~ (3,5)
DE 2 - (4,1) ~ (4,5)
DE 3 - (5,1) ~ (5,5)
In many cases the default decomposition will not suffice for higher rank
DistGrids (rank > 1). For this reason a decomposition descriptor
regDecomp argument is available during ESMF_DistGridCreate(). The
following call creates a DistGrid on the same 2D patch as before, but now with
a user specified regular decomposition of
DEs.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
regDecomp=(/2,3/), rc=rc)
The default DE labeling sequence follows column major order for the regDecomp argument:
-----------> 2nd dimension
| 0 2 4
| 1 3 5
v
1st dimension
By default grid points along all dimensions are homogeneously divided between
the DEs. The maximum element count difference between DEs along any dimension
is 1. The (min) (max) corners of the DE-local LR domains of the above
example are as follows:
DE 0 - (1,1) ~ (3,2)
DE 1 - (4,1) ~ (5,2)
DE 2 - (1,3) ~ (3,4)
DE 3 - (4,3) ~ (5,4)
DE 4 - (1,5) ~ (3,5)
DE 5 - (4,5) ~ (5,5)
The specifics of the patch decomposition into DE-local LR domains can be modified by the optional decompflag argument. The following line shows how this argument is used to keep ESMF's default decomposition in the first dimension but move extra grid points of the second dimension to the last DEs in that direction. Extra elements occur if the number of DEs for a certain dimension does not evenly divide its extent. In this example there are 2 extra grid points for the second dimension because its extent is 5 but there are 3 DEs along this index space axis.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
regDecomp=(/2,3/), decompflag=(/ESMF_DECOMP_DEFAULT,ESMF_DECOMP_RESTLAST/),&
rc=rc)
Now DE 4 and DE 5 will hold the extra elements along the 2nd dimension.
DE 0 - (1,1) ~ (3,1)
DE 1 - (4,1) ~ (5,1)
DE 2 - (1,2) ~ (3,2)
DE 3 - (4,2) ~ (5,2)
DE 4 - (1,3) ~ (3,5)
DE 5 - (4,3) ~ (5,5)
An alternative way of indicating the DE-local LR regions is to list the index space coordinate as given by the associated DistGrid patch for each dimension. For this 2D example there are two lists (dim 1) / (dim 2) for each DE:
DE 0 - (1,2,3) / (1)
DE 1 - (4,5) / (1)
DE 2 - (1,2,3) / (2)
DE 3 - (4,5) / (2)
DE 4 - (1,2,3) / (3,4,5)
DE 5 - (4,5) / (3,4,5)
Information about DE-local LR regions in the latter format can be obtained from the DistGrid object by use of ESMF_DistGridGet() methods:
allocate(dimExtent(2, 0:5)) ! (dimCount, deCount)
call ESMF_DistGridGet(distgrid, delayout=delayout, &
indexCountPDimPDe=dimExtent, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(terminationflag=ESMF_ABORT)
call ESMF_DELayoutGet(delayout, localDeCount=localDeCount, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(terminationflag=ESMF_ABORT)
allocate(localDeList(0:localDeCount-1))
call ESMF_DELayoutGet(delayout, localDeList=localDeList, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(terminationflag=ESMF_ABORT)
do localDe=0, localDeCount-1
de = localDeList(localDe)
do dim=1, 2
allocate(localIndexList(dimExtent(dim, de))) ! allocate list to hold indices
call ESMF_DistGridGet(distgrid, localDe=localDe, dim=dim, &
indexList=localIndexList, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(terminationflag=ESMF_ABORT)
print *, "local DE ", localDe," - DE ",de," localIndexList along dim=", &
dim," :: ", localIndexList
deallocate(localIndexList)
enddo
enddo
deallocate(localDeList)
deallocate(dimExtent)
The advantage of the localIndexList format over the min-/max-corner format is that it can be used directly for DE-local to patch index dereferencing. Furthermore the localIndexList allows to express very general decompositions such as the cyclic decompositions in the first dimension generated by the following call:
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
regDecomp=(/2,3/), decompflag=(/ESMF_DECOMP_CYCLIC,ESMF_DECOMP_RESTLAST/),&
rc=rc)
with decomposition:
DE 0 - (1,3,5) / (1)
DE 1 - (2,4) / (1)
DE 2 - (1,3,5) / (2)
DE 3 - (2,4) / (2)
DE 4 - (1,3,5) / (3,4,5)
DE 5 - (2,4) / (3,4,5)
Finally, a DistGrid object is destroyed by calling
call ESMF_DistGridDestroy(distgrid, rc=rc)
The examples of this section use the 2D DistGrid of the previous section to show the interplay between DistGrid and DELayout. By default, i.e. without specifying the delayout argument, a DELayout will be created during DistGrid creation that provides as many DEs as the DistGrid object requires. The implicit call to ESMF_DELayoutCreate() is issued with a fixed number of DEs and default settings in all other aspects. The resulting DE to PET mapping depends on the number of PETs of the current VM context. Assuming 6 PETs in the VM
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
regDecomp=(/2,3/), rc=rc)
will result in the following domain decomposition in terms of DEs
0 2 4
1 3 5
and their layout or distribution over the available PETs:
DE 0 -> PET 0
DE 1 -> PET 1
DE 2 -> PET 2
DE 3 -> PET 3
DE 4 -> PET 4
DE 5 -> PET 5
Running the same example on a 4 PET VM will not change the domain decomposition into 6 DEs as specified by
0 2 4
1 3 5
but the layout across PETs will now contain multiple DE-to-PET mapping with
default cyclic distribution:
DE 0 -> PET 0
DE 1 -> PET 1
DE 2 -> PET 2
DE 3 -> PET 3
DE 4 -> PET 0
DE 5 -> PET 1
Sometimes it may be desirable for performance tuning to construct a DELayout with specific characteristics. For instance, if the 6 PETs of the above example are running on 3 nodes of a dual-SMP node cluster and there is a higher communication load along the first dimension of the model than along the second dimension it would be sensible to place DEs according to this knowledge.
The following example first creates a DELayout with 6 DEs where groups of 2 DEs are to be in fast connection. This DELayout is then used to create a DistGrid.
delayout = ESMF_DELayoutCreate(deCount=6, deGrouping=(/(i/2,i=0,5)/), rc=rc)
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
regDecomp=(/2,3/), delayout=delayout, rc=rc)
This will ensure a distribution of DEs across the cluster resource in the following way:
0 2 4
1 3 5
SMP SMP SMP
The interplay between DistGrid and DELayout may at first seem complicated. The simple but important rule to understand is that DistGrid describes a domain decomposition and each domain is labeled with a DE number. The DELayout describes how these DEs are laid out over the compute resources of the VM, i.e. PETs. The DEs are purely logical elements of decomposition and may be relabeled to fit the algorithm or legacy code better. The following example demonstrates this by describing the exact same distribution of the domain data across the fictitious cluster of SMP-nodes with a different choice of DE labeling:
delayout = ESMF_DELayoutCreate(deCount=6, deGrouping=(/(mod(i,3),i=0,5)/), &
rc=rc)
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
regDecomp=(/2,3/), deLabelList=(/0,3,1,4,2,5/), delayout=delayout, rc=rc)
Here the deLabelList argument changes the default DE label sequence from column major to row major. The DELayout compensates for this change in DE labeling by changing the deGrouping argument to map the first dimension to SMP nodes as before. The decomposition and layout now looks as follows:
0 1 2
3 4 5
SMP SMP SMP
Finally, in order to achieve a completely user-defined distribution of the domain data across the PETs of the VM a DELayout may be created from a petMap before using it in the creation of a DistGrid. If for instance the desired distribution of a 2 x 3 decomposition puts the DEs of the first row onto 3 separate PETs (PET 0, 1, 2) and groups the DEs of the second row onto PET 3 a petMap must first be setup that takes the DE labeling of the DistGrid into account.The following lines of code result in the desired distribution using column major DE labeling by first create a DELayout and then using it in the DistGrid creation.
delayout = ESMF_DELayoutCreate(petMap=(/0,3,1,3,2,3/), rc=rc)
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
regDecomp=(/2,3/), delayout=delayout, rc=rc)
This decomposes the global domain into
0 2 4
1 3 5
and associates the DEs to the following PETs:
DE 0 -> PET 0
DE 1 -> PET 3
DE 2 -> PET 1
DE 3 -> PET 3
DE 4 -> PET 2
DE 5 -> PET 3
The examples of the previous sections showed how DistGrid objects with regular decompositions are created. However, in some cases a regular decomposition may not be specific enough. The following example shows how the deBlockList argument is used to create a DistGrid object with completely user-defined decomposition.
A single 5x5 LR domain is to be decomposed into 6 DEs. To this end a list is constructed that holds the min and max corners of all six DE LR blocks. The DE-local LR blocks are arranged as to cover the whole patch domain without overlap.
allocate(deBlockList(2, 2, 6)) ! (dimCount, 2, deCount) deBlockList(:,1,1) = (/1,1/) ! minIndex 1st deBlock deBlockList(:,2,1) = (/3,2/) ! maxIndex 1st deBlock deBlockList(:,1,2) = (/4,1/) ! minIndex 2nd deBlock deBlockList(:,2,2) = (/5,2/) ! maxIndex 2nd deBlock deBlockList(:,1,3) = (/1,3/) deBlockList(:,2,3) = (/2,4/) deBlockList(:,1,4) = (/3,3/) deBlockList(:,2,4) = (/5,4/) deBlockList(:,1,5) = (/1,5/) deBlockList(:,2,5) = (/3,5/) deBlockList(:,1,6) = (/4,5/) ! minIndex 6th deBlock deBlockList(:,2,6) = (/5,5/) ! maxInbex 6th deBlock
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
deBlockList=deBlockList, rc=rc)
By default the edges of all patches have solid wall boundary conditions. Periodic boundary conditions can be imposed by specifying connections between patches. For the single LR domain of the last section periodic boundaries along the first dimension are imposed by adding a connectionList argument with only one element to the create call.
Each connectionList element is a vector of (3 * dimCount + 2) integer numbers:
allocate(connectionList(3*2+2, 1)) ! (3*dimCount+2, number of connections)
and has the following format:
(/patchIndex_A, patchIndex_B, positionVector, orientationVector, repetitionVector/).
The following constructor call can be used to construct a suitable connectionList element.
call ESMF_DistGridConnection(connection=connectionList(:,1), &
patchIndexA=1, patchIndexB=1, &
positionVector=(/5, 0/), &
orientationVector=(/1, 2/), &
repetitionVector=(/1, 0/), rc=rc)
The patchIndexA and patchIndexB arguments specify that this is a connection within patch 1. The positionVector indicates that there is no offset between patchB and patchA along the second dimension, but there is an offset of 5 along the first dimension (which in this case is the length of dimension 1). This aligns patchB (which is patch 1) right next to patchA (which is also patch 1).
The orientationVector fixes the orientation of the patchB index space to be the same as the orientation of patchA (it maps index 1 of patchA to index 1 of patchB and the same for index 2). The orientationVector could have been omitted in this case which corresponds to the default orientation.
Finally, the repetitionVector idicates that this connetion element will be periodically repeated along dimension 1.
The connectionList can now be used to create a DistGrid object with the desired boundary conditions.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
deBlockList=deBlockList, connectionList=connectionList, rc=rc)
deallocate(connectionList)
This closes the patch along the first dimension on itself, thus imposing periodic boundaries along this direction.
Creating a DistGrid from a list of LR domains is a straight forward extension of the case with a single LR domain. The first four arguments of ESMF_DistGridCreate() are promoted to rank 2, the second dimension being the patch count index.
The following 2D patchwork domain consisting of 3 LR patches will be used in the examples of this section:
----------------------------------------> 2nd dim
|
| (1,11)-----(1,20)
| | |
| | |
| | |
| | |
| | |
| (10,11)---(10,20)
| (11,1)----(11,10)(11,11)---(11,20)
| | || |
| | || |
| | || |
| | || |
| | || |
| (20,1)----(20,10)(20,11)---(20,20)
|
|
v
1st dim
The first step in creating a patchwork global domain is to construct the minIndex and maxIndex arrays.
allocate(minIndex(2,3)) ! (dimCount, number of patches) allocate(maxIndex(2,3)) ! (dimCount, number of patches) minIndex(:,1) = (/11,1/) maxIndex(:,1) = (/20,10/) minIndex(:,2) = (/11,11/) maxIndex(:,2) = (/20,20/) minIndex(:,3) = (/1,11/) maxIndex(:,3) = (/10,20/)
Next the regular decomposition for each patch is set up in the regDecomp array. In this example each patch is associated with a single DE.
allocate(regDecomp(2,3)) ! (dimCount, number of patches) regDecomp(:,1) = (/1,1/) ! one DE regDecomp(:,2) = (/1,1/) ! one DE regDecomp(:,3) = (/1,1/) ! one DE
Finally the DistGrid can be created by calling
distgrid = ESMF_DistGridCreate(minIndex=minIndex, maxIndex=maxIndex, &
regDecomp=regDecomp, rc=rc)
The default DE labeling sequence is identical to the patch labeling sequence and follows the sequence in which the patches are defined during the create call. However, DE labels start at 0 whereas patch labels start at 1. In this case the DE labels look as:
2
0 1
Each patch can be decomposed differently into DEs. The default DE labeling follows the column major order for each patch. This is demonstrated in the following case where the patchwork global domain is decomposed into 9 DEs,
regDecomp(:,1) = (/2,2/) ! 4 DEs
regDecomp(:,2) = (/1,3/) ! 3 DEs
regDecomp(:,3) = (/2,1/) ! 2 DEs
distgrid = ESMF_DistGridCreate(minIndex=minIndex, maxIndex=maxIndex, &
regDecomp=regDecomp, rc=rc)
resulting in the following decomposition:
+-------+
| 7 |
| |
| 8 |
+-------+-------+
| 0 2 | |
| | 4 5 6 |
| 1 3 | |
+-------+-------+
DE 0 - (11,1) ~ (15,5)
DE 1 - (16,1) ~ (20,5)
DE 2 - (11,6) ~ (15,10)
DE 3 - (16,6) ~ (20,10)
DE 4 - (11,11) ~ (20,14)
DE 5 - (11,15) ~ (20,17)
DE 6 - (11,18) ~ (20,20)
DE 7 - (1,11) ~ (5,20)
DE 8 - (6,11) ~ (10,20)
The decompflag and deLabelList arguments can be used much like in the single LR domain case to overwrite the default grid decomposition (per patch) and to change the overall DE labeling sequence, respectively.
This section will be updated as the implementation of the DistGrid class nears completion.
INTERFACE:
! Private name; call using ESMF_DistGridCreate()
function ESMF_DistGridCreateRD(minIndex, maxIndex, regDecomp, &
decompflag, deLabelList, indexflag, connectionList, connectionTransList, &
delayout, vm, rc)
ARGUMENTS:
integer, intent(in) :: minIndex(:)
integer, intent(in) :: maxIndex(:)
integer, target, intent(in), optional :: regDecomp(:)
type(ESMF_DecompFlag), target,intent(in), optional :: decompflag(:)
integer, target, intent(in), optional :: deLabelList(:)
type(ESMF_IndexFlag), intent(in), optional :: indexflag
integer, target, intent(in), optional :: connectionList(:,:)
integer, target, intent(in), optional :: connectionTransList(:,:)
type(ESMF_DELayout), intent(in), optional :: delayout
type(ESMF_VM), intent(in), optional :: vm
integer, intent(out),optional :: rc
RETURN VALUE:
type(ESMF_DistGrid) :: ESMF_DistGridCreateRDDESCRIPTION:
Create an ESMF_DistGrid from a single logically rectangular (LR) patch with regular decomposition. A regular decomposition is of the same rank as the patch and decomposes each dimension into a fixed number of DEs. A regular decomposition of a single patch is expressed by a single regDecomp list of DE counts in each dimension.
The arguments are:
. The value of
deCount for a default DELayout equals petCount, i.e. the
default decomposition will be into as many DEs as there are
PETs and the distribution will be 1 DE per PET.