Implementation Details

In this Section, we provide basic implementation details of the Thread-Safe and Contexts-Domains extensions in Cray SHMEM.


The Cray DMAPP (Distributed Shared Memory Application) library is a low- level communication layer developed by Cray Inc. to provide support for logically shared, distributed memory programming models. It is designed to deliver the full hardware performance of the current Cray networks. DMAPP provides support for Remote Memory Access (RMA) between processes within a job in an one-sided manner. Cray’s implementation of OpenSHMEM and Partitioned Global Address Space (PGAS) compilers, such as Coarray Fortran [14] and Unified Parallel C (UPC) [7], are implemented on top of DMAPP.

All PUT, GET, and AMO communication operations are called Events. There are two types of Remote Memory Access mechanisms: Fast Memory Access (FMA), and Block Transfer Engine (BTE). FMA and BTE mechanisms initiate Put and Get events with FMA handling data transfers for small data sizes and BTE for large data sizes. FMA mechanism is also used to initiate AMO events.

For each process in a Cray SHMEM based application, DMAPP creates a local instance of a Communication Domain (CDM) and it obtains an identifier called an cdm_handle, for future reference to this CDM. In order to use the FMA and BTE mechanisms we configure the FMA and BTE descriptors and attach them to the CDM.

All communication operations on that PE are done through this CDM. DMAPP does not support shared CDMs and each PE should have a unique CDM. The number of FMA descriptors per node is limited. This is a hardware limitation. For example, on Cray-developed Aries [2] RDMA interconnect the number of FMA descriptors per node is fixed to 120.

The event notification mechanism is through Completion Queues (CQ). The event notification mechanism is the process to track the progress of communication operations. Each CDM has its own CQ and all events on that particular CDM are tracked through this CQ. By DMAPP design, there is 1-to-1 relation between CDM and FMA descriptors, as well as with CDM and CQ.

< Prev   CONTENTS   Source   Next >