Short Data Transfers
The functions uct-ugni-epjputshort and uct-ugni-ep-put-bcopy are the interfaces for small and medium data transfers. They use uGNI’s FMA interface, which is optimized for RDMA small data transfers as it copies the data from the source buffer to library buffers. The function uct_ugnLep_put_short returns after posting the FMA operation, and completion is learnt as required by the upper layer. For example, if this interface is used for implementing the shmem_put operation of the OpenSHMEM programming model, the OpenSHMEM layer will wait for global completion event by progressing the worker and flushing the endpoint. On the other hand, if this interface is used for shmem_put_nbi, the OpenSHMEM layer will return as soon as the operation is posted and waits for completion during the shmem_quiet or shmem_barrier calls. Thus, the interfaces along with the abstraction of endpoints, workers, and interfaces enable implementing various semantics.
Large Data Transfers
The function uct-ugnLep_puEzcopy implements the interface for large data transfers. It uses uGNI’s BTE interface, GNEPostRdrna(), with post type GNI_POST_ RDMAPUT or GNI_POST_RDMA_GET. The interface is optimized for RDMA large data transfers as it copies data from the source buffers to the network using DMA engines without CPU intervention. The interface has semantics similar to the short data transfer interfaces.
The mapping onto uGNI’s interface is straightforward when the data is 4- byte aligned and the data length is a multiple of 4-bytes. However, because of BTE’s Get interface’s constraints on the message length and alignment, a protocol was developed for unaligned messages. For example, when the data length is not a multiple of 4-bytes, we pad the data and transfer the data using the appropriate protocols as shown in Fig. 3.