Thread-Safe Implementation Details
As mentioned in Sect. 3, in a single threaded application, each PE has its own CDM and a corresponding CQ per CDM. In a multithreaded application, each registered thread gets its own CDM and a corresponding CQ. Only registerd threads can make OpenSHMEM calls.
If we over allocate the number of resources and register more threads than the number of CDMs per node, one or more threads are forced to share a CDM. For each thread, the cdm_handle that is uses to refer to its CDM is stored in Thread Local Storage (TLS). It is referenced during every communication operation from that thread.
The shmemx_thread_quiet routine for a particular thread waits for completion of all events in the associated CQ. In an over allocation scenario, if multiple threads share the same CQ, the shmemx_thread_quiet routine waits for completion of all events from all threads that share the CQ. This is because of the 1-to-1 relation between CDM and CQ.
Figure 5 shows the basic relation between threads and DMAPP network resources. In Fig. 5, we have three threads in PE and among them two are registered. These registered threads have their own CDMs with an allocated CQ. All events from these registered threads are tracked with their allocated CQs.