Message Passing

In a parallel environment, some form of inter-processor communications is required in order to exchange data and information between processors and to provide synchronization of the processors. Generally, there are two main approaches to inter-processor communication. One of them is called message passing between processors and its alternative is to use shared-address space where variables are kept in a common pool therefore they are globally available to all processors. Each paradigm has its own advantages and disadvantages.

In the shared-address space approach, the variables are globally accessible by all processors. Despite multiple processors operating independently, they share the same memory resources. Only one processor can access the shared memory location at a time. Thus, accessing the memory should be provided in a mutually exclusive fashion since accesses to the same variable at the same time by multiple processors might lead to inconsistent data. Shared-address space approach makes it simpler for the user to achieve parallelism but since the memory bandwidth is limited, severe bottlenecks are unavoidable with the increasing number of processors. Also, the user is responsible for providing the synchronization constructs in order to provide concurrent accesses.

In the message passing approach, there are independent cooperating processes (or processors). Each processor has a private local memory in order to keep the variables and data. If an exchange of the information is needed between the processors, the processors communicate and synchronize by passing messages which are simple send and receive instructions. With this method, each processor can access its own memory very rapidly. But users have to send and receive data among processors. The message passing paradigm is usually provided with the library extensions added to the sequential programming languages. PVM([9]), MPI([5]), P4([8]) are the most common message passing libraries and programs.

PVM refers to Parallel Virtual Machine, which is a software package that allows a programmer to create and access a parallel computing system. The components of such a system are the machines connected through the network(s). These machines might be in the same network as well as separated through the internet. Also,they may be homogenenous or heterogeneous in terms of the operating system running on those hosts. The idea is to bring together a variety of architectures under a centralized control. Thus a PVM user divides a problem into subtasks and assigns each subtask to one processor in the system.

PVM is based on the parallel message-passing model. Messages are exchanged between tasks via the connecting networks. If the communication is done between two different types of machines that do not have a common representation for the data, then data conversion is done automatically. Initialization and termination of a process are the user's responsibilities. The user should also use standard interface routines defined in PVM in order to exchange data and to synchronize with the other processes.

MPI stands for Message Passing Interface. It provides a standard for writing message-passing programs. It was designed for high performance on both massively parallel machines and on workstation clusters. It provides more than 100 functions as a library. It also defines an interface for Fortran and C. There are a couple of implementations of MPI on different architectures/systems. It also support heteregenous computing as PVM does. A comparison of MPI and PVM can be found in [3].

There are several other libraries in the literature and they have more or less the same procedures and usage. Some of them are commercial products but one can find free available libraries (such as PVM) too.