Kinds of Parallel Programming
Parallel programming has many uses, but a fully parallel program is not always feasible.
We'll cover the following...
There are many kinds of parallel programming, some are general and can be run on any hardware while others are specific to particular hardware architectures.
Two main paradigms we can talk about here are shared memory and distributed memory models. In shared memory models, multiple processing units have access to the same shared memory space. This is the case on our desktop or laptop with multiple CPU cores. In a distributed memory model, multiple processing units each have their own memory store and information is passed between them. This is the model that a networked cluster of computers operates with. We won’t be talking about clusters here, but some of the tools we’ll talk about (e.g. MPI) are easily used with clusters.
Types of parallel tasks
Broadly speaking we can separate a computation into two camps depending on how it can be parallelized. A so-called embarrassingly parallel problem is one for which ...