In
computer scienceComputer science is the study of the theoretical foundations of information and computation, and of practical techniques for their implementation and application in computer systems. It is frequently described as the systematic study of algorithmic processes that create, describe and transform...
, a
parallel algorithm, as opposed to a traditional
sequential (or serial) algorithmSequential algorithm - in general, any algorithm executed sequentially, but, specifically, one for decoding a convolutional code....
, is an
algorithmIn mathematics, computing, linguistics, and related subjects, an algorithm is an effective method for solving a problem using a finite sequence of instructions. Algorithms are used for calculation, data processing, and many other fields....
which can be executed a piece at a time on many different processing devices, and then put back together again at the end to get the correct result.
Some algorithms are easy to divide up into pieces like this. For example, splitting up the job of checking all of the numbers from one to a hundred thousand to see which are
primesIn mathematics, a prime number is a natural number which has exactly two distinct natural number divisors: 1 and itself. The first twenty-six prime numbers are:An infinitude of prime numbers exists, as demonstrated by Euclid around 300 BC. The number 1 is by definition not a prime number...
could be done by assigning a subset of the numbers to each available processor, and then putting the list of positive results back together.
Most of the available algorithms to compute
piPi or π is a mathematical constant whose value is the ratio of any circle's circumference to its diameter in Euclidean space; this is the same value as the ratio of a circle's area to the square of its radius. The symbol π was first proposed by the Welsh mathematician William Jones in 1706...
(π), on the other hand, cannot be easily split up into parallel portions.
In
computer scienceComputer science is the study of the theoretical foundations of information and computation, and of practical techniques for their implementation and application in computer systems. It is frequently described as the systematic study of algorithmic processes that create, describe and transform...
, a
parallel algorithm, as opposed to a traditional
sequential (or serial) algorithmSequential algorithm - in general, any algorithm executed sequentially, but, specifically, one for decoding a convolutional code....
, is an
algorithmIn mathematics, computing, linguistics, and related subjects, an algorithm is an effective method for solving a problem using a finite sequence of instructions. Algorithms are used for calculation, data processing, and many other fields....
which can be executed a piece at a time on many different processing devices, and then put back together again at the end to get the correct result.
Some algorithms are easy to divide up into pieces like this. For example, splitting up the job of checking all of the numbers from one to a hundred thousand to see which are
primesIn mathematics, a prime number is a natural number which has exactly two distinct natural number divisors: 1 and itself. The first twenty-six prime numbers are:An infinitude of prime numbers exists, as demonstrated by Euclid around 300 BC. The number 1 is by definition not a prime number...
could be done by assigning a subset of the numbers to each available processor, and then putting the list of positive results back together.
Most of the available algorithms to compute
piPi or π is a mathematical constant whose value is the ratio of any circle's circumference to its diameter in Euclidean space; this is the same value as the ratio of a circle's area to the square of its radius. The symbol π was first proposed by the Welsh mathematician William Jones in 1706...
(π), on the other hand, cannot be easily split up into parallel portions. They require the results from a preceding step to effectively carry on with the next step. Such problems are called inherently serial problems. Iterative
numerical methodsNumerical analysis is the study of algorithms for the problems of continuous mathematics .One of the earliest mathematical writings is the Babylonian tablet YBC 7289, which gives a sexagesimal numerical approximation of , the length of the diagonal in a unit square.Being able to compute the sides...
, such as
Newton's methodIn numerical analysis, Newton's method , named after Isaac Newton and Joseph Raphson, is perhaps the best known method for finding successively better approximations to the zeroes of a real-valued function...
or the
three-body problemThree-body problem has two distinguishable meanings:# In an extended modern sense, a three-body problem is a class of problems in classical or quantum mechanics that model the motion of three particles...
, are also algorithms which are inherently serial. Some problems are very difficult to parallelize, although they are recursive. One such example is the
depth-first searchDepth-first search is an algorithm for traversing or searching a tree, tree structure, or graph. One starts at the root and explores as far as possible along each branch before backtracking....
of
graphsIn computer science, a graph is an abstract data structure that is meant to implement the graph concept from mathematics.A graph data structure consists mainly of a finite set of ordered pairs, called edges or arcs, of certain entities called nodes or vertices...
.
Parallel algorithms are valuable because of substantial improvements in
multiprocessingMultiprocessing is the use of two or more central processing units within a single computer system. The term also refers to the ability of a system to support more than one processor and/or the ability to allocate tasks between them...
systems and the rise of multi-core processors. In general, it is easier to construct a computer with a single fast processor than one with many slow processors with the same
throughputIn communication networks, such as Ethernet or packet radio, throughput or network throughput is the average rate of successful message delivery over a communication channel. These data may be delivered over a physical or logical link, or pass through a certain network node...
. But processor speed is increased primarily by shrinking the circuitry, and modern processors are pushing physical size and heat limits. These twin barriers have flipped the equation, making multiprocessing practical even for small systems.
The cost or complexity of serial algorithms is estimated in terms of the space (memory) and time (processor cycles) that they take. Parallel algorithms need to optimize one more resource, the communication between different processors. There are two ways parallel processors communicate, shared memory or message passing.
Shared memoryIn computing, shared memory is memory that may be simultaneously accessed by multiple programs with an intent to provide communication among them or avoid redundant copies. Depending on context, programs may run on a single processor or on multiple separate processors...
processing needs additional
lockingIn computer science, a lock is a synchronization mechanism for enforcing limits on access to a resource in an environment where there are many threads of execution. Locks are one way of enforcing concurrency control policies.-Types:...
for the data, imposes the overhead of additional processor and bus cycles, and also serializes some portion of the algorithm.
Message passingMessage passing in computer science, is a form of communication used in parallel computing, object-oriented programming, and interprocess communication.- Overview :...
processing uses channels and message boxes but this communication adds transfer overhead on the bus, additional memory need for queues and message boxes and latency in the messages. Designs of parallel processors use special buses like crossbar so that the communication overhead will be small but it is the parallel algorithm that decides the volume of the traffic.
Another problem with parallel algorithms is ensuring that they are suitably
load balancedLoad balancing or load distribution may refer to:*Load balancing , balancing a workload amongst multiple computer devices*Load balancing , the storing of excess electrical power by power stations during low demand periods, for release as demand rises*Weight distribution, the apportioning of weight...
. For example, checking all numbers from one to a hundred thousand for primality is easy to split amongst processors; however, some processors will get more work to do than the others, which will sit idle until the loaded processors complete.
A subtype of parallel algorithms,
distributed algorithmsA distributed algorithm is an algorithm designed to run on computer hardware constructed from interconnected processors. Distributed algorithms are used in many varied application areas of distributed computing, such as telecommunications, scientific computing, distributed information processing,...
are algorithms designed to work in
cluster computingCluster Computing: the Journal of Networks, Software Tools and Applications is a journal for parallel processing, distributed computing systems, and computer communication networks....
and
distributed computingDistributed computing is a field of computer science that studies distributed systems. A distributed system consists of multiple autonomous computers that communicate through a computer network. The computers interact with each other in order to achieve a common goal...
environments, where additional concerns beyond the scope of "classical" parallel algorithms need to be addressed.
External links