Computing is usually defined as the activity of using and developing computer technology, computer hardware and computer software. It is the computer-specific part of information technology.... , optimization is the process of modifying a system to make some aspect of it work more efficiently or use fewer resources. For instance, a computer program
Computer program
Computer programs are Instruction for a computer. A computer requires programs to function. Moreover, a computer program does not run unless its instructions are executed by a Central processing unit; however, a program may communicate an Algorithm#Formalization of algorithms to people without running.... may be optimized so that it executes more rapidly, or is capable of operating with less memory storage or other resources, or draw less power. The system may be a single computer program
Computer program
Computer programs are Instruction for a computer. A computer requires programs to function. Moreover, a computer program does not run unless its instructions are executed by a Central processing unit; however, a program may communicate an Algorithm#Formalization of algorithms to people without running.... , a collection of computer
Computer
A computer is a machine that manipulates Data according to a list of Code .The first devices that resemble modern computers date to the mid-20th century , although the computer concept and various machines similar to computers existed earlier.... s or even an entire network such as the Internet
Internet
The Internet is a global network of interconnected computers, enabling users to share information along multiple channels. Typically, a computer that connects to the Internet can access information from a vast array of available server and other computers by moving information from them to the computer's local memory.... . See also algorithmic efficiency
Algorithmic efficiency
In computer science, efficiency is used to describe properties of an algorithm relating to how much of various types of resources it consumes. The two most frequently encountered are... for further discussion on factors relating to improving the efficiency of an algorithm.
ough the word "optimization" shares the same root as "optimal," it is rare for the process of optimization to produce a truly optimal system.
Discussion
Ask a question about 'Optimization (computer science)'
Start a new discussion about 'Optimization (computer science)'
Computing is usually defined as the activity of using and developing computer technology, computer hardware and computer software. It is the computer-specific part of information technology.... , optimization is the process of modifying a system to make some aspect of it work more efficiently or use fewer resources. For instance, a computer program
Computer program
Computer programs are Instruction for a computer. A computer requires programs to function. Moreover, a computer program does not run unless its instructions are executed by a Central processing unit; however, a program may communicate an Algorithm#Formalization of algorithms to people without running.... may be optimized so that it executes more rapidly, or is capable of operating with less memory storage or other resources, or draw less power. The system may be a single computer program
Computer program
Computer programs are Instruction for a computer. A computer requires programs to function. Moreover, a computer program does not run unless its instructions are executed by a Central processing unit; however, a program may communicate an Algorithm#Formalization of algorithms to people without running.... , a collection of computer
Computer
A computer is a machine that manipulates Data according to a list of Code .The first devices that resemble modern computers date to the mid-20th century , although the computer concept and various machines similar to computers existed earlier.... s or even an entire network such as the Internet
Internet
The Internet is a global network of interconnected computers, enabling users to share information along multiple channels. Typically, a computer that connects to the Internet can access information from a vast array of available server and other computers by moving information from them to the computer's local memory.... . See also algorithmic efficiency
Algorithmic efficiency
In computer science, efficiency is used to describe properties of an algorithm relating to how much of various types of resources it consumes. The two most frequently encountered are... for further discussion on factors relating to improving the efficiency of an algorithm.
General
Although the word "optimization" shares the same root as "optimal," it is rare for the process of optimization to produce a truly optimal system. The optimized system will typically only be optimal in one application or for one audience. One might reduce the amount of time that a program takes to perform some task at the price of making it consume more memory. In an application where memory space is at a premium, one might deliberately choose a slower algorithm
Algorithm
In mathematics, computing, linguistics and related subjects, an algorithm is a sequence of finite instructions, often used for calculation and data processing.... in order to use less memory. Often there is no “one size fits all” design which works well in all cases, so engineer
Engineer
An engineer is a person professionally engaged in a field of engineering. Engineers are concerned with developing economical and safe solutions to practical problems, by applying mathematics and scientific knowledge while considering technical constraints.... s make trade-off
Trade-off
A trade-off is a situation that involves losing one quality or aspect of something in return for gaining another quality or aspect. It implies a decision to be made with full comprehension of both the upside and downside of a particular choice.... s to optimize the attributes of greatest interest. Additionally, the effort required to make a piece of software completely optimal—incapable of any further improvement— is almost always more than is reasonable for the benefits that would be accrued; so the process of optimization may be halted before a completely optimal solution has been reached. Fortunately, it is often the case that the greatest improvements come early in the process.
'Levels' of optimization
Optimization can occur at a number of 'levels':
Design level
At the highest level, the design may be optimized to make best use of the available resources. The implementation of this design will benefit from a good choice of efficient algorithms
Algorithmic efficiency
In computer science, efficiency is used to describe properties of an algorithm relating to how much of various types of resources it consumes. The two most frequently encountered are... and the implementation of these algorithms will benefit from writing good quality code. The architectural design of a system overwhelmingly affects its performance. The choice of algorithm
Algorithm
In mathematics, computing, linguistics and related subjects, an algorithm is a sequence of finite instructions, often used for calculation and data processing.... affects efficiency more than any other item of the design.
In some cases, however, optimization relies on using fancier algorithms, making use of special cases and special tricks and performing complex trade-offs; thus, a fully optimized program can sometimes, if insufficiently commented, be more difficult for less experienced programmers to comprehend and hence may contain more faults
Software bug
A software bug is an error, flaw, mistake, failure, or fault in a computer program that prevents it from behaving as intended . Most bugs arise from mistakes and errors made by people in either a program's source code or its software architecture, and a few are caused by compilers producing incorrect code.... than unoptimized versions.
Source code level
Avoiding bad quality coding can also improve performance, by avoiding obvious slowdowns. After that, however, some optimizations are possible which actually decrease maintainability; some, but not all of them can nowadays be performed by optimizing compilers. For instance, using more indirection is often needed to simplify or improve a software, but that indirection has a cost.
Compile level
Use of an optimizing compiler tends to ensure that the executable program is optimized at least as much as the compiler can predict.
An assembly language is a low-level language for programming computers. It implements a symbolic representation of the numeric machine codes and other constants needed to program a particular CPU architecture.... designed for a particular hardware
Hardware
Hardware is a general term that refers to the physical cultural artifacts of a technology. It may also mean the physical components of a computer system, in the form of computer hardware.... platform will normally produce the most efficient code since the programmer can take advantage of the full repertoire of machine instructions. The operating system
Operating system
An operating system is an interface between hardware and applications; it is responsible for the management and coordination of activities and the sharing of the limited resources of the computer.... s of most machines have been traditionally written in assembler code for this reason.
With more modern optimizing compilers and the greater complexity of recent CPUs, it is more difficult to write code that is optimized better than the compiler itself generates, and few projects need resort to this 'ultimate' optimization step.
However, a large amount of code written today is still compiled with the intent to run on the greatest percentage of machines possible. As a consequence, programmers and compilers don't always take advantage of the more efficient instructions provided by newer CPUs or quirks of older models. Additionally, assembly code tuned for a particular processor without using such instructions might still be suboptimal on a different processor, expecting a different tuning of the code.
Just-in-time is an inventory strategy implemented to improve the return on investment of a business by reducing in-process inventory and its associated carrying costs.... compilers and Assembler programmers may be able to perform run time
Run time
Run time or runtime may refer to:*Run time, the length of a film or television program in minutes, usually with end credits included*Runtime, in computer science, the duration of a computer program's execution, from beginning to termination... optimization exceeding the capability of static compilers by dynamically adjusting parameters according to the actual input or other factors.
Platform dependent and independent optimizations
Code optimization can be also broadly categorized as platform dependent and platform independent techniques; while the latter ones are effective of most or all platforms,
platform dependent techniques use specific properties of one platform, or rely on parameters depending on the single platform or even on the single processor; writing or producing different versions of the same code for different processors might be thus needed.
For instance, in the case of compile-level optimization, platform independent techniques are generic techniques such as loop unrolling, reduction in function calls, memory efficient routines, reduction in conditions, etc., that impact most CPU architectures in a similar way. Generally, these serve to reduce the total Instruction path length
Instruction path length
In computer performance, the Instruction path length is the number of machine code instructions required to execute a section of a computer program.... required to complete the program and/or reduce total memory usage during the process. On the other side, platform dependent techniques involve instruction scheduling, instruction level parallelism, data level parallelism, cache optimization techniques, i.e. parameters that differ among various platforms; the optimal instruction scheduling might be different even on different processors of the same architecture.
Different algorithms
Computational tasks can be performed in several different ways with varying efficiency. For example, consider the following C
C (programming language)
C is a general-purpose computer programming language originally developed in 1972 by Dennis Ritchie at the Bell Telephone Laboratories to implement the Unix operating system.... code snippet whose intention is to obtain the sum of all integers from 1 to N:
int i, sum = 0;
for (i = 1; i <= N; i++)
sum += i;
printf ("sum: %d\n", sum);
The term arithmetic overflow or simply overflow has the following meanings.# In a digital computer, the condition that occurs when a calculation produces a result that is greater in magnitude than what a given processor register or Computer storage location can store or represent.... ) be rewritten using a mathematical formula like:
int sum = (N * (N+1)) >> 1; // >>1 is bit right shift by 1, which is
// equivalent to divide by 2 when N is
// non-negative
printf ("sum: %d\n", sum);
The optimization, sometimes performed automatically by an optimizing compiler, is to select a method (algorithm
Algorithm
In mathematics, computing, linguistics and related subjects, an algorithm is a sequence of finite instructions, often used for calculation and data processing.... ) that is more computationally efficient while retaining the same functionality. See Algorithmic efficiency
Algorithmic efficiency
In computer science, efficiency is used to describe properties of an algorithm relating to how much of various types of resources it consumes. The two most frequently encountered are... for a discussion of some of these techniques. However, a significant improvement in performance can often be achieved by solving only the actual problem and removing extraneous functionality.
Optimization is not always an obvious or intuitive process. In the example above, the ‘optimized’ version might actually be slower than the original version if N were sufficiently small and the particular hardware happens to be much faster at performing addition and looping operations than multiplication and division.
Trade-offs
Optimization will generally focus on improving just one or two aspects of performance: execution time, memory usage, disk space, bandwidth, power consumption or some other resource. This will usually require a trade-off - where one factor is optimized at the expense of others. For example, increasing the size of cache
Cache
In computer science, a cache is a collection of data duplicating original values stored elsewhere or computed earlier, where the original data is expensive to fetch or to compute, compared to the cost of reading the cache.... improves runtime performance, but also increases the memory consumption. Other common trade-offs include code clarity and conciseness.
There are instances where the programmer performing the optimization must decide to make the software more optimal for some operations but at the cost of making other operations less efficient. These trade-offs may sometimes be of a non-technical nature - such as when a competitor has published a benchmark
Benchmark
The term benchmark originates from the chiseled horizontal marks that surveyors made, into which an angle-iron could be placed to bracket a levelling rod, thus ensuring that the levelling rod can be repositioned in exactly the same place in the future.... result that must be beaten in order to improve commercial success but comes perhaps with the burden of making normal usage of the software less efficient. Such changes are sometimes jokingly referred to as pessimizations.
Bottlenecks
Optimization may include finding a bottleneck, a critical part of the code that is the primary consumer of the needed resource - sometimes known as a hot spot
Hotspot
A hot spot is a region of high or special activity within a larger area of low or normal activity. It may refer to:... . As a rule of thumb, improving 20% of the code is responsible for 80% of the results.
The Pareto principle states that, for many events, roughly 80% of the effects come from 20% of the causes.Business management thinker Dr. Joseph Moses Juran suggested the principle and named it after Italian economist Vilfredo Pareto, who observed that 80% of the land in Italy was owned by 20% of the population.... can be applied to resource optimization by observing that 80% of the resources are typically used by 20% of the operations. In software engineering, it is often a better approximation that 90% of the execution time of a computer program is spent executing 10% of the code (known as the 90/10 law in this context).
More complex algorithms and data structures perform well with many items, while simple algorithms are more suitable for small amounts of data—the setup, initialization time, and constant factors of the more complex algorithm can outweigh the benefit.
In some cases, adding more memory can help to make a program run faster. For example, a filtering program will commonly read each line and filter and output that line immediately. This only uses enough memory for one line, but performance is typically poor. Performance can be greatly improved by reading the entire file then writing the filtered result, though this uses much more memory. Caching the result is similarly effective, though also requiring larger memory use.
In writing and typography Readability is defined as reading ease, especially as it results from a writing style. Extensive research has shown that easy-reading text improves comprehension, retention, reading speed, and reading persistence.... and add code that is used only to improve the performance
Computer performance
Computer performance is characterized by the amount of useful work accomplished by a computer system compared to the time and resources used.Depending on the context, good computer performance may involve one or more of the following:... . This may complicate programs or systems, making them harder to maintain and debug. As a result, optimization or performance tuning is often performed at the end of the development stage
Development stage
A software release is the distribution of an initial or upgraded Software versioning of a computer software product. The software engineering and company doing the work decide on how to distribute the program or system, or changes to that program or system.... .
Donald Ervin Knuth is a renowned computer science and Emeritus of the Art of Computer Programming at Stanford University.Author of the seminal multi-volume work The Art of Computer Programming , Knuth has been called the "father" of the run-time analysis, contributing to the development of, and systematizing formal mathematical techn... made the following statement on optimization:
"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil."
(He also attributed the quote to Tony Hoare several years later, although this might have been an error as Hoare disclaims having coined the phrase.)
"Premature optimization" is a phrase used to describe a situation where a programmer lets performance considerations affect the design of a piece of code. This can result in a design that is not as clean as it could have been or code that is incorrect, because the code is complicated by the optimization and the programmer is distracted by optimizing.
An alternative approach is to design first, code from the design and then profile
Performance analysis
In software engineering, performance analysis, more commonly today known as profiling, is the investigation of a program's behavior using information gathered as the program executes .... /benchmark
Benchmark
The term benchmark originates from the chiseled horizontal marks that surveyors made, into which an angle-iron could be placed to bracket a levelling rod, thus ensuring that the levelling rod can be repositioned in exactly the same place in the future.... the resulting code to see which parts should be optimized. A simple and elegant design is often easier to optimize at this stage, and profiling may reveal unexpected performance problems that would not have been addressed by premature optimization.
In practice, it is often necessary to keep performance goals in mind when first designing software, but the programmer balances the goals of design and optimization.
Macros
Optimization during code development using macros takes on different forms in different languages.
C is a general-purpose computer programming language originally developed in 1972 by Dennis Ritchie at the Bell Telephone Laboratories to implement the Unix operating system.... and C++
C++
C++ is a general-purpose programming language. It is regarded as a middle-level language, as it comprises a combination of both high-level programming language and low-level programming language language features.... , macros are implemented using token substitution. Nowadays, inline functions can be used as a type safe alternative in many cases. In both cases, the inlined function body can then undergo further compile-time optimizations by the compiler, including constant folding, which may move some computations to compile time.
In computer science, functional programming is a programming paradigm that treats computation as the evaluation of function s and avoids program state and immutable object data.... languages macros are implemented using parse-time substitution of parse trees/abstract syntax trees, which it is claimed makes them safer to use. Since in many cases interpretation is used, that is one way to ensure that such computations are only performed at parse-time, and sometimes the only way.
Lisp is a family of computer programming languages with a long history and a distinctive, fully parenthesized syntax. Originally specified in 1958, Lisp is the second-oldest high-level programming language in widespread use today; only Fortran is older.... originated this style of macro , and such macros are often called “Lisp-like macros.” A similar effect can be achieved by using Template metaprogramming
Template metaprogramming
Template metaprogramming is a metaprogramming technique in which Generic programming are used by a compiler to generate temporary source code, which is merged by the compiler with the rest of the source code and then compiled.... in C++
C++
C++ is a general-purpose programming language. It is regarded as a middle-level language, as it comprises a combination of both high-level programming language and low-level programming language language features.... .
In both cases, work is moved to compile-time. The difference between C macros on one side, and Lisp-like macros and C++ Template metaprogramming
Template metaprogramming
Template metaprogramming is a metaprogramming technique in which Generic programming are used by a compiler to generate temporary source code, which is merged by the compiler with the rest of the source code and then compiled.... on the other side, is that the latter tools allow performing arbitrary computations at compile-time/parse-time, while expansion of C macros does not perform any computation, and relies on the optimizer ability to perform it. Additionally, C macros do not directly support recursion nor iteration, so are not Turing complete.
As with any optimization, however, it is often difficult to predict where such tools will have the most impact before a project is complete.
Automated and manual optimization
See also :Category:Compiler optimizations
Optimization can be automated by compilers or performed by programmers. Gains are usually limited for local optimization, and larger for global optimizations. Usually, the most powerful optimization is to find a superior algorithm
Algorithm
In mathematics, computing, linguistics and related subjects, an algorithm is a sequence of finite instructions, often used for calculation and data processing.... .
Optimizing a whole system is usually undertaken by programmers because it is too complex for automated optimizers. In this situation, programmers or system administrators explicitly change code so that the overall system performs better. Although it can produce better efficiency, it is far more expensive than automated optimizations.
In software engineering, performance analysis, more commonly today known as profiling, is the investigation of a program's behavior using information gathered as the program executes .... ) to find the sections of the program that are taking the most resources — the bottleneck. Programmers sometimes believe they have a clear idea of where the bottleneck is, but intuition is frequently wrong. Optimizing an unimportant piece of code will typically do little to help the overall performance.
When the bottleneck is localized, optimization usually starts with a rethinking of the algorithm used in the program: more often than not, a particular algorithm can be specifically tailored to a particular problem, yielding better performance than a generic algorithm. For example, the task of sorting a huge list of items is usually done with a quicksort
Quicksort
Quicksort is a well-known sorting algorithm developed by C. A. R. Hoare that, average performance, makes comparisons to sort n items. However, in the Best, worst and average case, it makes comparisons.... routine, which is one of the most efficient generic algorithms. But if some characteristic of the items is exploitable (for example, they are already arranged in some particular order), a different method can be used, or even a custom-made sort routine.
After one is reasonably sure that the best algorithm is selected, code optimization can start: loops can be unrolled (for lower loop overhead, although this can often lead to lower speed if it overloads the CPU cache
CPU cache
A CPU cache is a cache used by the central processing unit of a computer to reduce the average time to access computer storage. The cache is a smaller, faster memory which stores copies of the data from the most frequently used main memory locations.... ), data types as small as possible can be used, integer arithmetic can be used instead of floating-point, and so on.
Performance bottlenecks can be due to language limitations rather than algorithms or data structures used in the program. Sometimes, a critical part of the program can be re-written in a different programming language
Programming language
A programming language is a machine-readable artificial language designed to express computations that can be performed by a machine, particularly a computer.... that gives more direct access to the underlying machine. For example, it is common for very high-level
High-level programming language
In computing, a high-level programming language is a programming language with strong Abstraction from the details of the computer. In comparison to low-level programming languages, it may use natural language elements, be easier to use, or more Porting across platforms.... languages like Python
Python (programming language)
Python is a general-purpose high-level programming language. Its design philosophy emphasizes code readability. Python's core syntax and semantics are Minimalism , while the standard library is large and comprehensive.... to have modules written in C
C (programming language)
C is a general-purpose computer programming language originally developed in 1972 by Dennis Ritchie at the Bell Telephone Laboratories to implement the Unix operating system.... for greater speed. Programs already written in C can have modules written in assembly
Assembly language
An assembly language is a low-level language for programming computers. It implements a symbolic representation of the numeric machine codes and other constants needed to program a particular CPU architecture.... . Programs written in D can use the inline assembler
Inline assembler
In computer programming, the inline assembler is a feature of some compilers that allows very low level code written in Assembly Language to be embedded in a high level language like C or Ada programming language.... .
Rewriting pays off because of a general rule known as the 90/10 law, which states that 90% of the time is spent in 10% of the code, and only 10% of the time in the remaining 90% of the code. So putting intellectual effort into optimizing just a small part of the program can have a huge effect on the overall speed if the correct part(s) can be located.
Manual optimization often has the side-effect of undermining readability. Thus code optimizations should be carefully documented and their effect on future development evaluated.
The program that does the automated optimization is called an optimizer. Most optimizers are embedded in compilers and operate during compilation. Optimizers can often tailor the generated code to specific processors.
Today, automated optimizations are almost exclusively limited to compiler optimization
Compiler optimization
Compiler optimization is the process of tuning the output of a compiler to minimize or maximize some attribute of an executable computer program.... .
Eiffel is an International Organization for Standardization-standardized, object-oriented programming language designed to enable programmers to efficiently develop extensible, reusable, reliable software.... , Esterel
Esterel
Esterel is a synchronous programming language programming language for the development of complex reactive systems. The imperative programming style of Esterel allows the simple expression of parallel computing and preemption .... ) optimize their programs by using an intermediate language
Intermediate language
In computer science, an intermediate language is the language of an abstract machine designed to aid in the analysis of computer programs. The term comes from their use in compilers, where a compiler first translates the source code of a program into a form more suitable for code-improving transformations, as an intermediate step before gener... .
Grid computing is the application of several computers to a single problem at the same time -- usually to a scientific or technical problem that requires a great number of computer processing cycles or access to large amounts of data.... or distributed computing
Distributed computing
Distributed computing deals with hardware and software systems containing more than one processing element or Computer data storage element, Concurrent computing processes, or multiple programs, running under a loosely or tightly controlled regime.... aims to optimize the whole system, by moving tasks from computers with high usage to computers with idle time.
Time taken for optimization
Sometimes, the time taken to undertake optimization in itself may be an issue.
Optimizing existing code usually does not add new features, and worse, it might add new bugs
Software bug
A software bug is an error, flaw, mistake, failure, or fault in a computer program that prevents it from behaving as intended . Most bugs arise from mistakes and errors made by people in either a program's source code or its software architecture, and a few are caused by compilers producing incorrect code.... in previously working code (as any change might). Because manually optimized code might sometimes have less 'readability' than unoptimized code, optimization might impact maintainability of it also. Optimization comes at a price and it is important to be sure that the investment is worthwhile.
An automatic optimizer (or optimizing compiler, a program that performs code optimization) may itself have to be optimized, either to further improve the efficiency of its target programs or else speed up its own operation. A compilation performed with optimization 'turned on' usually takes longer, although this is usually only a problem when programs are quite large (but probably more than compensated for over many run time savings of the code).
In particular, for just-in-time compilers the performance of the run time
Run time
Run time or runtime may refer to:*Run time, the length of a film or television program in minutes, usually with end credits included*Runtime, in computer science, the duration of a computer program's execution, from beginning to termination... compile component, executing together with its target code, is the key to improving overall execution speed.
Quotes
“The order in which the operations shall be performed in every particular case is a very interesting and curious question, on which our space does not permit us fully to enter. In almost every computation a great variety of arrangements for the succession of the processes is possible, and various considerations must influence the selection amongst them for the purposes of a Calculating Engine. One essential object is to choose that arrangement which shall tend to reduce to a minimum the time necessary for completing the calculation.” - Ada Byron's notes on the analytical engine
Ada Byron's notes on the analytical engine
Ada Byron's notes on the analytical engine are a description and associated documents produced by Ada Lovelace, on Charles Babbage's design for a mechanical computer called the analytical engine.... 1842.
“More computing sins are committed in the name of efficiency (without necessarily achieving it) than for any other single reason - including blind stupidity.” - W.A. Wulf
“We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.” - Donald Knuth
Donald Knuth
Donald Ervin Knuth is a renowned computer science and Emeritus of the Art of Computer Programming at Stanford University.Author of the seminal multi-volume work The Art of Computer Programming , Knuth has been called the "father" of the run-time analysis, contributing to the development of, and systematizing formal mathematical techn...
“Bottlenecks occur in surprising places, so don't try to second guess and put in a speed hack until you have proven that's where the bottleneck is.” - Rob Pike
Rob Pike
Robert C. Pike is a software engineer and author. He is best known for his work at Bell Labs, where he was a member of the Unix team and was involved in the creation of the Plan 9 from Bell Labs and Inferno operating systems, as well as the Limbo programming language....
“The First Rule of Program Optimization: Don't do it. The Second Rule of Program Optimization (for experts only!): Don't do it yet.” - Michael A. Jackson
Michael A. Jackson
Michael Anthony Jackson is a British computer scientist, and independent computing consultant in London, England. He is also part-time researcher at AT&T Research, Florham Park, NJ, United States, and visiting research professor at the Open University in the United Kingdom....
Niklaus Emil Wirth is a Switzerland computer science, best known for designing several programming languages, including Pascal , and for pioneering several classic topics in software engineering....
