Rank reversals in decision making
Encyclopedia
This article describes the role rank reversals play in assessing the merits of decision-making methods. The issue of rank reversals lies at the heart of many debates in decision-making and multi-criteria decision-making, in particular.
Unlike most other computational procedures, it is hard to tell if a particular decision-making method has derived the correct answer or not. Such methods analyze a set of alternatives described in terms of some criteria. They determine which alternative is the best one, or they provide relative weights of how the alternatives perform, or just how the alternatives should be ranked when all the criteria are considered simultaneously. This is exactly where the challenge with decision making exists. Oftentimes it is hard, if not practically impossible, to determine whether a correct answer has been reached or not. With other computational methods, for instance with a job scheduling method, one can examine a set of different answers and then categorize the answers according to some metric of performance (for instance, a project's completion time). But this may not be possible to do with the answers derived by most decision making methods. After all, determining the best decision making method leads to a decision making paradox
.
Thus the following question emerges: How can one evaluate decision-making methods? This is a very difficult issue and may not be answered in a globally accepted manner.
A critical part in answering this fundamental question is played by what is known as rank reversals.
To see the above point more clearly, suppose that three candidates are evaluated for some job opening. Let us designate these candidates as A, B, and C. Suppose that some decision making method has determined that the best candidate for that job is person A, followed by B, who is followed by C. This is the first ranking and it is indicated as follows: A > B > C (where > means better than). Next, suppose that candidate B (who is not the best one) is replaced by an even worse candidate, say person D. That is, now we have B > D, and candidate B is replaced by D while candidates A and C remain in the pool of candidates with exactly the same characteristics as before. When the new set of alternatives (i.e., candidates A, D and C) are ranked together and by assuming that the criteria have exactly the same weights as before, then should not candidate A still be the best one? It turns out that under some decision making methods the best alternative may be different now. This is known as a rank reversal and it is one of the types of rank reversals.
The first type of rank reversal in the above context was observed by Belton and Gear in 1983 as part of a study of the analytic hierarchy process
(AHP). They first considered a simple decision problem comprised by 3 alternatives and 2 criteria. Next a copy of a non-optimal alternative was introduced. When the 4 alternatives (i.e., the previous 3 plus the copy) were evaluated, and under the assumption that the criteria weights are exactly the same as before, it was observed that now the indication of the best alternative can change. That is, a rank reversal may occur with the AHP. A few years later it was observed that the AHP, as well as a new variant to it that was introduced by Professor Thomas Saaty
(the inventor of the AHP) in response to the previous observation by Belton and Gear, may exhibit rank reversals when a non-optimal alternative is replaced by a worse one (and not a copy of an alternative as in Belton and Gear’s experiment).
The issue of rank reversals has captured the interest of many researchers and practitioners in the field of decision-making. It is something that continues to be considered controversial by many and is debated a lot.
(AHP) and its additive variants, the TOPSIS and ELECTRE
methods and their variants.
The weighted product model
(WPM) does not exhibit the previous types of rank reversals, due to the multiplication formula it uses. However, the WPM does cause rank reversals when it is compared with the weighted sum model
(WSM) and under the condition that all the criteria of a given decision problem can be measured in exactly the same unit. The same is true with all the previous methods as well. This is the Type 5 ranking reversal.
It is quite possible to define more types of rank reversals. One only needs to determine ways to alter a test problem and see how the ranking of the alternatives of the new problem differs from the original ranking of the alternatives of the original problem. Furthermore, the difference in rankings, somehow, should indicate the presence of undesirable effects.
Let us consider a simple example of buying a car. Suppose that there are two cars available to the decision maker: Car A and Car B. Car A is much cheaper than Car B but its overall quality is much less when compared to that for Car B. On the other hand Car B is more expensive than Car A but it is also of better quality. A decision maker who is concerned of the high price issue, may choose Car A over the better quality and more expensive Car B. Next suppose that the car dealer presents to the decision maker a third car, say Car C, which is way more expensive than Car B but now the overall quality of Car C is marginally higher than that of Car B. Under such a scenario, it is quite possible for a decision maker to alter his/her opinion and purchase Car B instead of Car A, even if he/she has not actually seen Car C.
Such events may take place with many rational decision makers. In other words, rank reversals may actually be possible in rational decision making. The issue of having rank reversals by rational decision makers has been studied extensively by Amos Tversky
. In other words, having rank reversals in certain occasions and of certain types may not be indicative to faulty decision making. However, the key question is how to be able to distinguish when rank reversals indicate that something is wrong or when they do not conflict rational decision making
. This is a highly debated issue and it is unlikely that a consensus will be reached in the decision making
community.
Unlike most other computational procedures, it is hard to tell if a particular decision-making method has derived the correct answer or not. Such methods analyze a set of alternatives described in terms of some criteria. They determine which alternative is the best one, or they provide relative weights of how the alternatives perform, or just how the alternatives should be ranked when all the criteria are considered simultaneously. This is exactly where the challenge with decision making exists. Oftentimes it is hard, if not practically impossible, to determine whether a correct answer has been reached or not. With other computational methods, for instance with a job scheduling method, one can examine a set of different answers and then categorize the answers according to some metric of performance (for instance, a project's completion time). But this may not be possible to do with the answers derived by most decision making methods. After all, determining the best decision making method leads to a decision making paradox
Decision making paradox
The word paradox comes from the Greek words "para" and "doksa" or "doxa"...
.
Thus the following question emerges: How can one evaluate decision-making methods? This is a very difficult issue and may not be answered in a globally accepted manner.
A critical part in answering this fundamental question is played by what is known as rank reversals.
Rank reversal
One way to test the validity of decision-making methods is to construct special test problems and then study the solutions they derive. If the solutions exhibit some logic contradictions (in the form of undesirable rank reversals of the alternatives), then one may argue that something is wrong with the method that derived them.To see the above point more clearly, suppose that three candidates are evaluated for some job opening. Let us designate these candidates as A, B, and C. Suppose that some decision making method has determined that the best candidate for that job is person A, followed by B, who is followed by C. This is the first ranking and it is indicated as follows: A > B > C (where > means better than). Next, suppose that candidate B (who is not the best one) is replaced by an even worse candidate, say person D. That is, now we have B > D, and candidate B is replaced by D while candidates A and C remain in the pool of candidates with exactly the same characteristics as before. When the new set of alternatives (i.e., candidates A, D and C) are ranked together and by assuming that the criteria have exactly the same weights as before, then should not candidate A still be the best one? It turns out that under some decision making methods the best alternative may be different now. This is known as a rank reversal and it is one of the types of rank reversals.
The first type of rank reversal in the above context was observed by Belton and Gear in 1983 as part of a study of the analytic hierarchy process
Analytic Hierarchy Process
The Analytic Hierarchy Process is a structured technique for organizing and analyzing complex decisions. Based on mathematics and psychology, it was developed by Thomas L...
(AHP). They first considered a simple decision problem comprised by 3 alternatives and 2 criteria. Next a copy of a non-optimal alternative was introduced. When the 4 alternatives (i.e., the previous 3 plus the copy) were evaluated, and under the assumption that the criteria weights are exactly the same as before, it was observed that now the indication of the best alternative can change. That is, a rank reversal may occur with the AHP. A few years later it was observed that the AHP, as well as a new variant to it that was introduced by Professor Thomas Saaty
Thomas Saaty
Thomas L. Saaty is Distinguished University Professor at the University of Pittsburgh, where he teaches in the Joseph M. Katz Graduate School of Business...
(the inventor of the AHP) in response to the previous observation by Belton and Gear, may exhibit rank reversals when a non-optimal alternative is replaced by a worse one (and not a copy of an alternative as in Belton and Gear’s experiment).
The issue of rank reversals has captured the interest of many researchers and practitioners in the field of decision-making. It is something that continues to be considered controversial by many and is debated a lot.
Different types of rank reversals
There are many different types of rank reversals, depending on how the alternatives in a problem are defined and evaluated. These types are described next as Type 1, Type 2, Type 3, Type 4, and Type 5.Rank reversals of Type 1
As stated earlier, one may introduce identical or near-identical copies of non-optimal alternatives and then check to see if the indication of the best alternative changes or not.Rank reversals of Type 2
Another way is to replace a non-optimal alternative with a worse one and then see if the indication of the best alternative changes or not.Rank reversals of Type 3
A different way is to run some tests as follows. First consider a problem with all the alternatives together and get a ranking. Next, decompose the original problem into a set of smaller problems defined on two alternatives at a time and the same criteria (and their weights) as before. Get the rankings of these smaller problems and check to see if they are in conflict with the ranking of the alternatives of the original (larger) problem.Rank reversals of Type 4
This is like the previous case, but now ignore the ranking of the original (larger) problem. Instead, check to see if the rankings of the smaller problems are in conflict with each other. For instance, suppose that the following 3 alternatives A, B, and C are considered. Next, suppose that some 2-alternative problems are solved and the rankings A > B, B > C, and C > A, are derived from these 2-alternative problems. Obviously, the above situation indicates a case of non-transitivity (or contradiction) as we get A > B > C > A.Rank reversals of Type 5
All previous types of rank reversals are known to occur with the analytic hierarchy processAnalytic Hierarchy Process
The Analytic Hierarchy Process is a structured technique for organizing and analyzing complex decisions. Based on mathematics and psychology, it was developed by Thomas L...
(AHP) and its additive variants, the TOPSIS and ELECTRE
ELECTRE
ELECTRE is a family of multi-criteria decision analysis methods that originated in Europe in the mid-1960s. The acronym ELECTRE stands for: ELimination Et Choix Traduisant la REalité ....
methods and their variants.
The weighted product model
Weighted product model
The weighted product model is a popular multi-criteria decision analysis / multi-criteria decision making method. It is similar to the weighted sum model . The main difference is that instead of addition in the main mathematical operation now there is multiplication...
(WPM) does not exhibit the previous types of rank reversals, due to the multiplication formula it uses. However, the WPM does cause rank reversals when it is compared with the weighted sum model
Weighted sum model
The weighted sum model is the best known and simplest multi-criteria decision analysis / multi-criteria decision making method for evaluating a number of alternatives in terms of a number of decision criteria. It is very important to state here that it is applicable only when all the data are...
(WSM) and under the condition that all the criteria of a given decision problem can be measured in exactly the same unit. The same is true with all the previous methods as well. This is the Type 5 ranking reversal.
It is quite possible to define more types of rank reversals. One only needs to determine ways to alter a test problem and see how the ranking of the alternatives of the new problem differs from the original ranking of the alternatives of the original problem. Furthermore, the difference in rankings, somehow, should indicate the presence of undesirable effects.
Are rank reversals always undesirable?
Decision-making methods are used to make decisions in many aspects of human activity. This is especially true with decisions that involve large amounts of money or decisions that may have huge impact on large numbers of people. Given the well-established fact that different methods may yield different answers when they are fed with exactly the same problem, the question is how to evaluate them. Rank reversals are at the very heart of assessing the merits of such methods. At the same time, they are at the center of many heated debates in this area. Many authors use them as means to criticize decision making methods or to better explain rational behavior.Let us consider a simple example of buying a car. Suppose that there are two cars available to the decision maker: Car A and Car B. Car A is much cheaper than Car B but its overall quality is much less when compared to that for Car B. On the other hand Car B is more expensive than Car A but it is also of better quality. A decision maker who is concerned of the high price issue, may choose Car A over the better quality and more expensive Car B. Next suppose that the car dealer presents to the decision maker a third car, say Car C, which is way more expensive than Car B but now the overall quality of Car C is marginally higher than that of Car B. Under such a scenario, it is quite possible for a decision maker to alter his/her opinion and purchase Car B instead of Car A, even if he/she has not actually seen Car C.
Such events may take place with many rational decision makers. In other words, rank reversals may actually be possible in rational decision making. The issue of having rank reversals by rational decision makers has been studied extensively by Amos Tversky
Amos Tversky
Amos Nathan Tversky, was a cognitive and mathematical psychologist, a pioneer of cognitive science, a longtime collaborator of Daniel Kahneman, and a key figure in the discovery of systematic human cognitive bias and handling of risk. Much of his early work concerned the foundations of measurement...
. In other words, having rank reversals in certain occasions and of certain types may not be indicative to faulty decision making. However, the key question is how to be able to distinguish when rank reversals indicate that something is wrong or when they do not conflict rational decision making
Decision making
Decision making can be regarded as the mental processes resulting in the selection of a course of action among several alternative scenarios. Every decision making process produces a final choice. The output can be an action or an opinion of choice.- Overview :Human performance in decision terms...
. This is a highly debated issue and it is unlikely that a consensus will be reached in the decision making
Decision making
Decision making can be regarded as the mental processes resulting in the selection of a course of action among several alternative scenarios. Every decision making process produces a final choice. The output can be an action or an opinion of choice.- Overview :Human performance in decision terms...
community.
Methods that have been verified to exhibit rank reversals
The following is just a partial list of multi-criteria decision making methods which have been confirmed to exhibit various types of rank reversals:- The analytic hierarchy processAnalytic Hierarchy ProcessThe Analytic Hierarchy Process is a structured technique for organizing and analyzing complex decisions. Based on mathematics and psychology, it was developed by Thomas L...
(AHP) and some of its variants. - The ELECTREELECTREELECTRE is a family of multi-criteria decision analysis methods that originated in Europe in the mid-1960s. The acronym ELECTRE stands for: ELimination Et Choix Traduisant la REalité ....
(outranking) method and its variants. - The TOPSIS method.
- The PROMETHEEProméthéeProméthée is an opera in three acts by the French composer Gabriel Fauré. The libretto, by Jean Lorrain and Ferdinand Hérold, is based on the Greek myth of Prometheus. Although designated a tragédie lyrique, the opera resists easy categorisation. It was intended as a large-scale work with spoken...
(outranking) method. - Multi-attribute utility theory (MAUT).