WINETASTER ON 11/2/93 WITH 9 JUDGES AND 9 WINES BASED ON RANKS, IDENT=N Copyright (c) 1995-2011 Richard E. Quandt, V. 1.65
A Vertical Tasting of Château Ducru Beaucaillou
FLIGHT 1: Number of Judges = 9 Number of Wines = 9
Identification of the Wine: The judges' overall ranking:
Wine A is Ch. Ducru Beaucaillou 1982 tied for 5th place Wine B is 1986 tied for 5th place Wine C is 1977 ........ 2nd place Wine D is 1979 ........ 7th place Wine E is 1978 ........ 9th place Wine F is 1976 tied for 3rd place Wine G is 1975 tied for 3rd place Wine H is 1970 ........ 8th place Wine I is 1966 ........ 1st place
The Judges's Rankings
Judge Wine -> A B C D E F G H I Ken 4. 6. 2. 7. 5. 8. 3. 9. 1. Dick 7. 3. 6. 9. 4. 5. 2. 8. 1. Norton 6. 9. 1. 8. 7. 3. 4. 5. 2. Burt 4. 2. 5. 7. 9. 6. 8. 3. 1. John 2. 5. 4. 6. 9. 1. 7. 8. 3. Orley 9. 8. 4. 3. 2. 1. 6. 5. 7. Bob 3. 8. 4. 6. 9. 5. 7. 2. 1. Ed 7. 4. 9. 3. 5. 6. 1. 8. 2. Frank 6. 3. 2. 5. 8. 7. 4. 9. 1.
Table of Votes Against Wine -> A B C D E F G H I
Group Ranking -> 5 5 2 7 9 3 3 8 1 Votes Against -> 48 48 37 54 58 42 42 57 19
( 9 is the best possible, 81 is the worst)
Here is a measure of the correlation in the preferences of the judges which ranges between 1.0 (perfect correlation) and 0.0 (no correlation):
W = 0.2407
The probability that random chance could be responsible for this correlation is quite small, 0.0268. Most analysts would say that unless this probability is less than 0.1, the judges' preferences are not strongly related. We now analyze how each taster's preferences are correlated with the group preference. A correlation of 1.0 means that the taster's preferences are a perfect predictor of the group's preferences. A 0.0 means no correlation, while a -1.0 means that the taster has the reverse ranking of the group. This is measured by the correlation R.
Correlation Between the Ranks of Each Person With the Average Ranking of Others
Name of Person Correlation R Norton 0.6360 Frank 0.6276 John 0.5774 Ken 0.5000 Dick 0.4017 Bob 0.2500 Burt 0.2333 Ed -0.0335 Orley -0.5167
The wines were preferred by the judges in the following order. When the preferences of the judges are strong enough to permit meaningful differentiation among the wines, they are separated by -------------------- and are judged to be significantly different.
1. ........ 1st place Wine I is 1966 --------------------------------------------------- 2. ........ 2nd place Wine C is 1977 3. tied for 3rd place Wine F is 1976 4. tied for 3rd place Wine G is 1975 5. tied for 5th place Wine B is 1986 6. tied for 5th place Wine A is 1982 7. ........ 7th place Wine D is 1979 8. ........ 8th place Wine H is 1970 9. ........ 9th place Wine E is 1978 We now test whether the ranksums AS A WHOLE provide a significant ordering. The Friedman Chi-square value is 17.3333. The probability that this could happen by chance is 0.0268 We now undertake a more detailed examination of the pair-wise rank correla- tions that exist between pairs of judges. First, we present a table in which you can find the correlation for any pair of judges, by finding one of the names in the left hand margin and the other name on top of a column. A second table arranges these correlations in descending order and marks which is significantly positive significantly negative, or not significant. This may allow you to find clusters of judges whose rankings were particularly similar or particularly dissimilar. Pairwise Rank Correlations Correlations must exceed in absolute value 0.70 for significance at the 0.05 level and must exceed 0.60 for significance at the 0.1 level Ken Dick Norton Ken 1.000 0.583 0.483 Dick 0.583 1.000 0.250 Norton 0.483 0.250 1.000 Burt 0.117 0.150 0.133 John 0.200 0.067 0.400 Orley -0.400 -0.250 0.150 Bob 0.167 -0.167 0.583 Ed 0.267 0.583 -0.217 Frank 0.767 0.517 0.333 Burt John Orley Ken 0.117 0.200 -0.400 Dick 0.150 0.067 -0.250 Norton 0.133 0.400 0.150 Burt 1.000 0.417 -0.633 John 0.417 1.000 -0.183 Orley -0.633 -0.183 1.000 Bob 0.650 0.450 -0.300 Ed -0.133 -0.150 -0.150 Frank 0.400 0.367 -0.417 Bob Ed Frank Ken 0.167 0.267 0.767 Dick -0.167 0.583 0.517 Norton 0.583 -0.217 0.333 Burt 0.650 -0.133 0.400 John 0.450 -0.150 0.367 Orley -0.300 -0.150 -0.417 Bob 1.000 -0.300 0.150 Ed -0.300 1.000 0.367 Frank 0.150 0.367 1.000 Pairwise correlations in descending order 0.767 Ken and Frank Significantly positive 0.650 Burt and Bob Significantly positive 0.583 Ken and Dick Not significant 0.583 Dick and Ed Not significant 0.583 Norton and Bob Not significant 0.517 Dick and Frank Not significant 0.483 Ken and Norton Not significant 0.450 John and Bob Not significant 0.417 Burt and John Not significant 0.400 Norton and John Not significant 0.400 Burt and Frank Not significant 0.367 John and Frank Not significant 0.367 Ed and Frank Not significant 0.333 Norton and Frank Not significant 0.267 Ken and Ed Not significant 0.250 Dick and Norton Not significant 0.200 Ken and John Not significant 0.167 Ken and Bob Not significant 0.150 Norton and Orley Not significant 0.150 Dick and Burt Not significant 0.150 Bob and Frank Not significant 0.133 Norton and Burt Not significant 0.117 Ken and Burt Not significant 0.067 Dick and John Not significant -0.133 Burt and Ed Not significant -0.150 Orley and Ed Not significant -0.150 John and Ed Not significant -0.167 Dick and Bob Not significant -0.183 John and Orley Not significant -0.217 Norton and Ed Not significant -0.250 Dick and Orley Not significant -0.300 Orley and Bob Not significant -0.300 Bob and Ed Not significant -0.400 Ken and Orley Not significant -0.417 Orley and Frank Not significant -0.633 Burt and Orley Significantly negative
COMMENT: The agreement among the tasters was very strong and the 1966 easily won the tasting, with no other wine being either significantly good or bad. However, one taster made an error and gave two wines the same rank. This occurred before WINETASTER® automatically checked whether every taster entered a valid set of rankings (remember that we do not permit ties). Consequently, we arbitrarily changed one of the duplicate ranks to the missing rank and the results reflect this alteration. In order to examine the effect of this, we shall rescore the results without that one taster; to follow below.
WINETASTER ON 11/29/93 WITH 8 JUDGES AND 9 WINES BASED ON RANKS, IDENT=N Copyright (c) 1995-2011 Richard E. Quandt, V. 1.65
Château Bucru Beaucaillou Tasting Continued
FLIGHT 1: Number of Judges = 8 Number of Wines = 9
Identification of the Wine: The judges' overall ranking:
Wine A is Chateau Ducru Beaucaillou 1982 ........ 6th place Wine B is 1986 ........ 5th place Wine C is 1977 ........ 2nd place Wine D is 1979 ........ 7th place Wine E is 1978 ........ 8th place Wine F is 1976 ........ 4th place Wine G is 1975 ........ 3rd place Wine H is 1970 ........ 9th place Wine I is 1966 ........ 1st place
The Judges's Rankings
Judge Wine -> A B C D E F G H I Ken 4. 6. 2. 7. 5. 8. 3. 9. 1. Dick 7. 3. 6. 9. 4. 5. 2. 8. 1. Norton 6. 9. 1. 8. 7. 3. 4. 5. 2. Burt 4. 2. 5. 7. 9. 6. 8. 3. 1. John 2. 5. 4. 6. 9. 1. 7. 8. 3. Orley 9. 8. 4. 3. 2. 1. 6. 5. 7. Ed 7. 4. 9. 3. 5. 6. 1. 8. 2. Frank 6. 3. 2. 5. 8. 7. 4. 9. 1.
Table of Votes Against Wine -> A B C D E F G H I
Group Ranking -> 6 5 2 7 8 4 3 9 1 Votes Against -> 45 40 33 48 49 37 35 55 18
( 8 is the best possible, 72 is the worst)
Here is a measure of the correlation in the preferences of the judges which ranges between 1.0 (perfect correlation) and 0.0 (no correlation):
W = 0.2505
The probability that random chance could be responsible for this correlation is quite small, 0.0419. Most analysts would say that unless this probability is less than 0.1, the judges' preferences are not strongly related. We now analyze how each taster's preferences are correlated with the group preference. A correlation of 1.0 means that the taster's preferences are a perfect predictor of the group's preferences. A 0.0 means no correlation, while a -1.0 means that the taster has the reverse ranking of the group. This is measured by the correlation R.
Correlation Between the Ranks of Each Person With the Average Ranking of Others
Name of Person Correlation R Frank 0.6527 Dick 0.5833 Ken 0.5439 Norton 0.3598 John 0.2167 Ed 0.1667 Burt 0.0753 Orley -0.4034
The wines were preferred by the judges in the following order. When the preferences of the judges are strong enough to permit meaningful differentiation among the wines, they are separated by -------------------- and are judged to be significantly different.
1. ........ 1st place Wine I is 1966 --------------------------------------------------- 2. ........ 2nd place Wine C is 1977 3. ........ 3rd place Wine G is 1975 4. ........ 4th place Wine F is 1976 5. ........ 5th place Wine B is 1986 6. ........ 6th place Wine A is 1982 7. ........ 7th place Wine D is 1979 8. ........ 8th place Wine E is 1978 --------------------------------------------------- 9. ........ 9th place Wine H is 1970 We now test whether the ranksums AS A WHOLE provide a significant ordering. The Friedman Chi-square value is 16.0333. The probability that this could happen by chance is 0.0419 We now undertake a more detailed examination of the pair-wise rank correla- tions that exist between pairs of judges. First, we present a table in which you can find the correlation for any pair of judges, by finding one of the names in the left hand margin and the other name on top of a column. A second table arranges these correlations in descending order and marks which is significantly positive significantly negative, or not significant. This may allow you to find clusters of judges whose rankings were particularly similar or particularly dissimilar. Pairwise Rank Correlations Correlations must exceed in absolute value 0.70 for significance at the 0.05 level and must exceed 0.60 for significance at the 0.1 level Ken Dick Norton Ken 1.000 0.583 0.483 Dick 0.583 1.000 0.250 Norton 0.483 0.250 1.000 Burt 0.117 0.150 0.133 John 0.200 0.067 0.400 Orley -0.400 -0.250 0.150 Ed 0.267 0.583 -0.217 Frank 0.767 0.517 0.333 Burt John Orley Ken 0.117 0.200 -0.400 Dick 0.150 0.067 -0.250 Norton 0.133 0.400 0.150 Burt 1.000 0.417 -0.633 John 0.417 1.000 -0.183 Orley -0.633 -0.183 1.000 Ed -0.133 -0.150 -0.150 Frank 0.400 0.367 -0.417 Ed Frank Ken 0.267 0.767 Dick 0.583 0.517 Norton -0.217 0.333 Burt -0.133 0.400 John -0.150 0.367 Orley -0.150 -0.417 Ed 1.000 0.367 Frank 0.367 1.000 Pairwise correlations in descending order 0.767 Ken and Frank Significantly positive 0.583 Dick and Ed Not significant 0.583 Ken and Dick Not significant 0.517 Dick and Frank Not significant 0.483 Ken and Norton Not significant 0.417 Burt and John Not significant 0.400 Burt and Frank Not significant 0.400 Norton and John Not significant 0.367 John and Frank Not significant 0.367 Ed and Frank Not significant 0.333 Norton and Frank Not significant 0.267 Ken and Ed Not significant 0.250 Dick and Norton Not significant 0.200 Ken and John Not significant 0.150 Dick and Burt Not significant 0.150 Norton and Orley Not significant 0.133 Norton and Burt Not significant 0.117 Ken and Burt Not significant 0.067 Dick and John Not significant -0.133 Burt and Ed Not significant -0.150 John and Ed Not significant -0.150 Orley and Ed Not significant -0.183 John and Orley Not significant -0.217 Norton and Ed Not significant -0.250 Dick and Orley Not significant -0.400 Ken and Orley Not significant -0.417 Orley and Frank Not significant -0.633 Burt and Orley Significantly negative
COMMENT: Rerunning the computations without the one person who misranked the wines still leaves very significant agreement in the group, with the further result that the 1970 is now rated significantly inferior, while 1966 remains significantly superior. The rank correlation between the previous result based on 9 tasters and the present one is 0.97, showing that the omission of one taster makes practically no difference in the ranking of the wines.
Return to previous page