WINETASTER ON 10/04/99 WITH 6 JUDGES AND 7 WINES BASED ON RANKS, IDENT=N
Copyright (c) 1995-99 Richard E. Quandt
FLIGHT 1:
Number of Judges = 6
Number of Wines = 7
Identification of the Wine: The judges' overall ranking:
Wine A is Chambertin 1981 ........ 3rd place
Wine B is Hospice de Beaune 1983 tied for 6th place
Wine C is Haute Cabrieres 1995 ........ 5th place
Wine D is Romanee-st-Vivant 1988 ........ 4th place
Wine E is Mount Eden Pinot Noir 1991 tied for 6th place
Wine F is Beaulieu Vineyard Reserve 1996 ........ 1st place
Wine G is Williams Selyem 1992 ........ 2nd place
The Judges's Rankings
Judge Wine -> A B C D E F G
John 3. 2. 7. 4. 6. 1. 5.
Grant 3. 7. 5. 6. 4. 1. 2.
Burt 2. 6. 4. 3. 7. 1. 5.
Frank 3. 6. 7. 4. 5. 1. 2.
Orley 7. 6. 4. 5. 2. 1. 3.
Dick 6. 4. 3. 5. 7. 2. 1.
Table of Votes Against
Wine -> A B C D E F G
Group Ranking -> 3 6 5 4 6 1 2
Votes Against -> 24 31 30 27 31 7 18
( 6 is the best possible, 42 is the worst)
Here is a measure of the correlation in the preferences of the judges which
ranges between 1.0 (perfect correlation) and 0.0 (no correlation):
W = 0.4643
The probability that random chance could be responsible for this correlation
is quite small, 0.0104. Most analysts would say that unless this
probability is less than 0.1, the judges' preferences are not strongly
related.
We now analyze how each taster's preferences are correlated with the group
preference. A correlation of 1.0 means that the taster's preferences are a
perfect predictor of the group's preferences. A 0.0 means no correlation,
while a -1.0 means that the taster has the reverse ranking of the group.
This is measured by the correlation R.
Correlation Between the Ranks of
Each Person With the Average Ranking of Others
Name of Person Correlation R
Frank 0.8108
Grant 0.6307
Burt 0.5045
Dick 0.3784
John 0.2703
Orley 0.1429
The wines were preferred by the judges in the following order. When the
preferences of the judges are strong enough to permit meaningful differentiation
among the wines, they are separated by -------------------- and are judged to be
significantly different.
1. ........ 1st place Wine F is Beaulieu Vineyard Reserve 1996
---------------------------------------------------
2. ........ 2nd place Wine G is Williams Selyem 1992
3. ........ 3rd place Wine A is Chambertin 1981
4. ........ 4th place Wine D is Romanee-st-Vivant 1988
5. ........ 5th place Wine C is Haute Cabrieres 1995
6. tied for 6th place Wine B is Hospice de Beaune 1983
7. tied for 6th place Wine E is Mount Eden Pinot Noir 1991
We now test whether the ranksums AS A WHOLE provide a significant ordering.
The Friedman Chi-square value is 16.7143. The probability that this could
happen by chance is 0.0104
We now undertake a more detailed examination of the pair-wise rank correla-
tions that exist between pairs of judges. First, we present a table in which you
can find the correlation for any pair of judges, by finding one of the names in the
left hand margin and the other name on top of a column. A second table arranges
these correlations in descending order and marks which is significantly positive
significantly negative, or not significant. This may allow you to find clusters
of judges whose rankings were particularly similar or particularly dissimilar.
Pairwise Rank Correlations
Correlations must exceed in absolute value 0.79 for significance at the 0.05
level and must exceed 0.71 for significance at the 0.1 level
John Grant Burt
John 1.000 0.179 0.500
Grant 0.179 1.000 0.464
Burt 0.500 0.464 1.000
Frank 0.536 0.821 0.571
Orley -0.107 0.571 -0.036
Dick 0.143 0.393 0.250
Frank Orley Dick
John 0.536 -0.107 0.143
Grant 0.821 0.571 0.393
Burt 0.571 -0.036 0.250
Frank 1.000 0.357 0.357
Orley 0.357 1.000 0.357
Dick 0.357 0.357 1.000
Pairwise correlations in descending order
0.821 Grant and Frank Significantly positive
0.571 Burt and Frank Not significant
0.571 Grant and Orley Not significant
0.536 John and Frank Not significant
0.500 John and Burt Not significant
0.464 Grant and Burt Not significant
0.393 Grant and Dick Not significant
0.357 Orley and Dick Not significant
0.357 Frank and Dick Not significant
0.357 Frank and Orley Not significant
0.250 Burt and Dick Not significant
0.179 John and Grant Not significant
0.143 John and Dick Not significant
-0.036 Burt and Orley Not significant
-0.107 John and Orley Not significant
COMMENT:
First, we note that in this tasting there were three French, three
American and one South African (Haute-Cabrieres) Pinot Noir wines.
All wines were considered extraordinarily good, but there was remark-
able agreement on the quality of the BV Reserve 1996 Pinot Noir.
The wine, in a very fancy bottle, is from Carneros and aged in French oak
barrels. The winner tasted like an outstanding red Burgundy. The winning
wine was incredibly rich and smooth. The wines were nevertheless very
close in terms of appeal, and it is interesting that the South African
held its own in a strong field.
Another thing we did in this tasting was to identify which wine came
from which country. Each judge was asked to record his guesses by
identifying each wine A, B, C, etc., with the letters A for American, F for
French, and S for South African. Each judge was required to identify
exactly three wines as American, three as French and one as South
African (otherwise one could automatically identify three wines
correctly by guessing every wine to be French or every wine to be
American).
The table below shows the identifications picked by the judges:
Identification table
Wine
A B C D E F G No.correct
John F F A S A F A 4
Grant F F F S A A A 5
Burt F A A F S F A 3
Frank F F F S A A A 5
Orley A F S F A F A 5
Dick A F F A S A F 2
No. correct 4 5 1 2 4 3 5
When can we say that a judge's identifications are statistically
significant? It can be calculated that on the assumption that
judges identify wines randomly, the number of correct identifica-
tions will occur with the following probabilities:
Number corrrectly Probability of
identified occurrence
0 0.043
1 0.136
2 0.257
3 0.321
4 0.129
5 0.107
6 0.0
7 0.007
Note that it is impossible ever to identify exactly six wines
correctly (because if six are correctly identified, the seventh
is automatically also identified correctly). In any event, to
be statistically significant (at the 0.114 level), a judge has to
identify at least five wines correctly. Note that if the number of
wines of type A, F and S is not 3, 3, 1 but some other numbers, the
statistical distribution above no longer applies but has to be
computed anew. For some other interesting examples, take a look at
Section 4 of Richard Quandt's paper.
Return to previous page