WINETASTER ON 10/04/99 WITH 6 JUDGES AND 7 WINES BASED ON RANKS, IDENT=N Copyright (c) 1995-99 Richard E. Quandt

FLIGHT 1: Number of Judges = 6 Number of Wines = 7
Identification of the Wine: The judges' overall ranking:
Wine A is Chambertin 1981 ........ 3rd place Wine B is Hospice de Beaune 1983 tied for 6th place Wine C is Haute Cabrieres 1995 ........ 5th place Wine D is Romanee-st-Vivant 1988 ........ 4th place Wine E is Mount Eden Pinot Noir 1991 tied for 6th place Wine F is Beaulieu Vineyard Reserve 1996 ........ 1st place Wine G is Williams Selyem 1992 ........ 2nd place
The Judges's Rankings
Judge Wine -> A B C D E F G John 3. 2. 7. 4. 6. 1. 5. Grant 3. 7. 5. 6. 4. 1. 2. Burt 2. 6. 4. 3. 7. 1. 5. Frank 3. 6. 7. 4. 5. 1. 2. Orley 7. 6. 4. 5. 2. 1. 3. Dick 6. 4. 3. 5. 7. 2. 1.
Table of Votes Against Wine -> A B C D E F G
Group Ranking -> 3 6 5 4 6 1 2 Votes Against -> 24 31 30 27 31 7 18
( 6 is the best possible, 42 is the worst)

Here is a measure of the correlation in the preferences of the judges which ranges between 1.0 (perfect correlation) and 0.0 (no correlation):
W = 0.4643

The probability that random chance could be responsible for this correlation is quite small, 0.0104. Most analysts would say that unless this probability is less than 0.1, the judges' preferences are not strongly related. We now analyze how each taster's preferences are correlated with the group preference. A correlation of 1.0 means that the taster's preferences are a perfect predictor of the group's preferences. A 0.0 means no correlation, while a -1.0 means that the taster has the reverse ranking of the group. This is measured by the correlation R.
Correlation Between the Ranks of Each Person With the Average Ranking of Others
Name of Person Correlation R Frank 0.8108 Grant 0.6307 Burt 0.5045 Dick 0.3784 John 0.2703 Orley 0.1429

The wines were preferred by the judges in the following order. When the preferences of the judges are strong enough to permit meaningful differentiation among the wines, they are separated by -------------------- and are judged to be significantly different.
1. ........ 1st place Wine F is Beaulieu Vineyard Reserve 1996 --------------------------------------------------- 2. ........ 2nd place Wine G is Williams Selyem 1992 3. ........ 3rd place Wine A is Chambertin 1981 4. ........ 4th place Wine D is Romanee-st-Vivant 1988 5. ........ 5th place Wine C is Haute Cabrieres 1995 6. tied for 6th place Wine B is Hospice de Beaune 1983 7. tied for 6th place Wine E is Mount Eden Pinot Noir 1991 We now test whether the ranksums AS A WHOLE provide a significant ordering. The Friedman Chi-square value is 16.7143. The probability that this could happen by chance is 0.0104 We now undertake a more detailed examination of the pair-wise rank correla- tions that exist between pairs of judges. First, we present a table in which you can find the correlation for any pair of judges, by finding one of the names in the left hand margin and the other name on top of a column. A second table arranges these correlations in descending order and marks which is significantly positive significantly negative, or not significant. This may allow you to find clusters of judges whose rankings were particularly similar or particularly dissimilar. Pairwise Rank Correlations Correlations must exceed in absolute value 0.79 for significance at the 0.05 level and must exceed 0.71 for significance at the 0.1 level John Grant Burt John 1.000 0.179 0.500 Grant 0.179 1.000 0.464 Burt 0.500 0.464 1.000 Frank 0.536 0.821 0.571 Orley -0.107 0.571 -0.036 Dick 0.143 0.393 0.250 Frank Orley Dick John 0.536 -0.107 0.143 Grant 0.821 0.571 0.393 Burt 0.571 -0.036 0.250 Frank 1.000 0.357 0.357 Orley 0.357 1.000 0.357 Dick 0.357 0.357 1.000 Pairwise correlations in descending order 0.821 Grant and Frank Significantly positive 0.571 Burt and Frank Not significant 0.571 Grant and Orley Not significant 0.536 John and Frank Not significant 0.500 John and Burt Not significant 0.464 Grant and Burt Not significant 0.393 Grant and Dick Not significant 0.357 Orley and Dick Not significant 0.357 Frank and Dick Not significant 0.357 Frank and Orley Not significant 0.250 Burt and Dick Not significant 0.179 John and Grant Not significant 0.143 John and Dick Not significant -0.036 Burt and Orley Not significant -0.107 John and Orley Not significant

COMMENT: First, we note that in this tasting there were three French, three American and one South African (Haute-Cabrieres) Pinot Noir wines. All wines were considered extraordinarily good, but there was remark- able agreement on the quality of the BV Reserve 1996 Pinot Noir. The wine, in a very fancy bottle, is from Carneros and aged in French oak barrels. The winner tasted like an outstanding red Burgundy. The winning wine was incredibly rich and smooth. The wines were nevertheless very close in terms of appeal, and it is interesting that the South African held its own in a strong field. Another thing we did in this tasting was to identify which wine came from which country. Each judge was asked to record his guesses by identifying each wine A, B, C, etc., with the letters A for American, F for French, and S for South African. Each judge was required to identify exactly three wines as American, three as French and one as South African (otherwise one could automatically identify three wines correctly by guessing every wine to be French or every wine to be American). The table below shows the identifications picked by the judges: Identification table Wine A B C D E F G No.correct John F F A S A F A 4 Grant F F F S A A A 5 Burt F A A F S F A 3 Frank F F F S A A A 5 Orley A F S F A F A 5 Dick A F F A S A F 2 No. correct 4 5 1 2 4 3 5 When can we say that a judge's identifications are statistically significant? It can be calculated that on the assumption that judges identify wines randomly, the number of correct identifica- tions will occur with the following probabilities: Number corrrectly Probability of identified occurrence 0 0.043 1 0.136 2 0.257 3 0.321 4 0.129 5 0.107 6 0.0 7 0.007 Note that it is impossible ever to identify exactly six wines correctly (because if six are correctly identified, the seventh is automatically also identified correctly). In any event, to be statistically significant (at the 0.114 level), a judge has to identify at least five wines correctly. Note that if the number of wines of type A, F and S is not 3, 3, 1 but some other numbers, the statistical distribution above no longer applies but has to be computed anew. For some other interesting examples, take a look at Section 4 of Richard Quandt's paper.
Return to previous page