The first paper is a methodological mess (see my blog post about why you cannot perform mathematical calculations on non-number data -- adding, subtracting, multiplying, and dividing). The author took rankings, turned them into arbitrary numbers, and then performed odd math to see if critics could taste the exact same wines and award the same grades.
Okay, let me take a second to say how statistically ridiculous this method is, and an how amazed I am that it got past peer review. If someone is asking people to rate their interests on a scale of one to five, and I like cotton candy 5 and liver 1, there is no reason to think I'll like liver flavored candy a 3. It makes no sense.
Despite the egregious abuse of math, it's pretty obvious by Hodgson's description of the data that the judges stink, at a major competition, with judges from the industry. I cannot say that there is any statistical validity to his conclusion, but there is lots of reason to be concerned.
The second paper makes me much happier as a statistician and much more nervous as a competition judge. Here, Hodgson uses a test called Cohen's Kappa, which is really cool because it looks at if the judge scored the same wine the same way, but also gives partial credit if he or she got close. This statistic showed that only 30 percent of judges could rate wine consistently. Many more judge fairly randomly. Hrm...
So I'm worried about doing a poor job, but then I'd be in good company. When I think about the wineries submitting samples, however, I feel a little better. For them, a competition is a no-lose proposition - they either get a medal to display in their tasting room, or they get nothing. There is no negative for a poor rating, only praise if they get it. So I'll just try to be contentious, and remember the advice in the excellent Fermentation Blog.
- Be aware that my personal tastes may not reflect great wine (an oaky Chardonnay can be excellent even if it isn't to my taste).
- Go back and review Wine Faults by John Hudelson.
- Stay hydrated and keep spitting.