Statistical Rant II - Price is an Indicator of Quality

In my last statistics post, I complained about a paper from the American Association of Wine Economists where the authors played fast and loose with statistical methods to come up  with meaningless results.  In this post, I want to address the setup and aims of the study.

In the interim, several blogs, the winediarist in particular, have held long and fairly tedious chains of comments about the study by people who have clearly not read it and do not understand statistics.

Here's my problem: it's a small bore student paper that sticks closely to it's data, applies the wrong test, but most importantly, doesn't appreciate the role of pricing in an economic system.  You see, price is a measure of quality along with all the measuring systems in the paper, especially in terms of 2005 Bordeaux data set, where the prices have been reacting to scores and consumer demand for about four years.  If you want to to measure point systems against release prices, well... that's another story... one that has nothing to do with consumer attitudes, but it might be interesting.

I urge anyone interested in the subject to read

Also a good read is

The field of wine pricing is rich, really rich.  There's cult wines that price according to perceived scarcity.  Grocery stores pricing is very sensitive to sales figures.  The three-tier system of distribution in this country makes price sensitive discounting an interesting subject.

The important thing to remember, however, is that price and ratings are both measuring the same thing - quality.  In other words, where the market is at equilibrium, better wines will cost more and command higher prices.  Short term disequilibrium will occur where a hidden gem hasn't been discovered yet or an old vintage is discounted to make room in the market for a new release, or a loss leader is advertised to get people into a store.  But in general, the better the wine is, the more it will cost.  And the more ratings points a wine garners from critics, the more it will cost.  Price and rating are both telling the same story (they are both y variables in a study with no x), so to compare them doesn't bring much new knowledge to the table.

I have some specific statistical objections in terms of the study's data that go along with this phenomena that we call multicollinearity.  For example, CT scores suffer from bias because the people who shelled out lots of money are less likely to rate a wine poorly than those who pay less.  Tanzer, at least in the hour I spent exploring his website, does not claim to taste blind and so his score will be biased at least in part by reputation, which is also collinear with price.  Last, the Parker data is mixed with barrel samples that aren't blind and post-release scores that are.  But one cannot draw any conclusions from this study because of the procedural mistakes.  It would be interesting to go back.

I'm actually writing a whole article on this based on Benjamin Lewin, MW writings in his two books and Tong magazine.  Stay tuned!

Customer Loyalty is Irrational

I was listening to NPR this evening on my way home, cause I'm a geek, and I heard a disturbing report.  A little while ago I called one of the banks that issues me a credit card, and asked them for a better rate.  I thought with interest rates going down, my credit ratings, and my payment history, there would be no problems.  But I was wrong!  They offered me only a tiny little decrease, and tonight I found out why.  The bank has created a statistic that measures my  likelyhood of shopping around for a better rate.  In other words, they are punishing me for being loyal.

But they should.  In economic terms, loyalty is irrational.  One should always be willing, for example, to abandon the mom and pop establishment in favor of a big box store that sells cheaper underwear.  Likewise if mom and pop beat the price, we should switch back.  The same is true of wines, except there it's a little harder.  Wine companies go to great lengths to convince us that their products are not interchangeable.  They also like to create exclusive clubs and allocation lists to attract our loyalty.

Times being what they are, however, I would like to give out some free advice to consumers and winemakers.  Consumers, be fickle.  Wineries, you are going to have to be very special to keep us.  Proliferation of internet tools is allowing us comparison of products more than ever before, and the economy is allowing access to "cult" wines like never before.  I can now compare prices of all the makers of Cabernet from the To-Kalon vineyard now.  I am able to order Pinot Noir from almost a dozen places that use the Shea Vineyard.  I can do searches on 90+ Parker wines below $20 on any number of online stores.  For all but the most scarce wine, this is a buyers market.

I'll pick on Araujo a little more, makers of the famed Eisele Estate Cabernet Sauvignon.  For three years, I paid $275 for 1 bottle of each vintage.  Then when they got a great vintage, they raised the minimum purchase to a level I could no longer afford.  I had thought that loyalty through more modest vintages should have meant something, but like my bank, they didn't budge.  But like my bank, they aren't the only game in town.  I can now get Araujo from other sources than their list...and cheaper too.  The winery charges its list members full retail markup on it's wine, but a trip to wine-searcher.com yields 11 stores that sell it for less.  Harlan, cheaper. Diamond Creek, cheaper.  Lokoya, you get the idea.

Wineries, you cannot charge full markup anymore.  I know that the lure of getting about four times more than a wholesaler offers is grand, but it pisses off your loyal customers.  And if a blockhead like me can abandon you, so can other irrational consumers.  And do something about those damn shipping costs!

Statistical Rant Part 1

subtitle - why respected organizations fail to observe basic math skills


Before I became a chef and wine geek, I had another career - I was a DC policy wonk.  I got my BA and MPA in Washington and I lived and breathed politics.  One of my greatest interests was the way studies were touted around town, and I made it my business to study lots of economics and statistics so that I could understand and either support or debunk as needed.  I don't have enough training to call myself either an economist or a statistician...but this post requires only the knowledge I learned in my first undergraduate course.

Last week, the American Association of Wine Economists, of which I am a member, released a new Working Paper titled A Buyer's Dilemma - Whose Rating Should a Wine Drinker Pay Attention To?  The blogoshpere has paid some attention to this paper, including Terroirist, Eric Levine of Cellartracker, and Vinography.

Basically the findings of the paper are that CellarTracker scores more closely track price point than Parker. Tanzer is sort of in the middle.  This is based on a collection of scores and prices for the 2005 vintage in Bordeaux.  The implication is that if you are looking at bang for buck, rely on Cellartracker scores (though the authors to their credit do not make this leap, confining their results to the data - 2005 Bordeaux).  (Disclaimer  -- I subscribe to all three content providers).  But the study is sadly and fatally flawed.  In my next post, I will go into several reasons why the content of the study may be misleading, but today I want to talk about math.

Bang for Buck


Everyone understands the buck part.  The Bang is the thing.  Bang, hedonic qualities, juicy goodness, wow factor, number of stars, number of points.  Bang is not quantifiable in mathematical terms.

All of math is based on the number line that was stuck to your desk in primary school.  1, 2, 3, 4, etc.  The distance between 1 and 2 is 1.  The distance between 3 and 7 is 4.  You get the idea.  There is no rating system in existence that can draw a number line and say the distance, in units of goodness, between wine A and wine B is 12.  Furthermore, I have never seen any of them make that claim.  Parker gives 50 points just for being alcohol; Tanzer clearly states that his system is based on wow factors and not quantification, and CellarTracker lets people grade in any manner they see fit.

No, wine scores are not, as we say in the industry, interval values, like those we find on a number line.  If they were, you could mix a glass with equal parts 92 and 93 point wines and the result would get a 92.5.  No, wine scores are ordinal values, where there is a difference between say, 4 and 5 stars, but we can't really quantify that gap.  Other examples of ordinal values include military ranks, top ten lists, and the finishing places of runners in a race.  The difference between the finishers could be thousandths of a second or hours.  The great consequence to social scientists here is that you cannot add, subtract, divide, and multiply ordinal values; and that is exactly what the authors of this paper have done.

The Desire to Quantify


I understand the desire, the need to make things into numbers and then line them up.  Social scientists are all alike in this respect, they want some data and they want it to say something about the world.  But data in the form of numbers is hard to come by, and way way more often than you might realize, a researcher will clasp onto a set of numbers and run with them despite the warning signs (read footnotes!).  And here I want to emphasize a problem with the toolbox of statistics.  In math, if you plug random stuff into a calculation, it usually won't work.  The equation won't balance, you get irrational numbers, division by zero, an obviously bizzare result, or something.  In statistics, when you plug your numbers into the equation, you always get an answer.  It is the primary job of the statistician no to do math, it is to understand that data, ask the correct questions, perform the apporpriate tests, and correctly understand their result. 

In this study, the authors used a t-test.  Also called a student's t-test, and one with great tradition in the alcoholic world, having been invented to help quality control at the Guinness Brewery in the early 1900s.  I cribbed the text from wikipedia to show what the test looks like:

where

Where s² is the unbiased estimator of the variance of the two samples, n = number of participants, 1 = group one, 2 = group two. Note that in this case, is not a pooled variance. For use in significance testing, the distribution of the test statistic is approximated as being an ordinary Student's t distribution with the degrees of freedom calculated using

That can look scary, but the the top equation is the key.  It is comparing the mean of two samples against a hypothetical "regular" mean.  But you cannot add, subtract, divide or multiply ordinal variables.

Wrong test, meaningless result, even if it churned out numbers that seemed to make sense.  And yes, any freshman in college can tell you this, and anyone who claims to be a wine economist should too.  Not that this study is unusual in making this error.  You see it all the time in studies about health care policy.  But that is another blog entry for a different blog.

WSET Aroma List

The Wine and Spirits Education Trust has a very strict method it teaches for tasting wine. I described the Advanced method many posts ago, and now I am reading about and learning the diploma method in anticipation of new classes starting this week.

In the beginning levels, they don't care much about specifics. Does the fruit taste red or black? Is there citrus? Are the flavors fresh or stewed?  The focus is on big picture and evaluating balance.  And then with each level, the taster is asked to be more and more specific until the diploma level. Here is the approved list of flavors and aromas:

Floral -- acacia, honeysuckle, chamomile, elderflowe, geranium, generic blossom

Perfume -- rose, violet, iris, neroli, honey, soap

Green Fruit -- apple (green/ripe), gooseberry, pear, custard apple, quince, grapey, amylic

Citrus Fruit -- grapefruit, lemon, lime, zest (orange/lemon)

Stone Fruit -- peach, apricot (fresh/dried), nectarine

Tropical Fruit -- banana, lychee, mano, melon, passion fruit, pineapple

Red Fruit -- redcurrant, cranberry, rapberry, strawberry, red cherry, plum (fresh/baked)

Black Fruit -- blackcurrant, blackberry, bramble, blueberry, black cherry

Unripeness -- capsicum, grass, white pepper, leafiness, tomato, potato

Herbaceous -- grass, asparagus, blackcurrant leaf, pyrazine

Herbal -- eucalyptus, mint, medicinal, lavender, fennel, dill

Cooked -- cabbage, tinned vegetables, black olive

Sweet Spice -- cinnamon, cloves, ginger, nutmeg, vanilla

Pungent -- black pepper, licorice, juniper

Autolytic -- yeast, biscuit, bread, toast, lees

Dairy -- butter, cheese, cream, yogurt

Mineral -- earth, petrol, kerosene, rubber, tar, smoke, stone/steel, wet wool

Oak -- vanilla, butterscotch, toast, cedar, charred wood, smoke, acrid, resinous

Kernal -- almond, marzipan, coconut, hazelnut, walnut, chocolate, coffee

Maturity -- vegetal, mushroom, hay, wet leaves, forest floor, tobacco, gamey, savoury

Anisoles -- mustiness, wet carboard, TCA

Brettanomyces -- animal, leather, meaty, wet plaster, vinyl, farmyard

Oxidation -- caramel, toffee, staleness, sherry, aldehydes

Volitile Acidity -- vinegar, solvents, nail polish remover

Reduction -- mercaptans, cabbage, eggs, sweat, rubber, onion, garlic, blocked drains

Other -- beetroot, rot, mold

And to answer your next question, yes, that's it. Any responses not on the list will not be graded. Which is good because it prevents bullshit like "it tastes like the white peach my first girlfriend fed me on a sticky summer....." But it also prevents things like noting the difference between bosc pears and anjou pears, which is huge. Also, being British, they have loads of stuff I've never smelled on this list, like blackcurrant leaf and custard apple. So look for my for my future post where I try to collect all these scents into my aroma kit.

Harvest Report from Virginia

Well if you believe in signs from God, you have to think he doesn't much care for wineries and vineyards in Virginia this year.  After reading lots of weather reports and talking to a few winemakers, my impression of this vintage is that it was quite a challenge to make good wine.

Spring -- Between 40 and 80 days after bud break in the spring, a vine will begin to flower.  The temperature during this period needs to be within a fairly narrow band, between about 58 and 68 F (15-20 C).  If spring is wet and cold as it was in Virginia, many flowers won't get fertilized or will do so at uneven rates.  As the flowers give way to fruit, the crop will be smaller where fertilization failed, and within clusters, there will be big gaps, small berries and big berries.  These 'chickens and hens' will cause problems later becayse they will ripen at different rates.  When a picker is evaluating a bunch and sees some grapes ready to burst with ripeness, and others still green and hard, the correct path is difficult to determine.  A machine harvester is often a better choice in these cases, as it will shake off only the berries with loose (ripe) pedicels, leaving the hard ones.  Also, on the brighter side, these 'shattered' clusters allow more airflow and have fewer problems with mildew and rot.

photo source: viragevineyards.wordpress.com

photo source: shutupandmakewine.wordpress.com


Summer -- This summer was hot and short.  Hot is okay, of course.  Vines need lots of sun and heat to produce great grapes.  The concern here is with heat spikes.  During a heat spike, a vine might shut down and stop photosynthesis.  Without photosynthesis, the grape will not accumulate sugars, and eventually will ripen without being sweet.  There are lots of ways to deal with heat spikes: cool water spritzing, shading with tarps of cheesecloth, big fans to keep air moving, but all are expensive.

Fall and Harvest -- Virginia had the effects of two hurricanes at the end of August and start of September.  In fact, some parts of the state had 10 days or more in a row of rain, with levels more than 4 inches above average.  Many growers brought in grapes in advance of the hurricane Irene, at sugar and ripeness levels below optimal.  Those who waited out Irene, but got caught picking in the 10 days of rain that started in September, picked grapes that were swollen with water and whose flavors will be dilute.  Some used the industrial equivalent of the Dyson Blade hand dryer to blow water off during harvest.  For varieties that ripen later in the month like Cabernet, mold and rot will be a major problem.  A berry that is split by rain swelling contains 40 times more 'organisms' than whole berries.  Whole clusters will form beards.

photo source: jimsloire.blogspot.com

photo source:  ppath.cornell.edu




Options are limited for this smount of soggyness in the vineyard.  There will surely be great pressure to rush the harvest in to limit the damage.  Then only rigorous sorting will prevent very funky wine.


This vintage is an excellent challenge to winemakers indeed and it will separate those trying to make great wine from those content to charge $20 per bottle of crap to tourists.  I suppose I am setting myself up for a future tasting report...shudder...but I am up to the task.


Meanwhile, the weather elsewhere in the wine world has been, um, sub-optimal.  Cold damp spring on the Sonoma Coast has lead to a record late fruit set complete with widespread shatter.  Hail has destroyed large parts of the Mosel and St. Emilion, and Australia continues with a multi-year drought.   

DC is the land of winos!

Hey all, this is just a re-post from bottlenotes, but it suprised the heck out of me!

The United States of Winos

The capital’s other spending habit

It’s no secret that California, New York, Washington State, and Oregon make most of the wine produced in the U.S.* But ever wonder which states drink the most of it? The list of heavy sippers may (or may not) surprise you:

State (Per Capita Consumption 2009 in gallons)**
• District of Columbia (6.6)
• New Hampshire (4.8)
• Massachusetts (4.1)
• Vermont (4.1)
• Nevada (3.8)
• Connecticut (3.6)
• Delaware (3.6)
• New Jersey (3.6)
• Rhode Island (3.4)
• California (3.4)
• Hawaii (3.4)

Income appears to be the best predictor for wine consumption. With the exception of Vermont (in 22nd place), these states all rank in the top 20 for median household income. New Hampshire (1), Connecticut (2), and New Jersey (4) are in the top five.***

However, most of these states happen to drink heartily in general. With the exception of Connecticut, California, and New Jersey, each ranks in the top 18 for alcohol consumption per capita.

Why is D.C. so far ahead of the pack? Foreign embassies and their ample entertaining budgets may explain this outlier. Nevada probably ranks higher than expected too because casinos are liberal with their “free” beverages.

So, no, Washington doesn’t seem to be drinking your tax dollars, however much they might be tempted to.


*U.S. Department of the Treasury Alcohol Tax and Tobacco Trade Bureau **Beer Institute
***U.S. Census Bureau, Current Population Survey, 2008, 2009, and 2010 Annual Social and Economic Supplement

New MWs Announced

The Institute of Masters of Wine has just announced an astounding 11 new Masters of Wine!  Usually, a class tends to be between 2 and 8 people, so this is quite an accomplishment to have so many pass the grueling tests that I hope someday to pass myself.  There are now about 300 MWs in the world.  The test consists of four three-hour written exams on winemaking, viticulture, business and economics, and issues in wine.  There are 3 12-wine blind tastings where you have to write a paper on each wine in about 2 hours.  And finally, there is a dissertation of original research you must write and defend.  It is considered to be the most difficult and prestigious wine  certification in the world.  You can read about the freshly minted MWs at http://www.mastersofwine.org/en/news/index.cfm/id/B66A4C2B-C36C-4EC8-9DB44800C896F55E.