Statistical Rant II - Price is an Indicator of Quality

In my last statistics post, I complained about a paper from the American Association of Wine Economists where the authors played fast and loose with statistical methods to come up  with meaningless results.  In this post, I want to address the setup and aims of the study.

In the interim, several blogs, the winediarist in particular, have held long and fairly tedious chains of comments about the study by people who have clearly not read it and do not understand statistics.

Here's my problem: it's a small bore student paper that sticks closely to it's data, applies the wrong test, but most importantly, doesn't appreciate the role of pricing in an economic system.  You see, price is a measure of quality along with all the measuring systems in the paper, especially in terms of 2005 Bordeaux data set, where the prices have been reacting to scores and consumer demand for about four years.  If you want to to measure point systems against release prices, well... that's another story... one that has nothing to do with consumer attitudes, but it might be interesting.

I urge anyone interested in the subject to read

Also a good read is

The field of wine pricing is rich, really rich.  There's cult wines that price according to perceived scarcity.  Grocery stores pricing is very sensitive to sales figures.  The three-tier system of distribution in this country makes price sensitive discounting an interesting subject.

The important thing to remember, however, is that price and ratings are both measuring the same thing - quality.  In other words, where the market is at equilibrium, better wines will cost more and command higher prices.  Short term disequilibrium will occur where a hidden gem hasn't been discovered yet or an old vintage is discounted to make room in the market for a new release, or a loss leader is advertised to get people into a store.  But in general, the better the wine is, the more it will cost.  And the more ratings points a wine garners from critics, the more it will cost.  Price and rating are both telling the same story (they are both y variables in a study with no x), so to compare them doesn't bring much new knowledge to the table.

I have some specific statistical objections in terms of the study's data that go along with this phenomena that we call multicollinearity.  For example, CT scores suffer from bias because the people who shelled out lots of money are less likely to rate a wine poorly than those who pay less.  Tanzer, at least in the hour I spent exploring his website, does not claim to taste blind and so his score will be biased at least in part by reputation, which is also collinear with price.  Last, the Parker data is mixed with barrel samples that aren't blind and post-release scores that are.  But one cannot draw any conclusions from this study because of the procedural mistakes.  It would be interesting to go back.

I'm actually writing a whole article on this based on Benjamin Lewin, MW writings in his two books and Tong magazine.  Stay tuned!

Customer Loyalty is Irrational

I was listening to NPR this evening on my way home, cause I'm a geek, and I heard a disturbing report.  A little while ago I called one of the banks that issues me a credit card, and asked them for a better rate.  I thought with interest rates going down, my credit ratings, and my payment history, there would be no problems.  But I was wrong!  They offered me only a tiny little decrease, and tonight I found out why.  The bank has created a statistic that measures my  likelyhood of shopping around for a better rate.  In other words, they are punishing me for being loyal.

But they should.  In economic terms, loyalty is irrational.  One should always be willing, for example, to abandon the mom and pop establishment in favor of a big box store that sells cheaper underwear.  Likewise if mom and pop beat the price, we should switch back.  The same is true of wines, except there it's a little harder.  Wine companies go to great lengths to convince us that their products are not interchangeable.  They also like to create exclusive clubs and allocation lists to attract our loyalty.

Times being what they are, however, I would like to give out some free advice to consumers and winemakers.  Consumers, be fickle.  Wineries, you are going to have to be very special to keep us.  Proliferation of internet tools is allowing us comparison of products more than ever before, and the economy is allowing access to "cult" wines like never before.  I can now compare prices of all the makers of Cabernet from the To-Kalon vineyard now.  I am able to order Pinot Noir from almost a dozen places that use the Shea Vineyard.  I can do searches on 90+ Parker wines below $20 on any number of online stores.  For all but the most scarce wine, this is a buyers market.

I'll pick on Araujo a little more, makers of the famed Eisele Estate Cabernet Sauvignon.  For three years, I paid $275 for 1 bottle of each vintage.  Then when they got a great vintage, they raised the minimum purchase to a level I could no longer afford.  I had thought that loyalty through more modest vintages should have meant something, but like my bank, they didn't budge.  But like my bank, they aren't the only game in town.  I can now get Araujo from other sources than their list...and cheaper too.  The winery charges its list members full retail markup on it's wine, but a trip to wine-searcher.com yields 11 stores that sell it for less.  Harlan, cheaper. Diamond Creek, cheaper.  Lokoya, you get the idea.

Wineries, you cannot charge full markup anymore.  I know that the lure of getting about four times more than a wholesaler offers is grand, but it pisses off your loyal customers.  And if a blockhead like me can abandon you, so can other irrational consumers.  And do something about those damn shipping costs!

Statistical Rant Part 1

subtitle - why respected organizations fail to observe basic math skills


Before I became a chef and wine geek, I had another career - I was a DC policy wonk.  I got my BA and MPA in Washington and I lived and breathed politics.  One of my greatest interests was the way studies were touted around town, and I made it my business to study lots of economics and statistics so that I could understand and either support or debunk as needed.  I don't have enough training to call myself either an economist or a statistician...but this post requires only the knowledge I learned in my first undergraduate course.

Last week, the American Association of Wine Economists, of which I am a member, released a new Working Paper titled A Buyer's Dilemma - Whose Rating Should a Wine Drinker Pay Attention To?  The blogoshpere has paid some attention to this paper, including Terroirist, Eric Levine of Cellartracker, and Vinography.

Basically the findings of the paper are that CellarTracker scores more closely track price point than Parker. Tanzer is sort of in the middle.  This is based on a collection of scores and prices for the 2005 vintage in Bordeaux.  The implication is that if you are looking at bang for buck, rely on Cellartracker scores (though the authors to their credit do not make this leap, confining their results to the data - 2005 Bordeaux).  (Disclaimer  -- I subscribe to all three content providers).  But the study is sadly and fatally flawed.  In my next post, I will go into several reasons why the content of the study may be misleading, but today I want to talk about math.

Bang for Buck


Everyone understands the buck part.  The Bang is the thing.  Bang, hedonic qualities, juicy goodness, wow factor, number of stars, number of points.  Bang is not quantifiable in mathematical terms.

All of math is based on the number line that was stuck to your desk in primary school.  1, 2, 3, 4, etc.  The distance between 1 and 2 is 1.  The distance between 3 and 7 is 4.  You get the idea.  There is no rating system in existence that can draw a number line and say the distance, in units of goodness, between wine A and wine B is 12.  Furthermore, I have never seen any of them make that claim.  Parker gives 50 points just for being alcohol; Tanzer clearly states that his system is based on wow factors and not quantification, and CellarTracker lets people grade in any manner they see fit.

No, wine scores are not, as we say in the industry, interval values, like those we find on a number line.  If they were, you could mix a glass with equal parts 92 and 93 point wines and the result would get a 92.5.  No, wine scores are ordinal values, where there is a difference between say, 4 and 5 stars, but we can't really quantify that gap.  Other examples of ordinal values include military ranks, top ten lists, and the finishing places of runners in a race.  The difference between the finishers could be thousandths of a second or hours.  The great consequence to social scientists here is that you cannot add, subtract, divide, and multiply ordinal values; and that is exactly what the authors of this paper have done.

The Desire to Quantify


I understand the desire, the need to make things into numbers and then line them up.  Social scientists are all alike in this respect, they want some data and they want it to say something about the world.  But data in the form of numbers is hard to come by, and way way more often than you might realize, a researcher will clasp onto a set of numbers and run with them despite the warning signs (read footnotes!).  And here I want to emphasize a problem with the toolbox of statistics.  In math, if you plug random stuff into a calculation, it usually won't work.  The equation won't balance, you get irrational numbers, division by zero, an obviously bizzare result, or something.  In statistics, when you plug your numbers into the equation, you always get an answer.  It is the primary job of the statistician no to do math, it is to understand that data, ask the correct questions, perform the apporpriate tests, and correctly understand their result. 

In this study, the authors used a t-test.  Also called a student's t-test, and one with great tradition in the alcoholic world, having been invented to help quality control at the Guinness Brewery in the early 1900s.  I cribbed the text from wikipedia to show what the test looks like:

where

Where s² is the unbiased estimator of the variance of the two samples, n = number of participants, 1 = group one, 2 = group two. Note that in this case, is not a pooled variance. For use in significance testing, the distribution of the test statistic is approximated as being an ordinary Student's t distribution with the degrees of freedom calculated using

That can look scary, but the the top equation is the key.  It is comparing the mean of two samples against a hypothetical "regular" mean.  But you cannot add, subtract, divide or multiply ordinal variables.

Wrong test, meaningless result, even if it churned out numbers that seemed to make sense.  And yes, any freshman in college can tell you this, and anyone who claims to be a wine economist should too.  Not that this study is unusual in making this error.  You see it all the time in studies about health care policy.  But that is another blog entry for a different blog.

WSET Aroma List

The Wine and Spirits Education Trust has a very strict method it teaches for tasting wine. I described the Advanced method many posts ago, and now I am reading about and learning the diploma method in anticipation of new classes starting this week.

In the beginning levels, they don't care much about specifics. Does the fruit taste red or black? Is there citrus? Are the flavors fresh or stewed?  The focus is on big picture and evaluating balance.  And then with each level, the taster is asked to be more and more specific until the diploma level. Here is the approved list of flavors and aromas:

Floral -- acacia, honeysuckle, chamomile, elderflowe, geranium, generic blossom

Perfume -- rose, violet, iris, neroli, honey, soap

Green Fruit -- apple (green/ripe), gooseberry, pear, custard apple, quince, grapey, amylic

Citrus Fruit -- grapefruit, lemon, lime, zest (orange/lemon)

Stone Fruit -- peach, apricot (fresh/dried), nectarine

Tropical Fruit -- banana, lychee, mano, melon, passion fruit, pineapple

Red Fruit -- redcurrant, cranberry, rapberry, strawberry, red cherry, plum (fresh/baked)

Black Fruit -- blackcurrant, blackberry, bramble, blueberry, black cherry

Unripeness -- capsicum, grass, white pepper, leafiness, tomato, potato

Herbaceous -- grass, asparagus, blackcurrant leaf, pyrazine

Herbal -- eucalyptus, mint, medicinal, lavender, fennel, dill

Cooked -- cabbage, tinned vegetables, black olive

Sweet Spice -- cinnamon, cloves, ginger, nutmeg, vanilla

Pungent -- black pepper, licorice, juniper

Autolytic -- yeast, biscuit, bread, toast, lees

Dairy -- butter, cheese, cream, yogurt

Mineral -- earth, petrol, kerosene, rubber, tar, smoke, stone/steel, wet wool

Oak -- vanilla, butterscotch, toast, cedar, charred wood, smoke, acrid, resinous

Kernal -- almond, marzipan, coconut, hazelnut, walnut, chocolate, coffee

Maturity -- vegetal, mushroom, hay, wet leaves, forest floor, tobacco, gamey, savoury

Anisoles -- mustiness, wet carboard, TCA

Brettanomyces -- animal, leather, meaty, wet plaster, vinyl, farmyard

Oxidation -- caramel, toffee, staleness, sherry, aldehydes

Volitile Acidity -- vinegar, solvents, nail polish remover

Reduction -- mercaptans, cabbage, eggs, sweat, rubber, onion, garlic, blocked drains

Other -- beetroot, rot, mold

And to answer your next question, yes, that's it. Any responses not on the list will not be graded. Which is good because it prevents bullshit like "it tastes like the white peach my first girlfriend fed me on a sticky summer....." But it also prevents things like noting the difference between bosc pears and anjou pears, which is huge. Also, being British, they have loads of stuff I've never smelled on this list, like blackcurrant leaf and custard apple. So look for my for my future post where I try to collect all these scents into my aroma kit.

Harvest Report from Virginia

Well if you believe in signs from God, you have to think he doesn't much care for wineries and vineyards in Virginia this year.  After reading lots of weather reports and talking to a few winemakers, my impression of this vintage is that it was quite a challenge to make good wine.

Spring -- Between 40 and 80 days after bud break in the spring, a vine will begin to flower.  The temperature during this period needs to be within a fairly narrow band, between about 58 and 68 F (15-20 C).  If spring is wet and cold as it was in Virginia, many flowers won't get fertilized or will do so at uneven rates.  As the flowers give way to fruit, the crop will be smaller where fertilization failed, and within clusters, there will be big gaps, small berries and big berries.  These 'chickens and hens' will cause problems later becayse they will ripen at different rates.  When a picker is evaluating a bunch and sees some grapes ready to burst with ripeness, and others still green and hard, the correct path is difficult to determine.  A machine harvester is often a better choice in these cases, as it will shake off only the berries with loose (ripe) pedicels, leaving the hard ones.  Also, on the brighter side, these 'shattered' clusters allow more airflow and have fewer problems with mildew and rot.

photo source: viragevineyards.wordpress.com

photo source: shutupandmakewine.wordpress.com


Summer -- This summer was hot and short.  Hot is okay, of course.  Vines need lots of sun and heat to produce great grapes.  The concern here is with heat spikes.  During a heat spike, a vine might shut down and stop photosynthesis.  Without photosynthesis, the grape will not accumulate sugars, and eventually will ripen without being sweet.  There are lots of ways to deal with heat spikes: cool water spritzing, shading with tarps of cheesecloth, big fans to keep air moving, but all are expensive.

Fall and Harvest -- Virginia had the effects of two hurricanes at the end of August and start of September.  In fact, some parts of the state had 10 days or more in a row of rain, with levels more than 4 inches above average.  Many growers brought in grapes in advance of the hurricane Irene, at sugar and ripeness levels below optimal.  Those who waited out Irene, but got caught picking in the 10 days of rain that started in September, picked grapes that were swollen with water and whose flavors will be dilute.  Some used the industrial equivalent of the Dyson Blade hand dryer to blow water off during harvest.  For varieties that ripen later in the month like Cabernet, mold and rot will be a major problem.  A berry that is split by rain swelling contains 40 times more 'organisms' than whole berries.  Whole clusters will form beards.

photo source: jimsloire.blogspot.com

photo source:  ppath.cornell.edu




Options are limited for this smount of soggyness in the vineyard.  There will surely be great pressure to rush the harvest in to limit the damage.  Then only rigorous sorting will prevent very funky wine.


This vintage is an excellent challenge to winemakers indeed and it will separate those trying to make great wine from those content to charge $20 per bottle of crap to tourists.  I suppose I am setting myself up for a future tasting report...shudder...but I am up to the task.


Meanwhile, the weather elsewhere in the wine world has been, um, sub-optimal.  Cold damp spring on the Sonoma Coast has lead to a record late fruit set complete with widespread shatter.  Hail has destroyed large parts of the Mosel and St. Emilion, and Australia continues with a multi-year drought.   

DC is the land of winos!

Hey all, this is just a re-post from bottlenotes, but it suprised the heck out of me!

The United States of Winos

The capital’s other spending habit

It’s no secret that California, New York, Washington State, and Oregon make most of the wine produced in the U.S.* But ever wonder which states drink the most of it? The list of heavy sippers may (or may not) surprise you:

State (Per Capita Consumption 2009 in gallons)**
• District of Columbia (6.6)
• New Hampshire (4.8)
• Massachusetts (4.1)
• Vermont (4.1)
• Nevada (3.8)
• Connecticut (3.6)
• Delaware (3.6)
• New Jersey (3.6)
• Rhode Island (3.4)
• California (3.4)
• Hawaii (3.4)

Income appears to be the best predictor for wine consumption. With the exception of Vermont (in 22nd place), these states all rank in the top 20 for median household income. New Hampshire (1), Connecticut (2), and New Jersey (4) are in the top five.***

However, most of these states happen to drink heartily in general. With the exception of Connecticut, California, and New Jersey, each ranks in the top 18 for alcohol consumption per capita.

Why is D.C. so far ahead of the pack? Foreign embassies and their ample entertaining budgets may explain this outlier. Nevada probably ranks higher than expected too because casinos are liberal with their “free” beverages.

So, no, Washington doesn’t seem to be drinking your tax dollars, however much they might be tempted to.


*U.S. Department of the Treasury Alcohol Tax and Tobacco Trade Bureau **Beer Institute
***U.S. Census Bureau, Current Population Survey, 2008, 2009, and 2010 Annual Social and Economic Supplement

New MWs Announced

The Institute of Masters of Wine has just announced an astounding 11 new Masters of Wine!  Usually, a class tends to be between 2 and 8 people, so this is quite an accomplishment to have so many pass the grueling tests that I hope someday to pass myself.  There are now about 300 MWs in the world.  The test consists of four three-hour written exams on winemaking, viticulture, business and economics, and issues in wine.  There are 3 12-wine blind tastings where you have to write a paper on each wine in about 2 hours.  And finally, there is a dissertation of original research you must write and defend.  It is considered to be the most difficult and prestigious wine  certification in the world.  You can read about the freshly minted MWs at http://www.mastersofwine.org/en/news/index.cfm/id/B66A4C2B-C36C-4EC8-9DB44800C896F55E.

Book Report -- Wine Faults: Causes, Effects, Cures

John Hudelson holds a PhD in Chemistry and has worked for decades in the wine industry.  He created this book from a PowerPoint presentation for wine judges on the difference between a flaw and a fault, but it grew bigger, eventaully becoming a very complete list indeed.

This book is not for the afficianado of wine.  It isn't for sommeliers, or most students.  It is for people with several chemistry classes under their belt who are going to make a living by making wine.  Parts of it are completely impenetrable and had me going to Wiki multiple times, and I still don't understand it!

For example, there are several chemical compounds in wine that can cause or speed up the production of volatile sulfur compounds (bad) in the absence of oxygen.  That is important because most winemakers are trying to avoid introducing oxygen into winemaking.  But there is a chart of half-reactions with their 'redox' potential expresed in volts.  It took a trip to Wiki to figure out why he is listing half-reactions and why that stuff is expressed in volts to begin with; and I never could fathom if the scale was useful or how common any of those reactions are.  Frustrating.

My big takeaway from this book is an appreciation of how complicated the winemaker's process can be.  I used to think of it as a tightrope act between, say, acid and flab, tannin and lack of structure, fruit and barrel characteristics, etc.  But it's more of a dance on a tightrope.  For example, when you start fermentation, every strain of yeast you might select works best under different conditions, conditions you cannot necessarily control.  If it's out of its sweet spot, that yeast will start producing nasty byproducts.  If it is in its sweet spot, it still might produce nasty flavors depending on all sorts of chemical factors.  So dance, motherf*%ker, and remember that nitrogen is bad in the vineyard, but vital in the fermentation tank!

Hudelson also does a great job explaining two issues in winemaking that have been eluding me, the issue of volatile acidity and the play between total acidity and pH.

Total acid is the measure all all the fixed (not volatile) acid in a wine. pH is the measure of the strength of thse acids. We don't really taste the power of acids as much as how much is there, so TA is important. A wine with 5 grams per liter of acid is fat and flabby, and at 9 grams per liter it is very sharp. pH is important for the stability of wine. 4.0 is very unstable and 3.0 is very stable.

Volatile Acidity is acid that is in gasseous form at room temperature.  All wine has a bit as a byproduct of fermentation, usually below the threshold of human perception (700mg/l).  It can seem like consumers are detecting more and more, but that isn't necessarily true.  Screwcaps and other modern closures are preventing VA from exchanging out of the wine as they would in semi-permeable cork.  So just decant or swish your glass and it'll go away.  In my classes, VA got caught up in conversations of sulfur compounds that smell similar and are indeed on the rise.  Also VA can convert to vinegar.

Consider that a pH falls, production of VA in the form of acetic acid rises.  If a high level of alcohol is present, that will convert to ethyl acetate...vinegar.  Lots of new world wines love the profile of low acids because they want a generous rich wine.  They are also picking their grapes later and later to get the most color and fruit flavors.  But picking late means more sugars which translate to more alcohol - vinegar city.

Hudelson also tells us how to get rid of vinegar precursors, but that is really really esoteric, and I think you get the idea.  Lots of information, much of it not useful to all but the hardcore.

The $500 Napa Cabernet

Way back when I was still a policy wonk and wine was just a hobby, I put my name on the waiting lists for all the highly allocated wines I could think of.  Wines like Kistler, Sine Qua Non, Sea Smoke, Screaming Eagle, you know what I mean.  I figured someday I might be able to afford such things, or even be able to flip a few for a quick profit.

Instead of waiting years, I got onto almost all of them when the economy tanked in 2008.  This is one of the reasons I have 500 bottles of wine and no children.  Well the grandaddy of them all came knocking this week, Harlan Estates.  I've purchased his second wine before and several of his wines under the Bond label, but this time, the offer was for a bottle of Harlan Estate 2009.  The cost, $500.

Now this is not a post to bash Bill Harlan and his wines; I am quite a fan.  Consider the words of Mr. Robert Parker Junior, "Harlan Estate might be the single most profound red wine made not just in California, but in the world."  And five hundred bucks for one of the top bottles in the world doesn't seem out of profile when you consider that first growth bordeaux runs $1000 to $1500.  Also consider that my $500 offer is a significant discount from the 'street' price of Harlan, usually between $600 and $900.

But I can't do it.  I cannot order one this year, or probably next.  For those reading, my birthday is coming up...  The amount of money got me thinking about cultish California producers I have spent money on this year.  There are many many options for someone willing to shell out $100 for a Napa Cab, and some really incredible wine.  If I had five hundred bucks, I think I'd buy these five bottles instead of just one Harlan...

Pahlmeyer Proprietary Red 2007
Helen Turley was the wine maker at Pahlmeyer when I started drinking the 1994 vintage.  Her onetime assistant Erin Green now leads the cellar and it just seems to get better and better.  BTW, she makes awesome Merlot and Pinot Noir for the Pahlmeyers too.  The tannins are super-rich and fine grained.  The flavors are, of course, full of fruit, but also minerals of excellent complexity.  Parker and others rain praise on Pahlmeyer, and I cannot believe it isn't more expensive.

Neal Family Vineyards Rutherford Dust Cabernet Sauvignon 2006
I first encountered Neal Cabernet in its basic Napa bottling, and I thought it was solid, but only one amongst many.  Once I tried their single vineyard offerings, however, I knew I had found something special.  My favorite is named Rutherford Dust, after the alluvial soil of the benchland.  Neal eschews any notion that their wine should be drunk early; they are just now releasing their 2006 and the 2005 bottling is still closed and monolithic.  Given time, however, it turns into liquid velvet.

Harris Estate Vineyards Lakeview Cabernet 2008

One of the greatest compliments you can get as a chef is for a person to spend a ton of money at your place, and then thank you for it.  On a trip to Napa a few years ago, the manager of the Bounty Hunter recommended I make the trip way up in the hills above Calistoga to visit Harris.  From the name, I was expecting a corporate setup with tasting room, but ended up sitting in Mike Harris' living room, drinking wine and petting his dog, Jake.  Then I spent a ton of money on his wine and thanked him for letting me do so.  The wines are opulent and powerful, full of pure dark fruit and brambly tannins.

Nickel & Nickel Rock Cairn Cabernet 2008
Nickel & Nickel is the 'single vineyard sites only' project of Far Niente.  If you visit the winery, they actually have boxes of dirt you can sniff from each of their projects.  And they are pretty serious about it, making over a dozen distinct bottlings from Napa. Rock Cairn is sourced from the southern edge of Oakville on a late ripening Western exposure.  It's a very dark wine, full of black cherries, old time red licorice (the kind they used to make with molasses), and tea.  It's approachable young, but has great acid and tannin for aging.

Heitz Cellars Martha's Vineyard Cabernet 2006
In 1999, Wine Spectator magazine named the 1974 vintage of Heitz Martha's Vineyard the greatest wine of the 20th Century.  Certainly one of the most storied sites in Napa, or anywhere else, Martha's vineyard is back after a long replanting project necessitated by phylloxera.  The minty, chocolaty Cabernet is loaded with glycerine and ages like Helen Mirren.