How reliable are wine judges?

Does cultural background influence wine judging? How does gender affect wine scoring? What about age?

Christian Wolf, the tasting director of Meininger Verlag, decided to find out.

“I was just interested, because we see a lot of things which say that women are tasting different to men, or there is a specific German way of describing wine or that people from Australia judge in a different way,” he said.

Wolf is the perfect person to test the proposition, given that he’s the organiser of MUNDUS VINI, one of Europe’s largest wine competitions.

The experiment

Twice a year, MUNDUS VINI brings more than 280 international judges together to taste in groups of six or seven. For the summer edition in 2019, Wolf created eight special groups: there was a table of women; a table of men; a table of Germans; a table with older jurors; a group of younger jurors; people drawn from the wine trade; a table of sommeliers; and a table of wine writers.

Each group was presented with a flight of wines that was also given to another table, while individual wines were doubled and given to different juries.

Regardless of age, gender or nationality, the wines received the same scores.

“It was very interesting to see that the scoring between those tables, young and old, male and female, was really very close,” said Wolf. 

There was one exception – the scores varied a lot when it came to Brettanomyces, with older judges penalising Brett-affected wines much more than younger judges. Wolf isn’t sure whether this is because older judges have the experience to recognise it, or because younger tasters are more open.

“Overall, it was interesting to see that the higher the professionalism of the people you are inviting, the less important is their age, their sex, is their origin or their profession,” said Wolf.

His thoughts echo a study by statistician Jeffrey Bodington, published in the Journal of Wine Economics in 2017, which analysed the results from 23 wine competitions. It found that male and female judges assigned much the same scores.

“I cannot find a statistically significant difference between the scores assigned to the same wines at the same time by male judges, female judges or judges of different nationality,” he wrote to Meininger’s.

As to the question of whether it’s education or experience that makes the difference, Bodington wrote, “I fear that a lot of experience may not be worth much all by itself. Running slowly for years doesn’t make you fast in a sprint. I suspect that good wine education may count for a lot more than years of experience.”

A different view

The results from MUNDUS VINI seem to contradict the work of statistician Robert Hodgson, who began to doubt the value of competition medals after he saw wines do well at one tasting, only to tank at another. In 2005, Hodgson convinced the organisers of the California State Fair wine competition to let him present panels not just with doubles of the same wine, but with triples – that is, judges were presented with the same wines three times. Hodgson found that “chance has a great deal to do with the awards that wines win,” as he later told The Guardian.

This has not been Wolf’s experience.

Wolf did quite a few experiments in 2019. In June 2019, he also organised MUNDUS VINI Nordic in conjunction with the German Wine Institute. Held in Copenhagen, it hosted more than 50 wine professionals from around Scandinavia, who tasted and rated around 660 wines from 200 German producers. Among those wines were some that had already won medals at the spring tasting.

“It was interesting to see that the results of the scoring were nearly the same,” said Wolf. He said that if a wine had won a Gold medal in the spring tasting, it would also win one in the Nordic competition.

The judging process

MUNDUS VINI uses a modified version of the OIV judging system, where judges focus on one wine at a time, rather than comparing them to others in a line up. The judges must also fill in a detailed sheet, rating the wines on a number of specific attributes. Wolf thinks the scoring system, often criticised for being bureaucratic, is the key to the consistent results. “You can’t just decide to give a wine 93 points,” he said. “You have to go through a process and focus on different aspects. When we look at the sheets later, we can see why a judge scored the wine the way they did.”

Wolf said the results are so consistent, MUNDUS VINI can give producers feedback on what judges thought about the wines. “We can show them where the wine was excellent, where it was good and where, in our opinion, it was not as good as it could be,” he said. The producers also get a spider graph of the wine’s attributes, so they can compare how their product performed compared to others.

Of course, the results from Wolf’s 2019 experiment are, as yet, simply interesting. To determine if he has really discovered something, the experiment needs to use a much bigger sample size. Wolf said it’s something he’s planning to do, with a view to eventually publishing the data.

Getting a big enough sample size won’t be a problem, fortunately, as MUNDUS VINI attracts 12,000 wines a year, from 45 countries.

The question at the heart of this may not be about individual tasters and whether they can reproduce their exact results in tasting after tasting – which is probably unlikely, given that humans aren’t machines – but whether the judging system itself can produce consistent results. 

So far, according to Wolf, it does.

Felicity Carter

MUNDUS VINI is organised by Meininger Verlag, the publisher of Meininger’s Wine Business International. Felicity Carter is a regular judge. She was also, apparently, part of the experiment, being placed on the all-female panel. She only found out when she wrote this piece.