Inside Ratings and Coffee Review
Readers occasionally call Coffee Review to task for rating coffees too high. (On the other hand, others ask why we’ve never rated a coffee higher than 97. That latter question I’ll save for another time and another blog.)
But in response to those harboring the “too high” suspicion, the first thing I would point out is that in general we only publish reviews of coffees that exceed 87 or 88. That doesn’t mean we don’t rate many more coffees lower than 87, often considerably lower. We just don’t review these many lower-rated coffees. We only publish reviews of the good ones because we generally prefer to reward excellence, not punish failure. If we do identify weakness we try to do it in a more general context in the introductory articles to our reviews, rather than naming names in the reviews themselves. For example, we pointed out the overall deteriorating quality of many Hawaii and Caribbean coffees in our March 2010 article “Island Coffees: Hawaii and the Caribbean,” and gave some reasons for that trend. However, we stopped short of printing reviews of the disappointments that supported that generalization, focusing instead on reviews of those coffees and producers that bucked that trend.
The other exceptions to the we-only-review-87-or-above approach are situations where consumers are limited in their choices. When we review mass-market coffees rather than specialty coffees, for example, we cover the whole range from dismal to pretty good, on the assumption that readers who buy these inexpensive coffees are limited to what they find on the supermarket shelf. Similarly, when we review single-serve pods or capsules that fit only one kind of machine we tend to review a range of whatever is available for that machine, regardless of rating.
To sum up, for a typical review article we sample an average of about 25 to 40 coffees. These coffees have already gone through a selection process. They have been nominated by roasters (or in some cases by consumers) as outstanding coffees, so they presumably represent some of the best options for a given origin or category. Nevertheless, we typically find no more than about one-third of those 25 to 40 coffees worthy of review.
Better, More Distinctive Coffees Mean Higher Ratings
A second development to keep in mind is that the many of the best specialty coffees have gotten better – often much better – over the past two or three years. The leading edge of specialty coffee production, composed largely of smaller, younger coffee companies and a newer generation of more savvy, worldly coffee growers, has raised the bar for quality and distinction in specialty coffee. The historical dynamic of this change is too complex to go into here, but it starts with consumers who are willing to pay higher prices for small, one-time-only lots of exceptional coffees; roasting companies that are willing to spend time and money ferreting out the best small lots of exceptional coffees; coffee growers who understand this situation and are willing to meet or exceed the expectations of both demanding roasting companies and aficionados; and finally the growth of institutions that foster competitiveness in coffee, like the Cup of Excellence and the Barista Guild, and development institutions that fund improvements in coffee production at origin aimed at helping producers meet higher expectations and thus merit higher prices for their coffees.
In other words, the high end of specialty coffee is simply producing better and more distinctive coffees than ever before, coffees that, when they hit their mark, deserve the recognition of high ratings.
The Macro-Lot Challenge
We at Coffee Review find ourselves concerned, however, that we may not be sufficiently honoring the middle range of specialty, those coffees from larger specialty roasters or more traditional smaller roasting companies that are sturdy, pleasing, reasonably distinctive, but perhaps only occasionally exceptional, yet offer a good value for their often modest price and are widely available. For coffees of this kind and price, a rating of 88 may be something to celebrate. We are trying to find more ways to focus on such coffees, along the lines of the “best wines under $15” approach to wine reviewing. For example, this year we are planning a review of single-origin coffees from larger lots of green coffees (at least one hundred 150-pound sacks), which we are calling “macro-lots” rather than the tiny, precious (and often high-priced) “micro-lots” that rightfully yet perhaps dauntingly tend to dominate the highest ratings at Coffee Review.
