Consumer Reports magazine is no longer a “Best Buy”
By Vindu Goel
Wednesday, February 28th, 2007 at 2:54 pm in 10 MOST POPULAR POSTS, Business, Journalism.
Digg this 
del.icio.us
(UPDATE: I discuss many of the comments on this post in a followup post here.)
Today, Consumer Reports is releasing its annual auto issue, which officially hits newsstands next week. (Mercury News Auto Editor Matt Nauman offers his take here.) Eagerly awaited by many car buyers, the issue analyzes all the latest car models, offers readers recommendations in various auto categories and provides survey results of the reliability of older models.
It’s one of the few subject areas where Consumer Reports is still useful. (Just don’t ask the magazine for advice on infant car seats, and be wary of its cost savings numbers on hybrid vehicles.)
In this age of information abundance, dedicated specialists and passionate individuals analyze virtually every product and service available and share their opinions with the world online.
And in most cases, those tens of thousands of engaged citizens provide shoppers with far more detailed, much more useful information than Consumer Reports’ white-coated lab technicians do.
I say this with a bit of sadness. My father has relied on Consumer Reports for product advice for 40 years. I was a subscriber myself until last year.
But I finally gave up on CR after I found that in category after category, its information was too sketchy, too basic and too unreliable to be genuinely useful.
Buying a coffee maker? Read the user reviews of the top sellers on Amazon.com to find out which ones leak and which ones keep your coffee hot.
Thinking about a new digital camera? Compare actual photos taken with different cameras at Digital Photography Review and dive as deep as you dare into the more-megapixel debate.
Trying to figure out how to save money on your next vacation? Read the articles by respected travel journalists (including a former CR editor) at Smarter Travel for tips on everything from hotel upgrades to frequent-flier seats.
All of them are better, more timely, more thorough sources of information than Consumer Reports. And they’re free!
“Today you can find experts on many of the things we cover at probably half a dozen to a dozen sites,” acknowledged Kevin McKean, editorial director of Consumer Reports in a recent telephone interview with me.
And “when I shop, I look at user reviews all the time,” he said. “You’ve got to have both.”
But McKean argues that only an advertising-free entity that employs experts can provide the neutral, truly informed reviews that put consumers first.
I beg to differ. Too often, Consumer Reports reviews are shallow and unreflective of the real-world experience of actual product users (used cars and cell-phone service are two notable exceptions).
For example, suppose you’re in the market for a new smartphone, as I was last fall.
Consumer Reports, in its classic, fill-in-the-little-bubbles format, gave top marks to Palm’s Treo 700p, one of the best — and most troublesome smartphones — available. (I bought one last fall and love it, but Verizon is shipping me Phone No. 4 as we speak.)
Consumer Reports’ spare August 2006 description of the phone, which can cost $500 before rebates, reads like a puffy Palm product brochure (actually, the brochure is much more informative — see for yourself). CR offered no discussion of the Treo 700w, a similar phone from Palm that runs Microsoft Windows and might be preferred by many users.
The negatives the magazine noted were minor: a small keypad, lack of voice dialing and inability to download music (which a normal user wouldn’t do anyway since you can easily copy MP3s to a memory card and pop it in the Palm).
Consumer Reports repeated the Treo 700p as a top-pick recommendation in a chart in January 2007.
Yet a quick look at the Treo 700p review at Cnet.com, the leading source for consumer electronics reviews, provides an extended discussion of the phone’s strengths and weaknesses and links to reviews of competing products. (Cnet ranks it seventh among smartphones.)
In addition to Cnet’s generally upbeat professional review, 51 users chimed in with their own experiences, including many who complained of the product’s repeated crashes, inability to hook up with Bluetooth headsets and problematic headset jack (all issues that I can personally attest to). Such complaints are common on other user sites, such as Everything Treo. Even a Google search pops up a lot of good reasons to be concerned.
Doesn’t anyone at Consumer Reports look at the Internet when preparing reviews? Wouldn’t incorporating or investigating some of those user complaints lead to better information for CR readers?
The world is still awaiting the magazine’s explanation for its recent fiasco on infant car seats, where it initially claimed that virtually every model available was dangerous. Alarmed federal regulators quickly found fatal flaws in the magazine’s analysis, which Consumer Reports retracted.
I wonder why an apparent massive safety failure of an entire category of products didn’t raise alarm bells at the magazine before publication. A basic “gut check” would have suggested a major testing failure by the magazine or huge holes in government safety standards. Either would seem to demand more double-checking on the magazine’s part.
“We were and are completely mortified that that happened,” McKean said. “But leaving that aside, people should have comfort in recommendations coming out of this place.”
McKean admits that Consumer Reports has been slow to account for real customer experiences in its reviews, although it does poll readers annually on their experience with a variety of products for those famous repair history charts.
The site is starting to launch message boards, but the traffic is minimal compared to the wide world of the Internet.
My other big beef with Consumer Reports is its decision to squander scarce resources reviewing things that are so subjective that any pretense of scientific review is ridiculous.
Remember CR’s conclusion a few weeks ago that McDonald’s coffee is better than Starbucks? Who are they kidding? I loathe Starbucks and all of its faux-Italian ambiance, but there’s no way that Mickey D’s makes a better cup of coffee. (I even did a taste test myself a couple of weeks ago to make sure that I wasn’t being biased against McDonald’s.)
But even if I’m just a misguided coffee snob, why is CR spending money on “trained tasters” to evaluate take-out coffee? Its readers are perfectly capable of figuring out on their own what kind of coffee they prefer or what brand of peanut butter tastes best to them. (McKean defended the coffee tasteoff: “It was a typical quick check of something we found interesting. But the testing that lies behind it was real.”)
If it really wants to serve consumers, Consumer Reports should direct its resources to big projects that individuals, small Web sites and yes, ad-supported media, can’t do well.
For example, why not take a comprehensive look at mortgages: how to shop for them, which major lenders are the best on fees and services, what to watch out for with mortgage brokers, how to avoid subprime loan traps, etc.? (According to CR’s subscription Web site, the last look at mortgages was a thin piece on adjustable-rate loans in 2003.)
While acknowledging my complaints, McKean says that data-hungry consumers like me aren’t his magazine’s core audience: time-starved ones are.
“I’m the average person just trying to get my job done. Every once in a while I buy a digital camera. I buy a car every three or four years. I just bought a 50-inch plasma TV. That’s who Consumer Reports serves par excellence,” he said.
Certainly the magazine is doing something right. Magazine circulation has grown from 3.9 million to 4.3 million over the past three years. And online subscriptions rose from 300,000 in 1997 to more than 2.6 million today.
McKean knows that CR won’t provide enough to satisfy readers on the two or three specific topics they happen to be passionate about. But “in the rest of my life, I still don’t want to be a fool. I want to make good choices.”
I want to make good choices, too. I just don’t think Consumer Reports helps me make them.
[Both comments and pings are currently closed.]
February 28th, 2007 at 3:56 pm
I’m a long-term Consumer Reports subscriber. Whenever I buy something (car, CRT TV, LCD TV, cell phone, cellular provider, DVR, handheld GPS, etc) I always review the Consumer Reports ratings first and then do my own research.
I haven’t been disappointed yet with anything that Consumer Reports has written. They’re unbiased, responsible, and own up to their mistakes immediately when they make them.
It’s really unfortunate that US automakers have consistently (and still) put quality and reliability at the bottom of their priorities. My family used to buy American cars. Then we bought one Nissan to replace an ailing Chevy and we’ve never bought an American car since. We didn’t read Consumer Reports back then, but we all wholeheartedly agree with their latest assessment.
February 28th, 2007 at 4:12 pm
Consumer Reports has been filled with yellow journalism for years - it’s about time people are waking up to the fact that the staff is horribly biased towards unprovable statements. Their Auto reviews (for example) might appeal to sheep who actually believe that only Toyota and Honda build reliable vehicles - but that’s a farce disproved daily by hundreds of online sites sharing tens of thousands of real world reports. They reek of inability and poor testing methodology. I recommend that everyone cancel their subscriptions to Consumer Reports.
February 28th, 2007 at 4:37 pm
I gave up on Consumer Reports when they tested tennis balls by putting them in a clothes dryer to test endurance of the nap. How silly is that? Any tennis player knows how different tennis balls behave. I guess no one at CR plays tennis.
February 28th, 2007 at 5:00 pm
I agree with much of what you say about CR, although I do find some value in their reviews sometimes. I’d check them if I were buying a new washing machine, for instance. But I’m frequently frustrated by the limited number of products they test in a category, and how often the ones they pick aren’t to be found near me. I dig into user reviews on various websites, but they can also drive you crazy with unrigorous opinions, and some reviewers seeming to be out to trash products for their own perverse reasons. It’s a consumer jungle out there!
February 28th, 2007 at 6:07 pm
I agree more with Mark J than with Vindu. Part of what everyone should keep in mind is that CR use to be the *only* (unbiased) source for the products they evaluated. Now there are additional review sources, but many of the online/blog reviews by individuals can be even less informative and helpful than CR. I use CR as a good foundation, and seek out other reviews when what I get from CR isn’t sufficient for my needs. I tire very quickly of the many sites that have reviews from individuals - too many times, they either gush glowingly or have an axe to grind. There are some notable exceptions, but the consumer has to be even more careful of a buyer when relying on what a small self-selected set of reviewers provides, each message in it’s own context.
March 1st, 2007 at 10:52 am
I’ve been shopping for High Def TVs. I started out at Consumer Reports, where they’ve apparently done pretty extensive testing. Toshiba dominated their ratings for 50 in. or larger projection HDTVs.
I went to cnet.com to do some follow up, and was surprised that not a single Toshiba came up as a highly-rated projection HDTV. I did find that eight of the top 20 50-in.or larger HDTVs were manufactured by sponsors of or advertisers with cnet.com
This is not a scientific study obviously, and the only way to judge in this case is to go and look at the TVs and make your judgment. And this article has given me ideas and other places to check out. But I will continue to pay for consumer reports online magazine to get the unbiased approach…it’s only like $15 per year anyway.
And I find user review flawed in the sense that you may read ten bad reviews at cnet, yet maybe tens of thousands of that product have been sold, so is it really a representative sample. Plus, I think people are more apt to write a review when angry then when satsified.
March 1st, 2007 at 12:33 pm
I admire CR’s desire to provide unbiased reviews. Unfortunately, the Internet has made their hard copy distribution obsolete. Checkbook Magazine faces the same problem too.
March 1st, 2007 at 5:40 pm
I gave up on CR looooong ago, when they dared to recommend what bicycle one should buy. When I first got into cycling, I had the advantage of being able to rely on the judgment and experience of knowledgeable friends. As I became more involved, I learned more. By the time I saw CR’s reviews, I knew enough to know what a good bike is, how little or much one should cost and what the LBS (local bike shop) can do to help us keep our bikes going. CR revealed pretty quickly that they knew absolutely NOTHING about what they were recommending. I quickly drew the conclusion that, if they know nothing about bikes, how can they claim to know anything about any product I know next to nothing about but need to learn in order to make a wise purchase? Online reviews I can take with a grain of salt. But recommendations from people I know (and know a bit about what they know) are my first line of defense.
March 2nd, 2007 at 4:01 am
I beg to differ. Many of the Internet “evaluations” are tained. I tried to leave honest and negative feedback on the RubberMaid Big Max Jr. shed on the Home Depot site (an exclusive product for them) and it never showed up. Nor did they respond to my email inquiring why. RubberMaid initiatlly brushed me off but when i complained again they sent me a new part and an accessory pack as consolation. Nonetheless, the fact that there were two defective parts and really poor finish on a $400 product won’t be known by consumer. Long live independent Consumer Reports.
March 2nd, 2007 at 8:28 am
As far as cars go, CR is spot on. My first car was a VW and my second a Mitsubishi and they broke exactly when and how CR said they would. My two Hondas are now 15 and 12 years old and have never had any problems - just as CR predicted. The money I spent on repairs on the VW and Mitsu would have bought a 300 year subsctription to CR (never mind the aggravation). Which would be money better spent?
CR is right. They serve very well the casual buyer of consumer items like cameras, cam-corders, vacuum cleaners and washing machines. I buy one of those every 10 years or so. The specialty sites give great info for the fan who buys a new digital camera every year (so who cares about long term reliability?) but with much more tech detail than I care to slog through or can understand.
I’m fairly tech savvy but if I’m looking for a digital camera or new TV, CR provides basic insights for the casual buyer and statistically valid reliability surveys I’d never find at the fan sites or user reviews.
March 2nd, 2007 at 1:26 pm
CR for the last few years has consistently gone for LOWER price over quality. That’s when I gave up on them. If an item was way better and cost more, it would receive a lower grade. Plus, as the bike rider said, if you knew a thing or two about a category, and then read their reviews, you would be horrified.
March 2nd, 2007 at 5:34 pm
Vindu’s piece is well-argued, interest and at least partially right. In the few areas where I believe myself to be an expert, CU falls short at times.
But the magazine really does seem to know who their customer is - someone who needs a quick and impartial overview. And since I only use the magazine if my knowledge isn’t very deep, I’m not even sure which Web sites or blogs offer honest and informed opinions.
When I do have some depth of knowledge, I know where to turn to tunnel in - and that’s not CU. Still, like someone else said, it sure comes in handy for times I’m buying a washing machine or dishwasher.
March 2nd, 2007 at 5:54 pm
Back in the day, there were two magazines, Consumer Reports and Consumers Research. As I recall, one was started by dissents who had left the other organization, but I could be wrong on that. But the important point was that there were two magazines offering competing viewpoints. It was a healty time.
I haven’t read CR in years, but when I did, I found the paint tests the magazine’s best feature. I’m never going to buy 20 brands of paint and test them side by side for coverage and fading — that’s a job for someone with a small budget and plenty of time.
I quit reading CR about the time it compared the handling of a high-end sports car to an SUV!!!
March 2nd, 2007 at 6:06 pm
Until recently I was a reporter working in a DC bureau. Over the summer Consumer Reports released their “nursing home quality monitor” to advise consumers which homes to consider and avoid in their states. We had to kill the story in our paper because there were too many errors in CR’s data, and they actually blasted one of our local homes in their press release as a horror house, and then included it on the list of “homes to consider” in the actual report itself. I’ve seen several inconsistencies like that in CR’s work.
They have long been a shoddy outfit.
March 2nd, 2007 at 6:35 pm
I, too, used to depend on CR to validate purchases, but dropped it when it seemed to have more and more about cars and less and less about everything else! I’ve not bought a new car since 1984. The internet review sites are far more valuable (and current) than CR.
March 2nd, 2007 at 8:44 pm
I still remember when CU told people they didn’t need anything bigger than a 19″ TV!
Really, they wrote that.
Or when they told people they don’t need electric door locks on cars.
And you can’t email them unless you pay extra per year.
Once a year you get the annual questionnaire & ballot for CU’s directors.
They always include a small piece of paper asking for donations to cover the approx. $700,000 cost of mailing the questionnaire & ballot.
With a circulation of around 3 million all they have to do is raise the subscription price by 40¢ a year to cover this.
March 3rd, 2007 at 10:49 am
Consumer Reports was enthusiastic about a Volkswagen we subsequently bought; it was a lemon. We have been driving Chevrolets since the VW, and we have been very happy.
After we bought a frontloading washer, I noticed some loads had a lot of twisting, particularly the flannel sheet loads and the loads with a lot of long-sleeved knits and knit pants. A reader wrote into Consumer Reports to complain about twisting, and the editors brushed it the complaint off. And they’ve never written about the problems frontloader owners have finding high-efficiency (low sudsing) detergents that are fragrance-free. Manufacturers of detergents tell machine owners just to use the regular detergent, but less. Washer manufacturers say never to use regular detergent. Consumer Reports says nothing.
The annual questionnaire is too complicated to fill out and doesn’t collect information that consumers might volunteer about products that CR editors don’t care about; or at least, that was the case the last time we bothered to look at the thing.
Vindu is right about mortgages. There are a lot of housing-related issues the magazine could address. They don’t; Fine Homebuilding, published by Taunton, does a better job of discussing construction-quality issues, energy efficiency and right-sizing of space.
Finally, Consumer Reports isn’t good at the “personal preference” issues like how people like coffee, or whether people want a really lightweight vacuum cleaner, or whether people just want to log on instead of having a stack of magazines in the family room.
March 3rd, 2007 at 11:00 am
I lost confidence in CR when they listed the Corvette and it’s fiberglass body panels as average in rust through. Shortly thereafter they gave different grades to two cars that were the same except for the name badge. (It may have been the Dodge Shadow and Plymouth Sundance.)
March 3rd, 2007 at 11:14 am
>>> For example, why not take a comprehensive look at mortgages
The more one knows about a particular topic, the more you realize how little CR knows about it, or how swayed they are by de rigeur media hysteria.
As a mortgage broker, I was mystified by some of their recommendations in their last home financing article. If one listened to their recommendations, you could easily find yourself sending the bank a couple of hundred extra dollars per month for no good reason.
Like every other consumer topic, CR is outdone by either industry experts or online consumer feedback (epinions.com, etc).
Ya want good mortgage advice, free? Check out The Mortgage Professor, Jack Guttentag. But don’t listen to CR.
March 3rd, 2007 at 3:18 pm
The supposed Amazon.com link about coffee makers goes to a page about Japanese sex slaves. Nice.
March 3rd, 2007 at 7:00 pm
Len,
Sorry about the bad link. It’s now fixed and sends you to Amazon.com’s best-selling coffee makers.
If you’re wondering what happened: I originally linked to Amazon.com’s home page, but for some reason Amazon rejected the link, so you got bounced back to the latest entry on my blog, which happened to be about the uproar over Japan’s enslavement of women during World War II.
I double-checked all the links in this post, and they all should work fine.
March 5th, 2007 at 6:13 am
I went to Digital Photography Revied and found most of the reviews were outdated. I got more from Consumer Reports digitqal clamera ratings which were up to date and more helpful.
March 5th, 2007 at 9:50 am
You lost me when you suggested using Amazon reader recommendations over Consumer Reports. Yeah, right.
March 5th, 2007 at 11:41 pm
Some readers might come to similar conclusions if they substituted “San Jose Mercury News” for “Consumer Reports.” Sadly, it’s not just good ol’ CR that must find a way to be useful and informative frequently enough to keep folks from switching to online venues that can satisfy special cravings better than a general-interest publication. One might argue that CR and newspapers are the publishing industry’s version of the 5-and-Dime.
March 13th, 2007 at 8:31 pm
I still subscribe to Consumer Reports online, but I agree that they’re falling behind.
If you’re willing to brave the vast number of ads - I’ve found http://www.consumersearch.com valuable. It acts as a review consolidator, and tends to give more current, broader advice.
March 15th, 2007 at 6:58 pm
[…] I was humbled and gratified by the many thoughtful comments I received from readers — on the blog and via email — in response to my recent post about Consumer Reports magazine. […]
March 22nd, 2007 at 2:44 pm
[…] I’ve criticized CR’s overreliance on its own testing and its failure to incorporate outside views in its ratings. CR said the outsiders that it called in to examine what went wrong — Kennerly H. Digges, former director of Vehicle Safety Research at the National Highway Traffic Safety Administration, and Brian O’Neill, former president of the Insurance Institute for Highway Safety — found its culture of insularity contributed to the car-seat fiasco: The series of misjudgments, Digges and O’Neill said, stemmed mainly from CR’s decision to develop and run the side-impact tests without extensive consultation with other experts. We took that step based on our decades-long experience with front-impact simulations — CR was among the first to test child seats this way, back in 1972–as well as our practice of limiting contact with government and industry to avoid influencing the independence of our judgment. […]
June 22nd, 2007 at 10:16 am
I will agree that some of CU’s reporting on tech is a bit wanting but remember who their prime demographic is, probably not a blog-reading internet user from San José.
We have followed their reports for vacuum cleaners and clothes washers with great success and substantial savings. Even their digital camera review was helpful in narrowing down the field of choices.
By your logic, Wine Spectator is also redundant and yet how many people use their numbers to guide their wine buying.
I like the idea that CU is independent and ad-free, as they old saying goes, “Follow The Money.”