My Review of Tool Reviews
The most common questions I get here at The Wood Whisperer are about tools. Which one should I get? Which is the best value? Which one will make me successful with the ladies? You know, the usual stuff. I always try to give the person my best recommendation based on their budget and the type/amount of woodworking they do. My recommendation usually emanates from the collective experiences of hundreds of woodworkers. Whether its personal emails, forum threads, or magazine/website reviews, its all part of the knowledge base. But I am quick to remind folks that they are receiving one person’s opinion and nothing more.
Now I hope it’s obvious to most of us that Amazon.com reviews and personal opinions should all be taken with a grain of salt. I always tell people to look for general trends from these sources, as opposed to specifics. From my experience, the angriest people are the loudest people, so you need to be a bit guarded when doing this type of research. But a review from a magazine tends to be held in a higher regard as more of a “final word”. The question on my mind, though, has been whether or not I should trust magazine reviews any more than a reasonable complaint from a forum or a nod of approval from a friend. It’s only natural that I would trust the magazines more. After all, a magazine has an entire staff of talented and knowledgeable woodworkers with years and years of experience at their disposal. But more and more lately, I am beginning to take magazine reviews off of the pedestal I had previously placed them. Why? Well, it’s a combination of things: flaws in testing procedures, questionable results, pseudo scientific testing methods, results that don’t coincide with my personal experiences, and finally, results that directly conflict with similar tests done in the past.
Now before I cite a few examples, I would like to tell you why I was motivated to write this article in the first place. Wood Magazine recently published a revised test that corrected some major mistakes made in their December/January tablesaw blade review. I read several forum threads where numerous people were upset because they made a purchase based on the original review. And now that this revised version is out, they are ready to throw out or sell their recently acquired blades. I have to admit, I was a little saddened to realize that so many folks are literally hanging on every word of this review. Some were disgusted and genuinely hurt at being “misled”. This really made me wonder if their trust was misplaced? Or are they just putting too much stock in a review that was never intended to be “the final word” in the first place?
So just for fun, I decided to waste a few hours on a Sunday morning comparing reviews from one magazine to another. I have a limited number of issues in my office and decided to limit my search to that lot. There are playoff games to watch, you know. Please remember, the point of this exercise is not to discredit these magazine reviews. They are very valuable and play a significant role in my own purchases as well as the recommendations I give to my viewers. I am just trying to make sure you see these reviews in the right light. If two magazines give the same tool two very different ratings, is one more right than the other? Well, possibly. But I personally believe that the person writing the article and doing the tests introduces their own “X-factor”, which can easily skew the results one way or another.
I found numerous examples, but here are just a few for your entertainment.
A Fine Woodworking review of miter saws reports on the Makita LS1013: “Because the miter scale rotates with the table, we found it awkward to get a straight view of the side-mounted angle pointer.“ A Popular Woodworking review says the following about the same model (and awards it the Editors Pick): “The miter scale on the table was mounted to the right (a less-standard location that’s convenient when you’re working on the left side of the saw) and was precise and easy to read.”
A Woodworkers Journal jigsaw review highlights the Festool PSB 300EQ as the hands- down winner referring to it as “top of the class in every way.” In a similar shootout, Fine Woodworking gave the tool a B+ for cut quality.
In their recent hybrid tablesaw review, Fine Woodworking awards the Woodtek the distinction of Best Overall and Best Value. No offense to Woodtek, but this was a bit surprising since in most other magazine reviews, Woodtek is treated like the uncoordinated kid who always gets picked last for kickball. So, what did Popular Woodworking say in a similar test of hybrid saws? Well, they didn’t say anything because the Woodtek never made it to the testing floor. They did, however, select Jet and the Steel City for the Editor’s Picks. Ironically, Fine Woodworking reported these two as having the highest rip fence deflection of the bunch and Steel City, specifically, was reported as having the worst rating for blade to miter slot parallelism.
And the last example comes from two articles published by Fine Woodworking. Many of you probably remember the ground-shaking article about clamping pressure in the December 2007 issue. Many of us were left hanging our heads in shame wondering how we would ever achieve the required clamping pressure on our joints. The calculations presented in this article were highly dependent on the max clamping pressures of each clamp type. The end result of an article like this, aside from mass hysteria, is people using those clamping pressures as a guideline for buying their clamps. And based on what I’ve seen recently with the Wood Magazine blade article, I have no doubt that some folks replaced or modified their clamp collections based on this article. Interestingly enough, Fine Woodworking did another article in October 2004 that stated the maximum clamping pressures of the most common clamps. Here’s a quick comparison of reported clamping pressures from both articles. 3/4” pipe clamps: 1050 lbs (2007) and 1200 lbs (2004). Parallel jaw clamps: 370 lbs (2007) and 600 lbs (2004). Aluminum bar clamps: 680 lbs (2007) and 800 lbs (2004). Steel bar clamps 1350 lbs (2007) and 2000+ lbs (2004). So who’s results are correct? The first author or the second author?
My mission here is NOT to pick on our friends at these magazines. I have a great respect for what they do and I believe their reviews can be incredibly useful and insightful. However, feed rates, variations in test boards, flaws in test apparatuses and theory, variability in the tool itself, and opinions, all lead to less than consistent results. These are inevitabilities that cannot be avoided and a publication should not be faulted for the observations they make. What I hope is that you begin to question all reviews and opinions and read them critically. And remember, nothing counts more than YOUR opinion and YOUR experience, so try to get some hands-on time before you make a purchase. When that’s not possible, be sure to include shootout reviews as just one of the many resources you consider before spending your hard-earned money.
Comments
20 Responses to “My Review of Tool Reviews”Speak Your Mind and Maybe Win a Prize!
Leave a comment and you could be a winner in our Wood Whisperer Giveaway!And if you want a pic to show with your comment, go get a gravatar!
Check it Out!
Premium Membership
Keep Up With TWW!
Shop our Affiliates
Latest Video
Proud to be Sponsored By:
Support our Advertisers
Marc,
Great read man! I was just going through this yesterday when deciding on a plunge router. I looked at the Amazon reviews like you mentioned, and found reviews at online sites that specialize in them like consumersearch. In the end I went to the stores, several, and played with all the routers. I looked at Crapsman (for kicks, not seriously :) Porter Cable and Dewalt. The amount of slop in the Craftsman was awful, and I expected it, but what was suprising was that the PC had the same amount :( I thought maybe it was a fluke so went to another store and found the same thing. It is like the bearings on the plunge rails were loose or something, not much, but it doesn’t have to be that much if you are trying to make a perfect, tight inlay. I expected that from the Craftsman, but not the PC’s. In the end, the Dewalt at every store was tight and felt well made so that, and that alone is why I spent my hard earned money on the router. I think that is what you almost have to do, use the reviews as a guide, but in the end YOU make the decision based on what YOU feel is best, not someone doing a review, just like you pointed out. Nice write-up!
Jason
Marc;
Another great post! Like Jason, I’ve been going through the same situation trying to decide which block plane to purchase. It seemed like every review I found had a different “best” plane. I had the same results when reading different forums. I ultimately made my decision after discussing planes with a couple of experienced users. This post just helps remind all of us that the reviews are nothing more (or less) than a guide.
As I believe this article illustrates, you are a wise man. That is why I ONLY listen to YOU. (wink, wink, nudge, nudge). Seriously, great article. As with every decision in life, a person has to do their homework. Caveat Emptor!
Thanks for doing all this research – I think it’s hilarious that people would consider throwing out or selling their blades that last month were so great and this month they’ve been told (lemmings!) that they aren’t! What are they going to do next month when minds are changed again? LOL!
Not that I’m a great woodworker (yet), but every name-brand new blade that I’ve put on my saw cuts fantastically better than the old duller name-brand blade I just took off – with only a few exceptions, the quality differences from one blade to another is just a tiny bit.
It sounds like blades from companies like Tenryu are worth the extra money, and maybe I’ll soon be at the level where I can tell the difference, but doing other things like switching to a segmented belt and solidifying the base of my saw have done more for cut quality than blade changes have.
Love your site man!
As magazines make decisions for the masses: meaning if i buy FWW magazine for great woodworking techniques, well why not beleive them when it comes to tool reviews? Right, can we be blamed if we like the magazines for which we get great ideas, and then take whole heartedly for the tools they say are the best!
A lot of people only have access to magazine articles for info. Personally i think the best reviews are from the internet woodworking forums. Why, cuz people the info based on real testing over long periods, not short periods. I can also agree that amazon.com is another OKAY (not great) resource for making decisions and getting tool review advice.
One thing that is for sure (and i am guilty 100% for doing this): a lot of big box stores carry a lot of quality tools, their return policies are like 30 days with store credit. I have tried before i actually buyed some tools like my routers and tablesaw. It is not something that should be practiced. It is wrong! But i found it the best way to find out what few tools “I” liked and needed. Or for that matter, what tools worked best for my needs.
So stick to the internet for advice or ask someone reputable on a particular podcast we have all know and come to love. TWW. LOL
lates
Hey – great writeup! Well written, wise, and even though your research examples were from a limited set of magazines, they were funny and insightful!
And you didn’t even mention the possibility that results could be skewed by advertising dollars. I’d very much like to believe that all of these magazine tests are done without regard to who’s giving them money, and I sure don’t have any evidence to suggest otherwise. But it’s just another reason to take all this stuff with a grain of salt…
When looking for tool reviews, I too prefer firsthand reports from forums or blogs whenever possible. If that isn’t conclusive, or if the “best” tool is a matter of what fits or feels best, I try to order from a company (like Lee Valley) that allows me to order a few similar planes, try them out, and return the one(s) I don’t like as much. This might be less common with power tools, but in general, I think a good return policy and the ability to try out a tool before purchasing are very valuable.
Thanks again!
I’m not sure I believe your article.
I will have to get a second opinion.
:)
Great article. I think all sources must be taken with a grain of salt. Think about it – even if you, yourself write a review about tools that you have tried – in a year, you might have changed your opinion.
I thought the Asa Christiana interview provided a lot of insight for how tools are tested, and inherent limitations of the magazine reviews.
http://thewoodwhisperer.com/ep.....hristiana/
He also had a good point about the magazines that do “informal tests.” Current issue of Wood Magazine, 182 March 2008, page 22 is a great example. There is no appleapple comparison whatsoever, no coherent theme, and half the text describes hardships the authors had that caused them to need the tool in the first place. It’s just a page filler. Amazon reviews are better because they don’t pretend to be taken seriously. (The dust collector review on pg 66 in the same issue is pretty good though).
It’s kind of funny that something so simple as measuring clamping pressure can show up with different numbers for the new test. But how do you measure it? But who tightens the clamp? My grandmother, or Hulk Hogan?
I can recommend getting used tools from Craigslist. If you don’t like the tool, you can turn around and sell it for about the same price to someone who might like it better.
My advice would be to do your homework, and DON’T READ ANY MORE REVIEWS AFTER YOU BUY THE THING!
Certain tools, like table saws and hand planes, require tuning prior to use. It makes no sense to me to try to use the tool out-of-the-box and then report the result as indicative of the quality of the tool. The only time it might make sense is for a tool where you don’t expect to have to adjust it because of the difficulty in making the adjustment–such as miter slot to blade parallelism on a contractor saw. When minor tuning is done first on each to bring it up to correct operating design we get a level playing field. Fortunately, we’re starting to see this in magazine testing. I’d like to see an unbiased evaluation of HP since we know manufacturers are now able to pretty much claim anything they want–one that reads the actual HP coming off of the arbor instead of the qualitative evaluations of how easily it cuts compared to the other test saws. Current tests are better than those of years ago but they still have room for improvement.
Mark: The Wood Magazine review of dust collectors I mentioned before actually report “Rated HP”, “Amp Rating” and even “Maximum Amps Pulled in Test” which I thought was very useful.
I agree that tuning needs to be done, but I’m not sure I agree about excluding miter slot blade parallelism. The review won’t be fair unless all parts that can be tuned up are tuned up. FWW usually reports when they had to do something, and then test as tuned up. They might even say “I had to file down such-and-such part to get it to fit properly” or something like that. But what they fixed is to some extent anecdotal since it’s just one machine/tool (unless it’s obviously a design flaw).
I’d like to see the face of the the local car dyno operator when you bring your table saw out there :)
I love the honesty of it all. Don’t forget too, that a magazine review is based on just one single machine. The wheels on that bandsaw might be out of alignment more than any other in the test, but it might be the only one out of a thousand that is.
I pay more attention to customer service comments in online reviews than the equipment. If the trend seems to be bad service, I will seriously consider scratching it off my list of candidates. Laguna comes to mind.
In the end, after extensive research, I tend to buy what I think will fit me best. Some don’t even make the mag reviews (Hitachi, Triton, el-cheapo).
That’s my 2 cents.
Another great one Marc. I always read the mag reviews, forums, customer reviews when buying a new tool. But I also pay attention to what I see/hear people using either on TV or the web, like say, Marc and Norm. I know NYW is sponsored by PC and Delta, but I remember reading an article on Norm and they mentioned that Norm will only use a tool that he likes, no matter what. I also buy brands that I already have assuming that I’ve had good luck with them.
Great article, Marc! I became skeptical of magazine reviews back in the early 80’s when a prominent automotive monthly named the Renault Alliance as ‘Car of the Year’. That Renault was among the worst turds to ever roll (and soon die) on US highways. It was painfully obvious Renault paid the magazine serious money to get that Car of the Year award. I worked with a Marine Major who bought an Alliance because it was Car of the Year. Boy, was he sorry!
While I don’t think woodworking magazines are as slimey as car magazines, they obviously have to consider they’re reviewing products which are often advertised only a few pages away. Perhaps a periodical like Woodworking would be more credible since it has no advertising. This concept seems to work well for Consumer Reports. The testers at CU have some bias, but nothing like a magazine running on advertising dollars.
I agree online forums are a good source of tool information, which includes the ever-important customer service factor. Like Amazon reviews, you see general trends mixed with a few extremes on either side.
Keep up the great work!
Reviews are a great starting point to make a buying decision – especially if you have no clue where to start.
Betsy at my local wood pusher has gone on for years about the Tenryu Gold Medal blades and the cost put me off. I had heard a lot of good things about the Forrest WW11 blade also and picked one up at the last woodshow. Nice blade but not a revelation. Then I received a Tenryu Gold Medal as a gift and did some quick comparisons on mahogany cut offs. The Tenryu kicked bumm. Especially over a tired Amana blade that was my workhorse for so long (and I was never all that happy with it).
Which leads up to my main complaint with tests. Rarely are the tools I’m interested in in the same test or included in any test. How often do you see Tenryu blades tested?
Fine Woodworking did a mitre box test a while back but the DeWalts were not listed. The winner – the Bosch unit – while a fine saw, was not compared to any of the DeWalt models I’m also looking at. Now with the Festool unit coming over I’m keen on seeing how it fairs. Not that I’d spend that much money on a mitre box saw unless I won the lottery. But hey, a horse race is a horse race so you still like to see how badly your choice gets beat (or not).
In the end I try to balance what I hear from folks that do similar work to what I’m doing, what the reviews have to say and what the wallet says I can afford. Some days I end up with the editors pick, sometimes I get the Value Tool Winner and other times I shake my head on why on earth they liked a certain tool – but hey, we all have our own way of doing things and some tools just fit us better than someone else.
Only we don’t write the articles.
Kudo’s to Wood Magazine for redoing the test. That was not cheap, but cheaper than dealing with the fall out if they hadn’t.
Excellent article Marc! You made some really great points and I think you enlightened a lot of folks :-) I would really like to see unbiased tool reviews but how can you tell?
I think that FWW does a good job because they try to level the playing field before testing. That’s an important point because tools can vary from one to the other within the same model. For expample, I bought a low angle block plane made by Groz at WoodCraft. I found it horribly out of square and almost impossible to tune. I brought it back and exchanged it for another of the same exact model. The new one was almost perfect out of the box. Go figure.
Bottom line is you have to take everything into consideration. Some reviews are outright biased. Others are not apparently so but there could be advertiser influences in play. There may be other intangibles that affect the reviewers’ opinions too (such as the tool rep taking them out to lunch or an expensive dinner or having a good relationship with them in general). Who knows?
Such information should just be used to help make an informed, educated decision. Ultimately it should come down to what suits you best and makes you the most satisfied.
Ciao,
Joe
For my two cents… I have to say that the user reviews on Amazon are great. They are not going to make your decision for you, but can be very helpful. For example, a couple years ago I was comparing routers, and for one brand I was considering, three different reviewers said that they had problems with the switch, as well as some other issues. All things being equal, which the comparisons were, I am not buying the one where there seems to be a string of people identifying similar problems. Also on the reviews, sometimes people will tell you about features that I would not have thought to look at, or even tell you of alternative items that can be purchased.
Some reviews are suspect. You know some can be immediately disregarded because they sound like they were written by an idiot. Others sound like they were submitted by somebody from the manufacturer. What you can do is look at the “see all of my reviews” link under the reviewer, to see what else they are reviewing. If they are reviewing all one brand, or bashing everything or praising everything, that has to factor into what you want to gain from the review. I believe some of the Grizzly items sound like they have some “professional reviews”. I could be wrong. Some people write really extensive reviews citing pros and cons. It is just like the comment card at the restaurant, except everybody gets to see it.
Anyhow, for my two cents, you have to read those reviews for any tool you will be depending upon.
Carry on…
I have bought a few items based on reviews and been fairly happy… Then I met Rigid tools. I bought those based on reviews in a contractor mag… HOLY COW. What crap. I have found that a lot of their new hand tools aren’t even made by them. Metabo makes there jig saw, campbell hausfeld makes their compressors, just a few examples. Now I do research like all of you have suggested. I have checked out a lot of wood working forums and ya’ll are by far the most enlightened group I have had the pleasure of reading. Marc puts a hip spin on WW i excited o get out there and buy or build something
In defense of RIGID, I own their jointer, table saw, and 12 Volt drill, and 5 in Random Orbit Sander and couldn’t be happier with any of them for the price. Clearly, your’e not buying a cabinet saw for $600, but I don’t know a saw that can beat it in the price range. I couldn’t speak to their compressors or jigsaw, but as a pipe organ builder, I use my power drill for hours and hours every day, and it has taken any number of drops from scaffolding onto hard church floors and still runs like a champ. With everybody making tools for everybody else, I’m not sure the name on the side of a tool means much anymore, best to judge each tool on its own merits.
I resemble your observations and appreciate you taking the time to review some previous articles. Funny how some of the topics are cyclic (I have a stack of older woodworking mags from about 10 years ago that have the same topics, just different tools).
The internet, for as wonderful a machine as it is, is not the end all resource. Neither are magazines. Hands on is great for initial impression, but tells nothing of longevity of tools or customer service one is likely to receive. Best bets are always to buy tools that hundred of other people have proven and have positive comment on. Amazon is great, but as others have mentioned the forums will get more real world response.
Keep it real.
Here are some tips on reading tool reviews. http://tool-rank.com/info/buye.....081024277/