My Review of Tool Reviews

Article - January 13, 2008

The most common questions I get here at The Wood Whisperer are about tools. Which one should I get? Which is the best value? Which one will make me successful with the ladies? You know, the usual stuff. I always try to give the person my best recommendation based on their budget and the type/amount of woodworking they do. My recommendation usually emanates from the collective experiences of hundreds of woodworkers. Whether its personal emails, forum threads, or magazine/website reviews, its all part of the knowledge base. But I am quick to remind folks that they are receiving one person’s opinion and nothing more.

Now I hope it’s obvious to most of us that Amazon.com reviews and personal opinions should all be taken with a grain of salt. I always tell people to look for general trends from these sources, as opposed to specifics. From my experience, the angriest people are the loudest people, so you need to be a bit guarded when doing this type of research. But a review from a magazine tends to be held in a higher regard as more of a “final word”. The question on my mind, though, has been whether or not I should trust magazine reviews any more than a reasonable complaint from a forum or a nod of approval from a friend. It’s only natural that I would trust the magazines more. After all, a magazine has an entire staff of talented and knowledgeable woodworkers with years and years of experience at their disposal. But more and more lately, I am beginning to take magazine reviews off of the pedestal I had previously placed them. Why? Well, it’s a combination of things: flaws in testing procedures, questionable results, pseudo scientific testing methods, results that don’t coincide with my personal experiences, and finally, results that directly conflict with similar tests done in the past.

Now before I cite a few examples, I would like to tell you why I was motivated to write this article in the first place. Wood Magazine recently published a revised test that corrected some major mistakes made in their December/January tablesaw blade review. I read several forum threads where numerous people were upset because they made a purchase based on the original review. And now that this revised version is out, they are ready to throw out or sell their recently acquired blades. I have to admit, I was a little saddened to realize that so many folks are literally hanging on every word of this review. Some were disgusted and genuinely hurt at being “misled”. This really made me wonder if their trust was misplaced? Or are they just putting too much stock in a review that was never intended to be “the final word” in the first place?

So just for fun, I decided to waste a few hours on a Sunday morning comparing reviews from one magazine to another. I have a limited number of issues in my office and decided to limit my search to that lot. There are playoff games to watch, you know. Please remember, the point of this exercise is not to discredit these magazine reviews. They are very valuable and play a significant role in my own purchases as well as the recommendations I give to my viewers. I am just trying to make sure you see these reviews in the right light. If two magazines give the same tool two very different ratings, is one more right than the other? Well, possibly. But I personally believe that the person writing the article and doing the tests introduces their own “factor”, which can easily skew the results one way or another.

I found numerous examples, but here are just a few for your entertainment.

A Fine Woodworking review of miter saws reports on the Makita LS1013: “Because the miter scale rotates with the table, we found it awkward to get a straight view of the side-mounted angle pointer. A Popular Woodworking review says the following about the same model (and awards it the Editors Pick): “The miter scale on the table was mounted to the right (a less-standard location that’s convenient when you’re working on the left side of the saw) and was precise and easy to read.”

A Woodworkers Journal jigsaw review highlights the Festool PSB 300EQ as the hands- down winner referring to it as “top of the class in every way.” In a similar shootout, Fine Woodworking gave the tool a B+ for cut quality.

In their recent hybrid tablesaw review, Fine Woodworking awards the Woodtek the distinction of Best Overall and Best Value. No offense to Woodtek, but this was a bit surprising since in most other magazine reviews, Woodtek is treated like the uncoordinated kid who always gets picked last for kickball. So, what did Popular Woodworking say in a similar test of hybrid saws? Well, they didn’t say anything because the Woodtek never made it to the testing floor. They did, however, select Jet and the Steel City for the Editor’s Picks. Ironically, Fine Woodworking reported these two as having the highest rip fence deflection of the bunch and Steel City, specifically, was reported as having the worst rating for blade to miter slot parallelism.

And the last example comes from two articles published by Fine Woodworking. Many of you probably remember the ground-shaking article about clamping pressure in the December 2007 issue. Many of us were left hanging our heads in shame wondering how we would ever achieve the required clamping pressure on our joints. The calculations presented in this article were highly dependent on the max clamping pressures of each clamp type. The end result of an article like this, aside from mass hysteria, is people using those clamping pressures as a guideline for buying their clamps. And based on what I’ve seen recently with the Wood Magazine blade article, I have no doubt that some folks replaced or modified their clamp collections based on this article. Interestingly enough, Fine Woodworking did another article in October 2004 that stated the maximum clamping pressures of the most common clamps. Here’s a quick comparison of reported clamping pressures from both articles. 3/4″ pipe clamps: 1050 lbs (2007) and 1200 lbs (2004). Parallel jaw clamps: 370 lbs (2007) and 600 lbs (2004). Aluminum bar clamps: 680 lbs (2007) and 800 lbs (2004). Steel bar clamps 1350 lbs (2007) and 2000+ lbs (2004). So who’s results are correct? The first author or the second author?

My mission here is NOT to pick on our friends at these magazines. I have a great respect for what they do and I believe their reviews can be incredibly useful and insightful. However, feed rates, variations in test boards, flaws in test apparatuses and theory, variability in the tool itself, and opinions, all lead to less than consistent results. These are inevitabilities that cannot be avoided and a publication should not be faulted for the observations they make. What I hope is that you begin to question all reviews and opinions and read them critically. And remember, nothing counts more than YOUR opinion and YOUR experience, so try to get some hands-on time before you make a purchase. When that’s not possible, be sure to include shootout reviews as just one of the many resources you consider before spending your hard-earned money.