Q&A on running good direct marketing tests

direct marketing testsTesting is at the core of direct marketing. It’s what makes direct marketing scientific and accountable.

Unlike mass market advertising, nearly every decision in direct marketing is (or should be) made by the results on a calculator.

But after working in this industry for many years, I’ve discovered that most people have a hard time wrapping their head around even the most basic testing concepts.

So let’s run through a few of these ideas in the form of question and answer.

Q: What’s the first step in any test?
A: Good testing starts with careful thinking. Before you rush into a test, take out a pen and paper and write down the answers to a few basic questions: Why am I testing? What are my objectives? What do I hope to learn? What questions do I want answered?

Which questions are the most important? How complete do my answers need to be? What will I measure with this test? What are the variables? What variables are most important? How will I design the test? How much time do I have for a test? What is my budget for testing? What restrictions are there?

Q: What should you test?
A: What you test should reflect what you believe will make the most difference in your results. For a new direct marketing program, you should focus on big issues such as the product itself, lists or media, marketing strategy, prices, and offers. For an established program, you may want to test offer enhancements, formats, creative execution, and premiums.

Just don’t make the mistake of testing microscopic items such as the tilt of a stamp when you’re not even sure about the best price for your product. Focus on the basics. That’s generally where you’ll make the most impact.

Q: What should be the goal of any given test?
A: Knowledge. Yes, you want to boost response and increase net income. However, building a body of knowledge about what people want and what they respond to is ultimately more important. Testing is about gaining knowledge. Profits come from the application of that knowledge.

Ideally, every test should answer one specific question. For example, “Will adding a lift letter signed by a well-known doctor increase response to our package?” Your rationale may be that since people respond to authority figures, such as doctors, adding a lift letter will increase credibility and boost response.

Your test, then, will answer your question and prove or disprove your hypothesis for a particular product marketed to a particular audience. This is the basis for your growing body of knowledge, the ultimate goal or “big picture” of all your testing.

Q: Why bother with testing when you can just copy what other companies are doing?
A: Every company, market, product, and offer is in some way unique. Learn from others, but consider it a starting point, anecdotal information at best. Most people don’t test as carefully as they should, so just because you frequently see a particular direct mail piece, ad, or broadcast spot doesn’t mean you can count on it being a winner.

Be particularly cautious about copying the ads of big companies whose primary income is generated by means other than direct marketing. This is where some of the very worst examples of direct response advertising are produced.

Q: How do you test a list before you’ve tested a direct mail package?
A: Good question. And how do you test a direct mail package before you’ve tested a list? It’s frustrating. But when you start out, that’s the fix you’re in.

So here’s the advice I give to novice clients: Create a basic advertisement to test the basics. This means you must start with a direct mail package or ad that is straightforward and as simple as possible, using proven formats and techniques and nothing too creative. This will allow you to confidently test lists, media, offers, prices, product configurations, and other big issues. Later, you can move on to control testing to boost results.

Q: When is the best time to test?
A: Yesterday. Today. Tomorrow. Always. You should be testing something with every mailing, every print ad, every broadcast spot, every e-mail. Otherwise you’re wasting valuable opportunities to learn and, therefore, cut costs and boost profits.

Q: How many elements should you test at a time?
A: One. Testing more than one element will make it impossible for you to know what makes a difference in results. If you’re testing price, change only the price. If you’re testing an insert, don’t add anything else. Of course you can certainly have more “cells” in your testing grid to test other elements simultaneously. But each cell must only test one thing.

If you constantly test elements in this way to refine a control, you are taking what I call the “Evolutionary Approach,” slowly and methodically growing a stronger and stronger control over time. If you lack the patience for this or need a big breakthrough, try the “Revolutionary Approach.” This is where you test a whole new idea, the more different the better. Most smart direct marketers use a combination of these methods.

Q: What numbers should you track?
A: That depends on what you want to find out. But the more data, the better. Keep detailed reports on the number of pieces mailed, the number of responses you get, response source, the conversion percentage, the income those responses generate, the average order, the percentage response, your income per thousand, your cost per order or cost per response, your net profit, returns, bad debt, and every other fact that you need in order to calculate specifically how your promotions perform.

Over time, this will be a gold mine of information.

Q: What should you do with test numbers after a test is complete?
A: Analyze your results in writing. You should have an official report for every mailing, ad, and broadcast spot. It should contain a description of the test, the purpose of the test, the components of your package or the elements of your ad or spot, your mailing or media plan, statistical data, complete results, a numerical and verbal analysis, and the action taken or advised as a result of the test.

Q: Besides numbers, what else should you analyze?
A: Look for the “why” in every test. Knowing what works is important. But knowing why it works is even more important. If a lift note boosts response, don’t just settle for “Well, lift notes seem to increase response, so we’ll use lift notes in all our packages.” People don’t respond to technique per se; they respond because a technique does something to persuade them.

Ask “Why?” Why does this particular lift note work? What does it say? What objections does it meet? What function does it serve? When you know the why – or can make a reasonably good guess – you can apply that knowledge intelligently.

Q: How reliable are test results? Should you retest?
A: Initial tests have a nasty tendency to be fluky. So you should always retest, especially when you get a positive result or a significant change in results. It’s tempting to instantly toss anything that fails and adopt anything that wins. But it often pays to be patient. Test it again and see if you get similar results before making a decision.

Q: Should you use test results to determine creative strategy?
A: Absolutely. Mathematical results are the best way to end debates on issues such as creative approaches, formats, and offers. If your creative people are afraid of results that don’t match their ideas, you should strongly consider hiring new people.

Gut instinct plays a big part in the creative process, but no one knows better what will work than your own prospects and customers. Results rule.

Subscribe to FREE Newsletter / Subscribe to blog by RSS or E-mail


4 Responses to “Q&A on running good direct marketing tests”

  1. Justin Norris on September 28th, 2010 5:05 pm

    Great rundown on testing. And a very helpful format for this post.

    Most of the a/b stuff I do is in email, but I find the same principles generally apply.

    It seems that after a while, with a particular list, you get one variable (like subject line or “from” name) to an optimal point and can’t improve it any more. Then you move on to other elements or work on improving other steps in the conversion funnel.

    Does it work the same with paper mail?

  2. Dean Rieck on September 28th, 2010 5:09 pm

    @Justin: Yes, it works more or less the same way in any medium. But the wrench in the works is that even when you’ve found what works best, you have to keep testing because it won’t work best forever. Controls “fatigue” and lose effectiveness over time. Not always, but often.

  3. Ted Grigg on September 29th, 2010 2:44 pm

    Nice post Dean.

    Here are a few ideas for consideration for high volume traditional media advertisers in particular.

    1. Concentrate your traditional media testing during your best season to help fund the testing. Your overall response will be higher for each test whether it is a failure or a success.

    2. I think the goal of testing is response and not refining learning. Too many elements contribute to the success or failure of a successful mailing. And picking the one that will make a big difference is a shot in the dark and takes too long to discover and rollout. For instance, a two page letter for one offer may be displaced by a one page letter for that same offer. But another offer may be the one pager that is displaced by a four page letter. So test whole packages trying to keep the focus or gross change. Tweaking may eventually yield a test breakthroughs (though rarely), but by the time you find and rollout it out, it no longer works.

    Another challenge here is that once you change the environment of one element that brought a breakthrough for one package or offer, you often find that it delivers much lower response rates when inserted into a new environment.

    We do learn from testing, but that is really a byproduct of the goal of creating new controls.

    3. Test lists primarily with your control package by comparing the results for the same quantities of your control lists. Use a “test grid” in order to test packages and lists simultaneously making sure always to include control lists as a benchmark.

    4. In direct mail, the primary response predictors should be tested in this order of priority.
    -Lists/list selects
    -Channel integration such as supporting freestanding inserts with TV or email announcing the arrival of a special offer coming in the mail and so forth
    -Offers that include premiums, pricing, contests surveys or whatever concepts you are testing to best the control
    -Seasonality if that is not known
    -Mailing frequency to the control lists including pre-announcement letters or postcards

    Take care.

  4. Direct Marketing Companies on November 24th, 2010 5:59 am

    Nice tips! When starting out your first direct mail campaign, remember to walk first and don’t over test. Many times, simply having 2-3 good test elements in place will allow you a foundation to start from, where you can continue to test over the course of multiple waves.
    This approach will also allow you to focus your efforts and test smaller mail quantities with more statistical significance.

FREE Newsletter
Get my monthly newsletter and a FREE 16-page Report: 99 Easy Ways to Boost Your Direct Mail Response!
Enter your main e-mail:
Past issues and more info.
Your privacy is guaranteed.