Testing is at the core of direct marketing. It’s what makes direct marketing scientific and accountable.
Unlike mass market advertising, nearly every decision in direct marketing is (or should be) made by the results on a calculator.
But after working in this industry for many years, I’ve discovered that most people have a hard time wrapping their head around even the most basic testing concepts.
So let’s run through a few of these ideas in the form of question and answer.
Q: What’s the first step in any test?
A: Good testing starts with careful thinking. Before you rush into a test, take out a pen and paper and write down the answers to a few basic questions: Why am I testing? What are my objectives? What do I hope to learn? What questions do I want answered?
Which questions are the most important? How complete do my answers need to be? What will I measure with this test? What are the variables? What variables are most important? How will I design the test? How much time do I have for a test? What is my budget for testing? What restrictions are there?
Q: What should you test?
A: What you test should reflect what you believe will make the most difference in your results. For a new direct marketing program, you should focus on big issues such as the product itself, lists or media, marketing strategy, prices, and offers. For an established program, you may want to test offer enhancements, formats, creative execution, and premiums.
Just don’t make the mistake of testing microscopic items such as the tilt of a stamp when you’re not even sure about the best price for your product. Focus on the basics. That’s generally where you’ll make the most impact.
Q: What should be the goal of any given test?
A: Knowledge. Yes, you want to boost response and increase net income. However, building a body of knowledge about what people want and what they respond to is ultimately more important. Testing is about gaining knowledge. Profits come from the application of that knowledge.
Ideally, every test should answer one specific question. For example, “Will adding a lift letter signed by a well-known doctor increase response to our package?” Your rationale may be that since people respond to authority figures, such as doctors, adding a lift letter will increase credibility and boost response.
Your test, then, will answer your question and prove or disprove your hypothesis for a particular product marketed to a particular audience. This is the basis for your growing body of knowledge, the ultimate goal or “big picture” of all your testing.
Q: Why bother with testing when you can just copy what other companies are doing?
A: Every company, market, product, and offer is in some way unique. Learn from others, but consider it a starting point, anecdotal information at best. Most people don’t test as carefully as they should, so just because you frequently see a particular direct mail piece, ad, or broadcast spot doesn’t mean you can count on it being a winner.
Be particularly cautious about copying the ads of big companies whose primary income is generated by means other than direct marketing. This is where some of the very worst examples of direct response advertising are produced.
Q: How do you test a list before you’ve tested a direct mail package?
A: Good question. And how do you test a direct mail package before you’ve tested a list? It’s frustrating. But when you start out, that’s the fix you’re in.
So here’s the advice I give to novice clients: Create a basic advertisement to test the basics. This means you must start with a direct mail package or ad that is straightforward and as simple as possible, using proven formats and techniques and nothing too creative. This will allow you to confidently test lists, media, offers, prices, product configurations, and other big issues. Later, you can move on to control testing to boost results.
Q: When is the best time to test?
A: Yesterday. Today. Tomorrow. Always. You should be testing something with every mailing, every print ad, every broadcast spot, every e-mail. Otherwise you’re wasting valuable opportunities to learn and, therefore, cut costs and boost profits.
Q: How many elements should you test at a time?
A: One. Testing more than one element will make it impossible for you to know what makes a difference in results. If you’re testing price, change only the price. If you’re testing an insert, don’t add anything else. Of course you can certainly have more “cells” in your testing grid to test other elements simultaneously. But each cell must only test one thing.
If you constantly test elements in this way to refine a control, you are taking what I call the “Evolutionary Approach,” slowly and methodically growing a stronger and stronger control over time. If you lack the patience for this or need a big breakthrough, try the “Revolutionary Approach.” This is where you test a whole new idea, the more different the better. Most smart direct marketers use a combination of these methods.
Q: What numbers should you track?
A: That depends on what you want to find out. But the more data, the better. Keep detailed reports on the number of pieces mailed, the number of responses you get, response source, the conversion percentage, the income those responses generate, the average order, the percentage response, your income per thousand, your cost per order or cost per response, your net profit, returns, bad debt, and every other fact that you need in order to calculate specifically how your promotions perform.
Over time, this will be a gold mine of information.
Q: What should you do with test numbers after a test is complete?
A: Analyze your results in writing. You should have an official report for every mailing, ad, and broadcast spot. It should contain a description of the test, the purpose of the test, the components of your package or the elements of your ad or spot, your mailing or media plan, statistical data, complete results, a numerical and verbal analysis, and the action taken or advised as a result of the test.
Q: Besides numbers, what else should you analyze?
A: Look for the “why” in every test. Knowing what works is important. But knowing why it works is even more important. If a lift note boosts response, don’t just settle for “Well, lift notes seem to increase response, so we’ll use lift notes in all our packages.” People don’t respond to technique per se; they respond because a technique does something to persuade them.
Ask “Why?” Why does this particular lift note work? What does it say? What objections does it meet? What function does it serve? When you know the why – or can make a reasonably good guess – you can apply that knowledge intelligently.
Q: How reliable are test results? Should you retest?
A: Initial tests have a nasty tendency to be fluky. So you should always retest, especially when you get a positive result or a significant change in results. It’s tempting to instantly toss anything that fails and adopt anything that wins. But it often pays to be patient. Test it again and see if you get similar results before making a decision.
Q: Should you use test results to determine creative strategy?
A: Absolutely. Mathematical results are the best way to end debates on issues such as creative approaches, formats, and offers. If your creative people are afraid of results that don’t match their ideas, you should strongly consider hiring new people.
Gut instinct plays a big part in the creative process, but no one knows better what will work than your own prospects and customers. Results rule.