Testing... What is Best Practice for Your Business?

by Stephen Lett

Testing is the way to the Promised Land. Without testing, I’m afraid you are up a creek without a paddle or lost at sea.  Some things are easier to test than others. However, just about everything can be tested. Don’t just assume that something will work. It is always best to test first.  The results might surprise you. This month, I want to discuss testing and what is best practice for your business.

Structuring a test properly means testing only one variable at a time. Don’t try to test too many things at once. Stay focused on keeping all elements the same, except for the one variable you’re testing or changing. For example, if you are testing two offers against one another and against a control group, be sure all variables and conditions are the same for each test group. What’s the difference between a control group and a test group? What you’ve been doing, for example, not offering a promotional incentive would constitute the control. Therefore, doing something different from the control, for instance, offering a promotional incentive to a group of customers or prospects would be the test group.

Also, the same catalog should be mailed to all test panels on the same mail date. The offer should be promoted in exactly the same way: If you’re promoting the offer on the front cover for one test panel, do the same for the other test panel. Promoting your offer on the cover for one group and on the inside order form for another group doesn’t constitute a valid test; the offer’s design must be the same for each group. Thus, the only variable in your testing should be the copy describing the offer.

Over the years, I have developed the following rules when testing orders. I have found through experience that following these simple rules insures an accurate, measurable result and a sound conclusion.

Simple Rules For Testing

1. Clearly define the purpose of the test; define the objective.
2. Prepare a pro-forma; do your financial analysis.
3. Always test against a control.
4. Only test one variable at a time.
5. Don’t test during your peak season (unless you need to).
6. Always re-test against a control or another offer.
7. Make sure your sample size gives you statistically valid results.
8. Source code the control group and test groups properly.
9. Read the results and act on what you see!

A/B Split Tests

A/B splits are the basis for all testing. An A/B split is when a group of buyers and/or prospects are divided into two equal groups on an every-other-name basis for testing purposes.  You need to be sure your sample size gives you statistically valid results. Which groups do we want to test to? Housefile? Prospects? Both? Once you have the segments identified that you want to test; the next step is to determine the quantity. As a general rule for most tests, we need at least 100 orders from any one group to have a valid “read” on results. But sometimes it depends on the leap you will make when rolling out. Assuming we are mailing smaller quantities and looking for a minimum of 100 orders per panel, at a 2% response rate, we need a minimum of 5,000 copies per segment or test cell. For example, if we are testing 3 segments of the housefile and 5 different outside lists, we need to print at least 40,000 copies for our “A” group (the control group) and the same for our “B” group (the test group). Next, we need to assign key codes to every segment for the control group and for the test group. This will enable us to track our results. Proper coding is a must! When writing our merge specifications, you need to be careful to select a cross section of the housefile segments and outside prospect lists so that you select a representative sample across all zip codes. Both groups need to be selected from the same list universe. Here again, you can only change one variable at a time in order to have a valid test. Now you are ready to mail. Both the “A” group and the “B” group need to be mailed on exactly the same day. We cannot have a valid test otherwise. Mailing results should be tracked at least weekly. The mailing needs to be at least 50% complete before any conclusions can be drawn.

When you do any split tests make sure you look at the net benefit of selective binding in order to keep the mailing in one zip stream to maximize your postal discount as opposed to separate zip streams which is more costly (generally at quantities of 300M or more it is more cost efficient to selective bind).  Ask your service bureau to give you the postage estimates both ways (one zip stream vs. two zip streams), to be sure where the cut-off is.  If the postage savings outweighs the price of selective binding, then selective binding is definitely the way to go. How you handle A/B splits is even more important today considering the higher mailing costs that became effective in May.

In my chart below, I have the housefile, rentals and cooperative database lists split into an offer vs. no offer group. In this case, the “no offer” is the control and the “offer” is the test. All of the various lists and/or housefile segments were source coded properly.

TEST STRUCTURE AND RESULTS
SEGMENT	QUANTITY	ORDERS	DEMAND	AVO	RR	RPC
Housefile No Offer	10,000	341	$19,691	$57.76	3.41%	$1.97
Housefile Offer	10,000	405	$24,579	$60.72	4.05%	$2.46
Rentals No Offer *	60,000	1,099	$65,538	$59.63	1.83%	$1.09
Rentals Offer *	60,000	1,063	$66,542	$62.61	1.77%	$1.11
* Tested 5 different lists at 12,000 each net.
Coops No Offer *	40,000	907	$50,000	$55.11	2.27%	$1.25
Coops Offer *	40,000	1,163	$61,600	$52.97	2.91%	$1.54
* Tested 4 different segments at 10,000 each net.

What’s shown are the rolled-up results or totals by offer vs. no offer. In all cases, the offer beat the control as measured by the RPC (revenue per catalog). Based on this information, we are able to draw a conclusion and roll-out this test with confidence.

Contribution Analysis

The results of any test should include a contribution to profit & overhead analysis. On the surface, the test results might look good. However, you need to know if the offer and or test can be cost justified after all expenses have been considered including the cost of the promo or offer. Always do your homework. Know how much you need to increase the response rate and/or average order size in order to cost justify making the offer. For any test to be successful, it must increase the contribution to profit and overhead.

Testing is the critical to your success as a cataloger. Testing is a way to separate opinions from facts. Don’t just assume something like an offer will work. Test. Test. And, re-test!