Test. Test. Test!!!

by Stephen Lett

Do promotional offers increase response rates or average order sizes to the house file? What about to prospects? Which offers work best? Do different weights or grades of paper increase results? What affect do various page presentations have on sales? I am sure you have heard a million times that testing is the key to success. It’s true. Most of these types of things can be (and should be) tested before they are implemented. Gut feel is not always enough. And, answers to these important questions can be found if you are willing to take the time to test. This month, we will discuss the importance of testing various offers and approaches before we act.

It is important to understand that not everything can be tested cost effectively. Creative, for example, is not always cost effective to test due to pagination and/or press changes. Sometimes, it’s simply not practical to test ideas. It is important to consider the cost to establish the test, any test, compared with the potential return and/or risk associated with not testing.

But most of the time, it is possible to test before you rollout. For example, promotional offers can easily be tested. Does free shipping work better than a free gift? Does a dollar amount off the order work better than a percentage off? It is not difficult to test these types of offers and you might be surprised by the results. Thinking about increasing (or decreasing) the basis weight or grade of paper you use? This is something that can also be tested. Obviously, we test various new lists before we rollout. Let’s say you want to try featuring merchandise on your covers instead of using non-product covers which you have been doing up until now. You should not make a change like this without first testing because your customer (and prospects) might not recognize you.

Knowing what to test is as important as knowing how to test.  When it is all said and done, you must be able to read the results and know you had a valid test. A test is not a test unless it is structured properly. And, proper structure means testing only one variable at a time. Everything else must remain the same. Don’t try to test too many different things at once and stay focused on keeping all elements the same except for the one variable you are testing or changing. When considering the set-up of the test, think about whether or not the test could have an impact on the subsequent drops.  If so, you might want to keep the test panels split for the subsequent drops to measure the impact over a period of time.
Let’s select something to test and walk through how we are going to structure our test in order to look back and know the affect it had on results. For purposes of this example, let’s assume we have not been featuring products on the cover of our catalog. We have a feeling that showing products on the cover might increase results. Chances are the house file will react differently to this than outside prospects. Here is how we might go about setting up this test:

1. Step #1 – Define the test. What do we want to learn? What is our objective? What end result do we expect? We need to be able to articulate the test and what we hope to accomplish.
2. Step #2 – Which groups do we want to test to? For example, is this a house file test or a test to prospects? In this case, both groups should be tested. A change in the design of the cover could have a different affect on the house file compared with prospects that have not purchased from the catalog previously. Our assumption might be that changing the image on the cover would affect the house file to a greater degree than prospects but we don’t know until we test it.
3. Step #3 – Once we have the segments identified that we want to test; the next step is to determine the quantity so that we can notify the printer of our intentions. As a general rule for most tests, we need at least 100 orders from any one group to have a valid “read” on results. But sometimes it depends on the leap you will make when rolling out. In our cover change example, assuming the total circulation is 3.0 million, you would never feel comfortable rolling on only a 5,000 test (a test of 5,000 would produce the 100 orders). In this case, you would probably want to test at least 5% to 10% of the total or 150,000 to 300,000. Assuming we are mailing smaller quantities and looking for a minimum of 100 orders per panel, at a 2% response rate, we need a minimum of 5,000 copies per segment or test cell. For example, if we are testing 3 segments of the house file and 5 different outside lists, we need to print at least 40,000 copies for our “A” group (the control group) and the same for our “B” group (the test group). If you test specific item treatment in the catalog, you need to consider the rate in which the item will be ordered.  For example, if you want to test featuring one item on a page for best sellers vs. your control layout, this could be viewed as a two part test.  (1) You want to see if there is a lift in overall sales with the new layout and (2) you want to see if the item sells better with more space.  Therefore, you need to determine if the test is large enough to read both goals.  Most likely, you will need a bigger base than 100 orders since not all of the orders will include the item(s) being tested.  For example, for the item(s) you are testing you might expect that .5% will order that specific item. If the test brings in 100 orders, you are likely to receive zero orders on that specific item. 
4. Step #4 – Next, we need to assign key codes to every segment for the control group and for the test group. This will enable us to track our results. Proper coding is a must!
5. Step #5 – It is time to write our merge/purge instructions. We need to be careful to select a cross section of our house file segments and outside prospect lists so that we pick a representative sample across all zip codes. Both groups need to be selected from the same list universe. Here again, we can only change one variable at a time in order to have a valid test. Our instructions need to outline how the test will be set-up and how the service bureau is going to make the selections and how they are going to send the output of the merge to the printer.
6. Step #6 – Now we are ready to mail. Both the “A” group and the “B” group need to be mailed on exactly the same day. We cannot have a valid test otherwise.
7. Step # 7 – Time to begin to read the results. Mailing results should be tracked at least weekly. Our mailing needs to be at least 50% complete in order to draw accurate conclusions. The percent "done" can be determined using historical weekly order curves.
8. Step #8 – Time to fish or cut bait! Assess the results. Did the test work or not? If so, it is time to roll it out. Don’t be afraid to take advantage of what you have learned.

So, how did our test do? Did featuring products on the cover (vs. atmosphere shots) increase results? Were the results we achieved different for the house file vs. to prospects? Here are the actual results of our test:

GOAL: To determine if products on the front cover increase results.
	CONTROL NO PRODUCTS		TEST PRODUCT COVER
	HOUSE	PROSPECTS	HOUSE	PROSPECTS
Quantity	6,132	52,000	6,133	52,000
Response Rate	7.00%	1.34%	4.85%	1.56%
Avg. Order Size	$84.71	$87.38	$86.79	$92.11
RPC *	$5.97	$1.17	$4.20	$1.44
* Revenue per Catalog.

In this case, the house file did indeed react differently than the prospects. The RPC to the house file was $5.97 with our control cover (no products on the cover) vs. $4.20 from the product cover test. Results were almost 42% better with our traditional non-product cover. However, for prospects, we achieved entirely different results. The RPC to prospects was $1.17 for the control cover and $1.44 for the test cover. This means the product cover increased the results to prospects by 23%.  Prospects wanted to see what we were selling on the cover. Previous buyers identified more with our non-product covers which they have seen time and time again. Does this mean we want to produce and circulate two different covers for each mailing; a non-product cover for the house file and a product cover for prospects? Probably not because it would be important to continue mailing the same type of cover the buyer responded too initially. But, it is good that we tested this before we made a “gut” decision to make a change like this one.

When evaluating the performance of the test, it is important to note the effect of the expiration date on the sales curve.  Most likely the test will have a faster curve than the control.  This has to be considered when making a judgment on the performance.

This is only one example of what might happen. In summary, most anything can and should be tested. Don’t just assume what results might be achieved. You could be surprised and the results might negatively impact your catalog. Yes, it does cost money to test but in the end, you will cost justify doing so.