# Size does matter when it comes to testing

You may be making the wrong marketing decisions if you're not using statistical analysis to read your test results. Statistical analysis is complex, but statistical models on the Internet make it easier.

Statistical analysis is important because if test sample sizes are not big enough, then the results may not represent the real world results when taking products or creative to market.

The first thing to look for when testing is difference in response. There are models that show positive and negative difference. For this article, positive differences will be used.

For example, an A/B split Web page test to 4,000 consumers (2,000 for each page) that generates a 2 percent response for page A and a 2.4 percent response for page B would give the appearance page B is 20 percent better than page A.

However, the 20 percent lift is an estimate based on a single experiment with a small sample size and relatively low response rates. The real world rollout may not actually be a 20 percent lift. In fact, there could be a decrease in response. An analogy is flipping a quarter 5 times. It may land on its head 4 out of the 5 times even though there is a 50/50 chance it will land on heads or tails. You never can be 100 percent certain of results because you are always just taking an estimate, unless the test was run on 100 percent of the population.

There is an inverse relationship between the sample size and the response rate. The higher the response rate, the lower the required sample size. The lower the response rate, the higher the required sample size to get the same level of confidence, or statistical validity, in a test.

When testing most new concepts, a company could start with a fairly small sample size and a 90 percent confidence level. In that case, the test and response rates above would require 5,000 views per page for it to reach the conclusion that page B is better than page A. A company could reach this level of confidence because if it repeated the test, based on statistical analysis, 90 percent of the time page B would have a higher response rate than page A. However, such a company may be taking some risk if it rolled out Page B because there would be a 10 percent margin for error that the result would not be replicated if tested again or rolled out. Another way to look at this is to say: 9 out of 10 times there should be a lift in response with page B… and 1 out of 10 times there may not be a lift in response. This "margin of error" is the same given in political polls and other surveys.

It is also important to understand that just because the stat model shows a 90 percent confidence level with a 20 percent response lift… it does not mean there will always be a 20 percent lift in response for page B. There may only be a 5 percent lift, or 30 percent lift, but there should be a lift, 90 percent of the time.

The second thing to look for testing is the real response lift. To validate the exact amount of the lift in response, run tests to an even larger sample size. In the example above, to be sure the real response lift was at least 10 percent (yielding an overall response rate of no less than 2.2 percent) using 2 percent response for page A and 2.4 percent for page B, a sample size of 16,000 page views would be required for each page to see that the 2.4 percent response rate of Page B would not drop below 2.2 percent... 90 percent of the time. Or, 1 out of 10 times there may be a difference less than 10 percent.

There are other statistical models and approaches as well as many other facets to statistics just as there are other factors that go into making business decisions - knowledge, experience, instinct.

Even with all the statistical models, some things may not replicate or may be inexplicable. To paraphrase Aristotle, 'It is very probable that many improbable things will happen.' But there are good statistical tools to help verify sample sizes and test results to help make the right decision, most of the time.

## How to prospect for new list business

 Best of... Weekly Daily Insider Events Weekly Hot List of the Day Sponsored Promotion Whitepaper of the Day DMNTech Roundup DMNTech Sponsored Promotion United States United Kingdom Canada Afghanistan Albania Algeria American Samoa Andorra Angola Anguilla Antarctica Antigua and Barbuda Argentina Armenia Aruba Australia Austria Azerbaijan Bahamas Bahrain Bangladesh Barbados Belarus Belgium Belize Benin Bermuda Bhutan Bolivia Bosnia and Herzegovina Botswana Bouvet Island Brazil British Indian Ocean Territory Brunei Darussalam Bulgaria Burkina Faso Burundi Cambodia Cameroon Cape Verde Cayman Islands Central African Republic Chad Chile China Christmas Island Cocos (Keeling) Islands Colombia Comoros Congo Congo (DRC) Cook Islands Costa Rica Côte D'Ivoire Croatia Cuba Cyprus Czech Republic Denmark Djibouti Dominica Dominican Republic Ecuador Egypt El Salvador Equatorial Guinea Eritrea Estonia Ethiopia Falkland Islands (Malvinas) Faroe Islands Fiji Islands Finland France French Guiana French Polynesia French Southern Territories Gabon Gambia Georgia Germany Ghana Gibraltar Greece Greenland Grenada Guadeloupe Guam Guatemala Guinea Guinea-Bissau Guyana Haiti Heard Island and Mcdonald Islands Honduras Hong Kong Hungary Iceland India Indonesia Iran Iraq Ireland Israel Italy Jamaica Japan Jordan Kazakhstan Kenya Kiribati Korea Kuwait Kyrgyzstan Laos Latvia Lebanon Lesotho Liberia Libya Liechtenstein Lithuania Luxembourg Macao Macedonia Madagascar Malawi Malaysia Maldives Mali Malta Marshall Islands Martinique Mauritania Mauritius Mayotte Mexico Micronesia Moldova Monaco Mongolia Montenegro Montserrat Morocco Mozambique Myanmar Namibia Nauru Nepal Netherlands Netherlands Antilles New Caledonia New Zealand Nicaragua Niger Nigeria Niue Norfolk Island North Korea Northern Mariana Islands Norway Oman Pakistan Palau Panama Papua New Guinea Paraguay Peru Philippines Pitcairn Poland Portugal Puerto Rico Qatar Reunion Romania Russia Rwanda Saint Helena Saint Kitts and Nevis Saint Lucia Saint Martin Saint Pierre and Miquelon Saint Vincent and The Grenadines San Marino Sao Tome & Principe Saudi Arabia Senegal Serbia-Montenegro Seychelles Sierra Leone Singapore Slovakia Slovenia Solomon Islands Somalia South Africa Spain Sri Lanka Sudan Suriname Svalbard and Jan Mayen Swaziland Sweden Switzerland Syria Taiwan Tajikistan Tanzania Thailand Timor-Leste Togo Tokelau Tonga Trinidad and Tobago Tunisia Turkey Turkmenistan Turks and Caicos Islands Tuvalu Uganda Ukraine United Arab Emirates Uruguay Uzbekistan Vanuatu Vatican City Venezuela Vietnam Virgin Islands Virgin Islands, British Wallis and Futuna Western Samoa Yemen Yugoslavia Zambia Zimbabwe

## Company of the Week

Brightcove is the world's leading video platform. The most innovative and respected brands confidently rely on Brightcove to solve their most demanding communication challenges because of the unmatched performance and flexibility of our platform, our global scale and reliability, and our award-winning service. With thousands of customers and an industry-leading suite of cloud video products, Brightcove enables customers to drive compelling business results.

Find out more here »

### Career Center

Check out hundreds of exciting professional opportunities available on DMN's Career Center.
Explore careers in digital marketing, sales, eCommerce, marketing communications, IT, data strategies, and much more. And don't forget to update your resume so employers can contact you privately about job opportunities.