Test and learn
Encyclopedia
Test and Learn is a set of practices followed by retailers, banks and other consumer-focused companies to test ideas in a small number of locations or customers to predict impact. The process is often designed to answer three questions about any tested program before rollout:
. Capital One has been aggressive about testing since the firm was founded, testing everything from product design to marketing to customer selection to collection policies. In a single year, the company performs tens of thousands of tests, allowing it to offer thousands of different types of credit cards to customers, based on the knowledge gained from their tests. Richard Fairbank, CEO of Capital One, called test and learn "a marketing revolution that can be applied to many businesses".
tools available today to support systematic testing within an organization. However, there is currently no one software package that covers all types of tests, and in some cases significant knowledge of statistics is still required for effective analysis.
Many companies, including Capital One
and eBay
, have developed experiment management software within the company. Alternatives to internal solutions include broad statistical analysis software tools such as SAS
or test and learn focused software from Applied Predictive Technologies
. There are also a variety of more narrowly tailored products, focused on particular problems or fields. In web analytics
, for example, firms such as Omniture
, WebTrends
, and Google
have created specialized software.
February 2009 article "How to Design Smart Business Experiments", Thomas Davenport discusses the Test and Learn Process, and describes how several companies have overcome these and other barriers to testing.
- What impact will the program have on key performance indicators if executed across the network or customer base?
- Will the program have a larger impact on some stores/customers than others?
- Which components of the idea are actually working?
Historical context
Test and Learn has been systematically applied as far back as 1988 by Capital OneCapital One
Capital One Financial Corp. is a U.S.-based bank holding company specializing in credit cards, home loans, auto loans, banking and savings products...
. Capital One has been aggressive about testing since the firm was founded, testing everything from product design to marketing to customer selection to collection policies. In a single year, the company performs tens of thousands of tests, allowing it to offer thousands of different types of credit cards to customers, based on the knowledge gained from their tests. Richard Fairbank, CEO of Capital One, called test and learn "a marketing revolution that can be applied to many businesses".
Examples
- Retail banks running different commercials in different markets.
- Fast food restaurants launching a new menu item or product for a limited time in select locations
- Big-box retailers evaluating “store of the future” programs
- Companies evaluating pricing strategies
- Retailers testing merchandising layouts across their door base
Test and learn in practice
- Wawa Food MarketsWawa Food MarketsWawa Inc. is a chain of convenience store/gas stations located in the Mid-Atlantic region of the United States. It operates in Pennsylvania, New Jersey, Delaware, Maryland, Virginia, and Florida. The company's corporate headquarters is located in Chester Heights, Pennsylvania, near the community...
, a Mid-Atlantic convenience store chain, uses Test and Learn across several aspects of its business:- Measuring the effect of introducing a new product
- Determining the impact of adding additional staff
- Understanding how the opening a new location would impact existing nearby locations
- SubwaySubway (restaurant)Subway is an American restaurant franchise that primarily sells submarine sandwiches and salads. It is owned and operated by Doctor's Associates, Inc. . Subway is one of the fastest growing franchises in the world with 35,519 restaurants in 98 countries and territories as of October 25th, 2011...
ran a promotion that lowered the price of its foot-long sandwiches to $5. Before deciding to launch the promotion nationally, Subway ran a test that compared locations offering the $5 foot-long sandwiches to a group of similar controlScientific controlScientific control allows for comparisons of concepts. It is a part of the scientific method. Scientific control is often used in discussion of natural experiments. For instance, during drug testing, scientists will try to control two groups to keep them as identical and normal as possible, then...
restaurants that were not offering the promotion. - Kraft FoodsKraft FoodsKraft Foods Inc. is an American confectionery, food and beverage conglomerate. It markets many brands in more than 170 countries. 12 of its brands annually earn more than $1 billion worldwide: Cadbury, Jacobs, Kraft, LU, Maxwell House, Milka, Nabisco, Oscar Mayer, Philadelphia, Trident, Tang...
, a manufacturer and distributor of packaged foods, uses testing to optimize the placement of its products on grocery store shelves:- Determining the effect of a coffee aisle re-design on sales of premium coffees
- Measuring the impact on product sales of a change to in-store sales representatives' responsibilities
- Pier 1 ImportsPier 1 ImportsPier 1 Imports Inc. is a Fort Worth, Texas based retailer specializing in imported home furnishings and decor, particularly furniture, table-top items, decorative accessories and seasonal decor. The chain operates over 1,000 stores under the name Pier 1 Imports in the United States, Canada, Mexico...
, a retail chain specializing in home decor and furnishings, uses testing to measure the effectiveness of search advertisingSearch advertisingIn Internet Marketing, Search Advertising is a method of placing online advertisements on Web pages that show results from search engine queries. Through the same search-engine advertising services, ads can also be placed on Web pages with other published ....
campaigns. - Test and Learn is also methodically used online to test impact of various web page layouts on user behavior. Tools such as GoogleGoogleGoogle Inc. is an American multinational public corporation invested in Internet search, cloud computing, and advertising technologies. Google hosts and develops a number of Internet-based services and products, and generates profit primarily from advertising through its AdWords program...
Website Analyzer provide every site with the ability to test various elements and their effect on user engagement.
Test and learn software
There are a variety of softwareComputer software
Computer software, or just software, is a collection of computer programs and related data that provide the instructions for telling a computer what to do and how to do it....
tools available today to support systematic testing within an organization. However, there is currently no one software package that covers all types of tests, and in some cases significant knowledge of statistics is still required for effective analysis.
Many companies, including Capital One
Capital One
Capital One Financial Corp. is a U.S.-based bank holding company specializing in credit cards, home loans, auto loans, banking and savings products...
and eBay
EBay
eBay Inc. is an American internet consumer-to-consumer corporation that manages eBay.com, an online auction and shopping website in which people and businesses buy and sell a broad variety of goods and services worldwide...
, have developed experiment management software within the company. Alternatives to internal solutions include broad statistical analysis software tools such as SAS
SAS Institute
SAS Institute Inc. , headquartered in Cary, North Carolina, USA, has been a major producer of software since it was founded in 1976 by Anthony Barr, James Goodnight, John Sall and Jane Helwig...
or test and learn focused software from Applied Predictive Technologies
Applied Predictive Technologies
Applied Predictive Technologies is a software company that provides business analytics software to other businesses, along with support, training, and consulting services for its software. APT's clients include Starbucks, Subway, Royal Bank of Canada, and more than 40 of the Fortune 500 across a...
. There are also a variety of more narrowly tailored products, focused on particular problems or fields. In web analytics
Web analytics
Web analytics is the measurement, collection, analysis and reporting of internet data for purposes of understanding and optimizing web usage....
, for example, firms such as Omniture
Omniture
Omniture is an online marketing and web analytics business unit owned by Adobe Systems. The Omniture Business Unit is based in Orem, Utah, with offices worldwide. It serves customers in 75 countries worldwide.- History :...
, WebTrends
WebTrends
Webtrends is a private company headquartered in Portland, Oregon, United States. It provides web analytics and other software solutions related to marketing intelligence...
, and Google
Google
Google Inc. is an American multinational public corporation invested in Internet search, cloud computing, and advertising technologies. Google hosts and develops a number of Internet-based services and products, and generates profit primarily from advertising through its AdWords program...
have created specialized software.
Criticism
Companies wanting to conduct Test and Learn must mobilize, track and analyze site and customer data at very granular levels. This can often be a massive and expensive undertaking. Increasingly however, exponential increase in computing power, and decrease in its cost, have made testing more accessible. In the Harvard Business ReviewHarvard Business Review
Harvard Business Review is a general management magazine published since 1922 by Harvard Business School Publishing, owned by the Harvard Business School. A monthly research-based magazine written for business practitioners, it claims a high ranking business readership among academics, executives,...
February 2009 article "How to Design Smart Business Experiments", Thomas Davenport discusses the Test and Learn Process, and describes how several companies have overcome these and other barriers to testing.
See also
- Clinical trials
- Scientific methodScientific methodScientific method refers to a body of techniques for investigating phenomena, acquiring new knowledge, or correcting and integrating previous knowledge. To be termed scientific, a method of inquiry must be based on gathering empirical and measurable evidence subject to specific principles of...
- Design of experimentsDesign of experimentsIn general usage, design of experiments or experimental design is the design of any information-gathering exercises where variation is present, whether under the full control of the experimenter or not. However, in statistics, these terms are usually used for controlled experiments...
- Evidence-based managementEvidence-based managementEvidence-based management is an emerging movement to explicitly use the current, best evidence in management decision-making. Its roots are in evidence-based medicine, a quality movement to apply the scientific method to medical practice....
- Business process reengineeringBusiness process reengineeringBusiness process re-engineering is the analysis and design of workflows and processes within an organization.According to Davenport a business process is a set of logically related tasks performed to achieve a defined business outcome....