Super Crunchers by Ian Ayres

explains, they “warm to the idea of using Offermatica to optimize their primary business drivers: page views and ad clicks.”
    Charities and political campaigns could also test which web designs optimize their contributions. In fact, charities have already started using off-line randomized trials to explore the wellsprings of giving. Experimental economists Dean Karlan and John List helped a non-profit advocacy group test the effectiveness of direct mailings much the same way as Credit Indemnity did. They sent out over 50,000 letters to past contributors asking for gifts. The letters differed in whether and how they offered a matching gift. Some of the letters offered no matching gift, some offered dollar-for-dollar matching, and some letters offered two- and even three-for-one matching. Dollar-for-dollar matching offers did increase giving by about 19 percent. The surprising result, however, was that the two-for-one and three-for-one matches didn’t generate any higher giving than the one-for-one match. This simple study gives charities a powerful new tool. A donor who wants the biggest bang for the buck would do a lot better to offer it as part of a one-for-one program.
    One thing we’ve seen over and over is that decision makers overestimate the power of their own intuitions. The intuitions make sense to us and we become wedded to them. Randomized testing is an objective way to see whether we were right. And testing is a road that never ends. Tastes change. What worked yesterday may not work tomorrow. A system of periodic retesting with randomized trials is a way to ensure that your marketing efforts remain optimized. Super Crunching is centrally about data-driven decisions. And ongoing randomized trials make sure that there is a continual supply of new data to drive decisions.
    Randomization—It’s Not Just for Breakfast
    You might think at this point that the power of randomization is just about marketing—optimizing direct-mail solicitations or web advertisements. But you’d be wrong. Randomized trials are also being used to help manage both employee and customer relationships.
    Kelly Cook, director of customer relationship management at Continental Airlines, used the coin-flipping approach to figure out how to build stronger customer loyalty. She wanted to see how best to respond when a passenger experienced what Continental euphemistically called a “transportation event.” This is the kind of event you don’t want to experience, such as having your flight severely delayed or canceled.
    Cook randomly assigned Continental customers who had endured transportation events to one of three groups. For the next eight months, one group received a form letter apologizing for the event. The second group received the letter of apology and compensation in the form of a trial membership in Continental’s President’s Club. And the third group, which served as a control, received nothing.
    When the groups were asked about their experience with Continental, the control group that didn’t receive anything was still pretty angry. “But the other groups’ reaction was amazement that a company would have written them unsolicited to say they were sorry,” Cook recalls. The two groups that received a letter spent 8 percent more on Continental tickets in the ensuing year. For just the 4,000 customers receiving letters, that translated to extra revenue of $6 million. Since expanding this program to the top 10 percent of Continental’s customers, the airline has seen $150 million in additional revenues from customers who otherwise would have had a good reason to look elsewhere.
    Just sending a letter without compensation was enough to change consumer perceptions and behavior. And the compensation of trial membership itself turned into a new source of profit. Thirty percent of customers who received a trial membership in Continental’s President’s Club decided to