Super Crunchers by Ian Ayres Page A

renew their membership after the trial period expired.
    Yet retailers beware. Customers can become angry if they learn that the same product is being offered at different prices. In September 2000, the press started running with the story of a guy who said that when he deleted the cookies on his computer (which identified him as a regular Amazon customer), Amazon’s quoted price for DVDs fell from $26.24 to $22.74. A lot of customers were suddenly worried that Amazon was rigging the Internet. The company quickly apologized, saying that the difference was the result of a random price test. In no uncertain terms CEO Jeff Bezos declared, “We’ve never tested and we never will test prices based on customer demographics.”
    He also announced a new policy with regard to random testing that might serve as a model for other companies that test. “If we ever do such a test again, we’ll automatically give customers who purchase a test item the lowest test price for that item at the conclusion of the test period—thereby ensuring that all customers pay the lowest price available.” Just because you test higher prices doesn’t mean that you have to actually charge them once a customer places an order.
    These randomized tests about prices are a lot more problematic than other kinds of randomization. When Offermatica randomizes over different visual elements of a web page, it’s usually about trying to improve the customer experience by removing unnecessary obstacles. The results of this kind of experiment are truly a win–win for both the seller and the buyers. But as we saw with CapOne, randomization can also be used to test how much money can be extracted from the consumer. Offermatica or AdWords could be used to run randomized tests about what kind of price the market will bear.
    More insidiously, sellers might run randomized tests on what kinds of terms to include in their consumer contracts. Some businesses are really pushing the ethical and legal envelope by testing whether to conceal warranty waivers on their website. If a company runs a randomized test and finds that disclosing the waiver on a top-level page substantially reduces their sales, they have an incentive to bury the offensive information. Matt Roche emphasizes, though, that even this kind of warranty testing is a two-edged sword that sometimes helps consumers. Randomized tests show that displaying certain types of express warranties increases sales. “Including the
Verisign
security certificate almost universally gives you a lift,” he said. “Tests show that consumers demand trust. Without this hard evidence, a lot of companies wouldn’t have been willing to give it.”
    Getting Into the Game
    Randomized trials are not some ivory tower daydream. Lots of firms are already using them. The puzzle is why more firms aren’t following the lead of CapOne and Jo-Ann Fabrics. Why isn’t Wal-Mart running randomized experiments? Wal-Mart is great at using what their consumers have done in the past to predict the future. Yet at least publicly they won’t admit to randomized testing. All too often information management is limited to historical data, to recent and not-so-recent information about past transactions. Business is now very good at tracking these kinds of data, but businesses as a group still haven’t gone far enough in proactively creating useful new data.
    The examples in this chapter show that it’s really not that hard. The Excel “=rand()” function on just about any computer will flip the coin for you. Any bright high schooler could run a randomized test and analyze it. The setup isn’t hard and the analysis is just a comparison of two averages, the average results for the “treated” and “untreated” groups. That’s really all Offermatica is doing when they tell you the average click-through rate for one web page versus another. (Okay, it’s