house to be ready? A man happily and comfortably living as a squatter in an abandoned warehouse? If we compare homelessness across different cities and states, the various jurisdictions may use different definitions. Even if the definition becomes standardized across jurisdictions, astatistic you encounter may not have defined homelessness the way that you would. One of the barriers to solving “the homelessness problem” in our large cities is that we don’t have an agreed-upon definition of what it is or who meets the criteria.
Whenever we encounter a news story based on new research, we need to be alert to how the elements of that research have been defined. We need to judge whether they are acceptable andreasonable. This is particularly critical in topics that are highly politicized, such as abortion, marriage, war, climate change, the minimum wage, or housing policy.
And nothing is more politicized than, well, politics. A definition can be wrangled and twisted to anyone’s advantage in public-opinion polling by asking a question just-so.Imagine that you’ve been hired by a political candidate to collect information on his opponent, Alicia Florrick. Unless Florrick has somehow managed to appeal to everyone on every issue, voters are going to have gripes. So here’s what you do: Ask the question “Is there anything at all that you disagree with or disapprove of, in anything the candidate has said, even if you support her?” Now almost everyone will have some gripe, so you can report back to your boss that “81 percent of people disapprove of Florrick.” What you’ve done is collected data on one thing (even a single minor disagreement) and swept it into a pile of similar complaints, rebranding them as “disapproval.” It almost sounds fair.
Things That Are Unknowable or Unverifiable
GIGO is a famous saying coined by early computer scientists: garbage in, garbage out. At the time, people would blindly put their trust into anything a computer output indicated because the output had the illusion of precision and certainty. If a statistic is composed of a series of poorly defined measures, guesses, misunderstandings, oversimplifications, mismeasurements, or flawed estimates, the resulting conclusion will be flawed.
Much of what we read should raise our suspicions. Ask yourself: Is it possible that someone can know this?A newspaper reports the proportion of suicides committed by gay and lesbian teenagers. Any such statistic has to be meaningless, given the difficulties in knowing which deaths are suicides and which corpses belong to gay versus straight individuals. Similarly, the number of deaths from starvation in a remote area, or the number of people killed in a genocide during a civil war, should be suspect. This was borne out by the wildly divergent casualty estimates provided by observers during the Iraq-Afghanistan-U.S. conflict.
A magazine publisher boasts that the magazine has 2 million readers. How do they know? They don’t. They assume some proportion of every magazine sold is shared with others—what they call the “pass along” rate. They assume that every magazine bought by a library is read by a certain number of people. The same applies to books and e-books. Of course, this varies widely by title. Lots of people bought Stephen Hawking’s A Brief History of Time . Indeed, it’s said to be the most purchased and least finished book of the last thirty years. Few probably passed it along, because it looks impressive to have it sitting there in the living room. How many readers does a magazine or book have? How many listeners does a podcast have? We don’t know. We know how many were sold or downloaded, that is all (although recent developments with e-books will probably be changing that long-standing status quo).
The next time that you read that the average New Zealander flosses 4.36 times a week (a figure I just made up, but it may be as accurate as any estimate), ask yourself: How could