Stat Spotting (9/2008)
A Field Guide to Identifying Dubious Data

•  1: Getting Started
              A. Spotting questionable numbers   B. Background  (B1 Statistical Benchmarks, B2 Severity and Frequency)

•  2: Varieties of dubious Data
       C. Blunders: C1. The slippery decimal point,  C2. Blotched transactions,  C3. Misleading graphs, C4. Careless calculations
       D. Sources: who counted and why?  D1. Big round numbers, D2. Hyperbole,   D3. Shocking claims    D4.  Naming the problem
       E. Definitions, what did they count?  E1. Broad definitions, E2. Expanding definitions, E3. Changing the definitions,  E4. The uncounted.
       F. Measurements: how did they count?  F1. Creating measures,  F2. Odd units of analysis,  F3. Loaded questions,
                                                             F4. Raising the Bar,       F5. Technical measures.
       G. Packaging: what are they telling us?  G1. Impressive formats,    G2. Misleading samples,       G3. Convenient time frames
                                                             G4. Peculiar Percentages,  G5. Selective comparisons,  G6. Statistical milestones
                                                             G7. Averages, G8. Epidemics, G9. Correlations,  G10. Discoveries.
       H. Debates: what if they disagree?    H1. Causality debates,  H2. Equality debates, H3. Policy debates.

• 3: Stat-Spotting on your own
       I. summary; common signs of dubious data
       J. Better data: some characteristics
       K. If you had no idea things were that bad, they probably aren't.
       L. Suggestions for those who want to continue stat-spotting.

• Acknowledgements, Notes and Index

Are four million women really battered to death by their husbands or boyfriends each year? Does a young person commit suicide every thirteen minutes in the United States? Is methamphetamine our number one drug problem today? Alarming statistics bombard our daily lives, appearing in the news, on the Web, seemingly everywhere. But all too often, even the most respected publications present numbers that are miscalculated, misinterpreted, hyped, or simply misleading. Following on the heels of his highly acclaimed Damned Lies and Statistics and More Damned Lies and Statistics, Joel Best now offers this practical field guide to help everyone identify questionable statistics. Entertaining, informative, and concise, Stat-Spotting is essential reading for people who want to be more savvy and critical consumers of news and information.

Stat-Spotting features:

* Pertinent examples from today's news, including the number of deaths reported in Iraq, the threat of secondhand smoke, the increase in the number of overweight Americans, and many more

* A commonsense approach that doesn't require advanced math or statistics

Review by Bernie Madison, Univ. of Arkansas:  "As we now swim in information, much of it bogus or biased, spotting dubious data is super important. In Stat-Spotting, Joel Best plays off the format of field guides to give readers good commonsense ways not only to sense bad data but also to understand what's wrong.  Broken up into short independent sections, the book is easy and enjoyable to read.  I will recommend it to my students, and to others, as a resource for critical consumers of numbers."

Review by Neil Lutsky, Carleton college: "Stat-spotting proposes to help readers become more critical consumers of statistical claims.  It is an important work that addresses a significant problem in contemporary society: thoughtlessness about numerical claims.  Joel Best provides a direct, accessible guide to critical readings of statistics." 

Review by Alan Jasper, Graduate Center,  City University of New York:  "If you ever scan the newspaper, watch the TV news, or surf the blogs, you should read this charming book.  If you're a journalist, read it twice."

More Damned Lies and Statistics (8/2004)
How Numbers Confuse Public Issues   PDF

•  Table of Contents

•  Preface: People Count

• Ch 1: Missing Numbers

• Ch 2: Confusing Numbers

• Ch 3: Scary Numbers

• Ch 4: Authoritative Numbers

• Ch 5: Magical Numbers

• Ch 6: Contentious Numbers

• Ch 7: Toward Statistical Literacy

Ch 7 Statistical Literacy: Excerpts

Shouldn't we be able to teach "statistical literacy"-basic skills for critically interpreting the sorts of statistics we encounter in everyday life?

what if we call statistical literacy a basic skill? Certainly a plausible argument exists for considering it in these terms. After all, we are talking about teaching people to be more critical, to be more thoughtful about what they read in the newspaper or watch in a news broadcast, to ask questions about claims from scientists, politicians, or activists. Being better able to assess such claims is certainly valuable; we might even argue that it is fundamental to being an informed citizen. Why not consider statistical literacy a basic skill?

"Why not consider statistical literacy a skill?  But this raises another question: what sort of basic skill is it?"   "This competition [between departments] means that teaching basic skills often is devalued."   "College instructors are well aware that substantial proportions of students have trouble reading -- let alone thinking critically about -- basic graphs and tables.  This is a very important skill because graphs and tables are certain to appear in much of the reading a student will need to do in the course of college.  And yet, no one wants to teach this skill, or at least spend much time doing so.   Many have the sense that students should already be proficient in these skills when they get to college (even though it is clear that many are not).  To many others, it seems to simple, too basic -- a waste of time for professors who would prefer to teach the more advanced topics in their disciplines."  "Thus statistic and mathematics instructors are unlikely to have any more interest in teaching statistical literacy than English professors have in teaching first-year composition.  Nor are other departments eager to teach this material.  I teach sociology courses, but I know that most sociology professors tend to dismiss statistical literacy as "not really sociology"; faculty in psychology and other disciplines probably have the same reaction. Statistical literacy falls between the stools on which academic departments perch."  "The lessons involved in teaching statistical literacy are not so terribly difficulty; rather, the difficulty lies in finding someone willing to teach them."

THE STATISTICAL LITERACY MOVEMENT

Despite these obstacles, a small educational movement advocating statistical literacy has emerged. Professor Milo Schield, director of the W. M. Keck Foundation Statistical Literacy Project at Augsburg College in Minneapolis, is the movement's leading voice. Schield operates the Statistical Literacy Web site (www.StatLit.org); for those interested in statistical literacy as an educational movement, the site includes a section on teaching. Although this is a promising development, the campaign to promote formal instruction in statistical literacy is in its early phases.

But perhaps statistical literacy doesn't have to be taught in classrooms. Recently, there seem to be increasing calls to promote statistical literacy outside the educational establishment. <snip>

In short, it may be true that "everyone" agrees that improving statistical literacy is desirable, but it isn't clear that they can agree on what statistical literacy means, what improving it might involve, or what the consequences of this improvement might be.

Even if no one opposes statistical literacy, serious obstacles remain. There is disagreement about which skills need to be taught, and, at least so far, no group has offered to take responsibility for doing the necessary teaching. Plenty of information is out there-any interested individual can learn ways to think more critically about statistics-but the statistical literacy movement has yet to convince most educators that they need to change what the educational system is doing.

As things stand, we constantly find ourselves exposed to lots of statistics. Some of those numbers are pretty good, but many aren't. As a result, we worry about things that probably aren't worth the trouble, even as we ignore things that ought to warrant our attention. Improving statistical literacy - if we can manage it - could help us tell the difference and, in a small way, make us wiser.

Reviews:

In this sequel to Best's Damned Lies and Statistics (2001), the premise is simple: there are vast quantities of statistics being bandied about in all walks of life, and we frequently rely on them to form our own opinions about things. Often, however, neither we nor the experts understand how those numbers work. "People need to agree about what to count before they can start counting," the author tells us, explaining why different people often disagree about the same statistics. Some journalists say child-abduction cases are up; others say they're down; but no one has bothered to agree on what they mean by child or abduction. Another problem: news media perpetuate inaccuracies by citing each other's statistics without checking for accuracy. This is why, for example, we keep hearing that 150 people die every year after being hit by falling coconuts. (In fact, there is no such statistic because no one tracks coconut deaths.) The book is packed with helpful tips for understanding statistics, and it even manages to make a usually dull topic entertaining. David Pitt Copyright © American Library Association. All rights reserved Review

"Through his devastating work on common myths about social problems, Joel Best has established himself as a brilliant observer of our national fads and scares. In this latest book, Best confronts yet more of the pseudo-statistics by which we are bamboozled day by day. One obvious question comes to mind. If he can deal with highly significant topics in such lucid and enjoyable prose, why can't other social scientists begin to match him?"—Philip Jenkins, author of The New Anti-Catholicism: The Last Acceptable Prejudice

"Joel Best continues to confront us with the delicious lunacy of statistical gaffes and fantasies. Whether discussing 'deaths from falling coconuts,' teenage bullying, or likelihood of contracting breast cancer, Best teaches us to avoid the dangers of statistical illiteracy. As his cogent and comic examples from the media amply demonstrate, there is much teaching yet to be done. While we like to believe that it is our opponents who are fools with figures, this volume demonstrates that liberals, conservatives, libertarians, lawyers, physicians, and educators fall in the same numerical traps."—Gary Alan Fine, co-author of Whispers on the Color Line: Rumor and Race in America

"Best provides us with another telling compendium of misleading statistics about a variety of topical issues. His approach to explicating them is lucid, instructive, and quite engaging."—John Allen Paulos, author of Innumeracy

About the Author: Joel Best is Professor and Chair of Sociology and Criminal Justice at the University of Delaware. He is the author of Damned Lies and Statistics: Untangling Numbers from the Media, Politicians, and Activists (California, 2001), Random Violence: How We Talk about New Crimes and New Victims (California, 1999), and Threatened Children: Rhetoric and Concern about Child-Victims (1990).

Reviews by Augsburg students: " I think that [reading] this [book] makes me a more informed person and one less easily duped.  I feel like I am less confused, now, by conflicting claims.  It wasn’t a book I would have read outside of class, but I’m glad I did read it."

Damned Lies and Statistics (5/2001)
Untangling Numbers from the Media, Politicians and Activists

Following excerpts chosen by Milo Schield

P. 10. “Statistics can become weapons in political struggles over social problems and social policy.”

P. 19. “Innumeracy is the mathematical equivalent of illiteracy; it is ‘an inability to deal comfortably with the fundamental notions of numbers and chance’.”

P. 26. "The lesson should be clear: Statistics -- even official statistics such as crime rates, unemployment rates, and census counts-- are products of social activity.”

P. 27. "All statistics are social products, the results of people's efforts."

P. 28. "There are three basic questions we should ask whenever we encounter a new statistic. 1. Who created this statistic? 2. Why was this statistic created? 3. How was this statistic created?" P. 28

P. 29. "Plan of the Book: The following chapters discuss some of the most common and important problems with the creation and interpretation of social statistics. Chapter 2 examines four basic sources of bad statistics: bad guesses, deceptive definitions, confusing questions, and biased samples. Chapter 3 looks at mutant statistics, at ways even good statistics can be mangled, misused, and misunderstood. Chapter 4 discusses the logic of statistical comparison and explores some of the most common errors in comparing two or more time periods, places, groups or social problems. Chapter 5 considers debates over statistics. Finally, chapter 6 examines three general approaches to thinking about statistics."

P. 30. "Chapter 1 argued that people produce statistics. Of course they do. All human knowledge -- including statistics -- is created thru people's actions; everything we know is shaped by our language, culture and society. Sociologists call the the social construction "of knowledge."

P. 31. "Our knowledge about society tends to be "softer" than our knowledge of the physical world."

P. 32. "Often the ways that people produce statistics are flawed: their numbers may be little more than guesses; or the figures may be the produce of poor definitions, flawed measurements or weak sampling. These are the four basic ways to create bad social statistics."

P. 33. "Every social problem has a dark figure because some instances (of crime, child abuse, poverty, or whatever) inevitably go unrecorded.

P. 34. "Because activists sincerely believe that the new problem is big and important, and because they suspect there is a very large figure of unreported or unrecorded cases, the activists' estimates tend to be high, to error on the side of exaggeration. Their guesses are far more likely to overestimate than under-estimate a problem size."

P. 36. "Once a number appears in one news report, that report is a potential source for everyone who becomes interested in the social problem of statistics; officials, experts, activists, and other reporters routinely repeat figures that appear in press reports. The number takes on a life of its own, and it goes through 'number laundering.'"

P. 39. "Definitions: Whenever examples substitute for definitions, there is a risk that our understanding of the problem will be distorted."

P. 39. "People promoting social problems sometimes do offer definitions. When they do so, the tend to prefer general, broad, inclusive definitions. "

P. 40. "This [broad versus narrow definitions] has obvious implications for social statistics, because broad definitions support much larger estimates of a problem's size."

P. 40. "No definition of a social problem is perfect, but there are two principal ways such definitions can be flawed... False positives (they mistakenly identify case as part of the problem)... False negatives (incorrectly identified as not being part of the problem)."

P. 44. "Statistics about social problems always depend on how we define the problem. The broader the definition, the bigger the statistic."

P. 44. "There are, then, two questions about definitions that ought to be asked whenever we encounter statistics about social problems. First, how is the problem defined? <snip> Second, is the definition reasonable?"

P. 45. "Measuring: Public attitudes toward most social issues are too complex to be classified as simple pros or cons, or to be measured by a single survey question."

P. 52. "Sampling: Virtually all social statistics involve generalizing from samples."

P. 53. "There are two problems with sampling -- one obvious, and the other more subtle. The obvious problem is sample size. <snip> The second issue: the representativeness of a sample is actually more important than sample size."

P. 54. "Statisticians can calculate the probability that such random samples represent the population... The real problem is that few samples are random."

P. 55. " Statistics about social problems are usually based on samples tht fall far short of randomness."

P. 57. "In short, the process of generalization is at the centre of sampling."

P. 58. "The difficulties of drawing accurate samples also invite another sort of generalizing claim. <snip> They [activists] may argue that the problem threatens 'everyone', that it affects 'people of all sorts,' even that it strikes at random."

P. 58. "In general, social problems are patterned... But people promoting social problems often find it advantageous to gloss over these problems, to imply that everyone shares the same risks and therefore we all have the same, substantial stake in solving the social problem."

P. 59. "Characteristics of Good Statistics: First, good statistics are based on more than guessing.... Second, good statistics are based on clear, reasonable definitions. ... Finally, good statistics are based on good samples."

Source: http://portal.tpu.ru:7777/SHARED/k/KITAEVA/statistics/book/Tab3/damnedstatistics.pdf

==========================================================================

TABLE OF CONTENTS

• Introduction: The Worst Social Statistic Ever

• Ch 1: The Importance of Social Statistics

• Ch 2: Soft Facts: Sources of Bad Statistics

• Ch 3: Mutant Statistics: Methods for Mangling Numbers

• Ch 4: Apples and Oranges: Inappropriate Comparisons

• Ch 5: Stat Wars: Conflicts over Social Statistics

• Ch 6: Thinking about Social Statistics: The Critical Approach