This article is from the source 'bbc' and was first published or seen on . It will not be checked again for changes.
You can find the current article at its original source at http://news.bbc.co.uk/go/rss/-/1/hi/magazine/7605118.stm
The article has changed 8 times. There is an RSS feed of changes available.
| Version 2 | Version 3 |
|---|---|
| The doubt lurking behind statistics | The doubt lurking behind statistics |
| (19 minutes later) | |
| Three million people suffered the winter vomiting bug last year, we were told. But that figure should have come with a health warning of its own, says Michael Blastland, in the final lesson of his six-part primer on understanding statistics in the news. | Three million people suffered the winter vomiting bug last year, we were told. But that figure should have come with a health warning of its own, says Michael Blastland, in the final lesson of his six-part primer on understanding statistics in the news. |
| The story: Health crisis as three million people went down last winter with Norovirus - winter vomiting disease. | The story: Health crisis as three million people went down last winter with Norovirus - winter vomiting disease. |
| The flaw: The authorities don't know how many had it. They don't even know how many went to the doctor with it. All they know is the 2,000 cases in the peak months that were confirmed in the lab. | The flaw: The authorities don't know how many had it. They don't even know how many went to the doctor with it. All they know is the 2,000 cases in the peak months that were confirmed in the lab. |
| Did this virus sweep the nation? They say that for every lab-confirmed case there are about 1,500 in the community, a ratio of 1:1,500. (Hence: 2,000 lab cases x 1,500 = three million). But how do they know that? They don't. They haven't a clue. | Did this virus sweep the nation? They say that for every lab-confirmed case there are about 1,500 in the community, a ratio of 1:1,500. (Hence: 2,000 lab cases x 1,500 = three million). But how do they know that? They don't. They haven't a clue. |
| The lesson: Lurking beneath many statistics is a factor that changes everything, but is seldom admitted: doubt. | The lesson: Lurking beneath many statistics is a factor that changes everything, but is seldom admitted: doubt. |
| There can be a lot⦠or a little. A number in the news can be the kind that hits the mark, or wouldn't hit a barn door. Statisticians - good ones - admit the extent of their doubt. Newspapers almost never do. | There can be a lot⦠or a little. A number in the news can be the kind that hits the mark, or wouldn't hit a barn door. Statisticians - good ones - admit the extent of their doubt. Newspapers almost never do. |
| Here's a technical term, but relax, it's a simple idea. This is the "confidence interval". All it means is how sure they are. | Here's a technical term, but relax, it's a simple idea. This is the "confidence interval". All it means is how sure they are. |
| Maybe the estimate was based on a small sample and they got a chance result, a fluke, so that the true number could be higher or lower. How much higher or lower? That's the interval, the range of possibilities within which they think the true answer might lie. | Maybe the estimate was based on a small sample and they got a chance result, a fluke, so that the true number could be higher or lower. How much higher or lower? That's the interval, the range of possibilities within which they think the true answer might lie. |
| MISSED PAST LESSONS? Lesson One: Our survey saysLesson Two: Myth of countingLesson Three: Percentages Lesson Four: Averages Lesson Five: Causation Back to vomit, what are the confidence intervals around the 1:1,500 ratio? How high or low could that ratio really be? | MISSED PAST LESSONS? Lesson One: Our survey saysLesson Two: Myth of countingLesson Three: Percentages Lesson Four: Averages Lesson Five: Causation Back to vomit, what are the confidence intervals around the 1:1,500 ratio? How high or low could that ratio really be? |
| Well, the low end of the confidence interval is a lot lower, 1:140. That is, each lab case could equal 140 in the community. | Well, the low end of the confidence interval is a lot lower, 1:140. That is, each lab case could equal 140 in the community. |
| The high end is an astonishing 1:17,000. This implies that there could have been not three million, but just 280,000 cases of winter flu (2,000 confirmed lab cases x 140 = 280,000)⦠| The high end is an astonishing 1:17,000. This implies that there could have been not three million, but just 280,000 cases of winter flu (2,000 confirmed lab cases x 140 = 280,000)⦠|
| Or 34 million. Thirty-four million people in the UK vomiting? Don't you think we might have noticed? That's a lot of doubt. | Or 34 million. Thirty-four million people in the UK vomiting? Don't you think we might have noticed? That's a lot of doubt. |
| Why so much? Because they first arrived at this somewhat imprecise ratio by looking at what happened years ago in a small town. | Why so much? Because they first arrived at this somewhat imprecise ratio by looking at what happened years ago in a small town. |
| The result can be hugely affected by chance They tried to find everyone with gastro-intestinal problems, whether people went to the doc or not. Then they found out how many cases were laboratory-confirmed and did their sums to find that all important ratio. | The result can be hugely affected by chance They tried to find everyone with gastro-intestinal problems, whether people went to the doc or not. Then they found out how many cases were laboratory-confirmed and did their sums to find that all important ratio. |
| And in that small town, how many lab cases were there? One. The sample was one. Really. I'd like to know who it was. There could be fame in his, or her vomit, of sorts. | And in that small town, how many lab cases were there? One. The sample was one. Really. I'd like to know who it was. There could be fame in his, or her vomit, of sorts. |
| And you can imagine that whether a single lab case turns up from your town is pretty much a matter of luck. Does a GP there send samples for lab analysis every time? | And you can imagine that whether a single lab case turns up from your town is pretty much a matter of luck. Does a GP there send samples for lab analysis every time? |
| Or do they, none of them, ever bother? The result can be hugely affected by chance. Hence the enormous confidence intervals: they couldn't be sure if this - the benchmark town - was typical or downright odd. | Or do they, none of them, ever bother? The result can be hugely affected by chance. Hence the enormous confidence intervals: they couldn't be sure if this - the benchmark town - was typical or downright odd. |
| That's why the authorities concede that when the 1:1,500 ratio was applied to last winter's lab cases, the resulting total (three million) could have been about 11 times too high, or 11 times too low. They even warned journalists that it was unreliable. Did any mention the doubt? | That's why the authorities concede that when the 1:1,500 ratio was applied to last winter's lab cases, the resulting total (three million) could have been about 11 times too high, or 11 times too low. They even warned journalists that it was unreliable. Did any mention the doubt? |
| Junk Rating: Five out of five. This is about as bad as it gets: abject ignorance masquerading as a crisis. | Junk Rating: Five out of five. This is about as bad as it gets: abject ignorance masquerading as a crisis. |
| Last lesson: Numbers are almost never accurate. What do you expect when life is so messy? What matters is whether they point us in roughly the right direction or possibly up a dark alley. So it's essential to know their degree of uncertainty. | Last lesson: Numbers are almost never accurate. What do you expect when life is so messy? What matters is whether they point us in roughly the right direction or possibly up a dark alley. So it's essential to know their degree of uncertainty. |
| This last lesson draws together several others, but it's worth one of its own because the news habitually ignores it, yet it makes all the difference. | This last lesson draws together several others, but it's worth one of its own because the news habitually ignores it, yet it makes all the difference. |
| Maybe a lot has happened in two years to turn Scotland's once gleeful capital into a pit of malcontents. Or maybe not It arises for many reasons - because counting is harder than it looks (see lesson two), because of the power of chance (see lesson five). And it accounts for many of the failings of surveys (see lesson one) and much else. | Maybe a lot has happened in two years to turn Scotland's once gleeful capital into a pit of malcontents. Or maybe not It arises for many reasons - because counting is harder than it looks (see lesson two), because of the power of chance (see lesson five). And it accounts for many of the failings of surveys (see lesson one) and much else. |
| A trivial example is the "grump league" that hit the news last week. Which bit of Britain is happiest? (Powys, if you must know). Which most miserable? (Edinburgh, allegedly). | A trivial example is the "grump league" that hit the news last week. Which bit of Britain is happiest? (Powys, if you must know). Which most miserable? (Edinburgh, allegedly). |
| Except that this was in flat contradiction of findings by pollsters in 2006 that only three places were happier than Edinburgh. | Except that this was in flat contradiction of findings by pollsters in 2006 that only three places were happier than Edinburgh. |
| Maybe a lot has happened in two years to turn Scotland's once gleeful capital into a pit of malcontents. Or maybe not. | Maybe a lot has happened in two years to turn Scotland's once gleeful capital into a pit of malcontents. Or maybe not. |
| Even the researchers said the numbers were not statistically significant. (Translation: the differences are so small, from such miniscule samples, that we can't be sure there's any value in them whatsoever). Told this, reporters carried on regardless. Hey, it's a laugh, they say. Who cares? | Even the researchers said the numbers were not statistically significant. (Translation: the differences are so small, from such miniscule samples, that we can't be sure there's any value in them whatsoever). Told this, reporters carried on regardless. Hey, it's a laugh, they say. Who cares? |
| Happy with that in Edinburgh?School league tables may not be accurate | Happy with that in Edinburgh?School league tables may not be accurate |
| But there are confidence intervals around many numbers that cause huge public argument. A school's place in league tables, for example. | But there are confidence intervals around many numbers that cause huge public argument. A school's place in league tables, for example. |
| The public hardly ever sees these confidence intervals, so when they were the basis of a story last month, well reported by BBC news online here (praise where it's due), it will have been a surprise to most that they even existed. | The public hardly ever sees these confidence intervals, so when they were the basis of a story last month, well reported by BBC news online here (praise where it's due), it will have been a surprise to most that they even existed. |
| But there's so much doubt about whether school results reflect the performance of the school or simply the annual ups and downs of ability in the intake, that the confidence intervals can be very wide indeed. | But there's so much doubt about whether school results reflect the performance of the school or simply the annual ups and downs of ability in the intake, that the confidence intervals can be very wide indeed. |
| So wide, in fact, as to mean, according to one critic, that it's hard to tell if a school is better or worse than average for nearly three quarters of all schools. We argue madly over numbers which, in truth, could be nowhere near the truth. | So wide, in fact, as to mean, according to one critic, that it's hard to tell if a school is better or worse than average for nearly three quarters of all schools. We argue madly over numbers which, in truth, could be nowhere near the truth. |
| Even Ofsted, the schools inspector, has said this makes CVA (contextual value added) league tables largely useless, although the problem applies, even if Ofsted doesn't say so, to every kind of league table, whether GCSE, A level or CVA. | Even Ofsted, the schools inspector, has said this makes CVA (contextual value added) league tables largely useless, although the problem applies, even if Ofsted doesn't say so, to every kind of league table, whether GCSE, A level or CVA. |
| The rough guide for interpreting confidence intervals is that the narrower they are, the more reliable the number. At bottom, here's a very elementary question: Are you sure? How sure? | The rough guide for interpreting confidence intervals is that the narrower they are, the more reliable the number. At bottom, here's a very elementary question: Are you sure? How sure? |
| Junk rating: two out of five. I wouldn't say school performance data is useless. But it really ought to acknowledge the degree of uncertainty. Watch out for the much-hyped arrival of hospital performance tables, which will face similar problems. | Junk rating: two out of five. I wouldn't say school performance data is useless. But it really ought to acknowledge the degree of uncertainty. Watch out for the much-hyped arrival of hospital performance tables, which will face similar problems. |
| Michael Blastland is the author, with Andrew Dilnot, of The Tiger That Isn't. | |
| Add your comment to this story using the form below: | |
| Good article! Statistics are a great passion of mine. Interesting that the 1:1,500 ratio is used so often when it is simply the result of one test in one small village, many moons ago. In business I find that for every massive business deal I clinch, two will get away. Not a bad ratio, I'm sure you'll agree, but I am fluent in the language of business. With this vomiting bug, I wonder what the ratio of people who actually had the virus to the number of people who gave it as an excuse for a sick day was. 1:1,500? More like 1:2,000! Neil Renwick, Cambridge | |
| Well in our family Brendan and I threw up for a day (Christmas Day), Annie for a week and Kerita escaped it. So from my reckoning that's 75% of the population and 45 million in the UK. Isn't it?Kevin, Oxford | |
| Thank you so much for your wonderful, illuminating articles. I'm most of the way through an Economics BSc, and was relieved to find I already knew about everything you went though, but one of my pet hates is journalism which misuses statistics, and as a bonus I can point my innumerate friends and family to your articles!Harriet R, Bristol, UK | |
| An excellent set of articles. Relieving a lot of anxiety of fellow data analysts everywhere, I think!Peter Carleton, London | |
| This has been an amusing and informative series. I now know, for example, that I have more feet than the average person. Having spent much of my life producing forecasts, the series lays bare the truth we forecasters have known for some time. 78.4% of statistics are thought up on the spot anyway!Alan Tuthill, London | |
| More please, the best thing to come out of BBC News for along timeDavid, London | |
| I can believe that 3 million people had this illness. I caught it in mid December, by Christmas, my girlfriend, mum, dad, sister and her partner all caught it. But none of us went to a doctor. | |
| The authorities have to try and get the best estimate they can with what little information they have been given. How would the BBC have calculated it?Andrew Moody, Sheffield | |
| The fact that the contextual value added league tables can't tell you whether most schools are better or worse than average does NOT make these tables useless. On the contrary, it is deeply reassuring to know that most schools perform quite similarly, as far as is possible to know. Ofsted ought to be ashamed of themselves. They should trumpet this fact every time the tables are mentioned.Hilary Curtis, London UK | |