News stories based on surveys, polls, studies and statistics are everywhere. Wouldn't it be good to have the mental agility to separate the wheat from the chaff? In the first of a weekly series, author Michael Blastland gives some hints at getting to grips with surveys.
News stories based on surveys, polls, studies and statistics are everywhere. Wouldn't it be good to have the mental agility to separate the wheat from the chaff? In the first of a weekly series, author Michael Blastland gives some hints at getting to grips with surveys.
It's August. A time not yet of mellow fruitfulness (Keats), but certainly of fruitcake journalism. Among the best (or should that be 'the worst'?) in the media is the steady stream of duff data.
It's August. A time not yet of mellow fruitfulness (Keats), but certainly of fruitcake journalism. Among the best (or should that be 'the worst'?) in the media is the steady stream of duff data.
So for some light holiday reading while school's out, here's a short selection of number nonsense, and how to see through it. Each week, I'll pick a theme and a recent example or two, and have a go at pulling them apart.
So for some light holiday reading while school's out, here's a short selection of number nonsense, and how to see through it. Each week, I'll pick a theme and a recent example or two, and have a go at pulling them apart.
Lesson one: Surveys.
Lesson one: Surveys.
The story: "Motorists turn to public transport as fuel price bites" - Daily Record
The story: "Motorists turn to public transport as fuel price bites" - Daily Record
"MORE than three in five drivers are turning to public transport due to high fuel prices, a survey has revealed. The survey by transport firm National Express found 61 per cent of car users are definitely or probably considering using public transport due to the rise in prices at the pumps."
"MORE than three in five drivers are turning to public transport due to high fuel prices, a survey has revealed. The survey by transport firm National Express found 61 per cent of car users are definitely or probably considering using public transport due to the rise in prices at the pumps."
The flaw: Suckered by a press release. This story gives the impression that motorists are leaving their cars at home, en masse, for - guess what? - services run by the company that did the survey. In fact, the surveyed motorists are not necessarily doing anything. They are "definitely or probably" thinking about doing something - which they might eventually do once, or often, or never.
The flaw: Suckered by a press release. This story gives the impression that motorists are leaving their cars at home, en masse, for - guess what? - services run by the company that did the survey. In fact, the surveyed motorists are not necessarily doing anything. They are "definitely or probably" thinking about doing something - which they might eventually do once, or often, or never.
The Lesson: Two points. The survey has apparently bundled together the more and the less inclined, the definites and the probables, so that we have no idea if only 1% of the 61% are definites, or if most are. And it gives no indication of how big a change in behaviour they are considering - every trip⦠or just one.
The Lesson: Two points. The survey has apparently bundled together the more and the less inclined, the definites and the probables, so that we have no idea if only 1% of the 61% are definites, or if most are. And it gives no indication of how big a change in behaviour they are considering - every trip⦠or just one.
The second point is to wonder what it means to say you are "probably" thinking about something. Are they thinking that they will probably do it, but haven't made up their minds? Maybe it means they are not sure what they are thinking, but if they had to guess what they are thinking then it would "probably" be something about petrol. Maybe it doesn't mean anything.
The second point is to wonder what it means to say you are "probably" thinking about something. Are they thinking that they will probably do it, but haven't made up their minds? Maybe it means they are not sure what they are thinking, but if they had to guess what they are thinking then it would "probably" be something about petrol. Maybe it doesn't mean anything.
And are they actually doing it, as the headline suggests? No idea.
And are they actually doing it, as the headline suggests? No idea.
Press-ganging the middle, or blurring qualitative differences, is a common device, even in more serious surveys. Imagine a survey on the EU. Completely hypothetical, but let's say it asks if you are "pro" or "anti". There are three choices 1: wholly for, 2 wholly against, 3 partly for and partly against. The result is that one third are wholly for, one third wholly against and one third in the middle.
Press-ganging the middle, or blurring qualitative differences, is a common device, even in more serious surveys. Imagine a survey on the EU. Completely hypothetical, but let's say it asks if you are "pro" or "anti". There are three choices 1: wholly for, 2 wholly against, 3 partly for and partly against. The result is that one third are wholly for, one third wholly against and one third in the middle.
If you want to make it look like a vote against, you report that two thirds ("well over half") were "wholly or partly" against the EU. If you want to make it look like a vote for, you report that 67 per cent were "wholly or partly" in favour.
If you want to make it look like a vote against, you report that two thirds ("well over half") were "wholly or partly" against the EU. If you want to make it look like a vote for, you report that 67 per cent were "wholly or partly" in favour.
It's common to try to bundle up the undecided or half-hearted opinions with those who feel more strongly in order to make it look as if you have a clear and passionate majority on your side.
It's common to try to bundle up the undecided or half-hearted opinions with those who feel more strongly in order to make it look as if you have a clear and passionate majority on your side.
Another problem with surveys that often puts a spanner in the works is who they ask. Take a survey earlier this year that was much in the news and said that one third of girls self harm - a shocking statistic.
Another problem with surveys that often puts a spanner in the works is who they ask. Take a survey earlier this year that was much in the news and said that one third of girls self harm - a shocking statistic.
Who was surveyed? A cross-sample of the population. Good. How did they do the survey? By internet and text message. Not so good. How many replied? About half. Which half? Maybe those most likely already to have a big interest in self-harm?
Who was surveyed? A cross-sample of the population. Good. How did they do the survey? By internet and text message. Not so good. How many replied? About half. Which half? Maybe those most likely already to have a big interest in self-harm?
The news is chock full of surveys. Many are blatant self-promotion and/or statistical garbage. When a women's magazine reports a survey of the nation's sexual habits, you might wonder whether it's really what the people of Great Britain are up to, or just a good way of selling a magazine. Is it rather a survey of what that select group of readers-of-spicy-magazines-who-like-filling-in-sex-surveys gets up to? Interesting maybe, but - let's put it politely - of limited social relevance.
The news is chock full of surveys. Many are blatant self-promotion and/or statistical garbage. When a women's magazine reports a survey of the nation's sexual habits, you might wonder whether it's really what the people of Great Britain are up to, or just a good way of selling a magazine. Is it rather a survey of what that select group of readers-of-spicy-magazines-who-like-filling-in-sex-surveys gets up to? Interesting maybe, but - let's put it politely - of limited social relevance.
This survey on self-harm, was more serious, but prone to the same problem. It came up with a number - one third of teenage girls - that was between about 60 and 200% higher than any recognised surveys done before on this issue.
This survey on self-harm, was more serious, but prone to the same problem. It came up with a number - one third of teenage girls - that was between about 60 and 200% higher than any recognised surveys done before on this issue.
All surveys can suffer from bias - deliberately or not - by picking up or missing too many of a vital group and simply extrapolating - as if everyone in the country is the same as the people who completed the survey. Were they?
All surveys can suffer from bias - deliberately or not - by picking up or missing too many of a vital group and simply extrapolating - as if everyone in the country is the same as the people who completed the survey. Were they?
Next week's lesson: Counting.
Next week's lesson: Counting.
Michael Blastland is the author, with Andrew Dilnot, of The Tiger That Isn't.
Michael Blastland is the author, with Andrew Dilnot, of The Tiger That Isn't.
Add your comments on this story, using the form below.
Add your comments on this story, using the form below.
