You can prove anything with statistics

[ZERO]

I suspect it is a false claim. Statistics are raw data and should lead to one conclusion?
It is a common statement when stats are used as evidence. What is the truth?

[Cuddles]

The trouble is that statistics are not raw data. Statistics is a method for analysing large data sets, raw or otherwise. There are plenty of ways of getting the statistics to say what you want. To start with, you could just make up the data. If you don't want to go for quite such blantant fraud, cherry-picking the data (or the subjects giving the data) and leading questions are always good. Once you have the data, there are many, many different ways of looking at it, so there's bound to be one that will lead to the answer you want.

For example, take some of the surveys on the Twin Towers, since this is a popular topic for manipulating the statistics. Let's say you want to show that the majority of American citizens think the government was responsible for this. Assuming it's a real poll and not just made up, the easiest way to get the correct answer is to pick the people you ask. Instead of randomly phoning people, go to a truther rally and ask your questions there. You're virtually guaranteed to get 100% thinking it was the government.

If you can't quite bring yourself to get that close to them, just pick your questions carefully. A good leading question would be something like:
Do you think the government could have done more to prevent the attacks?
With hindsight the answer is of course they could, and most people will answer something to that effect. The important thing is to have multiple choice answers such as:
a) No, they did everything possible.
b) Yes, security could have been a bit tighter.
c) Yes, the government is hopelessly incompetent.
d) Yes, the government did it on purpose.
e) Not sure.
Hardly anyone will pick a), which means you can proudly announce to the world that 84% of Americans think 9/11 was the governments fault, even though that's not what most of them actually meant.

The important thing with all statistics is that unless you can see the raw data, the collection method and the workings, it is all extremely untrustworthy and shouldn't be taken too seriously. Even with respectable polling companies, the questions are usually written, at least partially, by the people commissioning the poll. Polls just asking questions in the street, write-ins or, horror of horrors, online polls are essentially worthless. Statistics published in peer reviewed journals are likely to be better, and are at least open about the methods, but even then there are often arguments about the validity of the analysis used and possible biases in the population.

As with anything, don't believe something just because someone has written about it. Look for peer review, openness, replication and so on. Can you actually prove anything with statistics? Well, obviously not, since you can't prove something that isn't true. However, it's relatively easy to make it look like you've proved something to people who aren't skeptical enough.

[ZERO]

Thank you for that comprehensive answer.

Yes, I see statistics are not raw data but the proccessed result. Quite obvious.
What you say about polls is interesting. When I was 18 I was questioned by the Morgan/Gallup poll. Some of the multiple answer questions I wouldn't answer because they just didn't fit my thinking. The interviewer was quite annoyed.
Leading questions are ugly and I can see the contamination resulting.
So if stats are unreliable proof is difficult. However, If I understand you, similar results from independant sources can be viewed as accurate.

[Dr B]

Zero

You cannot say anything you want with statistics.....you can try, but if the populus is educated in statistics - they won't let you get away with it.

Idealistic I know - but it is true.

Also remember, facts never speak for themselves, they require someone to make cogent arguments for them.

Anyone who says to you, "ah well statistics are unreliable as you can say anything you like with them", does not really understand that it is the people who are unreliable - not necessarily the technique.

Also - keep in mind that stats do not prove anything, they merely give you a degree of confidence as to what the chances are on you accepting / rejecting the wrong idea. Stats themselves are based on assumptions - though these are not always unhelpful.

The point is - they are not concrete proof of anything - but this does not mean they are worthless.

[MRT]

Cuddles - strictly speaking, isn't your argument against biased data collection rather than statistics, as such? Properly applied, statistics will consistently answer whatever question you ask of it from the data you supply. As the old computing motto went 'rubbish in, rubbish out'.

The problem of biased data collection is a large part of skepticism is about, I would guess.

[bobdezon]

Statistics can be misrepresented though to alter the final conclusion. I remember a few years ago they purposely manipulated the unemployment figures (to much much lower). It turned out they had excluded a large percentage of the population because they didnt consider them to be unemployed. Remember those Y.T.S. Schemes? they never included the people in them in the unemployment data, and yet perversely included them in the employment figures. It also excluded under 18's from the unemployment figures, and then they could not claim any unemployment benefit etc.

[FarSideOfTheMoon]

Statistics can be misrepresented though to alter the final conclusion. I remember a few years ago they purposely manipulated the unemployment figures (to much much lower). It turned out they had excluded a large percentage of the population because they didnt consider them to be unemployed. Remember those Y.T.S. Schemes? they never included the people in them in the unemployment data, and yet perversely included them in the employment figures. It also excluded under 18's from the unemployment figures, and then they could not claim any unemployment benefit etc.

And don't even start about the pointless surveys sponsored by corporate Britain which appear daily in the Newspapers....one shudders to think how targeted the questioning is for them to achieve the results they require.

[bobdezon]

I like the adverts for makeup/womens face creams etc. 90% of woman in britain are using this product. Then if you read the small print at the bottom of the screen it says

"based on a survey of 3 women, 2 trannys, a stray cat and a spanner"

The mind boggles.

[tkingdoll]

ZERO, if you're interested in trying to figure out what statistics to be skeptical of, a good place to look is the sample size. This is the actual number of people who were asked the question.

Now, if your sample is large enough and taken from the general population and not just one type of person, then you can say it's likely to be representative of the population as a whole. This is standard. When something tells you what a certain percentage of the population does, of course that survey hasn't asked everyone in the country. It's asked a representative sample and then weighted up the figures to the UK population.

For example, if the UK population is 55% male and 45% female, and I ask 2000 people "what is your favourite drink?", then I need make sure that my sample is roughly 55% male and 45% female if I want to make any claims about the UK population from my data. Let's say I claim "90% of UK women drink Baileys!". Quite a big claim. So if we examine my sample and discover that of the 2000 people I asked, only 10 of them were women, then we know that only 9 out of 2000 people actually claim to drink Baileys. My sample is not representative of the population as a whole so I can't make any claims about the UK as a whole.

The same goes for other demographics like age.

When you are analysing survey data, unless you are looking at something very niche and specific, then as a rough guide you shouldn't trust any data with a sample size of less than 50. That's because once you start to break it down into demographics, you're talking about very small numbers of people who actually answered the question.

Imagine you're looking at people who buy KitKats in the UK. You have a nice healthy sample taken from a nationwide internet survey of 3000 people. You want to break it down by geographic region. Your data tells you that 1000 of those surveyed live in London. That's fine, a random sample of 1000 Londonites will probably give you an accurate picture of Londonites in general. If 40% of your London survey respondents say they eat KitKats, you can safely claim that 40% of Londonites eat KitKats. But if you pull out the data for a tiny village in the Midlands, where only one person did the survey, is that one person representative of the whole village? Probably not.

Bobdezon touched on this, beauty companies are VERY fond of studies with tiny sample sizes. Buyer beware! What 18 women in controlled conditions say about Dove shampoo is not likely to be what most women think.

[ZERO]

ZERO, if you're interested in trying to figure out what statistics to be skeptical of, a good place to look is the sample size. This is the actual number of people who were asked the question.

Now, if your sample is large enough and taken from the general population and not just one type of person, then you can say it's likely to be representative of the population as a whole. This is standard. When something tells you what a certain percentage of the population does, of course that survey hasn't asked everyone in the country. It's asked a representative sample and then weighted up the figures to the UK population.

For example, if the UK population is 55% male and 45% female, and I ask 2000 people "what is your favourite drink?", then I need make sure that my sample is roughly 55% male and 45% female if I want to make any claims about the UK population from my data. Let's say I claim "90% of UK women drink Baileys!". Quite a big claim. So if we examine my sample and discover that of the 2000 people I asked, only 10 of them were women, then we know that only 9 out of 2000 people actually claim to drink Baileys. My sample is not representative of the population as a whole so I can't make any claims about the UK as a whole.

The same goes for other demographics like age.

When you are analysing survey data, unless you are looking at something very niche and specific, then as a rough guide you shouldn't trust any data with a sample size of less than 50. That's because once you start to break it down into demographics, you're talking about very small numbers of people who actually answered the question.

Imagine you're looking at people who buy KitKats in the UK. You have a nice healthy sample taken from a nationwide internet survey of 3000 people. You want to break it down by geographic region. Your data tells you that 1000 of those surveyed live in London. That's fine, a random sample of 1000 Londonites will probably give you an accurate picture of Londonites in general. If 40% of your London survey respondents say they eat KitKats, you can safely claim that 40% of Londonites eat KitKats. But if you pull out the data for a tiny village in the Midlands, where only one person did the survey, is that one person representative of the whole village? Probably not.

Bobdezon touched on this, beauty companies are VERY fond of studies with tiny sample sizes. Buyer beware! What 18 women in controlled conditions say about Dove shampoo is not likely to be what most women think.

So look for sample size and how widespread the participants are in society. This leads me to a new question.

Facts and evidence underpin a realistic opinion. How do you find this for yourself? Or a better way of putting it, what are some simple research methods for a layman like myself? Teach me to fish.

[MRT]

Statistics can be misrepresented though to alter the final conclusion.

I agree. My argument is that the basic process of applying statistics in a standard way is a valid way of manipulating data. If it was not, much science would fall by the wayside. In proper statistics you start with valid data collection, then apply appropriate statistical tools and finally draw conclusions within the defined error limits.

Just as there is pseudoscience, there is pesudostatistics! In pseudostatistics, you can manipulate data collection to suit yourself, apply arbitrary innappropriate statistical tools and draw whatever conclusion you like from the results. This sort of thing is commonly done by lobby groups and others trying to influence opinion. It is amazing how surveys always seem to come up with a result that supports what a lobby group is lobbying about, isn't it? To spot pseudostatistics, always look at who is presenting the results and ask yourself what they would like them to be.

[FarSideOfTheMoon]

ZERO, if you're interested in trying to figure out what statistics to be skeptical of, a good place to look is the sample size. This is the actual number of people who were asked the question.

Now, if your sample is large enough and taken from the general population and not just one type of person, then you can say it's likely to be representative of the population as a whole. This is standard. When something tells you what a certain percentage of the population does, of course that survey hasn't asked everyone in the country. It's asked a representative sample and then weighted up the figures to the UK population.

For example, if the UK population is 55% male and 45% female, and I ask 2000 people "what is your favourite drink?", then I need make sure that my sample is roughly 55% male and 45% female if I want to make any claims about the UK population from my data. Let's say I claim "90% of UK women drink Baileys!". Quite a big claim. So if we examine my sample and discover that of the 2000 people I asked, only 10 of them were women, then we know that only 9 out of 2000 people actually claim to drink Baileys. My sample is not representative of the population as a whole so I can't make any claims about the UK as a whole.

The same goes for other demographics like age.

When you are analysing survey data, unless you are looking at something very niche and specific, then as a rough guide you shouldn't trust any data with a sample size of less than 50. That's because once you start to break it down into demographics, you're talking about very small numbers of people who actually answered the question.

Imagine you're looking at people who buy KitKats in the UK. You have a nice healthy sample taken from a nationwide internet survey of 3000 people. You want to break it down by geographic region. Your data tells you that 1000 of those surveyed live in London. That's fine, a random sample of 1000 Londonites will probably give you an accurate picture of Londonites in general. If 40% of your London survey respondents say they eat KitKats, you can safely claim that 40% of Londonites eat KitKats. But if you pull out the data for a tiny village in the Midlands, where only one person did the survey, is that one person representative of the whole village? Probably not.

Bobdezon touched on this, beauty companies are VERY fond of studies with tiny sample sizes. Buyer beware! What 18 women in controlled conditions say about Dove shampoo is not likely to be what most women think.

Those are good examples because they illustrate another pitfall to look out for - the quality of the sample beyond the simple demographics.

If you were interested in whether people drink Baileys or not, would your sample include people who don't drink alcohol, or people who don't go to pubs? What if you did your survey in the drinks aisle of a supermarket?

Similarly for KitKats, what if you did your survey in a corner shop? What about the part of the population who don't eat sugary snacks?

Advertisers use these tricks to introduce favourable bias to their results. How many times have you looked at a survey and thought those figures are just too high.

[Melanie]

Far Side... Cave ne ante ullas catapultas ambules

[median]

bobdezon wrote

Those are good examples because they illustrate another pitfall to look out for - the quality of the sample beyond the simple demographics.

Exactly, quality is important. What is also interesting is that you can have sample sizes too big as well as too small.
Variance is one to take into consideration as well.

[bobdezon]

That was farsideofthemoon's quote not mine, although its nice, I might claim it possesion is 0.9% of the law (and acorahs bread n butter)