Online testing of psychic - help needed.

[Pixel42]

I posted the following on the JREF board earlier today, and was advised that someone on this board might be able to help me.

I've been participating in a thread discussing psychic powers on another board (the general forum of a fan site, not a board specifically about the paranormal). A semi-professional psychic (tarot card reader) who has also been participating has sportingly agreed to a JREF-type test, to establish whether or not she can do successful readings over the internet. [She's quite open to the possibility that she's unconsciously picking up clues (i.e. using cold reading) to give her clients a cheap form of therapy, and there is nothing genuinely paranormal going on].

The test protocol we've come up with is:

1. Six volunteers will PM a trusted moderator with their first name, gender, age and location.

2. The mod will assign each an identifier (subject1, subject2 etc) and add two fictional subjects. He will then PM the list of identifiers and details to the psychic, who will do tarot card readings for each and post them, together with the identifier. She's agreed not to include anything which could be deduced from the minimal details she's been given on each subject, so the probability of the volunteers correctly identifying their own reading should be no better than chance.

3. Each volunteer will pick out the reading they think is theirs, and PM the mod with its identifier. The mod will post which subject picked which reading when he's received everyone's choice.

Where we need help is in calculating the probability of getting 1, 2, 3, 4, 5 and 6 correct choices out of 8 by chance (my maths is a little rusty), and deciding what the success criteria should be. Does anyone know what level of probability JREF require to be exceeded for an applicant to be deemed to have passed the preliminary test?

Any help would be appreciated.

I should make it clear that this is just a bunch of friends doing an interesting experiment in a spirit of genuine enquiry. A fair amount of trust is being placed in both the psychic and the moderator, so we understand that nothing will be conclusively proved, and our test isn't rigorous enough to be accepted by JREF as a real preliminary test. But a lot of interesting and thought-provoking discussion has already been generated, and we'd like to see it through.

[Admin]

Some first thoughts are:

Create a pool of targets from which to be selected.

This pool should be as homogenous as possible. i.e. make sure they are all the same sex and age etc. (at least make them similar in the info the reader will get).

Choose 6 subjects from a larger pool completely at random.

Number them randomly from 1 to 8 (if there are two dummies included).

Give the details to the reader who will prepare the 8 readings.

The intended targets of the readings will be noted by the person running the test.

Give all eight readings to each of the six participants.

They have to choose the reading which they think is the best match to themselves.

Once the results are in a score is calculated on how many people matched up the correct reading.

Scoring.

Odds of success can be set arbitrarily. In most preliminary trials of this type odds of 1000:1 against are usually set. This is high enough to make it unlikely for success to be a chance occurrence and the odds also allow for some slackness in the testing procedure.

In this case scoring 4 or more has odds of 1:337

Scoring 5 or more has odds of 1:6,096

I would say that a score of 5 or 6 would be needed to be considered as meaningful (worthy of further, more rigorous consideration).

_________________________________________________

Is that the sort of set up that describes this test?

If so, I'm sure that we can refine the details as required.

[Pixel42]

Thanks for the reply :)

All your suggestions are good, but the terms of the test have already been agreed, and the number of participants was determined by the number of volunteers, which was 6. The readings will be done tomorrow.

Everyone understands the limitations of the test, it's really just an experiment to generate some thought and discussion (it's succeeding).

I'll post the result early next week, if you're interested.

[Admin]

Absolutely.

These things are not scientifically rigorous of course but they are a great way of learning about testing and how to go about putting controls in place.

[tkingdoll]

Did the participants volunteer publicly? If so, she can do a lot of research on them all beforehand, this would give her an advantage if they are all different ages, nationalities, genders etc.

Regarding the readings she's giving, are they lifestyle readings (telling the participants what kind of person they are and how they live) or is she predicting their futures?

If the latter, how can she avoid mentioning specifics?

I'd be interested to see a typical reading of hers to see what sort of thing she'll be producing.

[Admin]

Did the participants volunteer publicly?

Hopefully we'll find that out - it's crucial that the volunteers are not known publicly.

It looks like it's already been sorted though so points like this will be missed.

[Pixel42]

Did the participants volunteer publicly?

No. They PM'd the moderator who assigned them anonymous identifiers. (point 1 in the protocol). Only he knows who they are (i.e. what the usernames they post under are - none of them are known to her personally).

Regarding the readings she's giving, are they lifestyle readings (telling the participants what kind of person they are and how they live) or is she predicting their futures?

The idea is to give facts which will enable them to recognise themselves. There's no point including information about their futures, because they don't know what's going to happen to them, so how would that help them pick out their reading?

I'd be interested to see a typical reading of hers to see what sort of thing she'll be producing.

If I have time I'll cut & paste one here for you once she's started posting them.

[tkingdoll]

No. They PM'd the moderator who assigned them anonymous identifiers. (point 1 in the protocol). Only he knows who they are (i.e. what the usernames they post under are - none of them are known to her personally).

Fair enough. I was imagining a forum environment where the request was made and people reply in the thread saying "I'm in, PM on it's way with my details" but obviously that didn't happen.

Just one question about controls - how are you ensuring the participants won't or haven't contacted the woman?

The idea is to give facts which will enable them to recognise themselves. There's no point including information about their futures, because they don't know what's going to happen to them, so how would that help them pick out their reading?

I was thinking of 'future' stuff like "you have been considering a new career in nursing. Now would be the right time to pursue this as it will be a fruitful development for you" etc.

We know roughly what our short-term future holds because we are planning for it, by and large.

For example, I run a business which is fairly new. So a statement like "your business will continue to grow and by this time next year you will have taken on three new staff." would leap out as 'meant' for me.

If I have time I'll cut & paste one here for you once she's started posting them.

I meant a typical reading that she's already done, rather than the ones she'll be producing for the test. If you haven't seen one, how do you know this is a demonstration of her usual ability?

[Pixel42]

The psychic has now posted her readings. Unfortunately, despite being specifically requested not to include the gender/age/country information she was given in the readings, she has used 'she' or 'he' in four of the eight readings, and made it obvious that two of the subjects live in the US.

I got the moderator to PM me with the numbers of male/female/American volunteers, and have worked out that instead of all 6 subjects having 8 readings they can choose from, each subject will now (after eliminating those which can't possibly be theirs because they are of people who are a different sex or nationality) have a maximum of 6 and a minimum of 4. Averaging that out at 5 - a bit dodgy to just average it, but it's the best I can do - and feeding that new figure into the formula means that the chances of getting each number of correct choices by chance is now:

1 in 10 of getting 3 or more (down from 1 in 34)

1 in 59 of getting 4 or more (down from 1 in 337)

1 in 624 of getting 5 or more (down from 1 in 6096)

1 in 15624 of getting all 6 (down from 1 in 262144)

So using the JREF threshold of beating odds of 1 in 1000, the success criteria is now all six correct.

We've decided to continue the experiment with the new success criteria. We're now waiting for all the volunteers to read them and get back to the moderator with their choice, which might take a few days. I'll let you know the result.

tkingdoll: Like I said this is just a few friends doing an interesting experiment. A fair amount of trust is being place in all the participants.

[Admin]

That's the problem of having a non-homogenous pool.

If you, or the webmaster, want to do another trial, let us know and we will be happy to help with the design protocol.

As you can see, it's quite easy for things to go wrong - even these informal tests need good controls otherwise they quickly go haywire.

[tkingdoll]

tkingdoll: Like I said this is just a few friends doing an interesting experiment. A fair amount of trust is being place in all the participants.

Oh, I understand that, and starting out informally is as good a way as any. The problem occurs once the experiment is complete, because either it won't be significant, and that presumably will be the end of it, or it will be significant, in which case you have to examine the possible reasons, which could be:

1) coincidence
2) psychic powers
3) cheating
4) other reasons

If you have eliminated the possibility of 3 in advance then you have much less work to do when you repeat the test.

[Pixel42]

The results are in.

No-one picked the correct reading.

[blubird]

LOL!

Maybe Derren Browns book Trick of the Mind is worth a read.

He did a fantastic experiment.

Basically he wrote a "psychic reading" for a group of people. He told them seperatly that the reading would contain really in depth information about them and would be far more informative than most "mediums" would tell them.
When he handed them out and the subjects had read them they all agreed each (what they thought were) individual and personal reading was almost totally accurate to them.

Then he let the cat out of the bag by saying hed written the exact same for everyone of them. But as he said it proved that anything can be made to fit especially if the person recieving wants it to.

[Araneus]

That experiment was done over fifty years ago.

http://en.wikipedia.org/wiki/Forer_effect