Skeptiko is a pro-parapsychology podcast which attracts a mix of ire and exasperation from the skeptical community due to its misleading title and the tendency of its host, Alex Tsakiris, to promote the show as being an agenda free exploration of ‘controversial science’. A quick listen to any episode of Skeptiko will confirm to an impartial listener that the host, far from being ‘agenda-free’, possesses a painfully evident bias which infusses the entire character and tone of the show. Tsakiris tends to fawn over guests who are on to promote forms of parapsychology he supports and is openly hostile to- the point of occasionally berating- skeptical guests.

In light of the above, and to try and redress some of the bias inherent in the show, I have decided that I will try and offer some skeptical commentary on the Skeptiko shows as Alex adds them and I get time to go through them. It is unlikely that this will have any impact on Alex or the fans of his show, but it might mean that someone who listens to Skeptiko without an agenda and then does a google search on the topic could come across an alternative perspective rather than falling further down the Skeptiko rabbit hole.

So without further adieu, here are my thoughts on ‘Skeptiko episode 100: Dr. Garret Moddel Brings Psi Research to University of Colorado Classroom‘.

The show begins as it often does with Alex extolling the academic credentials of his guest. This is not something unusual in the podcast world, or interviews more generally, but for Alex it frequently serves as the basis for later appeals to authority. In these oft repeated appeals Alex’s general argument is that if some smart academics promote parapsychology then obviously there MUST be something to it. The fact that smart academics across the world promote a whole host of dubious theories including things like intelligent design and aids denialism makes this a less than persuasive argument but that doesn’t seem to ever give Alex much pause for thought… as we shall see later.

For now turning to the ‘meat’ of the podcast, the first thing to say is that Dr. Garrett Moddel is certainly a smart guy and he has probably forgot more about physics than I will ever know. He is a legitimate scientific researcher working in quantum engineering and in the interview he comes across as a reasonable man who seems to have a genuine interest in exploring psi. None of this however, means that his arguments or the studies he presents are compelling and indeed I find that there a host of rather transparent problems besetting his theoretical positions and the parapsychology research he promotes.

An immediate warning sign that his judgement on this topic might be off is that he states early into the show that he was ‘blown away’ when he started looking at the existing research on psi. Now this is, in some respects, a subjective call but frankly, I would love to know what specific evidence it was that ‘blew his mind, as the evidence for psi which is most often cited by psi proponents almost always shows very small effect sizes. And these small effects also frequently disappear when replications are attempted or  better controls are employed. Dr. Moddel seems to freely admit these points at various points during the interview by acknowledging that psi researchers often have problems replicating findings and that psi effects in experiments seem to be very ethereal. Getting blown away by small effects that frequently hover around the border of statistical significance and which disappear with surprising regularity with replications, or when tighter controls are introduced, seems to me to be a strange position for an experienced researcher to take. But that was just the start of the surprises…

Dr. Moddel also discussed how he went about setting up his course in psi research at the university of Colorado. Here aside from criticising the universities apparent initial reluctance to approve the course he also provided details about what the course entails and some of the impressive projects created by students of the course. In this portion of the discussion there were a number of issues that jumped out at me. The first is that after Dr. Moddel explained his vision for the course:

The way I had designed the course, it was discussing psi research and a point-counterpoint discussion throughout the course between skeptical and proponents’ perspectives.

He then goes on to state that for the course “the two textbooks that I’m using are Dean Radin’s Entangled Minds, which is just a wonderful, wonderful book describing psi research and then also Chris Carter’s book on Parapsychology and the Skeptic“. This will immediately set off alarm bells to any skeptic as Dean Radin is one of the most prominent pro-psi advocates today and he is not noted for having anything approaching a balanced position on psi research.

Instead, Radin is one of the strongest advocates for the position that psi phenomena have already been proven and mainstream researchers are ignoring the amazing results because of their conservative bias and prejudices. Using his book as one of the two textbooks for the course might be understandable however if the second textbook provided students with the alternative skeptical perspective and covered some of the common methodological issues which beset psi research. Chris Carter’s book is not the kind of book to provide this information.

Chris Carter, although sharing the same name is not the same individual responsible for the X-Files, I previously made the mistake of assuming they were! However, while he is not famous pro-paranormal TV show creator, his position and clear bias in relation to psi can be seen remarkably clear in a previous episode of Skeptiko during which he declared to Alex that the “so-called skeptics are basically defending an outmoded philosophy of science that’s turned into an ideology”. He also explained how skeptics are motivated to challenge research due to their preconceptions “that these sort of phenomena don’t make any sense and challenge their world-view… they’re going to do anything they possibly can to dismiss evidence that challenges their preconceptions”. This betrays a fundamental lack of understanding of the skeptical position and it also indicates quite clearly how unsuitable Chris Carter’s book is for providing any sort of balance to Radin’s perspective.

So if these two books are the ‘core texts’ for the course I think the chances that the skeptical position gets a fair hearing in Dr. Moddel’s class are rather low. A text book discussing how to avoid flaws in statistical analysis or how to design properly controlled experiments would be a lot more valuable for would-be parapsychology researchers than two pro-parapsychology treatises and this becomes even more evident when the topic moved on to the student’s experiments…

The first experiment mentioned in the interview was described as follows:

One of them involves guessing hidden Zener cards. Zener cards are cards that are of five different types having a circle, square, wriggly line, a plus, and a star. What the student did is he got 100 Zener cards, 20 of each, shuffled them, turned them upside-down, and had the subject sit across the table from him so that the cards were blocked and they couldn’t see them because they were upside-down and also because he had a laptop display between the cards and the subject. He then raised one card at a time and had the subject guess which of the five they thought it was.

He went through the entire stack this way. Then he did this with a number of different subjects. He did it with 12 subjects and tabulated the results.

The actual ‘study’ can be found here.

Now reading through the study it seems that the experiment was designed to test the ability of the respondent to use psi to identify what was located on a card that could not be seen from their position. It was not designed to test whether they read the answer from the experimenter’s mind, as the experimenter was also supposed to be blinded to what was on the card. The blinding procedure used was as follows: the cards were shuffled, placed face down, then the experimenter picked up a card and held it up facing away from them and towards the subject.  A laptop on the desk between the experimenter and the subject blocked the subject’s view of the card and then after they guessed it was set face down on the table.

This is an ok procedure but it would be easy to improve. For example, if a person’s psi ability means they can ‘see’ through a laptop you would imagine that they could also ‘see’ through a coloured envelope so why not just limit all chances of accidental detection from the experimenter AND the subject by randomising the cards and getting a volunteer (not the experimenter) to place them in numbered envelopes then carry out the experiment as before. This might seem like an insignificant amendment but when you consider that the research is attempting to test whether a person can use psi rather than any other more mundane signal to gain information then controls and blinding become extremely important!

For example, imagine that the experimenter unconsciously noticed blurred reflections in the laptop screen as he held up the card. That introduces a potential influence that could lead to significant results that have nothing to do with the subjects possessing psi ability so it is essential that such influences be ruled out.

However, there is a bigger problem with the study- the extremely small amount of participants! For such a simple experiment 12 participants is really a very small sample and it also dramatically increases the chances that you will get a statistically significant result which would disappear with a larger amount of participants. However, instead of taking steps to remedy this problem the student then exasperated the problem by breaking the 12 respondents into four different groups meaning that there were now only three subjects in each group! The groups were described as follows:

1) Those who believe and were presented with pro-psi arguments; 2) those who don’t believe and were presented with pro; 3) those who do believe and were presented with anti-psi arguments; and 4) those who don’t believe and were presented with anti-psi arguments.

And the results?

… for the three groups that had a negative input regarding psi, either they didn’t believe or they were told that it was rubbish; they were pretty close to random in their guesses as to what these cards were. However, the believers who were presented with pro-psi arguments got 27% of the cards correct, whereas by random chance, they would have been expected to get 20% correct.

Finding that one group out of the four got a result that was 7% above random chance is hardly surprising. In fact when you have lax controls and a selection of incredibly small groups it is virtually guaranteed that somewhere in your data something will come out as statistically significant.  And it is also worth noting that there is no indication that the experimenter was blinded to which group they were testing. This is important because numerous studies have demonstrated that experimenter’s bias has the potential to significantly influence results by causing them to treat groups differently due to their beliefs/expectations. This again would have been very easy to control for by simply making another experimenter responsible for the grouping or, to be even more thorough, blinding all the experimenters until the study was completed by using randomised envelopes.

As per Dr. Moddel’s point that ‘by random chance they would have been expected to get 20% correct’ well yes and no. It is true the odds of selecting the correct card are 1/5 but that does not mean that we would expect everyone to get exactly 20% anymore than we would expect that 12 people flipping coins 100 times in a row to all get heads 50 times. We would expect some to be over 50 and some to be under and we would also expect that there will occasionally be some outliers! This is exactly what Dr. Moddell and his student found. They got three outliers in their twelve results and the two positive outliers happened to lie in their pro-psi, pro-psi argument group. Now this could be due to the fact that this group’s positive intentions meant they were able to use a psi ability to improve their guesses or it could be due to random chance.

Unfortunately we will never know because the only way to find out if the effect was robust would be to replicate the study with more participants but Dr. Moddell is against both replicating studies and increasing the number of subjects involved. By doing either of these he argues that the experimenters enthusiasm and interest would drop and that could cause the effect to disappear because it is reliant on both subjects and experimenters being enthusiastic. That’s one potential explanation… but the other much simpler explanation is that the effect will disappear because it is a random anomaly that only exists due to poor controls and small sample size. The boredom objection also doesn’t really hold up because it would be a simple matter to get a group of different group of students to replicate the study with a different group of participants. Hence, their boredom should be no greater than the original experimenter as they would be doing the study for the first time.

Discussing the dangers of boredom Dr. Moddell asserts that:

When you do these experiments, if you’re enthusiastic and interested, you put your intention and interest into it and they work. In fact, that’s probably why, as you’ll see, most of the experiments that my students carry out, work because they’re young and enthusiastic and probably most important, they only do one set of trials. They do enough so that it’s statistically significant, but they don’t do it to the point of boredom.

But what he seems to completely ignore is that getting biased researchers to do small one off studies is also the best way to get biased experimental results which report a chance random effect that is statistically significant but which is, in reality, meaningless. And if psi phenomenon really is so incredibly fleeting that it only amounts to things like improving people’s ability to guess the predict the correct card to 7% above chance as long as no-one in the vicinity has any doubt in psi and those involved haven’t heard anything negative about psi recently then I have to wonder how Dr. Moddell can justify statements like this:

It is truly mysterious and it’s appearing that the answer to understanding consciousness is going to involve psi phenomena. That psi is not just a little pimple on the large experience that we have in life, but instead, psi is one of the major phenomena that controls almost every activity and every thought that we’re involved with.

If psi disappears in the presence of skeptics, repeated measures or simply if enthusiasm dips due to boredom then how can it also be a ‘major phenomena that controls almost every activity’. It seems Dr. Moddell want’s to have his cake and eat it by simultaneously claiming that psi is a huge fundamental force and that, at the same time, it only produces small effects that disappear with the slightest hint of a bad vibe in the area.

At one point after explaining another experiment involving people using their intention to increase the reflection of light in which the experimenter again appears to have been unblinded and a very small sample is used Alex enthusiastically exclaimed:

But I think it speaks to a couple of things. One, clearly for anyone who has any doubts, you’re a guy who knows how to set up and run an experiment like that, or at least oversee someone else doing it. You have all the credentials and you’re a Fellow of the Optical Society of America. I don’t even know what that means.

The typical appeal to authority is here but the point that Alex misses is that Dr. Moddell’s ability to run a Physics experiment is not in question nor is his ability to employ scientific equipment properly. What is in question is whether he has the experience to design properly controlled experiments with human participants and to implement proper controls to rule out experimenter bias and/or control for non-psi effects. Physicists and electrical engineers do not typically spend much of their time during their studies learning about the unique challenges involved with conducting research with human subjects and facts such as Dr. Moddell being a Fellow of the Royal Optical Society bear absolutely no relevance to his ability to conduct good parapsychology research. Mistaking academic expertise in one area for expertise in parapsychology and how to conduct good experiments with humans is a mistake constantly made on Skeptiko discussions and it seems to me that the various expert’s lack of relevant expertise is often the key source for the problems in parapsychology research.

There is much more that could be said about dodgy statistics and poor controls but I think the final experimental example Dr. Moddell provided illustrates a large number of the problems that beset the approach to research that he and his students follow. In this case he described how students in the class tried to use ‘associative remote viewing’ to see into the mind of the experimenter, draw an image of what they saw and thus predict the future. That sounds strange (and it is) but stick with me. The experimenter was supposed to have decided on two different images with one being associated with the stock market going up the next day and one with the stock market going down the next day. He would then show the students one of the images at the end of the following day depending on what the stock market had done.

So to be clear… in this experiment the students were collectively supposed to be able to not only read the mind of the experimenter but also use the image they saw to predict his actions in the future and thus predict the activities of the stock market! Dr. Moddell claimed that using this method his class was able to correctly predict the future activity of the stock market seven times in a row. Setting aside for now the bizarre nature of the claim the first point to note is that since the students were not informed of the two images the experimenter chose a ‘judge’ was included in the experiment and this judge took the collected drawings and then decided on which of the two images the students drawings most resembled.

This sounds like an incredibly subjective judgement and it is also absolutely crucial that the ‘judge’ had no idea which image was associated with which outcome as otherwise his knowledge of what was going on with the stock market could easily influence the result. Furthermore, it is also crucial that the experimenter was kept unaware of the final outcome of the judges deliberations as otherwise they could unconciously alter which image refers to which outcome. Absolutely stringent controls would be needed here as otherwise the prediction would have been a self fulfilling prophecy and yet I can’t help being skeptical that this would have been the case.

Furthermore, although getting the correct result seven times is unlikely it is certainly not as amazing as Dr.Moddell and Alex seem to believe. After all the stock market is not something which each day has a 50/50 chance of going up or down. In some periods the stock market might be stable and growing and in others such as a recession it is quite obvious that its going to be falling more often than its growing. It may have also been the case that of the two images used one was more likely to represent a positive outcome than the other and hence the judge who was probably not an unbiased observer may have been unconciously effected in how he interpreted the students drawings.

In short for experiments like this one, which claim incredible effects (reading minds and predicting the future stock market), you really want to make sure that you have super tight controls in place and that the effect is actually there i.e. multiple replications and repeated independent samples. Dr. Moddel’s theoretical positions forbids things like replications however and also provides a nice escape hatch that if skeptics cannot replicate his results then it is their mindset which is responsible. We are therefore left with nothing but claims of extraordinary results which are never replicated and which are in all likelihood down to poor controls, inadequate samples and experimenter bias. Dr. Moddel acknowledged for instance that when he continued to try and reproduce the effect it disappeared.

I continued using this effect with a few of us afterwards, and we found that as we continued to do it, a little less formally than we had done it with the class, that the effect started going haywire.

This could be as Dr. Moddel believes because those involved became to interested in the outcome but it is also exactly what we would expect to see if the original experiments were statistical flukes or being influenced by some problem with the experimental setup.

The bottom line is it all sounds very flakey and despite possesing clear credentials as a physicist Dr. Moddel (and his students) do not seem to be very capable of designing or implementing well controlled parapsychology experiments. Instead what we get is yet another series of extraordinary claims from a random academic with a suite of poor evidence from individual studies to back them up and a host of dubious explanations to account for why skeptics or even other researchers will likely not be able to replicate their results.

The final point I would mention and one which Alex seemed to pay little attention to during the interview is that Dr. Moddel while stating his endorsements of various ‘alternative’ research topics also alludes to what appears to be an endorsement for aids denialism:

So the topics that we consider are psi phenomena, of course. UFOs, advanced energy concepts, so zero point energy, what used to be called cold fusion is now called low energy nuclear reactions, just second law of thermodynamics issues. We also look at cosmology and astronomy, issues there.

We look at phenomena such as AIDS and HIV. There are alternative theories as to how AIDS is caught. In fact, I think pretty soon you’re going to see mainstream changing around and realizing that the connection between HIV and AIDS that was originally supposed is not quite right.

Then advanced propulsion techniques. Even cryptozoology. There are a number of different topics that are pursued.

So to return to my original point and one Alex and fans of Skeptiko’s approach would do well to heed. Someone being an expert in one field does not mean they are therefore experts in completely unrelated fields. Dr. Moddel’s comments about aids denialism should indicate that he might not be particularly good at assessing medical research and likewise when he claims that “somebody with an open mind, looking at the literature, has got to accept that these psi phenomena exist” I remain skeptical.