2<V
Journal ff Paiapsycholt,~,,y, Vol. 53, December 1989
"FUTURE TELLING": A METAANALYSIS OF
FORCEDCHOICE PRECOGNITION
EXPERIMENTS, 19351987
BN, CHARLES 110NORTON AND DIANE G. FERRARI
ABSTRACT: We report a metaanaIysis of forcedchoice precognition experiments
published in the Englishlanguage parapsychological literature between 1935 and
1987. These studies involve attempts by subjects to predict the identity of target
stimuli selected randomly over intervals ranging from several hundred milli
seconds to one year following the subjects' responses. We retrieved 309 studies
reported by 62 investigators. Nearly two million individual trials were contributed
by more than 50,000 subjects. Study outcomes are assessed by overall level of sta
tistical significance and effect size. There is a small, but reliable overall effect (z
11.41, P = 6.3 X 102S) . Thirty percent of the studies (by 40 investigators) are
significant at the 5% significance level. Assessment of vulnerability to selective re
porting indicates that a ratio of 46 unreported studies averaging null results would
be required for each reported study in order to reduce the overall result to nonsig
nificance. No systematic relationship was found between study outcomes and eight
indices of research quality. Effect size has remained essentially constant over the
survey period, whereas research quality has improved substantially. Four moder
ating variables appear to covary significantly with study outcome: Studies using
subjects selected on (lie basis of prior testing performance show significantly larger
effects than studies using unselected subjects. Subjects tested individually by an
experimenter show significantly larger effects than those tested in groups. Studies
in which subjects are given trialbytrial or runscore feedback have significantly
larger effects than those with delayed or no subject feedback. Studies with brief
intervals between subjects' responses and target generation show significantly
stronger effects than studies involving longer intervals. The combined impact of
these moderating variables appears to be very strong. Independently significant
outcomes are observed in seven of the eight studies using selected subjects, who
were tested individually and received trialbytrial feedback.
Precognition refers to the noninferential prediction of future
events. Anecdotal clainis of' "future telling" have occurred through
> out human history in virtually every culture and period. Today such
0
L_
CL This work was funded by SRI International and the John E. Fetzer Foundation.
CL We wish to thank our PRL colleague George P. Hansen, who is primarily responsible
for retrieving die studies used in ihe ruciaanalysis. Weare grateful to Edwin C. May,
Jcssica IJus, aud to live anonymous reviewcrs al SRI lor valuable Comments ()It .111
earlier draft ofthis report. Valuable comments were also made by Ephrairn Schechter
and by three anonylUOUS referees. The division of authorship responsibility is as fol
lows: Honorton is responsible for the design of the metaanalysis, definition of study
codini, criteria. the actual analyses, and the report itself. F rr 6 r iflorl thp ;nr1;v;i1tin1
research reports in consukation with lionorton and/or Hansen.
00
Q
00
Q
Q
Q
Q
C*4
(1)
U)
LL
>
0
CL
CL
282 The I
Jonriml ofl'(iral).~Iwhology A J111telAnalYsIS of 1,orcedCho'ce Prrcownition Exl)rrimrnts 283
claillis arc generally 1)(Ii(V((l lo bc based ()It f'.'Iclors Stich as delusion,
i&LlOllality, MId SUI)CPStitiolLIS thinking. The concept of piccogni
t4IIi runs counter to accepted notions of' causality and appears to
(1,flict with current scientific theory. Nevertheless, over the past
I f'century a substantial number of experiments have been re
l1r]ted Claiming empirical support. f'Or the hypothesis of' precogni
U7
t0hi. Subjects in FOrcedchoice experiments, according to many re
0
I" ts, have correctly predicted to a statistically significant degree tile
If9mity (or order) of' target StIInUlI randomly selected at it later
lahe.
K_%7e perf'ormed it nietaanalysis of' FOrcedchoice precognition ex
0).
foollineilLS I)LIbliShed Ili the Englishlanguage research literature be
~;een 1935 and 1987. Four Illa*01 questions were addressed
J
t9ough this nietaanalysis: (1) Is there overall evidence f'Or iICCUrate
fget Identification (abovechance hitting) in experimental precog
dkion studies~ (2) What is the magnitude of the overall precognition
lect? (3) Is the observed effect related to variations in methodo
"i'al (Illailty (hat could allow .1 Illore conventional cxplaflafloll~ (1)
.s
)c
precognition perfornia rice vary systematically with potential
Q)d(Y;IIiIIg vallables, .tl(ll as dif'JCJ(lI((.s III Sill)*((( popillolloll",
i
s1rilitilus conditions, experimental seiting, knowledge of' results, and
00 
tole interval between sill)* Ind target generation?
ject response
00
0
0 DELINEATING 'riiE DOMAIN
U) Parapsychological research is still academically taboo, and it Is
C9
LGhlikely that there have been many dissertations and theses Ili this
that have escaped publication. Out retrieval of'studies f'Or this
Tetaanalysis is therefore based on the published literature. The
q0jdles include all forcedchoice precognition experiments appear
In the peerreviewed Englishlanguage para psychology jou rna Is:
Irnal of Parapsychology, Journal (and Proceeding~s) of lite Soclely fiff
P~I~lchical Research, Ic a
6 Journal of the Ainerican Soc'ely fior Psych * al Rest, rch,
kLropean Journal of ParapsycholoAy (Including lite Research Letter of'
& Utrecht University Parapsychology Laboratory), and abstracts of'
Kelrcviewed papCIS presellicd at J'arapsychological Assoclaiion
meetings published Ili Research in Paraps~ychology.
Ciitei"a
7 fior Inclusloll
Our 1(.Vlcw Is re.SII,IcWd lo fixedlellgill Studies III which sigillh
cance levels and ef'f'eci sizes kised on (fireci liffling can be calcil
1,11cd. Similes lising olitcollic variables other than direct liittiiig, such
its I'LlllSCOIe VarianCe and displacement ef'Fects, are Included Only it'
tile report provides relevant information on direct hits (i.e., numgr
of' trials, hits, and probability of a hit). Finally, we exclude stu'r's
conducted by two investigators, S. G. Soal and Walter J. Levy, wh~e
work has been unreliable.
Many published reports contain more than.one experimen
tar
experimental unit. In experiments involving multiple conclitiQ1
significance levels and effect sizes are calculated for each conditi
0
0
Outcome Measures
00
S2igwificance level. Significance levels (z scores) were calculatelpr
each study from the reported number of trials, hits, and probabit,ky
of' success using the normal approximation to the binomial disi
bution with continuity correction. Positive z scores indicate abq[Le
chance scoring, and negative z scores reflect belowchance scori
eCt Size. IkCaUse Most parapsychological CXPCriIIICIItS,
EY c
. J, Pat
Illarly those in tile older literature. have used the trial rather An
the sub'eci. as the sampling unit, we use it Lrialbased eSdillatooit'
9 1
effect size. The effect size (ES) for each study is the z score divi,14d
kb
by the square root of' the number of' trials in the study.' 0
00
General Characteristics of 1he Domain
We located 309 studies in 113 separate publications. These SRI
tes were contributed by 62 different senior authors and were peb
lished over a 53year period, between 1935 and 1987. Considellig
tile halfcentLIFY turiespan over which the precognition experim(Alts
were conducted, it is not surprising that the studies are verydiv e.
t
The database comprises nearly two million individual trials Lnd,
more than 50,000 subjects. Study sample sizes rangefrom 2_10to
297,060 trials (median = 1,194). The number of subjects ranles
from I to 29,706 (median = 16). The studies use a variety Of In I
odologies, ranging from guessing ESP cards and other card s
;ynIi&is
to automated random number generator experiments. The clontril
encompasses diverse stll)*ect populations: the most frequently i=d
. <
' Elsewhere (Honorton, 1985), we have used tile effect size index Cohen's h
(Cohen, 1977), and one referee has asked that we explain why we are now using
z/All". The answer is that h and zfN"2 yield virtually identical results, and ZIN` is,
computationally simpler. For tile present sample of 309 precognition studies, the
nican difference boween the two indices is .00047, and the standard devi;ition of tile
differencc is .0126: 1(308) = 0.312, p = .756, twotailed.The coriclafion between the
two indices is .97.
v
0
04
04
CD
CD
W
0)
00
CD
CD
c6
(D
Q
00
0
00
0
0
CM
W
U)
C9
0
LL
0
(1)
>
2
CL
CL
<
28,1 Thrjomnal (?/ Paiap~ychology
TABLL I
OVERALL SIGNIFICANcE LEvEl. AND EFFECT SIZE
Z ES
Mean 0.65 0.020
2.68 0.100
SD
Lower 95% confidence estimate 0.40 0.011
10 25
Combined z 11.4 1, p 6.3 x
"Failsafe N" = 14,268
i(ES) = 3.51, 308 df, p = .00025
population is students (in approximately 40% of the studies); the
least frequently used populations are the experimenters themselves
and animals (each used in about 5% of the studies).
Though a Few Studies tested subjects through the mail, more typ
ically subjects were tested in person, either individually or in groups.
Target selection methods included no randomization at all (studies
using "quasirandom" naturalistic events), informal methods includ
ing manual cardshuffling or dicethrowing, and fornial methods,
primarily random number tables or random number generators.
The time interval between the subjects' responses and target gen
eration varied from less than one second to one year.
OVERALL CUMULATION
Evidence for an overall effect is strong. As shown in the top part
of Table 1, the overall results are highly significant.' Lower bound
(onetailed) 95% confidence estimates of the mean z score and ES
are displayed in the bottom portion of Table 1.
Ninetytwo studies (30%) show significant hitting at the 5% level,
and significant outcomes are contributed by 40 different investiga
tors. The z scores correlate significantly with sample size: r(307)
.156, p ~ .003. The mean number oftrials for significant studies is
34% larger than the mean number of trials for nonsignificant stud

ies.
'The statistical analyses presented here were performed using SYSTAT (Wilk
inson, 1988). When I tests are reported on samples with unequal variances, they are
(Aculaird usilig 111C sepalair varialittN wilhis) gloups For 111V cri'm and (Irgiers of
freedom following Brownlee (1965). Unless otherwise specified, p levels are one
tailed. Combined z's are based on Stollffer's method (Rosenthal, 1984).
285
%0 0
:0
:,9
0
~0
1
.0
: 0 0
Q)
:0 0
8. 0
0
: 0
0 00
0 0.
1
C')O
0
0 0
CM
04
0 0
00
0
a
0
0
0
0.2 0.1  0.0 0.1 U 0.3 0.4 0.5
00
0
Mean effect size 00
0
l,'igtite 1. rylean effect size by investigator. N = (12 investigators. 0
0
ReplIcallon Across lnvestil~afors CM
0)
U)
Virtually the same picture emerges when the cumulation is 1#
investigator rather than study as the unit of'analysis; the combin2t
__6
z is 12.13, and 23 of the 62 investigators (37%) have overall 01W
comes significant at the 5% leveL The mean, (investigator)'effect siAQ
0
is 0.033 (SD = .093). LL
There is a significant difference in the mean ES across invesi~
gators, but it is surprisingly sinall: KrUskalWallis oneway ANOV
by ranks, X'(6 1) = 82.7 1, p = .034. The effect is clearly not due
E?
a few major contributors. If investigators contributing more thZW
three studies are eliminated, leaving 33 investigators, the combin
z is still 6.00 (1) = 1.25 X 10") and the mean ES is .028 (SD
.0m). Figure I shows the mean effect sizes by investigator.
liese results indicate substantial crossinvestigator replicability
and directly contradict the claim of critics such as Akers (1987) that
286
Dir'lon)tial o1'Paraps),rhf)1oAy
successfill parapsychological oulconics arc achleved by only a I'CW
investigators.
CD
I
Zbe P'iledravler Probleiii
CD
CD
A wellknown reporting blas exists throughout tile behavioral
to
.45ences favoring publication of "significant" studies (e.g., Sterling,
a59). The extreme view of this "filedrawer problem" is that "the
are filled with the 5% of tile studies that show Type I er
=s , while the filedrawers back at tile Iab are I'lled with the 95% of'
Studies that show nonsignificance. . . " (Ros tinthal, 1984, p. 108).
)*COglllZillg tile importance of' this pi
I oblem, the Parapsychological
tsociation in 1975 adopted an official policy against selective 1'e
rting of'positive results.' Examination of the parapsychological lit
ture shows that nonsignificant results are frequently published,
ILd, in the precognition database, 70% OF the studies have reported
Rnsignificam restilis. Neveiificless, 7r)(;',, ofilic precogn,11011 st(OR's
e published before 1975, and we must ask to what extent selec
atl., publication bias could account fol tile Cumulative effects we ob
Qll_"~ . . . .  .  .  __  
CO 'I'lic cenfral, sectioll of''Fabic I 11scs Rosenihal's (198,0 "I'lailsa[C
!R' Statistic to estimate the nu1llbCl_ Of' unreported Studies with Z
9)1,es averaging zero that would be necessary to reduce the known
aitabase to nonsign if icai ice. The filedrawer estimate indicates that
8er '16 U111CpOrIC(l Studies IDUA exist for each reporied stialy (o
i ificant level.
MdUce tile C11111111itti
IVC Outcome to a nonsigin I
a) A different approach to the filedrawer problem is described by
ratwes, Landman, and Williams (1984; personal C011111111111CatiOn
.&)in Dawes to Honorton,
'Juiv 14, 1988). Their truncated nornial
(1) 1
Wrve analysis, like Rosenthal s "fallsafe N," is based oil nornial
~LUFVC assulliPtiOlls. Their null hypothesis is that z scores above some
12itical level (e.g., z = 1.65, 1.96, etc.) are randomly sampled from
AUO, 1) above that critical level. The alternative to the null hypothes'
U is
it) that, because there is some real effect, the distribution of z's is
.gifted to the right of 10 and the z's will be larger than predicted by
te null. For a critical level of'z = 1.65, the expected mean z is 2.06
Ad tile variance is .14. In (lie precognition database, there are 92
sLlies xvith z's > 1.65. Their average is 3_61, not 2.06 its predicted
Ailal~ses_ iiidicate no significaiii difference in the 111agnitude 4reported stmlv
_11C IM , V
oIII( ollics b(lorc and afic' 1975. .1 '011 FS to, %todirs III iot to 1975 k 0,0.~ I (~ )
.0,M), and hn studics ICpoItCd 111creatter the mean is 0.017 (SD = .106): f(307)
0.28, p = .782, twotailed.
orcedCholCe Preco "I*oii E Periiiients 287
A AletaAiiahsis ofF gm I
by tile null hypothesis. Since file variance of' the nornial truncated
above 1.65 is .14, the test z (using the Central Limit Theorem) com
paring 3.61 to 2.06 is 39.84 [1.55 divided by (.14/92)"']. Here, p is
virtually zero. Similar results are found with CUL points of 1.96, 2.33, rl
and 2.58. CD
Q
On the basis of these analyses, we conclude that the cumulative a
significance of the precognition Studies cannot satisfactorily be ex,r'
(D
plained by selective reporting. C)
CD
C*4
C*4
CD
Oun.u,Az Ri.,i)uunON CD
00
Although the overall z scores and effect sizes cannot reasonablyr
. . . CD
be attributed to chance, inspection of the standard deviations in(:)
I
Table I indicates that the study outcomes are extremely heterogeto
neous. Given tile diversity of methods, subject populations, and4m
Othel study features that CIMIaCtelMC this Iesearcli domain, tills iso
not surprising.
The study outcomes are in fact extremely heterogeni~ous. Al~
though a major objective of this metaanalysis is to account for theo
Variability aCrOSS St tidies by blocking oil differences in study quality, 00
procedural features, and sampling characteristics, the databasec)
clearly contains extreme outliers. The z scores range from 5.1 toco
. C)
19.6, a 25sigma spread! The standardized index of kurtosis (,2) 1S8
9.47, Suggesting that the tails of' the distribution are much too lon6SD
C)
for a normal distribution. C*4
We eliminated the extreme outliers by performing it "10 percenta)
trim" on the study z scores (Barnett & Lewis, 19 ,78). This involvesu)
M
eliminating studies with z scores in the tipper and lower 10% ofthe(I)
distribution, and results in an adjusted sample of' 248 Studies. T 142
trimmed z scores range from  2.24 to 3.21 .(g, 1. 1). The re"
vised z scores and effect sizes are presented in Table 2. 0
LL
Elimination of extreme outliers reduces the combined z scores b YV
Iv one half, but the outcomes remain highly significantV
approximate,
Twentyfive percent of the studies (62/248) show overall significanto
hitting at the 5% level. Lower bound confidence estimates show thab"L
the mean z's and effect sizes are above 0 at the 95% confidence level CL
Elimination OfOULljelS reduces the total IlUmber ofinvestigatol's
fioin 62 to 57, but the' results remain basically the same when the
Mlitl)'SCS MC IMSC(l Oil inVCN1i9jJ10YS ratlIff thall studies. ']'lie COID
bined z is 6.84; IS of the 57 investigators (31.6%) have overall.sig
.288 TheJournal (?J'Parapsychology A MelaAnalTsis ofForcedChoice Precognition Eicperiinent,~ 289
TABLE 2
C) SIGNIFICANCE LEVII. AND EFFECT SIZE FOR TRINUMEI) SAMpI.I,_
Q ES
Q
%an
0.38 0.012
1.45 0.065
Ower 95% confidence estimate 0.23 0.005
C*4
C*4 Combined z 6.02, p 1. 1 x 10
CD
CD I(ES) = 290, 247 dj, p = .002
fficant outcomes at the 5% level. The mean (invest'
igator) ES is
Q;020 (SD = .05).
9For the trimmed sample, the difference in ES across investiga
s is not significant: Kruskal
Wallis oneway ANOVA by ranks,
56) = 59.34, p = .355. If investigators contri 1
more than
ee studies are eliminated, leaving 37 investigators, the combined
107
z a
,,s still 5.00 (p = 3.0 X and the mean ES 0.022 (SD
M6). Figure 2 shows the inean effccL size by Investigator.
Thus, elimination of' the outliers does not substantially affect the
c
lusions drawn from our analysis of the database as a whole.
Yeile clearly is a nonchance effect. In the remainder of this report,
40 use the trimmed sample to examine covariations in effect size
CD
T51 a variety of methodological and other study features.
CD
C)
C*4 STUDY QUALITY
(D
U)
cu Because target stimull in precognition experiments are selected
uSy after the subjects' responses have been registered, precognition
(D
s&dIes are usually not vulnerable to sensory leakage problems.
(ILher potential threats to validity must, how6er, be considered.
Toe problem of variations in research quality remains a source of
G&troversy in metaanalysis. Some metaanalysts advocate eliminat
low quality studies whereas others recommend empirically ac
c
111sing the impact of variations in quality on study outcome. Rosen
I t] .(1984) points out that the practice of' discarding studies is
eSivalent to assigning them weights of zero, and he recommends
5 1
~NNghLing study z scores in relation to ratings of research quality.
Study Quality Criteria
Ideally, the assessment of study quality should be performed by
knowledgeable specialists who are blind to the study outcomes. III
6 0
:00
00.
:0
0
0 0
0
L_ d
0. 0
6
0,
0
0
0
0
0
0 0
00
0
01
.00
:0 00
10 0
0
0
0 0 1
0
0
0 0
00
0.20 0.15 0.10 0.05 0.00 0.05 0.10 0.15 0.20 CD
00
Mean effect size CD
CD
Figure 2. Mean effect size by investigator for trimmed sample. N 57 ig
vestigators. C*4
practice, this is usually not feasible, particularly When, as in the preM
ent case, large numbers of studies are involved. For our analysis at
study quality, statistical and methodological variables are definez
and coded in terms of procedural descriptions (or their absence) 1%
the research reports. This approachwas used in an earlier metw"a
analysis of psi ganzfeld research (Honorton, 1985), and it led W
study quality ratings that were generally in agreement, r(26) = .76~2
p = 10", with independent "flaw" ratings by an outside critic (H
0
L_
man, 1985).
One point is given (or Withheld) for each of the following eigla
criteria:
specification of sample size. Does the investigator preplan the num
ber of trials to be included in the study or is the study vulnerable
to the possibility of optional stopping? Credit is given to reports that
explicitly's pecify the sample size. Studies involving group testing, in
which it is not feasible to specify the sample size precisely, are also
290 TheJournal of Parapsychology A MetaAnalysis of ForcedChoice PrecQgnition Experinients 291
given credit. No credit is given to studies in which tile sample size
Is elther not preplarined or not addressed in tile experimental re
9,)011.
V_
CD Prej)lanned analYsis. Is the method of statistical analysis, Including
the outcome (dependent variable) measure, preplan ned;) Credit is
CD .
to given to studies explicitly specifying the form of' analysis and the
a outcome measure. No credit is given to those not explicitly stating
C)
the form of the analysis or those in which the analysis is clearly post
*4
C
C*4 lloc.
C)
C) Randoinization inelhod. Credit is given fOr use of' randorn number
Lables, random number generators, and mechanical shufflers. No
00 credit is given for failure to randomize (i.e., use of "quasirandom
1
c) naturalistic events") or for Illf'OrIllill InCtIlOdS Such its hilildShUf"lling,
9 diecasting, and drawing lots.
to
ConlroLs. Credit is given to studies reporting randomness control
(L checks, such as random number generator (RNG) control series and
In . '
W empirical crosscheck controls.
Recording. One l)OIIIL Is allotted For automated recording Of' Lar
gets and responses, and another for duplicate recording.
Checking. One Point is ~Illotted for automated checking of'
00 matches between target and response, and another for duplicate
C)
checking of' lilts.
00
CD
CD
CD Study Quality Analysis
C*4
(D Each Study received it (ILMlity weight between 0 and 8 (inean
U) 3.3, SD = 1.8). We find no significant relationship between study
C9
(1) quality and E'S: r(2,16) = .081, p = .202, twotailed. This tclidency
(1) [or Study Outcomes to correlate posilively with study quality has tile
consequence that the qualitywelghted z score of' 6.26 is slightly
0 larger than the Univeighted z of"6.02. Table 3 shows the correlations
LL.
between effect size and each of the eight individual quality meas
.1
tires . The mean effect sizes by quality level are displayed graphl
>
0 cally in Figure 3.
L_
CL
The correlation between ES and study quality is also nonsignificant for the un
CL ~
trimmed sample of 309 studies: r(307) = ,060, p = .289. The qUalityWCigl1tCd Z
score is 7.38: p = 2.32 x 10". However, thFCC Of1hC indiVidLU11 (111;llily meaSures
are significantly related to pcHormanCC. COnLrolS and duplicate cliccking correLuc
significantly positively with ES, and randomization correlates significantly negatively
with ES. these correlations appear to be due to a few studies witli z scores that are
extreme outliers (z > 7). When the 10 studies witli z > 7 are eliminated, the signifi
cant correlations between quality and ES disappear.
rABLE 3
CORREIA FIONS BEITNVIAIN Eii,i;(; I SIZE AND QUALI I'Y MEASURES
Q
I
e r(246) V_
QUaliLy ineasui C)
CD
Saniple size specified in advance .100 CD
Preplanned analysis .001 V_
to
RandOMIZaLIon .011 C)
CD
Controls .058 C*4
ALILOinated recording .169 C*4
C)
DUpIiCaLe recording .047 C)
Autoniated checking .136
Duplicate checking .078
00
I
CD
Quality Extre7nes CD
I
Is there a tendency for extremely weak studies to show larger [L
effects than exceptionally "good" studies? Analysis on the extremes 0
of" tile quality ratings indicates that this is not the case.
This analysis, based on the untrimmed sample of 309 studies,
uses Studies with quality ratings outside tile iiiLerquartile range of6
the rating distribution (median = 4, Q, = 2, Q, = 5). There are 56 ' '
s (ratings of 01) and 35 "highquality" . 00
"lowquality" studie I studies CD
(ratings of 68). The highquality studies have effect sizes that are 00
CD
not significantly lower than the lowquality studies; the ES means
CD
are 0.017 (SD = 0.063) and 0.037 (SD 0.137), for the low and.(D
C)
highqUality studies, respectively: t(82) .92, p = .358, twoC*4
talled
U)
M
Quality Vari(zhon in Publication Sourctu
precognition ES is not significantly related to source of' publicaW
tion: KruskalWallis oneway ANOVA, X'(4). = 0.78, p = .942.
0
However, the sources of publication differ significantly in studyLL
X2
quality: KruskalWallis oneway ANOVA, (4) = 17. 19, p = .002.
This is due largely to the lower quality of studies published in the >
Journal of the Society for Psychical Research and in Research in Parapsy
cholo CL
gy. CL
Study Quality in Relation to Year of Publication
Precognition effect size has remained constant over a halfcen
tury of research, even though the methodological quality of the.re
292 71e.fournal qf Paraf)sychology
8 
7  13
6  7
5  45
4  63
3  35
9  41
31
8
00 0.10 0.05 000 0.05 0_10
CD
00
CD Mean effect size
CD
Fig2e 3. Precognition effect size in relation to study quality, with 95% con
fid "'e limits. N = 248 studies.
seatah has improved significantly during this period. The correla
tior&etween ES and year of publication is .071: t(307) 1.25,
p =Z213, twotailed. Study quality and year of publication are, how
ev9, positively and significantly correlated: r(246) = .282, p = 2 x
10t twotailed.
L6ritics of parapsychology have long believed that evidence for
'&sychological effects disappears as the methodological rigor in
ar
p
cre2mes. The precognition database does not Support this belief'.
0
CL
CL
< "REALTIME" ALTERNATIVES TO PRECOGNITION
Investigators have long been aware of the possibility that precog
nitioll effeCLS could be modeled without assuming either Lillie IeVcl
sal or backward causality. For example, outcomes from studies with
A AlelaAnal.ysis (?f ForredChoWe Precognition Experiynent,~ 293
Laigets based on indeterminate randorn number generators (RNGs)
could be due to a causal influence on the RNGa psychokinetic
(PK) effcctradier than information acquisition concerning its f'u
ture state. In experiments with targets based on prepared tables of
Fandoin numbers, the possibility exists that the experimenter or
Offier randornizer may be the actual psi source, unconsciously using
"realtime" ESP combined with PK to choose an entry point in the
random number sequence that will significantly match the "sub
ect's" responses. While the latter possibility may seem farfetched,
it cannot be logically eliminated if one accepts the existing evidence
for contemporaneous ESP and PK, and it has been argued that it is
less farfetched than the alternative of "true" precognition.
Morris (1982) discusses models of experimental precognition
based on "realtime" psi alternatives and methods for testing "true"
precognition. in general terms, these methods constrain the selec
Lion of the target sequence so as to eliminate nonprecognitive psi
intervention. In the most common procedure, attributed to Mangan
ate a set of numbers t
(1955), (lice are thrown to genet hat are math
ematically manipulated to obtain an entry point in the random num
ber table. This procedure is sufficiently complex "as to be appar
ently beyond the capacities of the human brain, thus ruling out PK
because the TKer' would not know what to do even via ESP" (Mor
ris, 1982, p. 329).
I'wo features of precognition study target determination proce
dures were coded to assess "realtime" psi alternatives to precogni
tion: method of determining random number table entry point and
use of Mangan's method.
Methods of eliminating "realtime" psi alternatives have not been
used in studies with random number generators and have only been
used in a small number of studies involving randomization by hand
shuffling. These analyses are therefore restricted to studies using
random number tables (N = 138).
Method qf Determining RNT Entry Point
The reports describe six different methods of obtaining entry.
points in random number tables. If the study outcomes were due to
subjects' precognitive functioning rather than to alternative psi
modes on the part of the experimenter or the experimenter's as
sistants, there should be no difference in mean effect size across the
various methods used to determine the entry point. Indeed, our
analysis indicates that the study effect sizes do not vary systernAi
CD
Q
CD
CD
T"
CD
CD
C4
C*4
CD
CD
00
CD
CD
I
a)
IL
co
CD
00
CD
CD
CD
CD
C*4
(D
U)
ca
(D
0
LL
'a
>
0
CL
CL
CD
I
'
CD
CD
CD
V_
to
CD
CD
C*4
C*4
CD
CD
00
I
Q
C)
(D
CL
0
00
CD
00
CD
CD
C)
C*4
(D
U)
M
75
W
0
LL
0
"
CL
CL
294 Vir.lournal of Parapsi,rhologn~
cally as, a filliclion of' method of' deferillining Ihe entry polill: Krus
kalWallis oneway ANOVA by ranks: X~'(5) = 7.32, p = .198.
Use (?f Mangan'S Method
We find no significant difference in ES between studies using
complex calculations of' the type introduced by Mangan to fix the
random number table entry point and those that do not use Such
calculations: 1(45) = 0.38, p = .370, twotailed.
MODERATING VARIABLES
The stability of' precognition study outcomes over a 50year pe
riod, which we described earlier, also bit([ news. It shows that
vestigators in this area have yet to develop sufficlent IIndelSLailding
of' the conditions underlying the occurrence (or detection) of" diese
ef'feCLS to reliably HlCre~ISC LIICIr IMIgniLLIde. We have identified Four
variables that appear to covary systematically with precognition ES:
(1) selected versus unselected subjects, (2) individual versus group
testing, (3) f'eedback level, and (4) time interval between subject re
sponse and target generation.
The analyses us& the raw study z scores and effect sizes; we
found IIlitt this results in Unif'Ornily more conservative eSLiIIIaLCS of'
relationships with moderating variables than when the analyses are
Z3
IXISed Oil (IWIlitywelghted z scores and ef'f'ect sizes.
Selected Versus Unselected Subiects
Our inetaanalysis identifies eight sub*ect populations: unspecl
9
fied subJect popt I la (ions, numures of' several dif'Ferent populations,
MI 11MIS, StUdellUS, Children, "VOILIMCCIS," CXI)Cl'lll]ClltCl(S), 2111d Se
lected subjects.
EYFect. size magnitude does not vary significantly across these
X2(1)
eight SLI[JeCt pOpUlalions; KrUskalWallis oneway ANOVA, i
10.90, P = .143. Eff'ect sizes by subject population are displayed in
Figure 4.
However, studies using subjects selected oil the basis of prior
performance in experiments or Pilot tests show significantly larger
effects than studies using unselected Subjects. As shown in Table 4,
60% of' the studies with selected subjects are significant at the 5%
'1
level. The mean z score f'or these Studies is 1.39 (SD = 1.40). J'he
ES IS SignifiGinfly highCr f'Or SCICCtedSUbjCCIS Studies thall fiff Stud
4 'Wela,411all'sIN
of ForcedCholre
Precognition
295
CD
Selected  25
V_
CD
Exptr  12 CD
CD
V_
VOILIIlteel to
26
CD
CD
C*4
Children  31 C*4
CD
0 CD
~n_ sludclits108
00
~F Animals 10
Cf) C)
C)
28
mixed
Unspec 18
 0.06 0.04
0.02 0.00
0.02 0.04
0.06 0.08
0~10
Mean effect 00
size
CD
00
Figure .1. (2
Precognition
effect size
by subject
population,
with 95%
confi
dence limits. C)
N = 248
studies.
C)
CD
ies with C*4
Unselected
Subjects.
The t test
of'the difference
ill niean
ES
, (D
is equivalent
to a pointbiserial
correlation
of' .198.
U)
Does this M
difference
result from
less SLIIngem
controls
in studies
with selected 7a)
subjects~
The answer
appears
to be "No."
The average
CILI~Illty W
of' studies
with selected
Subjects
Is higher
than studies
us'ing
0
TABi.i.,. LL
4
SELECYED
VERSUS UNSELE.CFED
SUBJECI'S
>
0
Selected L_
Unselected
CL
N studies CL
25 223
Combined
z 6.89 4.04
Studies with
p. < .05
60% 21%
Mean ES .051
.008
.075 .063
t(246) 3. 16, p .00 1
296 'The Jourrial of Parapsychology
Q TABLE 5
I
INDIVIDUAL VERSUS GROUP TESTING
Q
Q
Q Individual Group
Ir
CD N studies 97 105
Q
Q Combined Z 6.64 1.29
04 Studies with P < .05 30% 19%
N
(:) Mean ES .021 .004
Q
W SD,.s .060 .066
CY)
00 t (200) 1.89, p .03
9 unselected subjects: 1(27) = 1.51, p = .142, twotailed. This result
to
CD appears to reflect a general tendency toward increased rigor and
0 more detailed reporting in studies with selected subjects.
0
W
1 Individual Versus Group Testing
<
Subjects were tested in groups, individually, Or through the mail.
co Studies in which subjects were tested individually by an experimen
52 ter have a significantly larger mean ES than studies involving group
00 testing (Table 5).
Q
Q The t test of' the difference is equivalent to ;I poinLbiserial Cor
Q relation of .132, favoring individual testing. Of the studies with sub
Q
04 jects tested individually, 30% are significant at the 5% level.
a) The methodological quality of studies with subjects tested indi
U)
M vidually is significantly higher than that of studies involving group
testing: 1(137) = 3.08, p = .003, twotailed. This result is consistent
W with the conjecture that group experiments are frequently con
" ducted as "targets of opportunity" and may often be carried out
0 hastily in an afternoon without the preparation and planning that
LL
go into a study with individual subjects that may be conducted over
'a
(D a period of weeks or months.
> Thirtyfive studies were conducted through the mail. In these
0
L_ studies, subjects completed the task at their leisure and mailed their
CL
a responses to the investigator. These correspondence studies yield
outcomes similar to those involving individual testing. The com
bined z score is 2.66, with a mean ES of 0.018 (SD ~ .082). Ten
correspondence studies (25.7%) are significant at the 5% level.
, Eleven studies are unclassifiable with regard to experimental set
ting.
A MetaAnalysis of ForcedChoice Precogm 'it 'on Experiments 297
T,kBLI... 6
FEEDBACK RECEIVED BY SUBJECTS
Q
Feedback of Results
Q
None Delayed Run score Trialbytrial
N swdles 15 21 21 47
Combined z 1.30 2.11 4.74 6.98 Q
Q
Studies with p < .05 0.0% 19.0% 33.3% 42.6% 04
04
Mean ES .001 .009 .023 .035 Q
SD, .028 .036 .048 .072 Q
00
Feedback r1l.
Q
Q
I
A significant positive relationship exists CD
between the degree of
a)
feedback subjects receive about their performance[L
and precognitive
effect size (Table 6).
Subject feedback information is available
for 104 studies. These
'
Studies fall into four feedback categories:
no feedback,
delayed
feedback (usually notification by mail),
runscore feedback, and
.
trialbytrial feedback. We gave
these categories numerical values
00
between 0 and 3. Precognition effect size (Q)
correlates .231 with feed
back level (102 df, p = .009). Of the 47 00
studies involving trialby
Q
trial feedback, 20 (42.6%) are significant Q
at the 5% level. None of
the studies without subject feedback are Q
significant.
Q
Feedback level correlates positively though 04
not significantly with
research quality: r(102) = .173, p = .082, (D
twotailed. Inadequate
randomization is the most plausible source U)
of pote ntial artifacts in M
studies with trialbytrial feedback. We
performed a separate analy
sis on the 47 studies in this group. Studies
using formal methods of
randomization do not differ significantly
in mean ES froin'those
with informal randomization: t(15) = 0.67, 0
p = .590, twotailed.
LL
Similarly, studies reporting randomness control
data do not differ
significantly in ES from those not includinga)
randoniness controls:
>
t(42) = 0.79, p = .436, twotailed. 0
L_
CL
Time Interval CL
<
The interval between the Isubject's response
and target selection
ranges from less than one second to one year.
Information about
the time interval is available for 144 studies.
This information, h6w
00
C)
00
CD
CD
Q
Q
C*4
U)
C9
(1)

0
LL
,Cl
(1)
>
0
L_
CL
a
2~i8
Mollills
Weeks
7z
:1
Days
;:Z Hours
C
MillUtCS
Seconds
I%Iilllsc(:
1jujournal ol Parap,~I,chology
Me~ill el'FeCt. SiZe
Figure 5. Eflcct size by precognitioii intcr%,al, i%,itli 95X confidence limits
N = 144 Studies.
 analvsis of' the relationship between
ever, is often imprecise. Out
precognitive ES and time interval is therefore limited to seven broad
interval categories: milliseconds, seconds, Minutes, hours, days,
weeks, and months. (Effect sizes by precognition interval are dis
played in Figure 5.)
Although it IS C0l1f'0LllIdCd with degree of' Feedback, there is it
ificant. decline III precognition E'S over :Increasing teni )oral (its
sign I
tance: r(142) = .199, p = .017, Lwotalled. The largest effects oc
cur over the millisecond interval: N = 31 studies, combined z
6.03, mean ES = 0.045, SD = .073. The smallest effects occur over
periods ranging from it nionth to a year: N = 7, combined z
0.53, nican ES = 0.001, SD ~ .0,19.
Interestingly, the decline of' precognition pe r F01111 a lice Ovel in
creasing teniporill dISIallCeS results entirely I'l0111 Studies LISIng, 1111
1 J11riablellYp'S I,)ec(Wtiitioii ExI)rrintrilts 299
selccicd sul)'ects: )(122) = .235, p = .009,
twotailed. Studies with
selected SLIlJeCLS Show a nonsignificant Positive
relationship between
ES and time Interval: r(18) = .077, p = .745,(D
twotailed. Although I
the difference between these two correlationsT.
IS not Significant (Z =
C)
1.24), this suggests that the origin of the C)
decline over time may be
nionvatlonal rather than the result of some C)
intrinsic physical bound ,
ary condition. The relationship between precognitionCTD
ES and feed
C)
back also supports this cot 'ectUre. Nevertheless,C)
any finding SLIg
11
10 1
gesting potential boundary Conditions Oil C*4
the phenomenon S I Li d C*4
be vigorously pursued. Q
Q
Influence qf Moderating Variables in Combinatio7l
00
C)
The above analyses examine the impact of eachC)
moderating var
iable in isolation. III this final set of (6
analyses, we explore theirjoint
influence oil precognition performance. For
this purpose, we iden
ify two subgroups of studies. One subgroup
characterized by the
t
use of' selected subjects tested individuallyW
with trialbytrial feed
back. We refer to this as the Optimal group
(N = 8 studies). The
second group is characterized by the use of
unselected subjects
tested in groups with no feedback. We refer
to this as the Suboplinial
roup (N = 9 studies). 00
9 C)
The Optimal studies are contributed by four '
independent inves 00
gators and the Suboptimal studies are contributedCD
by two of the
ti
same four investigators. All of the Optimal Q
studies involve short pre Q
cognition time intervals (millisecond interval);C)
the Suboptimal stud *
C
ies involve longer intervals (Intervals of 4
weeks or months). All of the
Optimal studies and 5 of the 9 Suboptimal U)
studies use RNG meth
M
odology. The two groups do not differ significantly(1)
in average sam
ple size. The mean study quality for the Optimal
group is signifi
candy higher than that of the Suboptimal studies:
Optimal mean =
'
0) = "0
6.63, SD = 0.92; Suboptimal mean = 3.44, SD
= 0.53; t(1
8.63, P = 3.3 x 10_(~, LWOLailed. LL
The combined impact of the moderating variables'D
appears to be (1)
even of the 8 Optimal studies (87.5%) are >
quite strong (Table 7). S
independently significant at the 5% level, L0_
whereas none of the Sub
Dptimal studies are statistically significant.CL
All four investigators con CL
tributing studies to the Optimal group have
significant outcomes.'
' Ill the untrininied saniplcoi*309studies,
thereareatotalol'17 Optinialstudies.
flie inean ES is 0.117 (SD = .154), and the
combined z is 15.84. The percentage of
illdepelldvilily significant sludics is Virtlially
the sailleas it is ill ille trillimcd sallipic:
15 ofthe 17 studies (88.2%) are significant.
_0ANi 0.01 11.112 0.1111 0.02 UAH 11.06 0.08 0. 10
mi) TheJournal t#'11arapsycholoA'y
9
v
(~o
0
0
C*4
C*4
0
0
0)
00
c6
TABLE 7
IMPACT OF MODERATORS IN COMBINATION
"Optimal" studies "Suboptimal" studies
N Studies 8 9
Combined z 6.14 1.29
Studies with 87.5% 0.0%
< .05
Mean ES .055 .005
SD,, .045 .035
t(1 5) = 2.6 1, p .0 1
r = .559
These results are quite striking and suggest that future studies
combining these moderators should yield especially reliable effects.
SUMMARY AND CONCLUSIONS
Our metaanalysis of forcedchoice precognition experiments
00
0 confirms the existence of a small but highly significant precognition
00 effect. The effect appears to be replicable; significant outcomes are
0
0 reported by 40 investigators using a variety of methodological par
0 adigms and subject populations.
0
C4 The precognition effect is statistically very robust: it remains
(D highly significant despite elimination of' studies with z scores in the
U)
CU upper and lower 10% of the zscore distribution and when a third
of the remaining investigatorsthe major contributors of precog
nition studiesare eliminated.
Estimates ofthe "filedrawer" problem and consideration ofpara
0 psychological publication practices indicate that the precognition ef
LL
feCt Cannot plausibly be cxplained on the basis ofselective publica
(D tion bias. Analyses of precognition effect sizes in relation to eight
>
2 measures of research quality fall to support the hypothesis that the
observed effect is driven to any appreciable extent by methodolog
CL ical flaws; indeed, several analyses indicate that methodologically su
<
perior studies yield stronger effects than methodologically weaker
studies.
Analyses of parapsychological alternatives to precognition, al
though limited to the subset of studies using random number tables,
provide no support for the hypothesis that the effect results froin
A MHWAna~ysis qfhorccdChoice Precog7iition
Exl)eriuietlt~ 301
.he operation of contemporaneous ESP and
PK aL the time of ran
lomization.
Although the overall precognition effect CD
size is small, this does
iot imply that it has no practical consequences.'L
It is, for example,
)f' the same order of magnitude as effect 0
sizes leading to the early
'
ermination of 0
several ma ,)or medical research studies.
In 1981, the
'~atioiial Heart, Lung, and Blood Institute
discontinued its study of to
)ropranolol because the results were so favorable0
to the propranolol
reatinent that it would be unethical to continue0
placebo treatment 04
04
Kolata, 1981); the effect size was 0.04. 0
More recently, The Steering
~ommlttee of the Physicians' Health Study 0
Research Group (1988),
n a widely publicized report, terminated
its study of the effects of C0
ispirin in the prevention of heart attacks 00
for the same reason. The
Ispirin group suffered significantly fewer 0
heart attacks than a pla
:ebo control group; the associated effect 9
size was 0.03.
The most important outcome of' the metaanalysis(0
is the identi C0
ication of several moderating variables that
appear to covary sys
ematically with precognition performance.
The largest effects are
)bserved in studies using subjects selected
on the basis of prior test
'
)erf
Orniance, who are tested individually, and
who receive trialby
rial feedback. The outcomes of studies combining0
these factors con
rast sharply with the null outcomes associated.
with the combination .
00
f group testing, unselected subjects, and 0
no feedback of results. Be
ause the two groups of studies were conducted00
by a subset of the
aine investigators, it is unlikely.that the 0
observed difference in per 0
ormance is due to experimenter effects. Indeed,0
these outcomes
111derscore the importance of carefully examining0
differences in 04
ubject populations, test setting, and so (D
forth, before resorting to
acile "ex lanations" based on psimediated U)
experinienter effects or C9
p
he "elusiveness of psi."
The identification of these moderating variables(D
has important
nplications for our understanding of the
phenomena and pr9vides
clear direction for future research. The LO_
existence of moderating
ariables indicates that the precognition LL
effect is not merely an
nexplained departure from a theoretical chance'0
baseline, but (D
adier is an effect that covaries with factors>
known to influence
iore familiar aspects of human performance. 2
It should now be pos
:ble to exploit these moderating factors CL
to increase tile magnitude CL
nd reliability of precognition effects in <
new studies.
REFERENCES
XERS, C. (1987). Parapsychology is science,
but its findings are inconclusive.
Behavioml .and Brai?i Scie?ices, 10, 566568.
:w2 Thijotmial (~/ Pamp.%y(holiwy
Z~I
RARNI11 1, V., & LFAVIS, T (1978). Outlier~ iii statistired detta. Ncw 1~)ik: John
Wiley & Soils.
I "g, In sclence and engl
C)jWWNl.VF, K. A. (1965). Statistical thron, and melhodolo,
T_ fierilng. New Yoik: john Wiley & Solis.
(:)COHEN, J. (1977). Statistical power analisisjor the behavioral sciences. New York:
Q
Q Academic Press.
D~,,%A,Fs, R. M., LANDMAN, J., & WII,I.IA%lS,,]. (1984). Reply to KUIOSaWa. Amer
to
Q ican Psvchoh~gnst, 39, 7175.
Q
04 HONORTON, C. (1985). Nfetaanalysis of' psi ganzMd research: A response to
04 1 iynianjownal (I Pwapsychob)gy, 49, 51 92.
Q
Q HYMAN, R. (1985). The ganA&I psi experiment: A critical appraisal.fiYurnal
ffl'aral).~Tthology, 49, 350.
Kol.ATA, G. B. (1981). Drug found to help heart attack SLIIVIVOI'S. Science, 214,
00
r 77.1775.
CD NIANG.AN, G. L (1955). Evidciice of, displaccillent in .1 precogilition testjolonal
a
(!J'IlarapsTcholoe), 19, 35:11.
(D MORRIS, R. L 0982). Assessing experimental mipport I'or (rile precogill(LOU.
a
Journal of Parap~ychology, 46, 321336.
RoSENTHm., R. (1984). Xlelaanal.ylic procedures fin social reW17Ch. Beverly Hills,
CA: Sage.
RF.SE.ARCiIGROUP.
(1988). Preliminary report: Findings froin the aspirin component of* the
oligollig Physicians, Health Study. Alew P"ngy1andjournal o
00 J'Medicine, 318,
Q 262264.
00 STERLING, T D. (1959). PUI)IiCaLiOll decisions and their possible efl~cts oil
Q inferences drawn from tests of significanceor vice versa. Journal o/ the
C)
Q Anierican Statistical As.wciation, 54, 3034.
Q WILKINS()N, L. (1988). SYSTAT' 77w.%vNIrm Fx;oislon, IL: SYS~L`ff
04
(1)
U) CHRON01A)GICA1. LISTING OF STUDIES IN NILTAANAtA'SIS
CV
CARINGTON, W (1935). Preliminary experiments in precognitive guessing.jour
nal of the Socielyj6r Psychical Research, 29, 86104.
RiIINF. J. 13. (1938). Experiments hearing oil (lie precognition ll~ poillesis: 1.
0 Preshuffling card calling.journal of FarapsycholojU, 2, 3854.
LL RIJINEj B., B. M., & WOODRUFFJ L. (1938). Experiments bearing
'a oil the precognition hypothesis: 11. The role of' ESP in the shuffling of'
W
> CardsJournal of Parap,~Tch(;IoAry, 2, 119  131.
LO HUMPHREY, 11. M., & PRATY, 'I. G. (1941). A comparison of' five ESP test
CL procedUresJournal of Parap.~yrholo~gy, 5, 267293.
CL RIHNE, J. B. (1941). Experiments bearing upon the precognition hypothesis:
Ill. Mechanically selected cardsJournal of Parapsychology, 5, 157.
STUAWF, C. E. (1941). An analysis to determine a test predictive of'extracliance
scoring in CardCalling LeSLS. Journal of Barapsycholo( gy, 5, 99 137.
HUFMPFIRF~N,, B. Nf., & RIIINE,J. B~ (1942). A C0llfIl`llla(orySLLldy OfSAiellCe ill
piccognition tesisjownal (!f Rmap.~Tcholog'y, 6, 190219.
A IlletelAnalysis oj P'oicedCholce Precognition Expe),1111ents 303
. .1. 11. (19,12). Evidence of, precognition ill ill( covarialioll of' salience
RIIINF,
ratiosJournal. (Y'llarapsychology, 6, 1111413.
Ni(:oi.j. E ,, & CARINGTON, W. (1947). Some experiments in willed diethrow
ing. Proceedings (fthe Socirlyfin Pql'hiiill 1?1,SCaYTh, 48, 164 175.
THOUt,Ess, R. H. (1949). A co'niparative study of' performance in three psi
tasksJournal qf Pei ral).qch ology, 13, 263273.
BASTIN, E. W., & GREEN, J. M. (1953). Some experiments in precognition.
journal (?f Pa.Yaj)sycholoA'Y, 17, 1371,13.
MGMAHAN, E. A., & BAiFs, E. K. (1954). Report offillther Marchesi exper
iniculs. Journal ofilarap.%ychology, 18, 8292.
MANGAN, G. L. (1955). Evidence of displacement in a precognition test.journal
efllarap,~yrhoh~gy, 19, 3514.
Osis, K. (1955). Precognition over time intervals of one to thirtythree days.
.1omnal (!fflarap.~vrhology, 19, 8291.
N I El,SEN, W. (1956). Ali exploratory precognition experinient.journal ofBara
p.~ycholqry, 20, 3339.
NiEllSEN, W (1956). Mental states associated with success in precogn i Lion. jour
nal of Parapsychology, 20, 96109.
FAH LER, J. (1957). ESP card tests with and without hypnosis. Journal of Para
psychologry, 21, 179 185.
MANGAN, G. L. (1957). An ESP experiment with dualaspect targets involving
one trial a dayjournal of Parapsychology, 21, 273283.
acher
ANDERSON, M., & WHITE, R. (1958). A survey of'work on ESP and te
pupil attiLudesJournal of Parapsychology, 22, 246268.
NASH, G. B. (1958). Correlation bet,~%,eeii ESP and religious value. Journal of
Parapsychology, 22, 204209.
ANDERSON, M. (1959). A precognition experiment comparing time intervals of
a 1~%v days and one ycarjournal (Y'Rarapsycholo""y, 23, 8189.
ANDERSON, M., & GREGORY, E. (1959). A twoyear program of tests for clair
voyance and precognition with a class Of' J)Ublic school pupils. Journal of
Parapsychology, 23, 149177.
NASH, C. B. (1960). Can precognition occur diametrically;i journal of Parapsy
chology, 24, 2632.
FREEMAN, J. A. (1962). Ali experiment in precognition.journal ofParapsychology,
26, 123130. 1 .
RiIINE, .1. B. (1962). The precognition of'coniputer numbers in a public test.
Journal of Parapsychology, 26, 244251.
Ryzi., M. (1962). Training the psi faculty by hypnosis. Journal of the Society for
Psychical Research, 41, 234259
SANDERS, M. S. (1962). A comparison of' verbal and written responses in'a
precognition experimenLjournal of Parapsychology, 26, 2334.
FREEMAN, J. (1963). Boygirl differences in a group precognition test. journal
of Parapsychology, 27, 175181.
RAO, K. R. (1963). Studies in the' preferential effect: 11. A language ESP test
involving precognition and "iiiterveiition.",foiiilietlofP(i7(tpsychott~gy, 27,147
160.
C)
v
Q
Q
CD
to
Q
Q
04
04
Q
Q
W
00
I*
Q
C)
01)
a.
0
00
Q
00
Q
C)
C)
C)
04
a)
U)
77D
W
L_
0
LL
'a
(1)
>
0
L_
CL
CL
30.1 TheJournal ol'Par(II)sYchoh~An,
FREEMAN, .1. (1964). A precognition test with a highschool science clul).Journal
of Papal),~y(hol(~gy, 28, 214221.
0 FREENIAN, 'I., & NiFi%FN, W (1964). Precognition score deviations as related
.4 to anxiety levels. journal ol'Iarapsychology, 28, 239249.
0 SCHNIEIDLER, G. (1964). Ali experiment oil precognitive clairvoyance: Part. 1.
0 The main results. Journal of Parapsychology, 28, 114~
0
T FREEMAN, J. A. (1965). Differential response of' the sexes to contrasting ar
rangements of'ESP target material. journal ofFarapsychology, 29, 251258.
0 Osis, K., 8 FAHLER,
j. (1965). Space and time variables in ESP Journal of the
C*4 American Society for Psychical Research, 59, 130145.
C*4
0 FAHLER, J., & Osis, K. (1966). Checking for awareness of'hits in a precognition
experiment with hypnotized subjects. Journal of the American Society for
Psychical Research, 60, 340346.
00 FREEMAN, J. A. (1966). Sex differences and target arrangement: Highschool
I
booklet, tests of' precognition..1ournal of Parapsychology, 30, 227235.
9 Ro(;I,Rs, D. R (1966). Negative and positive affect and ESP rullscore variance.
to irnal of Parapsychology, 30, 151159.
0) Jol
(L Ro(.F.RS, 1). R, & CARPENTEP, .1. C. (1966). The decline of' variance of' ESP
scores within it testing session. journal of'Parapsychology, 30, 141150.
BRIER, B. (1967). A correspondence ESP experiment with lilghI.Q. sub'
< Jmirnal of Peuttp.~yhotogy, 31, 1,13 1,18. .1ects.
U Buziii,, D. E. (1967). Subject attitude and score variance in ESP tests.Journal
of Parapsychology, 31, 4350.
CO BuzBv, D. E. (1967). Precognition and a test of sensory perception.JoUrnal of
0 Parapsychology, 31, 135142.
00
0 FREEMAN, J. A. (1967). Sex differences, LargCL arrangement, and primary men
0 tal abilities. Journal of Parapsychology, 31, 271279.
CD HONORTON, C. (1967). Creativity and precognition scoring levei.journal of
0
C4 Parapsychology, 31, 2942.
(D CARPENTER, J. C. (1968). Two related studies oil mood and precognition run
U)
co score variance. Journal of Parapsychology, 32, 7589.
(1) DuVAL, R, & MONTREDON, E. (1968). ESP experiments with inice.journal of
Parapsychology, 32, 153166.
FEATHER, S. R., & BRIER, R. (1968). The possible effect of the checker in
0 precognition tests. Journal of Parapsychology, 32, 167  175.
LL FREEMAN, J. A. (1968). Sex differences and primary mental abilities in a group
13 precognition teSL. Journal of Parapsychology, 32, 176182.
0 NASH, C. S., & NASH, C. B. (1968). Effect of target selection, field dependence,
> and body concept on ESP performance.Journal of Parapsychology, 32, 248
2
CL 257.
CL RHINk, L. E. (1968). Note on an informal group test of ESP. Journal of Para
psychology, 32, 4753.
RYZL, M. (1968). Precognition scoring and attitude toward ESP.Journal of Para
psycholqgy, 32, 18.
Ryzi., M. (1968). Precognition scoring and aLLitude. journal oJ_ Parapsychology,
32, 183189.
A illrlaAnalj%i~ ofForcrelChoffe, PreceWnition ExIierinientV 305
CARPENTF.R,
.1. C. (1969). Further study oil it mood ad *jective check list and
ESP 11111SCOYC V;l1_ial)Ce. J0111Mil ly 1`o)_tJPs)1ChOlqgY, 33, 4856.
DL1VAL, 1), & MONTREDON, E. (1969). Precognition in mice: A confirmation.
Journal of Parapsychology, 33, 71 72. 0
FREEIVIAN,J. A. (1969). The psidifferential effect in a precognition test.journal L
ofFarapsychology, 33, 206212. C)
0
FREEMAN,J. A. (1969). A precognition experiment with science teachers.Journal 0
T_
of Parapsychology, 33, 307310.
to
JOHNSON, M. (1969). AttiLude and target differences in a group precognition o
test.journal of Parapsychology, 33, 324325. C)
CM
MONTREDON, E., & RoBINSON, A. (1969). Further precognition work with mice. C14
0
Journal of Parapsychology, 33, 162163. 0
ScH m I i)i, H. (1969). Precognition of a quantum process. Journal of Parapsy W
hology, 33, 99108. 0)
C
00
B FIN DER, H. (1970). Differential scoring of an outstanding subject on GESP and
clairvoyance.Journal offlarapsychology, 34, 272273.
FREEMAN,J. A. (1970). Sex differences in ESP response as shown by the Free(6
mall picturefigure Lest.Jaurnal of Parapsychology, 34, 3746. 0)
a.
FREEMAN, J. A. (1970). Tenpage booklet tests with elementaryschool children.c)
Journal qf 34, 192196. W
FREEMAN, J. (1970)~ Shift in scoring direction WithjUlliOrhighSC11001 students:
A SUIlunalY.J91111nal of Rarapsychology, 34, 275.
FREEMAN, J. A. (1970). Mood, personality, and attitude in precognition tests.
Journal of Prapsychology, 34, 322. 00
HARALDSSON, E. (1970). Subject selection in it machine precognition test.journalR
of Parapsychology, 34, 182191. 00
0
HARALUSSON, E. (1970). Precognition of a quantum process: A modified rep6
licationjournal of1larapsychology, 34, 329330. 0
NIEUSEIN, W (1970). Relationships between precognition scoring levelandpN
mood. Journal of Parapsychology, 34, 93  116.
(D
ScHNni),r, H. (1970). Precognition test with a highschool group.jountal ofu)
M
Parapsychology, 34, 70. (D
BELOFF, J., & BATE, D. (1971). An attempt to replicate the Schmidt. findings a)
Journal of the Societyfor Psychical Research, 46, 2131.
HONORTON, C. (1971). Automated forcedchoice precognition tests with a "sen0
sitive." journal of the American Socielyfor Psychical Research, 65, 476481. LL
MITCHFLL, E. D. (1971). An ESP Lest from Apollo 14. Journal of Farapsychologym
35,89107. (D
>
Scimmyr, H., & PANTAS, L. (1971). Psi teSLS with psychologically equivalent 2
conditions and internally different machines. Journal of Parapsychology, 35,CL
326327. . CL
STANFORD,.R. G. (1971). Extrasensory effects upon "memory." journal of the<
A werican Socielyfor Ps iTchical Research, 64, 161186.
STEILBERG, B. J. (1971). Investigation of' the paranormal gifts of the Dutch
sensitive' Lida Tjournal (Y'Parapsychology, 35, 219225.
ThrJoIlllial of Paral).~TcholoAry
3 ~
Tlvwl~I~Ss, R. 11. (1971). Experimeni% oil psi self( raining with Dr. S(iiinidt's
precognitive apparatus. Journal oj'lhe Socielyfor Psychical Research, 46, 15
0 21.
v~ HONORTON, C. (1972). Reported frequency ofdrearn recall and ESI~Jonrnal
0 qf the American SorielT./or Ps.Tchirwl Research, 66, 369374.
0 JOI INSON, M., & NORDBECK, B. (1972). Varia(ioll ill tile Scoring behavior of a
psychic" sul)ject.Journal of Parapsycholilgy, 36, 122132.
0 KELLY, 1". E ', & KAN11 IAMANI, B. K. (1972). A subjeci's efforts toward voluntary
C*4 control. Journal of Parapsychology, 36, 185  197.
04 SU iMiDT, I L, & PAN1 AS, L. (1972). 11si tests with internally difliLrent machines.
0 Journal (Y'Rarapsychohigy, 36, 222232.
0
X CRAIG, .1. G. 0973). 'I'll( effect of colaingemy oil precogililioll ill the raf.
0) Research in llezrap,%ychology 1972, 154156.
00
I FRFF.,,IAN,J. k. (1973).The pS'(JUiz: A new ESP test. Reseench 1`arap.~Tchology
0
0 1972, 132134.
(6 ARTLFY, B. (1974). Confirmation ofthe smallrodent precognit ion work.Jaurnal
0) of Parapsychology, 38, 238239.
IL
a HARRIS, S., & TERRY, J. (1974). Precognition in a waterdeprived Wistar rat.
w Journal of Parapsychology, 38, 239.
RANDALL, J. L. (1974). An extended series of' ESP and PK tests with three
English schoolboys. Journal of the Socielyfor Psychical Research, 47, 485494.
EYsENCK, H.J. (1975). Precognition ill rats.Joarnal of Parapsychology, 39, 222
00 227.
0 HARAimssoN, E. (1975). Reported dreani recall, 1)t,et:ogiiitive(iie;tiiis,;iii(I ESJ~
00 Research in Parapsychology 1974, 4748.
0 HONOPTON, C., RANism,, M., & CABnwo, C. (1975). Experimenter effects in
0 extrasensory perception.journal ofthe Anierican Socielyfor Psychical Research,
0
N 69, 135149.
(1) KANTHAMANI, 11., & IZAO, H. 11. (1975). lZesponse lendencies and stimulus
U) structure. Journal of Parapsychology, 39, 97105.
co
(V LEVIN,.J. 2N. (1975). A sviics ofpsl experinivins wid) geibils..1wirmll
39, 36336.5.
L_ C., & I 1ARRis, S. A. (1975). Pr(cognition ill waierdeprived rals.
0 Research in Parapsychology 1974, 81.
LL DAVI.S,J. W, & HAIGHT,J. (1976). Psi experiments with rats.Jaurnal If Para
psycholl,~,,y, 40, 5,155.
(1) S, J., & BREEDERVE.U), H. (1976). Possible influences ofbirth order oil
> JACOB,
2 ESP ability. Research Letter (Parapsychology Laboratory, University of'
M Utrecia). No. 7. 1020.
Nvvu~i.i.., R. C~ (1976). Someaspecis of' precognition lesillig. Research ill Para
psychology 1975, 2931.
DRuCKER, S. A., DRFwj.s, A. A., & RmuN, L. (1977). FISP ill relation to cognitive
development and IQ ill yowig childrenjournal of lite Anterican Socielyjor
Psychical Research, 71, 289298.
zi MetaAnalTsis offorcedCholce 11recognitiOn Experinients 307
HARALDSSON, E. (1977). ESPand the defense mechanism test (DMT): A further
validation. Europeanjounial of Parapsychology, 2, 104114.
SARGENT, C. L, (1977). An experiment involving a novel precognition task.
Journal of Parapsychology, 41, 275293.
BiERMAN, D. 1 (1978). Testing the "advanced wave" hypothesis: An attempted VL
replication. European finunal ty'Beirapsychology, 2, 206212.
BRAUD, W. (1979). Project Chicken Little: A precognition experiment involving
the SKYLAB space station. Europeaujournal of'Parapsychology, 3, 149165. V
HARALDSSON, E., & JOHNSON, M. (1979). ESP and the defense mechanism test
(DMT) Icelandic study No. Ill: A case of the experimenter effect? European a
C*4
Journal of Parapsychology, 3, If 20. C*4
O'BRIEN,J. T. (1979). Ali examiumion of(he checker efR.c(. Research ill Bala C)
psychology 1978, 153155.
CUMENS, D. B., & PHILLHs, D. T. (1980). Further studies of precognition in 0)
00
nlice. Research in Parapsychology 1979, 156. 1
HARALDSSON, E. (1980). Scoring in a precognition test as a function of the Q
frequency of reading on psychical phenomena and belief in ESP. Research 9
to
Letter (Parapsychology Laboratory, University of Utrecht), No. 10, 18. CY)
SARGENT, C., & HARLEY, T A. (1981). Three studies using a psipredictive Om
trait variable questionnaire. Journal of Parapsychology, 45, 199 214. 0
WINKELMAN, M. (1981). The effect of formal education on extrasensory abil I
ities: The Ozolco, study. Journal of Parapsychology, 45, 321336.
NASI I, C. B. (1982). ESP of present and future Largets.Journal of the Societyfo,
Psychical Research, 51, 374377.
00
THALBOURNE, M., BELOFF, J., & DELANOY, D. (1982). A test for the "extra a
verted sheep versus introverted goats" hypothesis. Research in Parapsychology 00
1981, 155156.
CRANDALL,J. E., & HITE, D. D. (1983). Psimissing and displacement: Evidence a
f6r improperly f6cused psi?Journal ofthe Atnerican Socielyftr Psychical.Re (D
search, 77, 209228. N
(D
S(:i iwAR rz, S. A., R. 1)
J. (1983). The Nlo i s PsiQ (es(: Prefilil U)
inary findings. Research in Parapsychology 1982, 103105.
.101INSON, M., & I IARALDSSON, E. (1981). ThC Defe'llse Mechanism _1~.M ;Is a
predictor ofESP scores: Icelandic studies I.V and Vjournal of1laraps)rhology, X
48, 185200. L_
0
TEDDER, W Coil] pu terbased longdistance ESP: Ali exploratory exLL
animation (RB/PS). Research in Parapsychology 1983, 100 101.
HESELTINE, C. L. (1985). PK success during structured and nonstructured a)
IZNG operation. Journal of Faral)sychology, 49, 155163. 0
HARALDSSON, E &JOHNSON, N4. (Hoiy rhe Defimse Mechanism Test (DMT) "
I CL
as a predictor ol'ESP peribraim ice: Icelandic studies Viand \711. ?esearch CL
I.n Poral)sy~holqgy 1985, 4344. <
VASSY, Z. (1986). Experimental study ofcomplexity dependence in precogni
tion.Journal ~)f Parapsychology, 50, 235270.
0
CL
CL
308 TheJounial of'Parapsychology
HONORTON, C. (1987). Precognition and realtime ESP performance in a com
I)LIter task with an exceptional subject. Journal oJI'arapsychology, 51, 291 
320.
Psychophysical Research Laboratories
P 0. Box 569
P/aInsboro, NJ 08536
0
CL
CL