2-<-V Journal ff Paiapsycholt,~,,y, Vol. 53, December 1989 "FUTURE TELLING": A META-ANALYSIS OF FORCED-CHOICE PRECOGNITION EXPERIMENTS, 1935-1987 BN, CHARLES 110NORTON AND DIANE G. FERRARI ABSTRACT: We report a meta-anaIysis of forced-choice precognition experiments published in the English-language parapsychological literature between 1935 and 1987. These studies involve attempts by subjects to predict the identity of target stimuli selected randomly over intervals ranging from several hundred milli- seconds to one year following the subjects' responses. We retrieved 309 studies reported by 62 investigators. Nearly two million individual trials were contributed by more than 50,000 subjects. Study outcomes are assessed by overall level of sta- tistical significance and effect size. There is a small, but reliable overall effect (z 11.41, P = 6.3 X 10-2S) . Thirty percent of the studies (by 40 investigators) are significant at the 5% significance level. Assessment of vulnerability to selective re- porting indicates that a ratio of 46 unreported studies averaging null results would be required for each reported study in order to reduce the overall result to nonsig- nificance. No systematic relationship was found between study outcomes and eight indices of research quality. Effect size has remained essentially constant over the survey period, whereas research quality has improved substantially. Four moder- ating variables appear to covary significantly with study outcome: Studies using subjects selected on (lie basis of prior testing performance show significantly larger effects than studies using unselected subjects. Subjects tested individually by an experimenter show significantly larger effects than those tested in groups. Studies in which subjects are given trial-by-trial or run-score feedback have significantly larger effects than those with delayed or no subject feedback. Studies with brief intervals between subjects' responses and target generation show significantly stronger effects than studies involving longer intervals. The combined impact of these moderating variables appears to be very strong. Independently significant outcomes are observed in seven of the eight studies using selected subjects, who were tested individually and received trial-by-trial feedback. Precognition refers to the noninferential prediction of future events. Anecdotal clainis of' "future telling" have occurred through- > out human history in virtually every culture and period. Today such 0 L_ CL This work was funded by SRI International and the John E. Fetzer Foundation. CL We wish to thank our PRL colleague George P. Hansen, who is primarily responsible for retrieving die studies used in ihe rucia-analysis. Weare grateful to Edwin C. May, Jcssica I-Jus, aud to live anonymous reviewcrs al SRI lor valuable Comments ()It .111 earlier draft ofthis report. Valuable comments were also made by Ephrairn Schechter and by three anonylUOUS referees. The division of authorship responsibility is as fol- lows: Honorton is responsible for the design of the meta-analysis, definition of study codini-, criteria. the actual analyses, and the report itself. F rr 6 r iflorl thp ;nr1;v;i1tin1 research reports in consukation with lionorton and/or Hansen. 00 Q 00 Q Q Q Q C*4 (1) U) LL > 0 CL CL 282 The I Jonriml ofl'(iral).~Iwhology A J111-tel-AnalYsIS of 1,orced-Cho'ce Prrcownition Exl)rrimrnts 283 claillis arc generally 1)(-Ii(-V(-(l lo bc based ()It f'.'Iclors Stich as delusion, i&LlOllality, MId SUI)CPStitiolLIS thinking. The concept of pi-ccogni- t4IIi runs counter to accepted notions of' causality and appears to (1,flict with current scientific theory. Nevertheless, over the past I f'-century a substantial number of experiments have been re- l1r]-ted Claiming empirical support. f'Or the hypothesis of' precogni- U7 t0hi. Subjects in FOrced-choice experiments, according to many re- 0- I" ts, have correctly predicted to a statistically significant degree tile If9mity (or order) of' target StIInUlI randomly selected at it later lahe. K_%7e perf'ormed it nieta-analysis of' FOrced-choice precognition ex- 0). fool-lineilLS I)LIbliShed Ili the English-language research literature be ~;een 1935 and 1987. Four Illa*01- questions were addressed J t9-ough this nieta-analysis: (1) Is there overall evidence f'Or iICCUrate fget Identification (above-chance hitting) in experimental precog- dkion studies~ (2) What is the magnitude of the overall precognition lect? (3) Is the observed effect related to variations in methodo- "i'al (Illailty (hat could allow .1 Illore conventional cxplaflafloll~ (1) .s )c precognition perfornia rice vary systematically with potential Q)d(-Y;IIiIIg vallables, .tl(ll as dif'JCJ-(-lI((.s III Sill)*(-(( popillolloll", i s1rilitilus conditions, experimental seiting, knowledge of' results, and 00 - tole interval between sill)* Ind target generation? ject response 00 0 0 DELINEATING 'riiE DOMAIN U) Parapsychological research is still academically taboo, and it Is C9 LGhlikely that there have been many dissertations and theses Ili this that have escaped publication. Out- retrieval of'studies f'Or this Teta-analysis is therefore based on the published literature. The q0jdles include all forced-choice precognition experiments appear- In the peer-reviewed English-language para psychology jou rna Is: Irnal of Parapsychology, Journal (and Proceeding~s) of lite Soclely fiff P~I~lchical Research, Ic a 6 Journal of the Ainerican Soc'ely fior Psych * al Rest, rch, kLropean Journal of ParapsycholoAy (Including lite Research Letter of' & Utrecht University Parapsychology Laboratory), and abstracts of' K-el-rcviewed papCI-S presellicd at J'arapsychological Assoclaiion meetings published Ili Research in Paraps~ychology. Ciitei"a -7 fior Inclusloll Our 1-(.Vlcw Is re.SII,IcWd lo fixed-lellgill Studies III which sigillh- cance levels and ef'f'eci sizes kised on (fireci liffling can be calcil- 1,11cd. Similes lising olitcollic variables other than direct liittiiig, such its I'Llll-SCOI-e VarianCe and displacement ef'Fects, are Included Only it' tile report provides relevant information on direct hits (i.e., numgr of' trials, hits, and probability of a hit). Finally, we exclude stu'r'-s conducted by two investigators, S. G. Soal and Walter J. Levy, wh~e work has been unreliable. Many published reports contain more than.one experimen tar experimental unit. In experiments involving multiple conclitiQ-1 significance levels and effect sizes are calculated for each conditi 0 0 Outcome Measures 00 S2igwificance level. Significance levels (z scores) were calculatelpr each study from the reported number of trials, hits, and probabit,ky of' success using the normal approximation to the binomial disi- bution with continuity correction. Positive z scores indicate abq[Le- chance scoring, and negative z scores reflect below-chance scori eCt Size. IkCaUse Most parapsychological CXPCriIIICIItS, EY c- . J, Pat Illarly those in tile older literature. have used the trial rather An the sub'eci. as the sampling unit, we use it Lrial-based eSdillatooit' 9 1 effect size. The effect size (ES) for each study is the z score divi,14d kb by the square root of' the number of' trials in the study.' 0 00 General Characteristics of 1he Domain We located 309 studies in 113 separate publications. These SRI- tes were contributed by 62 different senior authors and were peb- lished over a 53-year period, between 1935 and 1987. Considellig tile half-centLIFY turie-span over which the precognition experim(Alts were conducted, it is not surprising that the studies are verydiv e. t The database comprises nearly two million individual trials Lnd, more than 50,000 subjects. Study sample sizes range-from 2_10to 297,060 trials (median = 1,194). The number of subjects ranles from I to 29,706 (median = 16). The studies use a variety Of In I- odologies, ranging from guessing ESP cards and other card s ;ynIi&is to automated random number generator experiments. The clontril encompasses diverse stll)*ect populations: the most frequently i=d . < ' Elsewhere (Honorton, 1985), we have used tile effect size index Cohen's h (Cohen, 1977), and one referee has asked that we explain why we are now using z/All". The answer is that h and zfN"2 yield virtually identical results, and ZIN` is, computationally simpler. For tile present sample of 309 precognition studies, the nican difference boween the two indices is .00047, and the standard devi;ition of tile differencc is .0126: 1(308) = 0.312, p = .756, two-tailed.The cori-clafion between the two indices is .97. v-- 0 04 04 CD CD W 0) 00 CD CD c6 (D Q 00 0 00 0 0 CM W U) C9 0 LL -0 (1) > 2 CL CL < 28,1 Thrjomnal (?/ Paiap~ychology TABLL I OVERALL SIGNIFICANcE LEvEl. AND EFFECT SIZE Z ES Mean 0.65 0.020 2.68 0.100 SD Lower 95% confidence estimate 0.40 0.011 10 -25 Combined z 11.4 1, p 6.3 x "Fail-safe N" = 14,268 i(ES) = 3.51, 308 df, p = .00025 population is students (in approximately 40% of the studies); the least frequently used populations are the experimenters themselves and animals (each used in about 5% of the studies). Though a Few Studies tested subjects through the mail, more typ- ically subjects were tested in person, either individually or in groups. Target selection methods included no randomization at all (studies using "quasi-random" naturalistic events), informal methods includ- ing manual card-shuffling or dice-throwing, and fornial methods, primarily random number tables or random number generators. The time interval between the subjects' responses and target gen- eration varied from less than one second to one year. OVERALL CUMULATION Evidence for an overall effect is strong. As shown in the top part of Table 1, the overall results are highly significant.' Lower bound (one-tailed) 95% confidence estimates of the mean z score and ES are displayed in the bottom portion of Table 1. Ninety-two studies (30%) show significant hitting at the 5% level, and significant outcomes are contributed by 40 different investiga- tors. The z scores correlate significantly with sample size: r(307) .156, p ~ .003. The mean number oftrials for significant studies -is 34% larger than the mean number of trials for nonsignificant stud- - ies. 'The statistical analyses presented here were performed using SYSTAT (Wilk- inson, 1988). When I tests are reported on samples with unequal variances, they are (Aculaird usilig 111C sepalair varialitt-N wilhis) gloups For 111V cri'm and (Irgiers of freedom following Brownlee (1965). Unless otherwise specified, p levels are one- tailed. Combined z's are based on Stollffer's method (Rosenthal, 1984). 285 %0 0 :0 :,9 0 ~0 1 .0 : 0 0 Q) :0 0 8. 0 0 : 0 0 00 0 0. 1 C')O 0 0 0 CM 04 0 0 00 0 a 0 0 0 -0.2 -0.1 - 0.0 0.1 U 0.3 0.4 0.5 00 0 Mean effect size 00 0 l,'igtit-e 1. rylean effect size by investigator. N = (12 investigators. 0 0 ReplIcallon Across lnvestil~afors CM 0) U) Virtually the same picture emerges when the cumulation is 1# investigator rather than study as the unit of'analysis; the combin-2t __6 z is 12.13, and 23 of the 62 investigators (37%) have overall 01W comes significant at the 5% leveL The mean, (investigator)'effect siAQ 0 is 0.033 (SD = .093). LL There is a significant difference in the mean ES across invesi~ gators, but it is surprisingly sinall: KrUskal-Wallis one-way ANOV by ranks, X'-(6 1) = 82.7 1, p = .034. The effect is clearly not due E? a few major contributors. If investigators contributing more thZW three studies are eliminated, leaving 33 investigators, the combin z is still 6.00 (1) = 1.25 X 10") and the mean ES is .028 (SD .0m). Figure I shows the mean effect sizes by investigator. liese results indicate substantial cross-investigator replicability and directly contradict the claim of critics such as Akers (1987) that 286 Dir'lon)-tial o1'Paraps),rhf)1oAy successfill parapsychological oulconics arc achleved by only a I'CW investigators. CD I Zbe P'iledravler Probleiii CD CD A well-known reporting blas exists throughout tile behavioral to .45ences favoring publication of "significant" studies (e.g., Sterling, a59). The extreme view of this "filedrawer problem" is that "the are filled with the 5% of tile studies that show Type I er =-s , while the filedrawers back at tile Iab are I'lled with the 95% of' Studies that show nonsignificance. . . " (Ros tinthal, 1984, p. 108). )*COglllZillg tile importance of' this pi I -oblem, the Parapsychological tsociation in 1975 adopted an official policy against selective 1'e rting of'positive results.' Examination of the parapsychological lit ture shows that nonsignificant results are frequently published, ILd, in the precognition database, 70% OF the studies have reported Rnsignificam restilis. Nevei-ificless, 7r)(;',, ofilic precogn,11011 st(OR's e published before 1975, and we must ask to what extent selec- atl-., publication bias could account fol- tile Cumulative effects we ob- Qll_"~ -. . . .- - . - . - __ - - CO 'I'lic cenfral, sectioll of''Fabic I 11scs Rosenihal's (198,0 "I'lail-sa[C !R' Statistic to estimate the nu1llbCl_ Of' unreported Studies with Z 9)1,es averaging zero that would be necessary to reduce the known aitabase to nonsign if icai ice. The filedrawer estimate indicates that 8er '16 U111-CpOrIC(l Studies IDUA exist for each reporied stialy (o i ificant level. MdUce tile C11111111itti IVC Outcome to a nonsigin I a) A different approach to the filedrawer problem is described by ratwes, Landman, and Williams (1984; personal C011111111111CatiOn .&)in Dawes to Honorton, 'Juiv 14, 1988). Their truncated nornial (1) 1 Wrve analysis, like Rosenthal s "fall-safe N," is based oil nornial ~LUFVC assulliPtiOlls. Their null hypothesis is that z scores above some 12itical level (e.g., z = 1.65, 1.96, etc.) are randomly sampled from AUO, 1) above that critical level. The alternative to the null hypothes' U is it) that, because there is some real effect, the distribution of z's is .gifted to the right of 10 and the z's will be larger than predicted by te null. For a critical level of'z = 1.65, the expected mean z is 2.06 Ad tile variance is .14. In (lie precognition database, there are 92 sLlies xvith z's > 1.65. Their average is 3_61, not 2.06 its predicted Ailal~-ses_ iiidicate no significaiii difference in the 111agnitude 4reported stmlv _11C IM , V oIII( ollics b(lorc and afic' 197-5. .1 '011 FS to, %todirs III iot to 1975 k 0,0.~ I (~ ) .0,M), and hn studics ICpoItCd 111creatter the mean is 0.017 (SD = .106): f(307) 0.28, p = .782, two-tailed. orced-CholCe Preco "I*oii E Periiiients 287 A Aleta-Aiiahsis ofF gm I by tile null hypothesis. Since file variance of' the nornial truncated above 1.65 is .14, the test z (using the Central Limit Theorem) com- paring 3.61 to 2.06 is 39.84 [1.55 divided by (.14/92)"']. Here, p is virtually zero. Similar results are found with CUL points of 1.96, 2.33, rl and 2.58. CD Q On the basis of these analyses, we conclude that the cumulative a significance of the precognition Studies cannot satisfactorily be ex-,r' (D plained by selective reporting. C) CD C*4 C*4 CD Oun.u-,Az Ri.,i)uunON CD 00 Although the overall z scores and effect sizes cannot reasonablyr- . . . CD be attributed to chance, inspection of the standard deviations in(:) I Table I indicates that the study outcomes are extremely heteroge-to neous. Given tile diversity of methods, subject populations, and4m Othel- study features that CIMI-aCtel-MC this I-esearcli domain, tills iso not surprising. The study outcomes are in fact extremely heterogeni~ous. Al--~ though a major objective of this meta-analysis is to account for theo Variability aCrOSS St tidies by blocking oil differences in study quality, 00 procedural features, and sampling characteristics, the databasec) clearly contains extreme outliers. The z scores range from -5.1 toco . C) 19.6, a 25-sigma spread! The standardized index of kurtosis (,-2) 1S8 9.47, Suggesting that the tails of' the distribution are much too lon6SD C) for a normal distribution. C*4 We eliminated the extreme outliers by performing it "10 percenta) trim" on the study z scores (Barnett & Lewis, 19 ,78). This involvesu) M eliminating studies with z scores in the tipper and lower 10% ofthe-(I) distribution, and results in an adjusted sample of' 248 Studies. T 142 trimmed z scores range from - 2.24 to 3.21 .(g, 1. 1). -The re-" vised z scores and effect sizes are presented in Table 2. 0 LL Elimination of extreme outliers reduces the combined z scores b YV Iv one half, but the outcomes remain highly significantV approximate, Twenty-five percent of the studies (62/248) show overall significanto hitting at the 5% level. Lower bound confidence estimates show thab"L the mean z's and effect sizes are above 0 at the 95% confidence level CL Elimination OfOULljel-S reduces the total IlUmber ofinvestigatol's fi-oin 62 to 57, but the' results remain basically the same when the Mlitl)'SCS M-C IMSC(l Oil inVCN1i9jJ10YS ratlIff thall studies. ']'lie COID- bined z is 6.84; IS of the 57 investigators (31.6%) have overall.sig- .288 TheJournal (?J'Parapsychology A Mela-AnalTsis ofForced-Choice Precog-nition Eicperiinent,~ 289 TABLE 2 C) SIGNIFICANCE LEVII. AND EFFECT SIZE FOR TRINUMEI) SAMpI.I,_ Q ES Q %an 0.38 0.012 1.45 0.065 Ower 95% confidence estimate 0.23 0.005 C*4 C*4 Combined z 6.02, p 1. 1 x 10 CD CD I(ES) = 2-90, 247 dj-, p = .002 fficant outcomes at the 5% level. The mean (invest' igator) ES is Q-;020 (SD = .05). 9For the trimmed sample, the difference in ES across investiga- -s is not significant: Kruskal -Wallis one-way ANOVA by ranks, 56) = 59.34, p = .355. If investigators contri 1 more than ee studies are eliminated, leaving 37 investigators, the combined 10-7 z a ,,s still 5.00 (p = 3.0 X and the mean ES 0.022 (SD M6). Figure 2 shows the inean effccL size by Investigator. Thus, elimination of' the outliers does not substantially affect the c lusions drawn from our analysis of the database as a whole. -Yeile clearly is a nonchance effect. In the remainder of this report, 40 use the trimmed sample to examine covariations in effect size CD T51 a variety of methodological and other study features. CD C) C*4 STUDY QUALITY (D U) cu Because target stimull in precognition experiments are selected uSy after the subjects' responses have been registered, precognition (D s&dIes are usually not vulnerable to sensory leakage problems. (ILher potential threats to validity must, how6er, be considered. -Toe problem of variations in research quality remains a source of G&troversy in meta-analysis. Some meta-analysts advocate eliminat- low quality studies whereas others recommend empirically ac- c 111sing the impact of variations in quality on study outcome. Rosen- I t] .(1984) points out that the practice of' discarding studies is eSivalent to assigning them weights of zero, and he recommends 5 1 ~NNghLing study z scores in relation to ratings of research quality. Study Quality Criteria Ideally, the assessment of study quality should be performed by knowledgeable specialists who are blind to the study outcomes. III 6 0 :00 00. :0 0 0 0 0 L_ d 0. 0 6 0, 0 0 0 0 0 0 0 00 0 01 .00 :0 00 10 0 0 0 0 0 1 0 0 0 0 00 -0.20 -0.15 -0.10 -0.05 0.00 0.05 0.10 0.15 0.20 CD 00 Mean effect size CD CD Figure 2. Mean effect size by investigator for trimmed sample. N 57 ig vestigators. C*4 practice, this is usually not feasible, particularly When, as in the preM ent case, large numbers of studies are involved. For our analysis at study quality, statistical and methodological variables are definez and coded in terms of procedural descriptions (or their absence) 1% the research reports. This approach-was used in an earlier metw"a analysis of psi ganzfeld research (Honorton, 1985), and it led W study quality ratings that were generally in agreement, r(26) = .76~2 p = 10", with independent "flaw" ratings by an outside critic (H 0 L_ man, 1985). One point is given (or Withheld) for each of the following eigla criteria: specification of sample size. Does the investigator preplan the num- ber of trials to be included in the study or is the study vulnerable to the possibility of optional stopping? Credit is given to reports that explicitly's pecify the sample size. Studies involving group testing, in which it is not feasible to specify the sample size precisely, are also 290 TheJournal of Parapsychology A Meta-Analysis of Forced-Choice PrecQgnition Experinients 291 given credit. No credit is given to studies in which tile sample size Is elther not preplarined or not addressed in tile experimental re 9,)011. V_ CD Prej)lanned analYsis. Is the method of statistical analysis, Including the outcome (dependent variable) measure, preplan ned;) Credit is CD . to given to studies explicitly specifying the form of' analysis and the a outcome measure. No credit is given to those not explicitly stating C) the form of the analysis or those in which the analysis is clearly post *4 C C*4 lloc. C) C) Randoinization inelhod. Credit is given fOr use of' randorn number Lables, random number generators, and mechanical shufflers. No 00 credit is given for failure to randomize (i.e., use of "quasi-random 1 c) naturalistic events") or for Illf'OrIllill InCtIlOdS Such its hilild-ShUf"lling, 9 die-casting, and drawing lots. to ConlroLs. Credit is given to studies reporting randomness control (L checks, such as random number generator (RNG) control series and In . ' W empirical cross-check controls. Recording. One l)OIIIL Is allotted For automated recording Of' Lar- gets and responses, and another for duplicate recording. Checking. One Point is ~Illotted for automated checking of' 00 matches between target and response, and another for duplicate C) checking of' lilts. 00 CD CD CD Study Quality Analysis C*4 (D Each Study received it (ILMlity weight between 0 and 8 (inean U) 3.3, SD = 1.8). We find no significant relationship between study C9 (1) quality and E'S: r(2,16) = .081, p = .202, two-tailed. This tclidency (1) [or Study Outcomes to correlate posilively with study quality has tile consequence that the quality-welghted z score of' 6.26 is slightly 0 larger than the Univeighted z of"6.02. Table 3 shows the correlations LL. between effect size and each of the eight individual quality meas- .1 tires . The mean effect sizes by quality level are displayed graphl- > 0 cally in Figure 3. L_ CL The correlation between ES and study quality is also nonsignificant for the un- CL ~ trimmed sample of 309 studies: r(307) = -,060, p = .289. The qUality-WCigl1tCd Z score is 7.38: p = 2.32 x 10". However, thFCC Of1hC indiVidLU11 (111;llily meaSures are significantly related to pcHormanCC. COnLrolS and duplicate cliccking correLuc significantly positively with ES, and randomization correlates significantly negatively with ES. these correlations appear to be due to a few studies witli z scores that are extreme outliers (z > 7). When the 10 studies witli z > 7 are eliminated, the signifi- cant correlations between quality and ES disappear. rABLE 3 CORREIA FIONS BEITNVI-A-IN Ei--i,-i;(; I- SIZE AND QUALI I'Y MEASURES Q I -e r(246) V_ QUaliLy ineasui C) CD Saniple size specified in advance .100 CD Preplanned analysis .001 V_ to RandOMIZaLIon .011 C) CD Controls .058 C*4 ALILOinated recording .169 C*4 C) DUpIiCaLe recording .047 C) Autoniated checking .136 Duplicate checking .078 00 I- CD Quality Extre7nes CD I Is there a tendency for extremely weak studies to show larger [L effects than exceptionally "good" studies? Analysis on the extremes 0 of" tile quality ratings indicates that this is not the case. This analysis, based on the untrimmed sample of 309 studies, uses Studies with quality ratings outside tile iiiLerquartile range of6 the rating distribution (median = 4, Q, = 2, Q, = 5). There are 56 ' ' s (ratings of 0-1) and 35 "high-quality" . 00 "low-quality" studie I studies CD (ratings of 6-8). The high-quality studies have effect sizes that are 00 CD not significantly lower than the low-quality studies; the ES means CD are 0.017 (SD = 0.063) and 0.037 (SD 0.137), for the low- and.(D C) high-qUality studies, respectively: t(82) -.92, p = .358, two-C*4 talled U) M Quality Vari(zhon in Publication Sourctu precognition ES is not significantly related to source of' publica-W tion: Kruskal-Wallis one-way ANOVA, X'(4). = 0.78, p = .942. 0 However, the sources of publication differ significantly in studyLL X2 quality: Kruskal-Wallis one-way ANOVA, (4) = 17. 19, p = .002. This is due largely to the lower quality of studies published in the > Journal of the Society for Psychical Research and in Research in Parapsy- cholo CL gy. CL Study Quality in Relation to Year of Publication Precognition effect size has remained constant over a half-cen- tury of research, even though the methodological quality of the.re- 292 71e.fournal qf Paraf)sychology 8 - 7 - 13 6 - 7 5 - 45 4 - 63 3 - 35 9 - 41 31 8 00 -0.10 -0.05 0-00 0.05 0_10 CD 00 CD Mean effect size CD Fig2e 3. Precognition effect size in relation to study quality, with 95% con- fid "-'e limits. N = 248 studies. seatah has improved significantly during this period. The correla- tior&etween ES and year of publication is -.071: t(307) 1.25, p =Z213, two-tailed. Study quality and year of publication are, how- ev9, positively and significantly correlated: r(246) = .282, p = 2 x 10t two-tailed. L6ritics of parapsychology have long believed that evidence for '&sychological effects disappears as the methodological rigor in ar p cre2mes. The precognition database does not Support this belief'. 0 CL CL < "REAL-TIME" ALTERNATIVES TO PRECOGNITION Investigators have long been aware of the possibility that precog- nitioll effeCLS could be modeled without assuming either Lillie I-eVcl-- sal or backward causality. For example, outcomes from studies with A Alela-Anal.ysis (?f Forred-ChoWe Precognition Experiynent,~ 293 Lai-gets based on indeterminate randorn number generators (RNGs) could be due to a causal influence on the RNG-a psychokinetic (PK) effcct-radier than information acquisition concerning its f'u- ture state. In experiments with targets based on prepared tables of Fandoin numbers, the possibility exists that the experimenter or Offier randornizer may be the actual psi source, unconsciously using "real-time" ESP combined with PK to choose an entry point in the random number sequence that will significantly match the "sub- ect's" responses. While the latter possibility may seem far-fetched, it cannot be logically eliminated if one accepts the existing evidence for contemporaneous ESP and PK, and it has been argued that it is less far-fetched than the alternative of "true" precognition. Morris (1982) discusses models of experimental precognition based on "real-time" psi alternatives and methods for testing "true" precognition. in general terms, these methods constrain the selec Lion of the target sequence so as to eliminate nonprecognitive psi intervention. In the most common procedure, attributed to Mangan -ate a set of numbers t (1955), (lice are thrown to genet hat are math- ematically manipulated to obtain an entry point in the random num- ber table. This procedure is sufficiently complex "as to be appar- ently beyond the capacities of the human brain, thus ruling out PK because the TKer' would not know what to do even via ESP" (Mor- ris, 1982, p. 329). I'wo features of precognition study target determination proce- dures were coded to assess "real-time" psi alternatives to precogni- tion: method of determining random number table entry point and use of Mangan's method. Methods of eliminating "real-time" psi alternatives have not been used in studies with random number generators and have only been used in a small number of studies involving randomization by hand- shuffling. These analyses are therefore restricted to studies using random number tables (N = 138). Method qf Determining RNT Entry Point The reports describe six different methods of obtaining entry. points in random number tables. If the study outcomes were due to subjects' precognitive functioning rather than to alternative psi modes on the part of the experimenter or the experimenter's as- sistants, there should be no difference in mean effect size across the various methods used to determine the entry point. Indeed, our analysis indicates that the study effect sizes do not vary systernAi- CD Q CD CD T" CD CD C-4 C*4 CD CD 00 CD CD I a) IL co CD 00 CD CD CD CD C*4 (D U) ca (D 0 LL 'a > 0 CL CL CD I ' CD CD CD V_ to CD CD C*4 C*4 CD CD 00 I- Q C) (D CL 0 00 CD 00 CD CD C) C*4 (D U) M 75 W 0 LL 0 " CL CL 294 Vir.lournal of Parapsi,rhologn~ cally as, a filliclion of' method of' deferillining Ihe entry polill: Krus- kal-Wallis one-way ANOVA by ranks: X~'(5) = 7.32, p = .198. Use (?f Mangan'S Method We find no significant difference in ES between studies using complex calculations of' the type introduced by Mangan to fix the random number table entry point and those that do not use Such calculations: 1(45) = 0.38, p = .370, two-tailed. MODERATING VARIABLES The stability of' precognition study outcomes over a 50-year pe- riod, which we described earlier, also bit([ news. It shows that vestigators in this area have yet to develop sufficlent I-Indel-SLailding of' the conditions underlying the occurrence (or detection) of" diese ef'feCLS to reliably HlCre~ISC LIICIr IMIgniLLIde. We have identified Four variables that appear to covary systematically with precognition ES: (1) selected versus unselected subjects, (2) individual versus group testing, (3) f'eedback level, and (4) time interval between subject re- sponse and target generation. The analyses us& the raw study z scores and effect sizes; we found I-Ilitt this results in Unif'Ornily more conservative eSLiIIIaLCS of' relationships with moderating variables than when the analyses are Z3 IXISed Oil (IWIlity-welghted z scores and ef'f'ect sizes. Selected Versus Unselected Subiects Our ineta-analysis identifies eight sub*ect populations: unspecl- 9 fied subJect popt I la (ions, numures of' several dif'Ferent populations, MI 11MIS, StUdellUS, Children, "VOILIMCCI-S," CXI)Cl'lll]ClltCl-(S), 2111d Se- lected subjects. EYFect. size magnitude does not vary significantly across these X2(1) eight SLI[JeCt pOpUlalions; KrUskal-Wallis one-way ANOVA, i 10.90, P = .143. Eff'ect sizes by subject population are displayed in Figure 4. However, studies using subjects selected oil the basis of prior performance in experiments or Pilot tests show significantly larger effects than studies using unselected Subjects. As shown in Table 4, 60% of' the studies with selected subjects are significant at the 5% '1 level. The mean z score f'or these Studies is 1.39 (SD = 1.40). J'he ES IS SignifiGinfly highCr f'Or SCICCted-SUbjCCIS Studies thall fiff Stud- 4 'Wela-,411all'sIN of Forced-Cholre Precognition 295 CD Selected - 25 V_ CD Exptr - 12 CD CD V_ VOILIIlteel-- to 26 CD CD C*4 Children - 31 C*4 CD 0 CD ~n_ sludclits108 00 ~F Animals 10 Cf) C) C) 28 mixed Unspec 18 - 0.06 0.04 0.02 0.00 0.02 0.04 0.06 0.08 0~10 Mean effect 00 size CD 00 Figure .1. (2 Precognition effect size by subject population, with 95% confi- dence limits. C) N = 248 studies. C) CD ies with C*4 Unselected Subjects. The t test of'the difference ill niean ES , (D is equivalent to a point-biserial correlation of' .198. U) Does this M difference result f-rom less SLI-Ingem controls in studies with selected 7a-) subjects~ The answer appears to be "No." The average CILI~Illty W of' studies with selected Subjects Is higher than studies us'ing 0 TABi.i.,. LL 4 SELECYED VERSUS UNSELE.CFED SUBJECI'S > 0 Selected L_ Unselected CL N studies CL 25 223 Combined z 6.89 4.04 Studies with p. < .05 60% 21% Mean ES .051 .008 .075 .063 t(246) 3. 16, p .00 1 296 'The Jourrial of Parapsychology Q TABLE 5 I INDIVIDUAL VERSUS GROUP TESTING Q Q Q Individual Group Ir- CD N studies 97 105 Q Q Combined Z 6.64 1.29 04 Studies with P < .05 30% 19% N (:) Mean ES .021 .004 Q W SD,.s .060 .066 CY) 00 t (200) 1.89, p .03 9 unselected subjects: 1(27) = 1.51, p = .142, two-tailed. This result to CD appears to reflect a general tendency toward increased rigor and 0- more detailed reporting in studies with selected subjects. 0 W 1 Individual Versus Group Testing < Subjects were tested in groups, individually, Or through the mail. co Studies in which subjects were tested individually by an experimen- 52 ter have a significantly larger mean ES than studies involving group 00 testing (Table 5). Q Q The t test of' the difference is equivalent to ;I poinL-biserial Cor- Q relation of .132, favoring individual testing. Of the studies with sub- Q 04 jects tested individually, 30% are significant at the 5% level. a) The methodological quality of studies with subjects tested indi- U) M vidually is significantly higher than that of studies involving group testing: 1(137) = 3.08, p = .003, two-tailed. This result is consistent W with the conjecture that group experiments are frequently con " ducted as "targets of opportunity" and may often be carried out 0 hastily in an afternoon without the preparation and planning that LL go into a study with individual subjects that may be conducted over 'a (D a period of weeks or months. > Thirty-five studies were conducted through the mail. In these 0 L_ studies, subjects completed the task at their leisure and mailed their CL a responses to the investigator. These correspondence studies yield outcomes similar to those involving individual testing. The com- bined z score is 2.66, with a mean ES of 0.018 (SD ~ .082). Ten correspondence studies (25.7%) are significant at the 5% level. , Eleven studies are unclassifiable with regard to experimental set- ting. A Meta-Analysis of Forced-Choice Precogm 'it 'on Experiments 297 T,kBLI... 6 FEEDBACK RECEIVED BY SUBJECTS Q Feedback of Results Q None Delayed Run score Trial-by-trial N swdles 15 21 21 47 Combined z -1.30 2.11 4.74 6.98 Q Q Studies with p < .05 0.0% 19.0% 33.3% 42.6% 04 04 Mean ES -.001 .009 .023 .035 Q SD, .028 .036 .048 .072 Q 00 Feedback r1l.- Q Q I A significant positive relationship exists CD between the degree of a) feedback subjects receive about their performance[L and precognitive effect size (Table 6). Subject feedback information is available for 104 studies. These ' Studies fall into four feedback categories: no feedback, delayed feedback (usually notification by mail), run-score feedback, and . trial-by-trial feedback. We gave these categories numerical values 00 between 0 and 3. Precognition effect size (Q) correlates .231 with feed- back level (102 df, p = .009). Of the 47 00 studies involving trial-by- Q trial feedback, 20 (42.6%) are significant Q at the 5% level. None of the studies without subject feedback are Q significant. Q Feedback level correlates positively though 04 not significantly with research quality: r(102) = .173, p = .082, (D two-tailed. Inadequate randomization is the most plausible source U) of pote ntial artifacts in M studies with trial-by-trial feedback. We performed a separate analy- sis on the 47 studies in this group. Studies using formal methods of randomization do not differ significantly in mean ES froin'those with informal randomization: t(15) = 0.67, 0 p = .590, two-tailed. LL Similarly, studies reporting randomness control data do not differ significantly in ES from those not includinga) randoniness controls: > t(42) = 0.79, p = .436, two-tailed. 0 L_ CL Time Interval CL < The interval between the Isubject's response and target selection ranges from less than one second to one year. Information about the time interval is available for 144 studies. This information, h6w- 00 C) 00 CD CD Q Q C*4 U) C9 (1) - 0 LL ,Cl (1) > 0 L_ CL a 2~i8 Mollills Weeks 7z :1 Days ;:Z Hours C MillUtCS Seconds I%Iilllsc(: 1jujournal ol Parap,~I,chology Me~ill el'FeCt. SiZe Figure 5. Eflcct size by precognitioii intcr%,al, i%,itli 95X confidence limits N = 144 Studies. - analvsis of' the relationship between ever, is often imprecise. Out precognitive ES and time interval is therefore limited to seven broad interval categories: milliseconds, seconds, Minutes, hours, days, weeks, and months. (Effect sizes by precognition interval are dis- played in Figure 5.) Although it IS C0l1f'0LllIdCd with degree of' Feedback, there is it ificant. decline III precognition E'S over :Increasing teni )oral (its sign I tance: r(142) = -.199, p = .017, Lwo-talled. The largest effects oc- cur over the millisecond interval: N = 31 studies, combined z 6.03, mean ES = 0.045, SD = .073. The smallest effects occur over periods ranging from it nionth to a year: N = 7, combined z 0.53, nican ES = 0.001, SD ~ .0,19. Interestingly, the decline of' precognition pe r F01-111 a lice Ovel- in- creasing teniporill dISIallCeS results entirely I'l-0111 Studies LISIng, 1111- -1 J11ria-blellYp'S I,)-ec(Wtiitioii ExI)rrintrilts 299 selccicd sul)'ects: )-(122) = -.235, p = .009, two-tailed. Studies with selected SLIlJeCLS Show a nonsignificant Positive relationship between ES and time Interval: r(18) = .077, p = .745,(D two-tailed. Although I the difference between these two correlationsT.- IS not Significant (Z = C) 1.24), this suggests that the origin of the C) decline over time may be nionvatlonal rather than the result of some C) intrinsic physical bound- , ary condition. The relationship between precognitionCTD ES and feed- C) back also supports this cot 'ectUre. Nevertheless,C) any finding SLIg- 11 10 1 gesting potential boundary Conditions Oil C*4 the phenomenon S I Li d C*4 be vigorously pursued. Q Q Influence qf Moderating Variables in Combinatio7l 00 C) The above analyses examine the impact of eachC) moderating var- iable in isolation. III this final set of (6 analyses, we explore theirjoint influence oil precognition performance. For this purpose, we iden- ify two subgroups of studies. One subgroup characterized by the t use of' selected subjects tested individuallyW with trial-by-trial feed- back. We refer to this as the Optimal group (N = 8 studies). The second group is characterized by the use of unselected subjects tested in groups with no feedback. We refer to this as the Suboplinial roup (N = 9 studies). 00 9 C) The Optimal studies are contributed by four ' independent inves- 00 gators and the Suboptimal studies are contributedCD by two of the ti same four investigators. All of the Optimal Q studies involve short pre- Q cognition time intervals (millisecond interval);C) the Suboptimal stud- * C ies involve longer intervals (Intervals of 4 weeks or months). All of the Optimal studies and 5 of the 9 Suboptimal U) studies use RNG meth- M odology. The two groups do not differ significantly(1) in average sam- ple size. The mean study quality for the Optimal group is signifi- candy higher than that of the Suboptimal studies: Optimal mean = ' 0) = "0 6.63, SD = 0.92; Suboptimal mean = 3.44, SD = 0.53; t(1 8.63, P = 3.3 x 10_(~, LWO-Lailed. LL The combined impact of the moderating variables'D appears to be (1) even of the 8 Optimal studies (87.5%) are > quite strong (Table 7). S independently significant at the 5% level, L0_ whereas none of the Sub- Dptimal studies are statistically significant.CL All four investigators con- CL tributing studies to the Optimal group have significant outcomes.' ' Ill the untrininied saniplcoi*309studies, thereareatotalol'17 Optinialstudies. flie inean ES is 0.117 (SD = .154), and the combined z is 15.84. The percentage of illdepelldvilily significant sludics is Virtlially the sailleas it is ill ille trillimcd sallipic: 15 ofthe 17 studies (88.2%) are significant. _0ANi -0.0-1 -11.112 0.1111 0.02 UAH 11.06 0.08 0. 10 mi) TheJournal t#'11arapsycholoA'y 9 v-- (~o 0 0 C*4 C*4 0 0 0) 00 c6 TABLE 7 IMPACT OF MODERATORS IN COMBINATION "Optimal" studies "Suboptimal" studies N Studies 8 9 Combined z 6.14 -1.29 Studies with 87.5% 0.0% < .05 Mean ES .055 .005 SD,, .045 .035 t(1 5) = 2.6 1, p .0 1 r = .559 These results are quite striking and suggest that future studies combining these moderators should yield especially reliable effects. SUMMARY AND CONCLUSIONS Our meta-analysis of forced-choice precognition experiments 00 -0 confirms the existence of a small but highly significant precognition 00 effect. The effect appears to be replicable; significant outcomes are 0 0 reported by 40 investigators using a variety of methodological par- 0 adigms and subject populations. 0 C4 The precognition effect is statistically very robust: it remains (D highly significant despite elimination of' studies with z scores in the U) CU upper and lower 10% of the z-score distribution and when a third of the remaining investigators-the major contributors of precog- nition studies-are eliminated. Estimates ofthe "filedrawer" problem and consideration ofpara- 0 psychological publication practices indicate that the precognition ef- LL feCt Cannot plausibly be cxplained on the basis ofselective publica- (D tion bias. Analyses of precognition effect sizes in relation to eight > 2 measures of research quality fall to support the hypothesis that the observed effect is driven to any appreciable extent by methodolog CL ical flaws; indeed, several analyses indicate that methodologically su < perior studies yield stronger effects than methodologically weaker studies. Analyses of parapsychological alternatives to precognition, al- though limited to the subset of studies using random number tables, provide no support for the hypothesis that the effect results froin A MHW-Ana~ysis qfhorccd-Choice Precog7iition Exl)eriuietlt~ 301 -.he operation of contemporaneous ESP and PK aL the time of ran- lomization. Although the overall precognition effect CD size is small, this does -iot imply that it has no practical consequences.'L It is, for example, )f' the same order of magnitude as effect 0 sizes leading to the early ' ermination of 0 several ma ,)or medical research studies. In 1981, the '~atioiial Heart, Lung, and Blood Institute discontinued its study of to )ropranolol because the results were so favorable0 to the propranolol reatinent that it would be unethical to continue0 placebo treatment 04 04 Kolata, 1981); the effect size was 0.04. 0 More recently, The Steering ~ommlttee of the Physicians' Health Study 0 Research Group (1988), n a widely publicized report, terminated its study of the effects of C0 ispirin in the prevention of heart attacks 00 for the same reason. The Ispirin group suffered significantly fewer 0 heart attacks than a pla- :ebo control group; the associated effect 9 size was 0.03. The most important outcome of' the meta-analysis(0 is the identi- C0 ication of several moderating variables that appear to covary sys- ematically with precognition performance. The largest effects are )bserved in studies using subjects selected on the basis of prior test ' )erf Orniance, who are tested individually, and who receive trial-by- rial feedback. The outcomes of studies combining0 these factors con- rast sharply with the null outcomes associated. with the combination . 00 f group testing, unselected subjects, and 0 no feedback of results. Be- ause the two groups of studies were conducted00 by a subset of the aine investigators, it is unlikely.that the 0 observed difference in per- 0 ormance is due to experimenter effects. Indeed,0 these outcomes 111derscore the importance of carefully examining0 differences in 04 ubject populations, test setting, and so (D forth, before resorting to acile "ex lanations" based on psi-mediated U) experinienter effects or C9 p he "elusiveness of psi." The identification of these moderating variables(D has important -nplications for our understanding of the phenomena and pr9vides clear direction for future research. The LO_ existence of moderating ariables indicates that the precognition LL effect is not merely an nexplained departure from a theoretical chance'0 baseline, but (D adier is an effect that covaries with factors> known to influence iore familiar aspects of human performance. 2 It should now be pos- :ble to exploit these moderating factors CL to increase tile magnitude CL nd reliability of precognition effects in < new studies. REFERENCES XERS, C. (1987). Parapsychology is science, but its findings are inconclusive. Behavioml .and Brai?i Scie?ices, 10, 566-568. :w2 Thijotmial (~/ Pamp.%y(holiwy Z~I RARNI-11 1, V., & LFAVIS, T (1978). Outlier~ iii statistired detta. Ncw 1~)ik: John Wiley & Soils. I "g, In sclence and engl- C)jWWNl.VF, K. A. (1965). Statistical thron, and melhodolo, T_ fierilng. New Yoi-k: john Wiley & Solis. (:)COHEN, J. (1977). Statistical power analisisjor the behavioral sciences. New York: Q Q Academic Press. D~,,%A,Fs, R. M., LANDMAN, J., & WII,I.IA%lS,,]. (1984). Reply to KUI-OSaWa. Amer- to Q ican Psvchoh~gnst, 39, 7-1-75. Q 04 HONORTON, C. (1985). Nfeta-analysis of' psi ganzMd research: A response to 04 1 iynianjownal (I Pwapsychob)gy, 49, 51 -92. Q Q HYMAN, R. (1985). The ganA&I psi experiment: A critical appraisal.fiYurnal ffl'aral).~Tthology, 49, 3-50. Kol.ATA, G. B. (1981). Drug found to help heart attack SLII-VIVOI'S. Science, 214, 00 r- 77.1-775. CD NIANG.AN, G. L (1955). Evidciice of, displaccillent in .1 precogilition testjolonal a (!J'IlarapsTcholoe), 19, 35-:1-1. (D MORRIS, R. L 0982). Assessing experimental mipport I'or (rile precogill(LOU. a- Journal of Parap~ychology, 46, 321-336. RoSENTHm., R. (1984). Xlela-anal.ylic procedures fin- social reW17-Ch. Beverly Hills, CA: Sage. RF.SE.ARCiIGROUP. (1988). Preliminary report: Findings froin the aspirin component of* the oligollig Physicians, Health Study. Ale-w P"ngy1andjournal o 00 J'Medicine, 318, Q 262-264. 00 STERLING, T D. (1959). PUI)IiCaLiOll decisions and their possible efl~cts oil Q inferences drawn from tests of significance-or vice versa. Journal o/ the C) Q Anierican Statistical As.wciation, 54, 30-34. Q WILKINS()N, L. (1988). SYSTAT' 77w.%vNIrm Fx;oislon, IL: SYS~L`ff 04 (1) U) CHRON01A)GICA1. LISTING OF STUDIES IN NILTA-ANAtA'SIS CV CARINGTON, W (1935). Preliminary experiments in precognitive guessing.jour- nal of the Socielyj6r Psychical Research, 29, 86-104. RiIINF. J. 13. (1938). Experiments hearing oil (lie precognition ll~ poillesis: 1. 0 Pre-shuffling card calling.journal of FarapsycholojU, 2, 38-54. LL RIJINEj B., B. M., & WOODRUFFJ L. (1938). Experiments bearing 'a oil the precognition hypothesis: 11. The role of' ESP in the shuffling of' W > CardsJournal of Parap,~Tch(;IoAry, 2, 119 - 131. LO HUMPHREY, 11. M., & PRATY, 'I. G. (1941). A comparison of' five ESP test CL procedUresJournal of Parap.~yrholo~gy, 5, 267-293. CL RIHNE, J. B. (1941). Experiments bearing upon the precognition hypothesis: Ill. Mechanically selected cardsJournal of Parapsychology, 5, 1-57. STUAWF, C. E. (1941). An analysis to determine a test predictive of'extra-cliance scoring in Card-Calling LeSLS. Journal of Barapsycholo( gy, 5, 99- 137. HUFMPFIRF~N,, B. Nf., & RIIINE,J. B~ (1942). A C0llfIl`llla(orySLLldy OfSAiellCe ill pi-ccognition tesisjow-nal (!f Rmap.~Tcholog'y, 6, 190-219. A Illetel-Analysis oj P'oiced-Cholce Precognition Expe),1111ents 303 . .1. 11. (19,12). Evidence of, precognition ill ill(- covarialioll of' salience RIIINF, ratiosJournal. (Y'llarapsychology, 6, 111-1413. Ni(:oi.j. E ,, & CARINGTON, W. (1947). Some experiments in willed die-throw- ing. Proceedings (fthe Socirlyfin- Pql'hii-ill 1?1,SCaYTh, 48, 164- 175. THOUt,Ess, R. H. (1949). A co'niparative study of' performance in three psi tasksJournal qf Pei ral).qch ology, 13, 263-273. BASTIN, E. W., & GREEN, J. M. (1953). Some experiments in precognition. journal (?f Pa.Y-aj)sycholoA'Y, 17, 137-1,13. MGMAHAN, E. A., & BA-i-Fs, E. K. (1954). Report offill-ther Marchesi exper- iniculs. Journal ofilarap.%ychology, 18, 82-92. MANGAN, G. L. (1955). Evidence of displacement in a precognition test.journal efllarap,~yrhoh~gy, 19, 35-14. Osis, K. (1955). Precognition over time intervals of one to thirty-three days. .1om-nal (!fflarap.~vrhology, 19, 82-91. N I El,SEN, W. (1956). Ali exploratory precognition experinient.journal ofBara- p.~ycholqry, 20, 33-39. NiEll-SEN, W (1956). Mental states associated with success in precogn i Lion. jour- nal of Parapsychology, 20, 96-109. FAH LER, J. (1957). ESP card tests with and without hypnosis. Journal of Para- psychologry, 21, 179- 185. MANGAN, G. L. (1957). An ESP experiment with dual-aspect targets involving one trial a dayjournal of Parapsychology, 21, 273-283. acher- ANDERSON, M., & WHITE, R. (1958). A survey of'work on ESP and te pupil attiLudesJournal of Parapsychology, 22, 246-268. NASH, G. B. (1958). Correlation bet,~%,eeii ESP and religious value. Journal of Parapsychology, 22, 204-209. ANDERSON, M. (1959). A precognition experiment comparing time intervals of a 1~%v days and one ycarjournal (Y'Rarapsycholo""y, 23, 81-89. ANDERSON, M., & GREGORY, E. (1959). A two-year program of tests for clair- voyance and precognition with a class Of' J)Ublic school pupils. Journal of Parapsychology, 23, 149-177. NASH, C. B. (1960). Can precognition occur diametrically;i journal of Parapsy- chology, 24, 26-32. FREEMAN, J. A. (1962). Ali experiment in precognition.journal ofParapsychology, 26, 123-130. 1 . RiIINE, .1. B. (1962). The precognition of'coniputer numbers in a public test. Journal of Parapsychology, 26, 244-251. Ryzi., M. (1962). Training the psi faculty by hypnosis. Journal of the Society for Psychical Research, 41, 234-259 SANDERS, M. S. (1962). A comparison of' verbal and written responses in'a precognition experimenLjournal of Parapsychology, 26, 23-34. FREEMAN, J. (1963). Boy-girl differences in a group precognition test. journal of Parapsychology, 27, 175-181. RAO, K. R. (1963). Studies in the' preferential effect: 11. A language ESP test involving precognition and "iiiterveiition.",foiii-lietlofP(i7-(tpsychott~gy, 27,147- 160. C) v-- Q Q CD to Q Q 04 04 Q Q W 00 I*- Q C) 01) a. 0 00 Q 00 Q C) C) C) 04 a) U) 77D W L_ 0 LL 'a (1) > 0 L_ CL CL 30.1 TheJournal ol'Par(II)sYchoh~An, FREEMAN, .1. (1964). A precognition test with a high-school science clul).Journal of Papal),~y(hol(~gy, 28, 214-221. 0 FREENIAN, 'I., & NiFi-%FN, W (1964). Precognition score deviations as related .4 to anxiety levels. journal ol'Iarapsychology, 28, 239-249. 0 SCHNIEIDLER, G. (1964). Ali experiment oil precognitive clairvoyance: Part. 1. 0 The main results. Journal of Parapsychology, 28, 1-14~ 0 T-- FREEMAN, J. A. (1965). Differential response of' the sexes to contrasting ar- rangements of'ESP target material. journal ofFarapsychology, 29, 251-258. 0 Osis, K., 8-- FAHLER, j. (1965). Space and time variables in ESP Journal of the C*4 American Society for Psychical Research, 59, 130-145. C*4 0 FAHLER, J., & Osis, K. (1966). Checking for awareness of'hits in a precognition experiment with hypnotized subjects. Journal of the American Society for Psychical Research, 60, 340-346. 00 FREEMAN, J. A. (1966). Sex differences and target arrangement: High-school I- booklet, tests of' precognition..1ournal of Parapsychology, 30, 227-235. 9 Ro(;I-,Rs, D. R (1966). Negative and positive affect and ESP rull-score variance. to irnal of Parapsychology, 30, 151-159. 0) Jol (L Ro(.F.RS, 1). R, & CARPENTEP, .1. C. (1966). The decline of' variance of' ESP scores within it testing session. journal of'Parapsychology, 30, 141-150. BRIER, B. (1967). A correspondence ESP experiment with lilgh-I.Q. sub' < Jmirnal of Peuttp.~y-hotogy, 31, 1,13- 1,18. .1ects. U Buziii,, D. E. (1967). Subject attitude and score variance in ESP tests.Journal of Parapsychology, 31, 43-50. CO BuzBv, D. E. (1967). Precognition and a test of sensory perception.JoUrnal of 0 Parapsychology, 31, 135-142. 00 0 FREEMAN, J. A. (1967). Sex differences, LargCL arrangement, and primary men- 0 tal abilities. Journal of Parapsychology, 31, 271-279. CD HONORTON, C. (1967). Creativity and precognition scoring levei.journal of 0 C4 Parapsychology, 31, 29-42. (D CARPENTER, J. C. (1968). Two related studies oil mood and precognition run- U) co score variance. Journal of Parapsychology, 32, 75-89. (1) DuVAL, R, & MONTREDON, E. (1968). ESP experiments with inice.journal of Parapsychology, 32, 153-166. FEATHER, S. R., & BRIER, R. (1968). The possible effect of the checker in 0 precognition tests. Journal of Parapsychology, 32, 167 - 175. LL FREEMAN, J. A. (1968). Sex differences and primary mental abilities in a group 13 precognition teSL. Journal of Parapsychology, 32, 176-182. 0 NASH, C. S., & NASH, C. B. (1968). Effect of target selection, field dependence, > and body concept on ESP performance.Journal of Parapsychology, 32, 248- 2 CL 257. CL RHINk-, L. E. (1968). Note on an informal group test of ESP. Journal of Para- psychology, 32, 47-53. RYZL, M. (1968). Precognition scoring and attitude toward ESP.Journal of Para- psycholqgy, 32, 1-8. Ryzi., M. (1968). Precognition scoring and aLLitude. journal oJ_ Parapsychology, 32, 183-189. A illrla-Analj%i~ ofForcrel-Choffe, PreceWnition ExIierinientV 305 CARPENTF.R, .1. C. (1969). Further study oil it mood ad *jective check list and ESP 11111-SCOYC V;l1_ial)Ce. J0111-Mil ly 1`o)_tJPs)1ChOlqgY, 33, -48-56. DL1VAL, 1), & MONTREDON, E. (1969). Precognition in mice: A confirmation. Journal of Parapsychology, 33, 71- 72. 0 FREEIVIAN,J. A. (1969). The psi-differential effect in a precognition test.journal L ofFarapsychology, 33, 206-212. C) 0 FREEMAN,J. A. (1969). A precognition experiment with science teachers.Journal 0 T_ of Parapsychology, 33, 307-310. to JOHNSON, M. (1969). AttiLude and target differences in a group precognition o test.journal of Parapsychology, 33, 324-325. C) CM MONTREDON, E., & RoBINSON, A. (1969). Further precognition work with mice. C14 0 Journal of Parapsychology, 33, 162-163. 0 ScH m I i)-i-, H. (1969). Precognition of a quantum process. Journal of Parapsy- W hology, 33, 99-108. 0) C 00 B FIN DER, H. (1970). Differential scoring of an outstanding subject on GESP and clairvoyance.Journal offlarapsychology, 34, 272-273. FREEMAN,J. A. (1970). Sex differences in ESP response as shown by the Free-(6 mall picture-figure Lest.Jaurnal of Parapsychology, 34, 37-46. 0) a. FREEMAN, J. A. (1970). Ten-page booklet tests with elementary-school children.c) Journal qf 34, 192-196. W FREEMAN, J. (1970)~ Shift in scoring direction WithjUlliOr-high-SC11001 students: A SUIlunal-Y.J91111-nal of Rarapsychology, 34, 275. FREEMAN, J. A. (1970). Mood, personality, and attitude in precognition tests. Journal of Prapsychology, 34, 322. 00 HARALDSSON, E. (1970). Subject selection in it machine precognition test.journalR of Parapsychology, 34, 182-191. 00 0 HARALUSSON, E. (1970). Precognition of a quantum process: A modified rep-6 licationjournal of1larapsychology, 34, 329-330. 0 NIEUSEIN, W (1970). Relationships between precognition scoring level-andpN mood. Journal of Parapsychology, 34, 93 - 116. (D ScHNni),r, H. (1970). Precognition test with a high-school group.jountal ofu) M Parapsychology, 34, 70. (D BELOFF, J., & BATE, D. (1971). An attempt to replicate the Schmidt. findings -a) Journal of the Societyfor Psychical Research, 46, 21-31. HONORTON, C. (1971). Automated forced-choice precognition tests with a "sen-0 sitive." journal of the American Socielyfor Psychical Research, 65, 476-481. LL MITCHFLL, E. D. (1971). An ESP Lest from Apollo 14. Journal of Farapsychologym 35,89-107. (D > Sci-mmyr, H., & PANTAS, L. (1971). Psi teSLS with psychologically equivalent 2 conditions and internally different machines. Journal of Parapsychology, 35,CL 326-327. . CL STANFORD,.R. G. (1971). Extrasensory effects upon "memory." journal of the< A werican Socielyfor Ps iTchical Research, 64, 161-186. STEILBERG, B. J. (1971). Investigation of' the paranormal gifts of the Dutch sensitive' Lida Tjournal (Y'Parapsychology, 35, 219-225. ThrJoIll-lial of Paral).~TcholoAry 3 ~ Tlvwl~I--~Ss, R. 1-1. (1971). Experimeni% oil psi self-( raining with Dr. S(iiinidt's pre-cognitive apparatus. Journal oj'lhe Socielyfor Psychical Research, 46, 15- 0 21. v~ HONORTON, C. (1972). Reported frequency ofdrearn recall and ESI~Jonrnal 0 qf the American Soriel-T./or Ps.Tchirwl Research, 66, 369-374. 0 JOI INSON, M., & NORDBECK, B. (1972). Varia(ioll ill tile Scoring behavior of- a psychic" sul)ject.Journal of Parapsycholilgy, 36, 122-132. 0 KELLY, 1". E ', & KAN-1-1 IAMANI, B. K. (1972). A subjeci's efforts toward voluntary C*4 control. Journal of Parapsychology, 36, 185 - 197. 04 SU iMiDT, I L, & PAN1 AS, L. (1972). 11si tests with internally difliLrent machines. 0 Journal (Y'Rarapsychohigy, 36, 222-232. 0 X CRAIG, .1. G. 0973). 'I'll(- effect of colaingemy oil precogililioll ill the raf. 0) Research in llezrap,%ychology 1972, 154-156. 00 I- FRFF.,,IAN,J. k. (1973).The pS'(JUiz: A new ESP test. Reseench 1`arap.~Tchology 0 0 1972, 132-134. (6 ARTLFY, B. (1974). Confirmation ofthe small-rodent precognit ion work.Jaurnal 0) of Parapsychology, 38, 238-239. IL a HARRIS, S., & TERRY, J. (1974). Precognition in a water-deprived Wistar rat. w Journal of Parapsychology, 38, 239. RANDALL, J. L. (1974). An extended series of' ESP and PK tests with three English schoolboys. Journal of the Socielyfor Psychical Research, 47, 485-494. EYsENCK, H.J. (1975). Precognition ill rats.Joarnal of Parapsychology, 39, 222- 00 227. -0- HARAimssoN, E. (1975). Reported dreani recall, 1)t,et:ogiiitive(ii-e;tiiis,;iii(I ESJ~ 00 Research in Parapsychology 1974, 47-48. 0 HONOPTON, C., RANism,, M., & CABnwo, C. (1975). Experimenter effects in 0 extrasensory perception.journal ofthe Anierican Socielyfor Psychical Research, 0 N 69, 135-149. (1) KANTHAMANI, 11., & IZAO, H. 1-1. (1975). lZesponse lendencies and stimulus U) structure. Journal of Parapsychology, 39, 97-105. co (V LEVIN,.J. 2N. (1975). A svi-ics ofpsl experinivins wid) gei-bils..1wirmll 39, 363-36.5. L_ C., & I 1ARRis, S. A. (1975). Pr(cognition ill waier-deprived rals. 0 Research in Parapsychology 1974, 81. LL DAVI.S,J. W, & HAIGHT,J. (1976). Psi experiments with rats.Jaurnal If Para- psycholl,~,,y, 40, 5,1-55. (1) S, J., & BREEDERVE.U), H. (1976). Possible influences ofbirth order oil > JACOB, 2 ESP ability. Research Letter (Parapsychology Laboratory, University of' M Utrecia). No. 7. 10-20. Nvvu~i.i.., R. C~ (1976). Someaspecis of' precognition lesillig. Research ill Para- psychology 1975, 29-31. DRuCKER, S. A., DRFwj.-s, A. A., & RmuN, L. (1977). FISP ill relation to cognitive development and IQ ill yowig childrenjournal of lite Anterican Socielyjor Psychical Research, 71, 289-298. zi Meta-AnalTsis offorced-Cholce 11recog-nitiOn Experinients 307 HARALDSSON, E. (1977). ESPand the defense mechanism test (DMT): A further validation. Europeanjounial of Parapsychology, 2, 104-114. SARGENT, C. L, (1977). An experiment involving a novel precognition task. Journal of Parapsychology, 41, 275-293. BiERMAN, D. 1 (1978). Testing the "advanced wave" hypothesis: An attempted VL replication. European finu-nal ty'Beirapsychology, 2, 206-212. BRAUD, W. (1979). Project Chicken Little: A precognition experiment involving the SKYLAB space station. Europeaujournal of'Parapsychology, 3, 149-165. V- HARALDSSON, E., & JOHNSON, M. (1979). ESP and the defense mechanism test (DMT) Icelandic study No. Ill: A case of the experimenter effect? European a C*4 Journal of Parapsychology, 3, If- 20. C*4 O'BRIEN,J. T. (1979). Ali examiumion of(he checker efR.c(. Research ill Bala- C) psychology 1978, 153-155. CUMENS, D. B., & PHILLHs, D. T. (1980). Further studies of precognition in 0) 00 nlice. Research in Parapsychology 1979, 156. 1 HARALDSSON, E. (1980). Scoring in a precognition test as a function of the Q frequency of reading on psychical phenomena and belief in ESP. Research 9 to Letter (Parapsychology Laboratory, University of Utrecht), No. 10, 1-8. CY) SARGENT, C., & HARLEY, T A. (1981). Three studies using a psi-predictive Om trait variable questionnaire. Journal of Parapsychology, 45, 199- 214. 0 WINKELMAN, M. (1981). The effect of formal education on extrasensory abil- I ities: The Ozolco, study. Journal of Parapsychology, 45, 321-336. NASI I, C. B. (1982). ESP of present and future Largets.Journal of the Societyfo, Psychical Research, 51, 374-377. 00 THALBOURNE, M., BELOFF, J., & DELANOY, D. (1982). A test for the "extra- a verted sheep versus introverted goats" hypothesis. Research in Parapsychology 00 1981, 155-156. CRANDALL,J. E., & HITE, D. D. (1983). Psi-missing and displacement: Evidence a f6r improperly f6cused psi?Journal ofthe Atnerican Socielyftr Psychical.Re- (D search, 77, 209-228. N (D S(:i iwAR rz, S. A., R. 1) J. (1983). The Nlo i s Psi-Q (es(: Prefilil- U) inary findings. Research in Parapsychology 1982, 103-105. .101INSON, M., & I IARALDSSON, E. (1981). ThC Defe'llse Mechanism _1~.M ;Is a predictor ofESP scores: Icelandic studies I.V and Vjournal of1laraps)rhology, X 48, 185-200. L_ 0 TEDDER, W Coil] pu ter-based long-distance ESP: Ali exploratory ex-LL animation (RB/PS). Research in Parapsychology 1983, 100- 101. HESELTINE, C. L. (1985). PK success during structured and nonstructured a) IZNG operation. Journal of Faral)sychology, 49, 155-163. 0 HARALDSSON, E- &JOHNSON, N4. (Hoiy -rhe Defimse Mechanism Test (DMT) " I CL as a predictor ol'ESP peribraim ice: Icelandic studies Viand \711. ?esearch CL I.n Poral)sy~holqgy 1985, 43-44. < VASSY, Z. (1986). Experimental study ofcomplexity dependence in precogni- tion.Journal ~)f Parapsychology, 50, 235-270. 0 CL CL 308 TheJounial of'Parapsychology HONORTON, C. (1987). Precognition and real-time ESP performance in a com- I)LIter task with an exceptional subject. Journal oJI'arapsychology, 51, 291 - 320. Psychophysical Research Laboratories P 0. Box 569 P/aInsboro, NJ 08536 0 CL CL