iw I Annro:a For Release 2000/08/08 so 9 CIA-RDP96-00789ROO2200410001-2 Final Report--Objectlve 1, Task I Covering the Period 31 July 1987 to 30 September 1968 META-ANALYSIS OF FORCED-CHOICE PRECOGNITION EXPERIMENTS CHARLES HONORTON DIANE B. FERRARI GEORGE HANSEN, Psychophysloal Research Laboratories By: EDWIN C. MAY SRI International Prepared for: Peter J. McNells, DSW CONTRACTING OFFICER'S TECHNICAL REPRESENTATIVE n CONTRACT DAMD17-85-C-5130 SRI Project 1291 Approved by: MURRAY J. BARON, Director Geosclence and Englneerlng Center 333 Ravenswood Ave. * Menlo Park, CA 94025 d For ReleAl~ 12Vft6V008 TtffA ~kb-o~ (inter %-266~gqkdb~Z&44ff001-2 December 1988 ....i Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Ow I INTRODUMON We have subcontracted to Psychophysical. Research Laboratories (PRL) to conduct a meta-analysis of the forced-choice precognition literature. Mr. Honorton, the director, has met the requirements of the subcontract. Attached, is the deliverable from PRL. Ow Ow =0 and Mw Approved For Release 2000/08/08 CIA-RDP96-00789ROO2200410001-2 I. Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 I-W I~= I Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 I m Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 TABLE OF CONTENTS ABSTRACT' ................................. I OBJECTIVES ................................ 2 DELINEATING THE DOMAIN ..................... 3 Source of Studies .......................... 3 Criteria for Inclusion ........................ 3 Outcome Measures ......................... 3 General Characteristics of the Domain .............. 4 OVERALL CUMULATION ....................... 6 Replication across Investigators .................. 7 The Filedrawer Problem ...................... 8 OUTIJER ELIMINATION ........................ 10 STUDY QUALITY .................... 13 ......... Study Quality Criteria ...... 13 .................. Study Quality Analysis ....................... 14 Quality Extremes .......................... 15 Quality Variation in Publication Sources ............. 15 Study Quality in relation to Year of Publication .........16 "REALTIME" ALTERNATIVES TO PRECOGNITION ...... 17 Method of Determining RNT Entry Point ............ 18 low Use of Mangan's Method ...................... 18 MODERATING VARIABLES ...................... 19 Selected versus Unselected Subjects ............... 19 Individual versus Group Testing .................. 21 Feedback ............................... 22 Time Interval ............................. 24 Influence of Moderating Variables in Combination ....... 26 DISCUSSION ................................ 27 REFERENCES ............................... 29 CHRONOLOGICAL LISTING OF STUDY REFERENCES .... 30 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 ABSTRACT We report a meta-analysis of forced-choice precognition experiments published in low English-language parapsychology journals between 1935 and 1987. These studies involve attempts by subjects to predict the identity or order of target stimuli selected randomly over intervals ranging from several hundred milliseconds to one year following the subject's responses. The database includes 309 studies reported by 62 senior authors. Nearly 2 million individual trials were contributed by more than 50,000 subjects. Study outcomes are assessed in terms of overall level of statistical significance and effect size. We find a small, but consistent, and highly significant overall tendency for directional hitting (z = 12.14). Analysis based on investigators'predictions of conditions associated with hitting and missing yields a much stronger result (z = 24.23). Thirty percent of the studies low (and 39% of the investigators) have directional outcomes that are significant at the 5% significance level. Assessment of the vulnerability of this database to selective reporting of positive results indicates that a ratio of 50 unreported studies averaging nun results would be required for each reported study in order to reduce the overall significance of the observed outcomes to nonsignificance. . Mai No systematic relationship exists between study outcomes and eight indices of research quality. Magnitude of effect has remained essentially constant over the survey period, while research quality has improved substantially. do Four moderating variables appear to covary significantly with study outcome: ~ Studies using subjects selected on the basis of prior testing performance show significantly larger effects than studies involving unselected sub- Ow jects. ~ Subjects tested individually by an experimenter show significantly larger aw effects than those tested in groups. ~ Studies in which subjects are given trial-by-trial or run-score feedback have significandy larger effects than those with limited or no subject feedback. ~ Studies with brief intervals between subjects' responses and target generation show significantly stronger effects than studies involving longer intervals. The combined impact of these moderating variables appears to be very strong. A nearly perfect replication rate is observed in the subset of studies using selected subjects, who are tested individually and receive trial-by-trial feedback. PsychophysIcal Research Laboratorles Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Usp CIA-RDP96 0789R002200410001-2 Approved Kf WRe I ~ s NPO/08/&Ice PrecogngO e -a ys orcea on Experirnents 2 OBJECTIVES Precognition refers to the noninferential prediction of future events. Anecdotal claims of "future knowing" have occurred throughout human history, in virtually every culture and period. Today, such claims are generally believed to be based on factors such as delusion, irrationality, and superstitious thinking. The concept of precognition runs counter to accepted notions of causality and appears to conflict with current scientific theory. Nevertheless, over the past half-century, a substantial number of experiments have been reported by more than 60 investigators claiming empirical support for the hypothesis of precognition. Subjects in forced-choice experiments, according to many reports, have correctly predicted to a statistically significant degree the identity (or order) of target low stimuli randomly selected at a later time. We performed a meta-analysis of forced-choice precognition experiments published in the English-language research literature between 1935 and 1987. Five major questions were addressed through this meta-analysis: ~ Is there overall evidence for accurate target identification (above chance hitting) in experimental precognition studies? ~ Is there overall evidence that investigators can accurately predict tendencies toward hitting and missing? ~ What is the magnitude of the overall (directional and predicted) precognition effect? ~ Is the observed effect related to variations in metbodologi- cal quality that could allow a more conventional explanation? ~ Does precognition performance vary systematically with Ito potential moderating variables, such as differences in sub ject populations, stimulus conditions, experimental setting, knowledge of results, and time interval between subject response and target generation? Psychophysical Research Laboratorles Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 I L Approved FoertaF~ele 0 /rO -RDP96-00789ROO2200410001-2 0 CV& CIA anam$2 01V holce PrecognItIon ExperIffiergli 3 DELINEATING THE DOMAIN Source of Studies Parapsychological research is still academically taboo and it is unlikely that there have been many dissertations and theses in this area that have escaped publication. Our retrieval of studies for this meta-analysis is therefore based on the published literature. The studies include all forced-choice precognition experiments appearing in the peer-reviewed English-language parapsychology journals: Journal of Paraps ychology, Journal (and Pioceedings) of the Societyfor Psychical Research, Journal of the American SocietyforPsychical Research, European Journal ofParapsychology (including the Research Letter of the Utrecht University Parapsychology Laboratory), and Research in Parapsychology. Crlterla for Inclusion Our review is restricted to fixed length studies in which significance levels and effect sizes based on direct hitting can be calculated. Studies using outcome variables other than direct hitting, such as run-score variance and displacement effects, are included only if the report provides relevant information on direct hits (i.e., number of trials, hits, and probability of a hit). Finally, we exclude studies conducted by two investigators, S. G. Soal and Walter J. Levy, whose work has been unreliable. Many published reports contain -more than one experiment or experimental unit. Experiments involving multiple conditions are treated as separate study units. Outcome Measures Signiflcance Levels: We calculated two significance estimates for each study. The &rectionalz-score (zdir) measures the subjects'success in scoring in the direction of their intention. Theprediciedz-score (Zpred) measures the investigator's success in predicting the relative strength or direction of the outcome through conditional comparisons, experimental manipulations, or correlations; above chance scoring (hitting) is assumed in single condition experiments unless psi-missing is explicitly predicted. Predicted z's have Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 'Meta-analysis of Forced-Choice Precogriftion Experlments 4 Positive signs if the study outcome supports the investigator's hypotheses and have negative signs if the outcome is opposite the investigator's hypotheses. Ile use of these two measures allows us to assess both overall accuracy (hitting) and lawfulnes 's (predictability). Effect Sizes: Most parapsychological experiments, particularly those in the older literature, have used the trial rather than the subject as the sampling unit. Thus, we must use a trial-based estimator of effect size. The effect size for each study is the z-score divided the square root of the number of trials in the study. As with significance levels, we have two effect sizes for each study. One reflects overall directional hitting (ESdir) and the other is based on the investigators' predictions of hitting or missing (ESpred). General Characteristics of the Domain We located 309 studies in 113 separate publications. These studies were contributed by 62 different senior authors and were published over a 52-year period, between 1935 and 1987. Considering the half-centw-y time-span over which the precognition experiments were conducted, it is not surprising that the studies are quite diverse. Ile data base comprises nearly 2 million individual trials and more than 50,000 subjects. Study sample sizes range from 25 to 297,060 trials (median 1, 194). The number of subjects ranges from I to 29,706 (median 16). Ile studies employ a variety of methodologies, ranging from guessing Zener cards and other card symbols, to automated random number generator experiments (Figure 1). The domain encompasses diverse subject populations: the most frequently used population is students (used in approximately 40% of the studies); the least frequently used populations are the experimenters themselves and animals (each used in about 5% of the studies). Though a few studies tested subjects through the mail, more typically subjects were tested in person, either individually or in groups. Target selection methods range from manual card-shuffling or dice-tbrowing to the use of random number tables or random number generators. The time-interval between the subjects' responses and target generation varies from less than one second to one year. Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 I I IM low F Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 5 FIG. I.: PRECOGNITION TESTING METHODS TOP LEFI7: Zener cards. TOP RIGHT: Subject tested with 4-choice random number generator. BOTTOM RIGHT: Block diagram for 4-choice RNG. BOTTOM LEFT-. Apparatus used for small rodent anticipation of shock experiment. UA W4 -n4" D. 9w-uw Golf 4 El. 0. a SYK GM P-W Psychophysical Research Laboratories Approved For Release 2000/08/08 CIA-RDP96-00789ROO2200410001-2 twd ww Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysls of Forced-Choice Precognition Experiments 6 OVERALL CUMULATION Evidence for overall directional hitting and for successful prediction of hitting and missing tendencies is strong. 1 As shown in the top part of Table 1, the overall results are highly significant. The mean predicted z is twice as large as the mean directional z, indicating the advantage of making focused predictions (and the lawfulness implied by being able to do so). Thirty percent of the studies show overall significant hitting at the 5% level. Nearly Wit 40% are significant on the basis of the investigators' predictions. TABLE 1: Overall Precognition Significance Levels Zdir Zpred Mean 0.69 1.38 Standard Deviation 2.65 236 Combined (Stouffer) z 12.14 24.23 -27 -52 P, 6 x 10 4x10 Filedrawer Estimate 16,529 66,687 Lower 95% Confidence Estimate0.44 1.16 Lower 99% Confidence Estimate0.34 1.07 Lower 99.9% Confidence Estimate0.22 0.97 Lower-bound confidence estimates of the mean z-scores displayed in the bottom portion of Table 1 indicate that the mean directional andpredicted z-scores are well above zero at the 95%,99%, and 99.9% confidence levek Significance levels, not surprisingly, are related to sample size. The correlation (r) is 0.151 for the directional z's (307 dfp = ON4), and for the 'The statistical analyses presented here were performed using SYSTAT (Wilkinson, 1988). When 1-tests are reported on samples with unequal variances, they are calculated using the separate variances within groups for the error and degrees of freedom following Brownlee (1965). Unless otherwise specified, p-levels are one-tailed. Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 1001 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analOs of Forced-Choice Precognition Experiments 7 predicted zs, r is 0.242 (307 df, p = 8.4 x 10 -6 ). Directionally significant studies have a mean sample size that is 37% larger than the mean for directionally nonsignificant studies. Using the predicted z-score criterion, significant studies have a mean sample size that is more than double that of the nonsignificant studies. The effect size analysis is presented in Table 2. Both directional and predicted outcomes are significantly above zero, and again, the mean predicted effect size is twice as large as the directional mean ES. TABLE 2: Overall Effect Sizes ESdir ESpmd Mean 0.022 0.041 Standard Deviation 0.098 0.092 t(308) 4.01 7.88 Pt 4 x 10-5 3xio -14 Lower 95% Confidence Limit 0.012 0.032 Lower 99% Confidence Limit 0.008 0.029 Lower 99.9% Confidence Limit 0.005 0.025 MW Replication across Investigators aw Virtually the same picture emerges when the cumulation is by investigator rather than study as the unit of analysis. Ile combined zs are 12.71 for directional outcomes and 22.12 for predicted outcomes. Twenty-four of the 62 investigators (39%) have directional outcomes significant at the 5% level, and 39 investigators (63%) have significant predicted outcomes. The mean (investigator) directional effect size is 0.036 (sd = .091), and the mean predicted ES is 0.050 (sd = .087). These results indicate a substantial level of cross-investigator replicability and directly contradict the claim of critics such as Akers (1988) that aw PsychophysIcal Research Laboratorles Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 OWN Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analySis of Forced-Choice Precognition Experiments 8 successful parapsychological outcomes are achieved by only a few investigators. The Filedrawer Problem A well-known reporting bias exists throughout the behavioral sciences favoring publication of "significant" studies (e.g., Sterling, 1959). The extreme view of this "filedrawer problem," as Robert Rosenthal describes it, "is that the journals are filled with the 5% of the studies that show type I errors, while the filedrawers back at the lab are filled with the 95% of the studies that show nonsignificance. . ." (Rosenthal, 1984, p. 108). Recognizing the importance of this problem, the Parapsychological Association in 1975 adopted an official policy against selective reporting of positive results. Examination of the parapsychological literature shows that nonsignificant results are frequently published and in the precognition database, 60% to 70% of the studies have reported nonsignificant results. Nevertheless, 75% of the precognition studies were published before 1975, and we must ask to what extent selective publication bias could account for the cumulative effects we observe. 2 The central section of Table 1 uses Rosenthal's (1984) filedrawer statistic to estimate the number of unreported studies with z-scores averaging zero that would be necessary to reduce the known database to nonsignificance. Ile filedrawer estimate suggests that over 50 unreported studies must exist for each reported study to r 'educe the cumulative hitting (directional) outcomes to a nonsignificant level. For the predicted outcomes, the filedrawer ratio is more than 200: 1. Another approach to the filedrawer problem is described by Robyn I Ow Dawes (Dawes, Landman and Williams, 1984; personal communication to Honorton, July 14, 1988). Dawes calculates the expected mean z and variance for various significance levels on the assumption that reported significant outcomes reflect nothing more than type I error. He then tests ism 2Analyses indicate no significant differences in the magnitude of reported study outcomes before and after 1975. The mean directional z-score for studies prior to 1975 is 0.719 (sd = 2.6) and for studies reported thereafter the mean is 0.605 (sd = 2.81) (t - 0325, 307 df, p = .746, 2-tailed). For predicted z-scores, the comparable values are 1.43 (sd = 2.29) and 1.22 (sd = 2.60); t = 0.675, 307 df, p = .675, 2-tailed). PsychophysIcal Research Laboratorles Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 9 IOEI lim the difference between the observed and expected values. Applying this method to the precognition domain, it is extremely unlikely that the reported significance levels are just type I error. For the 5% significance level, for "ample, the mean observed and expected dfrectional z-scores are 3.59 and 2.06, respectively. The observed mean is significantly larger than the expected value (z = 4.10, p = .000021). For the 0.5% significance level, the observed and expected means are 4.97 and 2.87 (z = 7.0,p = 2.7 x 10-12 ). Based on these analyses, we conclude that the cumulative significance of the precognition studies cannot plausibly be attributed to selective reporting. Psychophysical Research Laboratories Approved For Release 2000/08/08 CIA-RDP96-00789ROO2200410001-2 SW Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 10 OUTLIER ELIMINATION Although the overall significance levels and effect sizes for this database cannot reasonably be attributed to chance, inspection of the standard deviations in Tables 1 and 2 indicates that the study outcomes are extremely heterogeneous. Given the diversity of methods, subject populations, and other study features that characterize this research domain, this is not surprising. MW The study outcomes are in fact extremely heterogeneous. Although a major objective of this meta-analysis is to account for the variability across studies by blocking on differences in study quality, procedural features, and sampling characteristics, the database clearly contains extreme outliers. The directional z-scores range from -5.06 to 19.6, a 25-sigma spread! The standardized index of kurtosis ) is 9.86 (p < 10 -6 ), suggesting that the tails of the distribution are much too long for a normal distribution. We have eliminated the extreme outliers by performing a "10-percent trim" on the study z-scores (Barnett and Lewis, 1978). This involves aw eliminating studies having z-scores in the upper and lower 10% of the distribution, and results in an adjusted sample of 248 studies. The directional z-scores for the adjusted sample range from -2.11 to 3.20 (g2 = -1.1). The revised significance levels and effect sizes are presented in Tables 3 and 4. Elimination of extreme outliers has reduced the combined significance levels by approximately one-half, but the outcomes remain highly significant. Twenty-five percent of the studies show overall significant hitting at the 5% level, and 28% are significant based on the investigators' predictions. Lower bound confidence estimates show that the directional and predicted z's are above 0 at the 99.9 % confidence level. PsychophysIcal Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 L Relepsr CIA -RDP96-00789 R00220041 0001 -2 Approved Vr ~ ~ IRO/08/0& 3 P a-ana ys s orced ice rec nftlon Ex erimerits W p L L TABLE 3: Signiflcance Levels for Adjusted Sample Zdir Zpred L Mean 0.43 0.79 Standard Deviation 1.42 1.25 Combined (Stotiffer) z 6.69 1239 L PZ 1.93x 10"' 1.36 X 10-17 Lower 95% Confidence Estimate 0.28 0.66 L Lower 99% Confidence Estimate 0.22 0.61 Lower 99.9% Confidence Estimate 0.15 0.55 Table 4 presents effect size estimates for the adjusted sample . Both the directional and predicted effect sizes remain significantly above zero. TABLE 4: Effect Sizes for Adjusted Sample ESdft ESpred Mean 0.016 0.027 Standard Deviation 0.070 0.066 t(247) 3.60 6.44 Pt 1.92 x 104 2.4 x 10 -8 Lower 95% Confidence Limit 0.009 0.020 Lower 99% Confidence Limit 0.006 0.017 Lower 99.9% Confidence Limit 0.002 0.014 low i0w L Elimination of outliers reduces the total number of investigators from 62 to 57, but the results remain basically the same when the analyses are based on investigators rather than studies. The combined (Stouffer) z's are 7.37 for directional outcomes and 11.68 for predicted outcomes. Twenty one of the 57 investigators (36.8%) have directionally significant outcomes at the Psychophysical Research Laboratorles Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 12 5% level and 30 investigators (52.6%) have significant predicted outcomes. The mean (investigator) directional effect size is 0.023 (sd = .052), and the meanpredicted effect size is 0.028 (sd = .047). Both results remain above 0 on lower-bound 99.9% confidence estimates. Thus, elimination of the outliers does not substantially affect the conclusions drawn from our analysis of the database as a whole. There clearly is a nonchance effect. In the remainder of this report, we use the adjusted sample to examine covariations in magnitude of effect and a variety low of methodological and other study features. Mi low Psychophysical Research Laboratorles im Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 13 STUDY QUALITY Study Oualfty Criteria Since target stimuli in precognition experiments are selected only after the subjects' responses have been registered, precognition studies are usually not vulnerable to sensory leakage problems. Other potential threats to validity that must be, however, considered. For our analysis of study quality, statistical and methodological variables are defined and coded in terms of procedural descriptions (or their absence) in the research reports. One point is given (or withheld) for each of the following eight criteria: Specification of Sample Size. Does the investigator preplan the number of trials to be included in the study or is the study vulnerable to the possibility MW of optional stopping? Credit is given to reports that explicitly specify the sample size. Studies involving group testing, in which it is not feasible to l specify the sample size precisely, are also given credit. No credit is given to ow studies in which the sample size is either not preplanned or not addressed in the experimental report. P~eplanned Analysis. Is the method of statistical analysis, including the outcome (dependent variable) measure, preplanned? Credit is given to studies explicitly specifying the form of analysis and the outcome measure. No credit is given to those not explicitly stating the form of the analysis or those in which the analysis is clearly post hoc. Randomization Method. Credit is given for use of random number tables, random number generators, and mechanical shufflers, but not for hand shuffling, die casting, or drawing lots. Controls. Credit is given to studies reporting randomness control checks, such as random number generator (RNG) control series and empirical cross-check controls. Recording. One point is allotted for automated recording of targets and responses and another for duplicate recording. Checking. One point is allotted for automated checking of matches between target and response and another for duplicate checking of hits. PsychophysIcal Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 1.0 im NOW Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 14 Study Quality Analysis Each study received a quality weight between 0 and 8 (mean = 3.3, sd = 1.8). We find no relationship between study quality and effect size for either the directional (r246 = .029, p = .646, 2-tailed) or predicted (r2A6 = .006, p = .919, 2-tailed) effect sizes. Nor are any of the eight individual quality measures significantly related to effect size (Table 5). TABLE 5: Point-biserial Correlations Quality Measure ESdir ESpred Sample size specified in -.146 -.017 advance Preplanned analysis -.042 -.002 Randomization -.085 -.051 Controls .036 .004 Automated recording .139 -.016 Duplicate recording .054 .074 Automated checking .105 -.023 Duplicate checking .015 .035 The mean effect sizes by quality level are displayed graphically in Figure 2 (directional outcomes) and Figure 3 (predicted outcomes). FIGURE 2: Directional Outcomes in relation to Study Quality (95% Confidence Limits.) 0.100 0.070 0.0150 0.025 O.OHDO -0.025 -0.050 -0.075 -0.100 --------- ---------- -4 (12) (34) (33) (52) (71) (23) (6) (13) (4) 1 1 1 1 0. 1. 2. 3. 4. 5. 6. 7. 8. SrrTJI>'Y QUAL17Y Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved Pr Relepsr 2 -RDP9%,007ffi9RQ02200410001-2 , s 00/08/0S: CIN em-ana ys s orced olce recogn on Expenments 15 FIGURE 3: Predicted Outcomes in relation to Study Quality 0.100 0.075 0.050 0.020 0.000 -0-025 -0.050 -0.075 (95% Confidence Limits.) -------- I ---- --------- 1 MW 1. P- 3. 5. 6. S'rUD'Y" QUATATY" Quality Extremes Is there a tendency for extremely weak studies to show larger effects than exceptionally "good" studies? The grouped data presented in Figures 2 and 3 suggest that this is not the case and analysis on the extremes of the quality ratings indicates that the methodologically superior studies actually have somewhat larger mean effect sizes than studies with weaker methodology. This analysis uses studies with quality ratings outside the interquartile range of the rating distribution (median 3, Q1 = 2, Q3 = 4). There are 46 studies at each extreme ("low quality" ratings of 0-1, "high quality" = ratings of 5-8). The high quality studies have larger effect sizes than the low quality studies in both the directional and predicted analyses. For the directional analysis, the effect size means are 0.034 (sd = 0.061) and 0.016 (sd = 0.09 1), for the high and low quality studies respectively (t = -1.09, 90 df, p = .278, 2-tailed). For the predicted analysis, the effect size means are 0.038 (sd = 0.059) and 0.023 (sd = 0.089), for the high and low quality studies respectively (t = -0.90, 90 df, p = .368, 2-tailed). Quality Variation In Publication Sources Study quality does vary significantly across the five publication sources. Although neither significance level nor effect size are significantly related to source of publication, the five journals do vary significantly in quality PsychophysIcal Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 16 low (Kruskal-Wallis one-way ANOVA, chi-square = 11.78,4 dfp = .019). This outcome is due to the substantially lower quality of studies appearing in the Journal of the Society for Psychical Research. Study Quality In relation to Year of Publication Precognition effect sizes have remained constant over a half-century of research, even though the methodological quality of the research has improved significantly during this period. The correlation between directional effect size and year of publication is -.050 (t246 = -0.79, p = .429). The result is nearly identical for the predicted ES (r246 = -.059, p = .358). Study quality and year of publication are, however, positively and significantly correlated (r246 = .239, p = 7.2 x 10-5 ). See Figure 4. 1 Critics of parapsychology have long believed that evidence for parapsychological effects disappears as the methodological rigor increases. The precognition database does not support this belief. FIGURE 4: (a) Directional Effect Sizes in relation to Year of Publica tion, (b) Study Quality in relation to Year of Publication Least Squares Fit with 95% Confidence Limits 0.6 0.4 02 0.0 (a) or 19M 1940 19M 1060 1970 10W 1990 7 (b) 6 4 2 . .. . ...... 1900 1940 1960 L960 1WM 19W 19M YEAR OF PUBUCA77ON YEAR OF PUBIWATION Psychophysical Research Laboratorles Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 I law Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 17 "REAL-TIME" ALTERNATIVES TO PRECOGNITON Investigators have long been aware of the possibility that precognition effects could be modelled without assuming either time reversal or backward causality. For example, outcomes from studies with targets based on indeterminate random number generators (RNGs) could be due to a causal influence on the RNG - a psychokinetic (PK) effect - rather than information acquisition concerning its future state. In experiments with targets based on prepared tables of random numbers, the possibility exists that the experimenter or other randomizer may be the actual psi source, unconsciously using "real-time" ESP combined with PK to choose an entry point in the random number sequence that will significantly match the dd subject's" responses. While this latter possibility may seem farfetched, it cannot be logically eliminated if one accepts the existing evidence for contemporaneous ESP and PY, and it has been argued that it is less farfetched than the alternative of "true" precognition. Morris (1982) discusses models of experimental precognition based on 44 real-time" psi alternatives and methods for testing "true" precognition. In general terms these methods constrain the selection of the target sequence so as to eliminate nonprecognitive psi intervention. In the most common procedure, attributed to Mangan (1955), dice are thrown to generate a set of numbers which are mathematically manipulated to obtain an entry point in the random number table. This procedure is sufficiently complex "as to be apparently beyond the capacities of the human brain, thus ruling out PK because the'PKer'would not know what to do even via ESP" (Morris, 1982, p. 329). Two features of precognition study target determination procedures were coded to assess "real-time" psi alternatives to precognition: ~ Method of determining random number table entry point, ~ Use of Mangan's method. Methods of eliminating "'real-time" psi alternatives have not been employed in studies with random number generators and have only been Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 18 used in a small number of studies involving randomintion by hand shuffling. These analyses are therefore restricted to studies using random number tables (N = 137). Method of Detennining RNT Entry Point The reports describe six different methods of obtaining entry points in random number tables. If the study outcomes were due to subjects' precognitive functioning rather than to alternative psi modes on the part of the experimenter or the experimenter's assistants, there should be no difference in mean effect size across the various methods used to determine the entry point. Indeed, our analysis indicates that the study effect sizes do not vary systematically as a function of method of determining the entry point (Kruskal-Wallis one-way ANOVA by Ranks: chi-square = 8.29,5 df, p = .14 1). Use of Mangan's Method We find no significant difference in effect size between studies using complex calculations of the type introduced by Mangan to fix the random number table entry point and those that do not use such calculations (t = 0.92, df = 77, p = .359, 2-tailed). Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 L Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 19 L MODERATING VARIABLES The stability of precognition study outcomes over a 50-year period is also bad news. It shows that investigators in this area have yet to develop sufficient understanding of the conditions underlying the occurrence (or detection) of these effects to reliably increase their magnitude. We have identified four variables that appear to covary systematically with magnitude of precognition performance: ~ Selected versus unselected subjects ~ Individual versus group testing ~ Feedback level 9 Time interval between subject response and target genera- tion We are interested only in factors associated with hitting; therefore, our analyses are based on the directional study outcomes only. The analyses use the raw study significance levels and effect sizes; this results in uniformly more conservative estimates of relationships with moderating variables than when the analyses are based on quality-weighted significance levels and effect sizes. ago Selected versus Unselected Subjects Our meta-analysis identifies eight subject populations: * Unspecified subject populations 9 Mixtures of several different populations * Animals * Students * Children 0 "Volunteers" o Experimenter(s) o Selected subjects Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 20 Iwo Effect size magnitude varies significantly across these eight subject low populations (Kruskal-Wallis one-way ANOVA, chi-square = 15.71, 7 df, P = .028). Significance levels and effect sizes by subject population are low displayed in Figures 5 and 6. FIGURE 5: Signiflcance Level by Subject Population (95% Confidence Limits.) 2.0 25 do --- -- ------ -- -------- -- ----------------- -2.0 Ok" OUBJECY PC>PUL.AV1C31N 0.0 0.0 -0.6 FIGURE 6: Effect Size by Subject Population (95% Confidence Limits.) low ---------------- 1 J-- -1 1 SUBJECY PC>PUL.AV1C>N Psychophysical Research Laboratories Approved For Release 2000/08/08 CIA-RDP96-00789ROO2200410001-2 0."1 so 0.100 0.076 0.080 0.028 0.000 -0.0*6 -0-060 0.07 -0.1 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 21 The difference across subject populations largely results from the superiority of studies with selected subjects: Studies using subjects selected on the basis of prior performance in experiments or pilot tests show larger effects than studies using unselected subjects. As shown in Table 6, 60% Nei percent of the studies with selected subjects are significant at the 5% level. The mean z-score for these studies is 1.41 (sd = 1.36). The magnitude of effect size is significantly higher for selected subjects studies than for studies with unselected subjects. The t-test of the difference in mean effect size is equivalent to a point-biserial correlation of .186. TABLE 6: Selected versus Unselected Subjects Subjects N studies Stouffer Z Mean ESdir SD % SIG.05 Selected 25 7.05 0.055 0.072 60.0% Unselected 223 4.70 0.012 0.068 21.0% tm = 2.97,P = .0015 Does this difference result from less stringent controls in studies with selected subjects? The answer appears to be "No." The average quality of studies with selected subjects is higher than studies using unselected subjects (t27 = 2.05, p = .051, 2-tailed). This result appears to reflect a general tendency toward increased rigor and more detailed reporting in studies with selected subjects. Individual versus Group Testing Subjects were tested in groups, individually, or through the mail. Studies in which subjects were tested individually by an experimenter have a significantly larger mean effect size than studies involving group testing (Table 7). The t-test of the difference is equivalent to a point-biserial correlation of .234, favoring individual testing. Of the studies with subjects tested individually, 30.0% are significant at the 5% level. Now Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 ow Approved For Release 2000108/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Cholce Precogriftlon Experlments 22 The methodological quality of studies with subjects tested individually is significantly higher than that of studies involving group testing (tJ43 = 3.5, p = .001, 2-tailed). This result is consistent with the conjecture that group experiments are frequently conducted as "'targets of opportunity," and may often be carried out hastily in an afternoon without the preparation and planning that goes into a study with individual subjects that may be conducted over a period of weeks or months. TABLE 7: Individual versus Group Testing Test Setting Nstudies StoufferZ Mean ESdir SD % SIG.05 Individual 98 7.24 0.029 0.074 30.6% Group 104 1.49 0.005 0.064 18.3% 12oo = 2.40, p = .0085 ism Thirty-five studies were conducted through the mail. In these studies, subjects completed the task at their leisure and mailed their responses to the investigator. These correspondence studies yield outcomes similar to those involving individual testing. The combined z-score is 3.01, with a mean effect size of 0.021 (sd = .079). Ten correspondence studies (28.6%) are significant at the 5% level. Eleven studies are unclassifiable with regard to experimental setting. Feedback low A significant positive relationship exists between the degree of feedback subjects receive about their performance and precognitive effect size (Table Subject feedback information is available for 95 studies. These studies fall into four feedback categories: No feedback, delayed feedback (usually notification by mail), run-score feedback, and trial-by-trial feedback. We gave these categories numerical values between 0 and 3. Precognition effect size coffelates.258 with feedback level (103 dfp = .004). Of the 48 studies Psychophysical Research Laboratorles Approved For Release 2000/08108: CIA-RDP96-00789ROO2200410001-2 Approved Vr UPO/08/pe : 'FIVDP96-007~91RO02200410001-2 gigelegs e c 9 ri -ana s 6~ddd ho ce recognftion xperiments 23 x involving trial-by-trial feedback, 21 (43.8%) are significant at the 5% level. None of the studies without subject feedback are significant. TABLE 8: Feedback Received by Subjects Feedback Nstudies StoufferzMean ESdir SDEs %SIG. Level 05 No Feedback 15 0.00 -0.002 0.0270.0% Delayed 21 2.27 0.009 0.03523.8% Run-score 21 4.80 0.024 0.04733.3% Trial-by-trial48 7.59 0.048 0.09443.8% While trial-by-trial feedback is associated with the largest effect sizes and significance levels, there is no evidence that subjects'performance improved over time. Feedback level correlates positively though not significantly with research quality (rim = .134, p = .145). Inadequate randomization is the most plausible source of potential artifacts in studies with trial-by-trial feedback. We therefore performed a separate analysis on the 48 studies in this group, blocking on the randomization and control quality measures. Studies with optimal randomization do not differ significantly in either mean significance level or mean effect size from those with suboptimal randomization. For significance levels, t is 0.74 with 46 df (P = .465, 2-tailed). For ES, t is 0.89 with 14 df (p = .525, 2-tailed). Similarly, studies reporting randomness control data do not differ significantly in either significance level or effect size from those not including randomness controls. For significance levels, t is 0.25 with 46 df (p = .803, 2-tailed). For ES, t is 1. 19 with 46 df (p = .241, 2-tailed). MON Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000108/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 24 "nme interval The interval between the subject's response and target selection ranges from less than one second to one year. Information about the time interval is available for 145 studies. This information, however, is often imprecise. Our analysis of the relationship between precognitive effect size and time interval is therefore limited to seven broad interval categories: milliseconds, seconds, minutes, hours, days, weeks, and months. Although it is confounded with the feedback variable, there is a significant decline in precognition significance levels and effect size over increasing temporal distance. Using significance levels, r is -.270 with 143 df (P = .00 1, 2-tailed). Using effect size r is -.206 (p .013, 2-tailed). The largest effects occur over the millisecond interval (N 31 studies, Stoufferz = 6.12, mean ES = 0.046, sd = .072). The smallest effects occur over periods ranging low from a week to a month (N = 17, Stouffer z = -.36, mean ES = -0.004, sd .032). Significance levels and effect sizes by precognitive interval are displayed in Figures 7 and 8. (The intervals are labelled numerically: 1 = msec., 2 = sec., 3 = min., 4 = hr., 5 = days, 6 = weeks, and 7 = months). Curiously, this finding results entirely from studies using unselected subjects (rim = -.238, p = .008, 2-tailed). Studies with selected subjects show a nonsignificant positive relationship betweem ES amd time interval (r18 = .081,p = .734, 2-tailed) and the difference between these two correlations is itself significant (z = 2.58,p = .01, 2-tailed). This suggests that the origin of the decline over time may be motivational rather than the result of some intrinsic physical boundary condition. The relationship between precognitive effect size and feedback also supports this conjecture. Nevertheless, any finding suggesting potential boundary conditions on the phenomenon should be vigorously pursued. Psychophysical Research Laboratories Approved For Release 2000/08108 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 25 FIGURE 7: Significance Level by Precognitive Interval (95% Confidence Limits.) 2.0 1A5 1.0 0.0 -0.5 -1.0 -2.0 2. 3. 5. 6. 7. ]PIRECOONITIVE 11%TTERVAIL Pq ~__3 FIGURE 8: Effect Size by Precognitive Interval (95% Confidence Limits.) low 0.10 0.08 0.043 0.04 0.02 0.00 -0.02 -0.04 _-0.08 -0.08 -0.10 ------------------------ I ------------- (31) (21) (19) (35) (15) (17) (7) 1 1 1 ------------------------------ 1. Z. 3. 4. 5. 0. 7. PRECOONITIVE IN'rERVAL Psychophysical Research Laboratorles Approved For Release 2000/08/08 CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 26 Influence of Moderating Variables In Combination The above analyses examine the impact of each moderating variable in isolation. In this final set of analyses, we explore their joint influence on Now precognition performance. For this purpose, we identify two subgroups of studies. One subgroup is characterized by the use of selected subjects tested individually with trial-by-trial feedback. We refer to this as the Optimal group (N = 8 studies). ne second group is characterized by the use of unselected subjects tested in groups with no feedback. We refer to this as the Suboptimal group (N = 9 studies). The Optimal studies are contributed by 4 independent investigators and the Suboptimal studies are contributed by 2 of same 4 investigators. All of the Optimal studies involve short precognitive time intervals (interval 1) while the Suboptimal studies involve longer intervals (intervals 5 and 6). All of the Optimal studies and 5 of the 9 Suboptimal studies use RNG methodology. The two groups do not differ significantly in average sample size. The mean study quality for the Optimal group is significantly higher than that of the Suboptimal studies (Optimal mean = 6.63, sd = 0.92; -6 Suboptimal mean = 3.44, sd = 0.53; t = 8.63, 10 df, p = 3.3 x 10 dim 2-tailed). Ile combined impact of the moderating variables appears to be quite no strong: 7 of the 8 Optimal studies (87.5%) are independently significant at the 5% level, while none of the Suboptimal studies are statistically significant. All four investigators contributing studies to the Optimal group now have significant outcomes. The mean z-score for the Optimal group is 2.17 (sd = 0.55) and for the Suboptimal group the mean z is -0.37 (sd = 1.05). ne difference is highly significant (t = 6.13, 12 df, p 2 x 10-5). The Optimal studies are also significantly less variable (F(7,8) 3.67, p = .046). In terms of effect sizes, the Optimal group is 9 times larger than the Suboptimal group (mean ES = 0.055, sd = 0.045 for the Optimal studies, and 0.006, sd 0.033) for the Suboptimal studies; this difference is also significant (t 2.60, 15 df, p = .0 1). These findings suggest that future studies combining these moderators should yield especially promising outcomes. Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 low Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice PrecWnftlon Experiments 27 DISCUSSION Our meta-analysis of forced-choice precognition experiments confirms low the existence of a small but statistically highly significant precognition effect. Most importantly, the effect appears to be replicable; significant confirmations are reported by two dozen investigators using a variety of methodological paradigms and subject populations. Estimates of the "filedrawer"' problem and consideration of parapsychological publication practices indicate that the precognition effect cannot be plausibly explained on the basis of selective publication bias. Analyses of precognitive effect sizes in relation to eight measures of research quality fail to support the hypothesis that the observed effect is driven to any appreciable extent by methodological artifacts; indeed, several of the analyses indicate that methodologically superior studies yield stronger effects than methodologically weaker studies. Analyses of parapsychological alternatives to precognition, although limited to the subset of studies using random number tables, provide no support for the hypothesis that the effect results from the operation of contemporaneous ESP and PK at the time of randomization. The most important outcome of the meta-analysis is the identification of several moderating variables that appear to covary systematically with precognition performance. The largest effects are observed in studies using subjects selected on the basis of prior test performance, who are tested individually, and receive trial-by-trial feedback. The outcomes of studies combining these factors contrast sharply with the null outcomes associated with the combination of group testing, unselected subjects, and no feedback of results. The identification of these moderating variables has important implications for our understanding of the phenomena and provides a clear direction for future research. The existence of moderating variables indicates that the precognition effect is not merely an unexplained departure from a theoretical chance baseline, but is rather an effect that covaries with factors known to influence more familiar aspects of human performance. It Iwo should now be possible to exploit these moderating factors to increase the magnitude and reliability of precognition effects in new studies. Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 I'm Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysls of Forced-Choice Precognition Experiments 28 Im V;Wle the overall precognition effect size is small, this does not imply that 1, am it has no practical consequences. It is, for "ample, of the same order of magnitude as effect sizes leading to the early termination of several major medical research studies. In 1981, the National Heart, Lung, and Blood Institute discontinued its study of propranolol because the results were so favorable to the propranolol treatment that it would be unethical to continue low placebo treatment (Kolata, 1981); the effect size is 0.04. More recently, The Steering Committee of the Physicians' Health Study Research Group low (1988), in a widely publicized report, terminated its study of the effects of aspirin in the prevention of heart attacks for the same reason. Ile aspirin group suffered 45% fewer heart heart attacks than a placebo control group; the associated effect size is 0.03. The search for mechanisms underlying the phenomenon would be advanced considerably if it were possible to compare the magnitude of the precognition effect with the effect sizes in "real-time" ESP studies involving similar testing methods. Tart (1983) claims a robust and highly significant difference favoring "real-time" ESP in a small subset of forced-choice precognition and "real-time" ESP studies. However, his analysis is limited to 85 statistically significant studies (53 studies of "real-time" ESP and 32 precognition studies). Confirmation of this finding through comparative analysis of all retrievable "real-time" and precognition studies would have great value in efforts to model the phenomena and, also, for developing more effective research methods. Furthermore, although it is frequently claimed that ESP is independent of distance, we believe the evidence usually put forward in support of this claim is very weak and that a more satisfactory conclusion can only be reached through assessment of all of the evidence. For these reasons, we recommend that priority be given to a comprehensive meta-analysis of "real-time" ESP studies. Psychophys/cal Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000108/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forcod-Cholee PrecognItlon Experiments 29 REFERENCES Akers, C. (1987). Parapsychology is science, but its findings are incon- clusive. Behavioral and Brain Sciences, 10, 566-568. Barnett, V., & Lewis, T. (1978). Outliers in statistical data. New York: John Wiley & Sons, Inc. Brownlee, K A. (1965). Statistical theory and methodology in science and engineering. New York: John Wiley & Sons, Inc. Dawes, R., Landman, J., & Williams, M. (1984). Reply to Kurosawa. American Psychologist, 38, 74-75. Kolata, G. B. (1981). Drug found to help heart attack survivors. Science, 214,774-775. Morris, R. L (1982). Assessing experimental support for true precogni- tion. Journal of ParapsycholoSy, 46, 321-336. Rosenthal, R. (1984). Meta-analytic procedures for social research. Bever- ly Hills, CA: Sage. Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance--or vice versa. Journal of the American Statistical Association, 54, 30-34. Tart, C. T. (1983). Information acquisition rates in forced-choice ESP ex- periments: precognition does not work as well as present-time ESP. Journal of theAmerican Societyfor Psychical Research, 77,293-310. The Steering Committee of the Physicians' Health Study Research Group. (1988). Preliminary report: Findings from the aspirin com- ponent of the ongoing Physicians' Health Study. New England Journal of Medicine, 318,262-264. Wilkinson, L (1988). SYSTA T- 77te system for statistics. Evanston, IL: SYSTAT, inc. Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Foroed-Cholce PrecognItlon Experlments 30 CHRONOLOGICAL LISTING OF META-ANALYSIS STUDY REFERENCES 1 CARINGTON, W. (1935). Preliminary experiments in precognitive guessing. Journal of the Society for Psychical Research, 29, 86-104. 2-5 RHINE, J. B. (1938). Experiments bearing on the precogni tion hypothesis: I. Pre-shuffling card calling. Journal of Parapsychology, 2, 38-54. 6-11 RHINE, J. B., SMrM, B. M., & WOODRUFF, J. L (1938). Experiments bearing on the precogni- tion hypothesis: H. The role of ESP in the shuffling of cards. Journal of Parapsychology, 2,119-131. 12 HUMPHREY, B. M., & PRATT, J. G. (1941). A com- parison of five ESP test procedures. Journal of Parapsychology, 5, 267-293. 13-16 RHINE, J. B. (1941). Experiments bearing upon the precog- nition hypothesis: III. Mechanically selected cards. Journal of Parapsychology, 5, 1-57. 17 STUART, C. E. (1941). An analysis to determine a test predictive of extra-chance scoring in card Im calling tests. Journal of Parapsychology, 5, 99- 137' 18-21 HUMPHREY, B. M., & RHINE, J. B. (1942). A confir- matory study of salience in precognition tests. Journal of Parapsychology, 6, 190-219. 22-25 RHINE, J. B. (1942). Evidence of precognition in the covariation of salience ratios. Journal of Para- psychology, 6, 111-143. 26 NICOI, J. F., & CARINGTON, W. (1947). Some experi ments in willed die-throwing. Proceedings of the Societyfor Psychical Research, 48, 164-175. Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 31 27-28 THOULESS, R. H. (1949). A comparative study of perfor- mance in three psi tasks. Journal of Parapsy- chology, 13,263-273. 29 - 32 - BAS FIN, E. W., & GREEN, J. M. (1953). Some experi- ments in precognition. Journal of Parapsy- chology, 17, 137-143. 33 MCMAHAN, E. A~, & BATES, E. K (1954). Report of further marchesi experiments. Journal of P arapsychology, 18, 82-92. 34 MANGAN, G. L (1955). Evidence of displacement in a precognition test. Journal of Parapsychology, 19,35-44. 35-36 OSIS, K (1955). Precognition over time intervals of one to thirty-three days. Journal of Parapsychology, 19,82-91. 37 NIELSEN, W. (1956). An exploratory precognition experi- ment. Journal of Parapsychology, 20, 33-39. 38-39 NIELSEN, W. (1956). Mental states associated with suc- cess in precognition. Journal of Parapsy- chology, 20, 96-109. 40-41 FAHLER, J. (1957). ESP card tests with and without hyp- nosis. Journal of Parapsychology, 21, 179-185. 42-44 MANGAN, G. L (1957). An ESP experiment with dual- aspect targets involving one trial a day. Jour- nal of Parapsychology, 21, 273-283. 45-46 ANDERSON, M., & WHITE, R. (1958). A survey of work on ESP and teacher-pupil attitudes. Journal of Parapsychology, 22, 246-268. 47 NASH, C. B. (1958). Correlation between ESP and religious value. Journal of Parapsychology, 22, 204-209. Psychophysical Research Laboratorles Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 L Meta-analysis of Forced-Choloe PrecognItion Experlments 32 L 48-49 ANDERSON, M. (1959). A precognition experiment L com- paring time intervals of a few days and one year. Journal of Parapsychology, 23, 81-89. L 50-53 ANDERSON, M., & GREGORY, E. (1959). A two-year program of tests for clairvoyance and precog- nition with a class of public school pupils. L Journal of Parapsychology, 23, 149-177. 54 NASH, C. B. (1960). Can precognition occur diametrical- ly? Journal of Parapsychology, 24, 26-32. 55 FREEMAN, J. A. (1962). An experiment in precognition. Journal of Parapsychology, 26, 123-130. 56 RHINE, J. B. (1962). The precognition of computer num- bers in a public test. Journal of Parapsy- t w o chology, 26, 244-251. 57 RYZI, M. (1962). Training the psi faculty by hypnosis. Journal of the Society for Psychical Research , 41,234-252. 58-60 SANDERS, M. S. (1962). A comparison of verbal and writ- ten responses in a precognition experiment. Journal of Parapsychology, 26, 23-34. 61-64 FREEMAN, J. (1963). Boy-girl differences in a group precognition test. Journal of Parapsychology, 27,175-181. 65-68 RAO, K R. (1963). Studies in the preferential effect H. A language ESP test involving precognition and "intervention". Journal of Parapsychology 27 , , 147-160. 69-70 FREEMAN, J. (1964). A precognition test with a high- school science club. Journal of Parapsy- chology, 28, 214-22 1. 71-73 FREEMAN, J., & NIELSEN, W. (1964). Precognition score deviations as related to anxiety levels. Journal of Parapsychology, 28, 239-249. Now Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 33 74-82 SCHMEIDLER, G. (1964). An experiment on precognitive clairvoyance. Part I. The main results. Jour- nal of Parapsychology, 28, 1-14. 83-90 FREEMAN, J. A. (1965). Differential response of the sexes to contrasting arrangements of ESP target material. Journal of Parapsychology, 29, 25 1- 258. 91-92 OSIS, K, & FABLER, J. (1965). Space and time variables in ESP. Journal of the American Societyfor Psychical Research, 59, 130-145. 93-94 FABLER, J., & OSIS, K (1966). Checking for awareness of hits in a precognition experiment with hyp- notized subjects. Journal of the American Societyfor Psychical Research, 60,340-346. 95-102 FREEMAN, J. A. (1966). Sex differences and target arran- gement: High-school booklet tests of precog- nition. Journal of Parapsychology, 30, 227-235. 103 ROGERS, D. P. (1966). Negative and positive affect and ESP run-score variance. Journal of Parapsy- chology, 30, 15 1-15 9. 104 ROGERS, D. P., & CARPENTER, J. C. (1966).. The decline of variance of ESP scores within a testing session. Joumal of Parapsychology, 30, 141-150. 105 BRIER, B. (1967). A correspondence ESP experiment with high-I.Q. subjects. Journal of Parapsychology, 31,143-148. 106-107 BUZBY, D. E. (1967). Precognition and a test of sensory perception. Journal of Parapsychology, 31, 135-142. 108-109 BUZBY, D. E. (1967). Subject attitude and score variance in ESP tests. Journal of Parapsychology, 3 1, 43-50. Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 1W Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 34 low 110-113 FREEMAN, J. A. (1967). Sex differences, target arrange- Imi ment, and primary mental abilities. Journal of Parapsychology, 31, 271-279. 114-118 HONORTON, C. (1967). Creativity and precognition scor- ing level. Journal of Parapsychology, 31, 29- 42. 119-120 CARPENTER, J. C. (1968). Two related studies on mood and precognition run-score variance. Joumal of Parapsychology, 32, 75-89. 121 DUVAJ, P., & MONTREDON, E. (1968). ESP experi- ments with mice. Journal of ParqMchology, 32, 153-166. 122-129 FEATHER, S. R., & BRIER, R. (1968). The possible effect of the checker in precognition tests. Journal of Parapsychology, 32, 167-175. 130-137 FREEMAN, J. A. (1968). Sex differences and primary mental abilities in a group precognition test. Journal of Parapsychology, 32, 176-182. 138-139 NASH, C. S., & NASH, C. B. (1968). Effect of target selec- tion, field dependence, and body concept on ESP performance. Journal of Parapsychology, 32P 248-257. 140-143 RHINE, L E. (1968). Note on an informal group test of ESP. Journal ofParapsychology, 32, 47-53. 144-146 RYZI, M. (1968). Precognition scoring and attitude toward ESP. Journal of Parapsychology, 32, 1-8. f 147-148 RYZI, M. (1968). Precognition scoring and attitude. Jour- nal of Parapsychology, 32, 183-189. 149-150 CARPENTER, J. C. (1969). Further study on a mood adjec- tive check list and ESP run-score variance. Journal of Parapsychology, 33, 48-56. I Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 35 151 DUVAL, P., & MONTREDON, E. (1969). Precognition in mice: A confirmation. Journal ofParapsy. chology, 33, 71-72. 152-155 FREEMAN, J. A. (1969). The psi-differential effect in a precognition test. Journal of Parapsychology, 33,206-212. 156 FREEMAN, J. A~ (1969). A precognition experiment with science teachers. Journal of Parapsychology, 33,307-310. 157-158 JOHNSON, M. (1969). Attitude and target differences in a group precognition test. Journal of Parapsy- not chology, 33, 324-325. 159 MONTREDON, E., & ROBINSON, A. (1969). Further law precognition work with mice. Journal of Para- psychology, 33, 162-163. L 160-162 SCHMIDT, H. (1969). Precognition of a quantum process. Journal of Parapsychology, 33, 99-108. 163 BENDER, H. (1970). Differential scoring of an outstanding subject on GESP and clairvoyance. Journal of Parapsychology, 34, 272-273. do 164-165 FREEMAN, J. (1970). Shift in scoring direction with junior- high-school students: A surnmary. Journal of Parapsychology, 34, 275. 166-169 FREEMAN, J. A. (1970). Ten-page booklet tests with elementary-school children. Journal of Para- psychology, 34, 192-196 170-171 FREEMAN, J. A. (1970). Mood, personality, and attitude in precognition tests. Journal of Parqpsy- chology, 34,. 322 172-175 FREEMAN, J. A. (1970). Sex differences in ESP response as shown by the Freeman picture-figure test. Journal of Parapsychology, 34, 37-46. Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved or Relqase3$00/08 8 : Cl~-RDP96-00JXNR002200410001-2 C L 4eta-analysis orccN-Rolce recognftlon penments 36 L 176-179 HARALDSSON, E. (1970). Subject selection in a machine precognition test. Journal of Parapsychology, 34,182-191. 180 HARALDSSON, E. (1970). Precognition of a quantum process: A modified replication. Journal of Parapsychology, 34, 329-330. 181-200 NIELSEN, W. (1970). Relationships between precognition scoring level and mood. Journal of Parapsy- chology, 34, 93-116. 201-202 SCHMIDT, H. (1970). Precognition test with a high-school group. Journal of Parapsychology, 34, 70. 203-204 BELOFF, J., & BATE, D. (1971). An attempt to replicate the Schmidt findings. Journal of the Society for Psychical Research, 46, 21-3 1. 205-206 HONORTON, C. (1971). Automated forced-choice l precog- ow nition tests with a "sensitive". Journal of the American Society for Psychical Research, 65, 476-481. 207 MITCHELL~ E. D. (1971). An ESP test from Apollo 14. Joumal of Parapsychology, 35, 89-107. 208-209 SCHMIDT, H., & PANTAS, L (1971). Psi tests with psy- chologically equivalent conditions and inter- nally different machines. Journal of Parapsychology, 35, 326-327. 210 STANFORD, R. G. (1971). Extrasensory effects upon of memory". Journal of the American Society for Psychical Research, 64, 161-186. 211-212 STEILBERG, B. J. (1971). Investigation of the paranormal gifts of the Dutch sensitive Lida T. Journal of Parapsychology, 35, 219-225. 213 THOULESS, R. H. (1971). Experiments on psi self-train- l ing with Dr. Schmidt's pre-cognitive ap- ow paratus. Joumal of the Society for Psychical Research, 46, 15-2 1. Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 low Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 37 214-215 CRAIG, J. G. (1972). Ile effect of contingency on precog- nition in the rat. Research in Parapsychology, 154-156. 216-217 FREEMAN, J. A. (1972). The psi quiz: A new ESP test. Research in Parapsychology, 132-134. 218-220 HONORTON, C. (1972). Reported frequency of dream recall and ESP. Journal of the American Societyfor Psychical Research, 66, 369-374. 221 JOHNSON, M., & NORDBECK, B. (1972). Variation in the scoring behavior of a "psychic" subject. Journal of Parapsychology, 36, 122-132. 222 KELLY, E. F., & KANTHAMANI, B. K. (1972). A subject's efforts toward voluntary control. Journal of Parapsychology, 36, 185-197. 223-226 SCHMIDT, H., & PANTAS, L (1972). Psi tests with inter- nally different machines. Journal of Parapsy- chology, 36,222-232. law 227 ARTLEY, B. (1974). Confirmation of the small-rodent precognition work. Journal of Parapsychology, 38,238-239. 228-230 HARALDSSON, E. (1974).. Reported dream recall, precognitive dreams, and ESP. Research in Parapsychology, 47-48. 231 HARRIS, S., & TERRY, J. (1974) Precognition in a water- deprived Wistar rat. Journal of Parapsy- chology, 38, 239. 232 RANDAII, J. L (1974). An extended series of ESP and PK tests with three English schoolboys. Jour- nal of the Societyfor Psychical Research, 47, 485494. 233 TERRY, J. C., & HARRIS, S. A. (1974). Precognition in low water-deprived rats. Research in Parapsy- chology, 8 1. L INN Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 38 234-235 EYSENCK, H. J. (1975) Precognition in rats. Journal of Parapsychology, 39, 222-227. 236-237 HONORTON, C., RAMSEY, M., & CABIBBO, C. (1975). Experimenter effects in extrasensory percep- tion. Journal of the Ametican Societyfor Psychical Research, 69, 135-149. LOW 238-243 KANTHAMANI, H., & RAO, H. H. (1975). Response ten- dencies and stimulus structure. Journal of Parapsychology, 39, 97-105. 244 LEVIN, J. A. (1975). A series of psi experiments with ger- bils. Journal of Parapsychology, 39, 363-365. 245-248 NEVILLE, R. C. (1975). Some aspects of precognition test- ing. Research in Parapsycholov, 29-3 1. 249-251 DAVIS, J. W., & HAIGHT, J. (1976). Psi experiments with rats. Journal of Parapsychology, 40, 54-55. 252-256 JACOBS, J., & BREEDERVELD, H. (1976). Possible in- fluences of birth order on ESP ability. Res l am Letter, No. 7, 10-20. 257-263 DRUCKER, S. A., DREWES, A. A., & RUBIN, L (1977). ESP in relation to cognitive development and IQ in young children. Journal of the American Societyfor Psychical Research, 71, aw 289-298. 264-268 HARALDSSON, E. (1977). ESP and the defense mechanism test (DMT): A further valida- tion. European Journal of Parapsychology, 2, 104-114. 269-272 SARGENT, C. L (1977). An experiment involving a novel precognition task. Journal of Parapsychology, 41,275-293. 273-276 BIERMAN, D. J. (1978). Testing the "advanced wave" hypothesis: An attempted replication. European Journal of Parapsychology, 2, 206- 212. Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Cholce Precognftion Experlments 39 277-278 O'BRIEN, J. T. (1978). An examination of the checker ef- fect. Research in Parapsychology, 153-155. 279 BRAUD, W. (1979). Project chicken little: A precognition experiment involving the SKYLAB space sta- tion. European Journal of Parapsychology, 3, 149-165. 280-281 CLEMENS, D. B., & PHIUIPS, D. T. (1979). Further studies of precognition in mice. Research in Parapsychology, 156. 282 HARALDSSON, E., & JOHNSON, M. (1979). ESP and the defense mechanism test (DMT) Icelandic study No. 111. A case of the experimenter ef- fect? European Journal of Parapsychology, 3, 11-20. 283-285 HARALDSSON, E. (1980). Scoring in a precognition test as a function of the frequency of reading on psychical phenomena and belief in ESP. Res Letter, No. 10, 1-8. 286 SARGENT, C., & HARLEY, T. A. (1981). Three studies using a psi-predictive trait variable question- naire. Journal of Parapsychology, 45, 199-214. 287-290 THALBOURNE, M., BELOFF, J., & DELANOY, D. (1981). A test for the "extraverted sheep ver- sus introverted goats" hypothesis. Research in Parapsychology, 155-156. 291-292 WINKELMAN, M. (1981). The effect of formal education on extrasensory abilities: The Ozolco study. Journal of Parapsychology, 45, 321-336. 293 NASH, C. B. (1982). ESP of present and future targets. low Journal of the Societyfor Psychical Research, 51, 374-377. 294 SCHWARTZ, S. A~, & DEMATTEI, R. J. (1982).. The Mobius psi-Q test: Preliminary findings. Re- search in Parapsychology, 103-105. Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 40 295-296 CRANDALL, J. E., & HITE, D. D. (1983).. Psi-missing lift and displacement: Evidence for improperly focused psi? Journal of the American Society for Psychical Research, 77, 209-228. 1 M , TEDDER, W. (1983). Computer-based long distance 297 ESP: An exploratory examination (RB/PS). Re- search in Parapsychology, 100-101. 298 JOHNSON, M., & HARALDSSON, E. (1984). The defense mechanism test as a predictor of ESP scores. Icelandic studies IV and V. Journal of Parapsychology, 48, 185-200. 299 HARALDSSON, E., & JOHNSON, M. (1985). The defense mechanism test (DMT) as a predic- tor of ESP performance: Icelandic studies VI and VII. Research in Parapsychology, 43-44. 300-303 HESELTINE, G. L. (1985). PK success during structured and nonstructured RNG operation. Journal of Parapsychology, 49, 155-163. 304-308 VASSY, Z. (1986). Experimental study of complexity de- pendence in precognition. Journal of Parapsy- chology, 50, 235-270. 309 HONORTON, C. (1987). Precognition and real-time ESP performance in a computer task with an ex- ceptional subject. Journal of Parapsychology, 51, 291-320. MW Psychophysical Research Laboratories Approved For Release 2000/08/08 CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 International Qltnter 0 January 31, 1989 Ms. Jean V. Smith. Contracting Officer Department of the Army U.S. Army Medical Research And Development Command Ft. Detrick, Frederick, Maryland 21701-5014 Dear Ms. Smith: Enclosed please find one copy of the Project 1291 Final Report, Objective I, Task 1, "Meta-Analysis Of Forced-Choice Precognition Experiments." You now have one copy of each of the following nine FY 1988 report deliverables: Project 1291, Contract Number DAMD 17-83-C-3106 FY 1988 Report Deliverables Objective A, Task 3, Final Technical Report, "Enhanced Human Performance Investigation" Annual Administrative Report, "Enhanced Human Performance Investigation" Objective B, Task 1, Final Report, "Mass Screening For Psychoenergetic Talent Using A Remote Viewing Task" Objective D, Task 1, Final Report, "Neurophysiological Correlates To Remote Viewing" Objective E, Tasks I and 2, Final Report, "Feedback And Target Dependencies In Remote Viewing Experiments" Objective E, Task 3, Final Report, "The Effects Of Hypnosis On Remote Viewing Quality" Objective E, Task 4, Final Report, "Forced-choice Remote Viewing" Objective F, Task 1, Final Report, "Applications Of Fuzzy Sets To Remote Viewing Analysis" Objective I, Task 1, Final Report, "Meta-Analysis Of Forced-Choice Precognition Experiments" Sincerely, ather' A. Flowers Adthe Projec Coordinator ProietcCc Enclosure cc: Dr. Murray J. Baron Mr. James 0, Dolen Dr dwin C. May SG1J SRI International 333 Ravenswood Ave. 9 Menlo Park, CA 94025 e (415) 326-6200 * TWX: 910-373-2046 Telex: 334486 Facsimile: (415) 326-5512 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 aw I" Ow aw NNW aw aw Ow Ow mow Ow Ow .MW ,w Ali Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2200410001-2 mow