iW \hjLernationall/ rJved For Release 2000/08/08 CIA-RDP96-00788ROO1800080001-5 0 A REMOTE VIEWING EVALUATION PROTOCOL (U) December 1982 Final Report REVISED JULY 1983 MANAGING TOTAL QUALITY 3M QUALITY MANAGEMENT SERVICES ell 612/778-7560 --Nmmm-- 0e-14 This document consists of 48 pages. Coe 333 Ravenswood Ave. Menlo Park, CA 94025 TWX- 910-373-20,16 Plex 334 -486 Approved For Release 2000/08/08 CIA-RDP96-00788ROO1800080001-5 Approved For Release 24M 4ftfifWFW()0788R001 800080001-5 CONTENTS (U) LIST OF ILLUSTRATIONS (U) . . . . . . . . . . . . . . . . . . . . LIST OF TABLES (U) . . . . . . . . . . . . . . . . . . . . . . . I OBJECTIVE (U) . . . . . . . . . . . . . . . . . . . . . . . II INTRODUCTION (U) . . . . . . . . . . . . . . . . . . . . . . III ANALYSIS PROTOCOL (U) . . . . . . . . . . . . . . . . . . . IV CONCLUSIONS (U) . . . . . . . . . . . . . . . . . . . . . . Appendix A--EVALUATION OF FOUR COORDINATE REMOTE VIEWINGS (U). Appendix B--SUMMARY OF EVALUATION TECHNIQUE (U) . . . . . . . .... REFERENCES (U . . . . . . . . . . . . . . . . . . . . . . . . . . UNCLASSIFIED iv 2 4 11 12 43 44 Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 UM.LAbb1HtV Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 FIGURES (U) I Sample RV Assessment Form (U) . . . . . . . . . . 5 . . . . . . A-1 Transcript 1 (U) . . . . . . . . . . . . . . . . . 14 . . . . . A-2 Transcript 2 (U) . . . . . . .. . . . . . . . . . 15 . . . . . . A-3 Transcript 3 (U) . . . . . . . . . . . . . . . . . 16 . . . . . A-4 Transcript 4 (U) . . . . . . . . . . . . . . . . . 17 . . . . . A-5 Target Site Hang6 (U) . . . . . . . . . . . . . . 19 . . . . . . A-6 Target Site Fernando de Noronha (U) . . . . . . . 20 . . . . . . A-7 Target Site Inverness (U) . . . . . . . . . . . . 21 . . . . . . A-8 Target Site Punkaharju (U) o . . . . . . . . . . . 22 . . . . . UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 Approved For Release 2UM ' IL 2^646SIMR-00788 ROO 1800080001-5 TABLES (U) I Task-Defined Relevance ScaleTarget for Elements (U) . . . . 2 Quality Assessment Scale . . . . . . . . . . (U) . . . . . . 3 Numerical Score Conversion (U). . . . . . . . . Table . . . A-1 Target Element Relevance forHangb (U) . . . . Ratings . . . A-2 Target Element Relevance forInverness (U) Ratings A-3 Target Element Relevance forFernando de Ratings Noronha (U) . . . . . . . . . . . . . . . . . . . . . . . . . A-4 Target Element Relevance forPunkaharju (U) . Ratings . . . . A-5 RV Assessment Forms for Calibration of Remote Viewing (U). . A-6 A Rank Ordering of Weighted (U) . . . . . . . Averages . . . iv UNCLASSIFIED 6 8 8 23 23 24 24 25 41 Approved For Release 2000/08/08 CIA-RDP96-00788ROO1800080001-5 Approved For Release 20fJfJ&fi"[MWO788 ROO 1800080001-5 I OBJECTIVE (U) (U) The objective of this task was to develop an evaluation procedure to assess the relative quality of a set of different remote viewing (RV) responses. UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 Approved For Release 2000/0.8/08 : CIA-.RDP96-00788ROO1800080001-5 UNCLASSIFIED II INTRODUCTION (U) (U) In addressing the remote viewing (RV) evaluation question, we consider two separate requirements: 6 Absolute evaluation of a single response for a single unknown target. a Relative evaluation of a set of responses for a series of known targets. The first of these is of the most interest in an operational setting. As part of the RV enhancement task, we have considered this problem in two ways. First, by conducting an operational RV session between twwo calibra- tion RV sessions, a tentative a priori assessment of operational efficacy can be determined. The evaluation is made on the basis of performance during the calibration sessions, and on the basis of adherence to a pre- determined session structure. (U) A second technique for an a priori evaluation was explored as part of the FiscalYear 1982 program in an audio-linguistic task. This task provided indications that careful linguistic analysis, when coupled with technical audio analysis, could yield an assessment in the absence of knowledge about the target. (U)# Various techniques have been used in the past 1* in an attempt to solve the relative evaluation problem. The most common of these was the simple rank ordering of all responses, as assessed against all possible targets used in an experimental series. In this procedure, a judge is presented with n RV transcripts and n target sites, His task is to arrange (U) References are listed at the end of this report. 2 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 Approved For Release 20071~8 CIA-RDf 6-00788R001 800080001-5 (U) the transcripts in order of the best to least match for each of the n targets. A simple numerical counting procedure is then used to estimate the likelihood that the judge's transcript/target-matches are by chance alone.2 This early technique contained little systematic structure for determining the final order of matches. The first step toward systematizing the rank order judging procedure was to preprocess the raw data in the transcript by "concep- tualizing" both the verbal and the pictorial responses. Conceptualizing a transcript requires an analyst to paraphrase the transcript into a list of coherent statements. This concept list is then compared and scored concept-by-~concept to each of the targets in the experiment. The resulting scores are;averaged for each response, and all responses are rank-ordered on the basis of these scores.a This improved analysis procedure was applied to a number of experiments within the Technology Transfer Taskt,,4,: (U) The problem with the above technique is that there are no guide- r so lines as to how the analyst should paraphrase the transcript; furthermore, the method in which the concepts are to be assessed against the targets remains undefined. The purpose of the Evaluation Task in FY 1982 was to identify a procedure that corrected these deficiencies. Approved For Release 2000/08/08: CIA-RDP96-00788 ROO 1800080001-5 Approved For Release 2000 1MR-AS-SHUDD788RO01 800080001-5 III ANALYSIS PROTOCOL (U) M To quantify the analysis procedure, we have divided the task into four separate areas: subject response, target/task definition, quality assessment, and numerical analysis. Figure I is a sample RV Assessment Form that has been designed to emphasize the separation of the analysis tasks. Each of the parts of the form are described below. (U) The subject's response should be prepared for analysis without any knowledge of either the target site or the overall task. The aim of this method of response preparation is to reduce a possibly redundant, rambling response to a coherent set of concepts. To meet this requirement we have developed a set of initial guidelines to the conceptualization procedure. (U) A concept is defined as a paraphrase of a single idea that has been expressed in the RV verbal or drawing response. That coherent idea should not be fragmented into component parts. For example, a response might be of thp form, "I see a large, textured, gray building." The single concept that expresses this idea should be "large, textured, gray building," rather than four separate concepts--one for each word in the phrase. Each concept should be entered under the "Transcript Concept" column in the RV Assessment Form. M) For this initial evaluation technique, a particular concept should be used only once in the analysis. (Some weighting factor propor- tional to concept frequency could be utilized, but, for the initial attempt, only unique concepts are used.) If in the construction of the transcript concept list a concept later in the transcript is a duplicate of an earlier one, it should be so noted by placing the concept number of the original concept in the "D" column. 4 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 7--M -71~1'= r 0 (D -n 0 (D (D M Q Q Q Q 00 Q 00 0 > 0 -V 6 CD 4 00 00 X Q Q L 00 Q Q Q 00 Q Q Q 73, 01 C z > CA Begin Time: End Time: Length of Session: ZO M E. U E- Target: Method of Targeting: Session: Viewer: Date: Class: concept Quality D P 'ale Numberranscript Concept lement of Target Vance(Q) core (R) 1 2 3 4 5 6 7 9 10 11 12 13 14 15 Score of Concepts 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Normalized PXRAscoreScore 0 0 0.00 0.36 2 2 0.71 3 3 1.07 4 4 1.43 5 5 1.79 6 6 2.14 7 2.50 9 2.86 10 3.21 12 10 3.5 7 is 11 - 3.93 16 12 4.29 20 13 4.64 25 14 5.00 Weighted Average 0 0 (D (D 0) 6 Q -4 00 00 X Q Q -L 00 Q Q Q 00 Q CD CD FIGURE 1 SAMPLE RV ASSESSMENT FORM (U) Approved For Release N/W8 S-MSAI FJEQ6-00788RO01 800080001-5 (U) To utilize an analysis procedure that is capable of quantitative assessmentp it is necessary to define, in advance, what the goals of the assessment are. In the RV Assessment Form, columns "Element of Target" and "Relevance" are provided to clearly define the goal of the analysis. In the ideal situation, an RV target should be completely specified in advance. A target typically consists of a number of target elements, each of which may have varying relevance with regard to the overall RV task. For any given target, an independent list of target elements should be prepared. The selection of what constitutes a target elementtis left completely to the discretion of the task coordinator. The target element must be selected with little regard to task relevance (target element relevance is accounted for later). Because an RV target consists, in principle, of an essentially infinite number of possible elements, discre- tion needs to be exercised in the selection process. (U) For each target element identified for the site, the task coordinator must define a relevance rating. This rating allows the coordinator to tailor the analysis to the task requirements. Table I shows the scale that is used for the target element relevance rating. Table 1 W TASK-DEFINED RELEVANCE SCALE FOR TARGET ELEMENTS (U) Rating Relevance Scale I A targetelementof trivial interest 2 A targetelementof minor interest 3 A targetelementof intermediate interest 4 A targetelementof major interest 5 A targetelementof key interest 6 UNCLASSIFIED Approved For Release 2000/08108 : CIA-RDP96-00788RO01 800080001 -5 Approved For ReleaseUNQAU.1r4AQ-00788RO01 800080001-15 (U) For each transcript concept on the RV Assessment Form (Figure 1), the analyst should attempt to find the element on the list of target elements that he/she considers to be the best match. The analyst should be quite liberal in the concept/element matching (i.e., the quality of the match should be considered at this point in the analysis). If he/she is able to identify a target element that might be considered a match to the given concept, a I is placed in the "p" (present) column on the assessment form. If no element can be identified, a 0 is placed in the "p" column. After making a target element identification, the selected target element, and its corresponding overall relevance rating should be entered in the appropriate columns on the assessment form. (U) Having identified a corresponding target element for each con- cept, it is now appropriate to assess the quality of the match. The quality assessment is done on the basis of how well the single concept in question matches the selected target element. The judgement is to be made without regard to any other issues, such as importance of the concept to the transcript, or importance (relevance) of the target element to the target. Table 2 shows the quality assessment scale that is used for this part of the analysis. The appropriate quality score from Table 2 is entered in the "Quality" column on the RV Assessment Form for each concept for which a matching target element has been identified. (U) An intermediate numerical score is computed for each concept from the relevance and quality (Tables I and 2) evaluation as follows: S' = P X R x Q where P is the value in the " p" column (0 or 1); R is the relevance evaluation; and 0 is the quality assessment. S' can assume values ranging between 0 and 25. Table 3 demonstrates how to determine the final score, S, for a given value of S1 for each concept. The conversion table is 7 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788RO01 800080001 -5 Approved For Releas qIfff&6-00788R001 800080001 -5 U"CUAIS Table 2 (U) QUALITY ASSESSMENT SCALE (U) Rating- Discrimination Scale I Poor description; only one or two aspects of the material match. 2 Fair description; a few aspects of the material match, but a large ambiguity exists. 3 Reasonable description; many aspects of the material match, but there remains some ambiguity. 4 Good description; a large number of aspects of the material matches, but it is possible to conceive of material that would be a better match. 5 Excellent description; all or nearly all aspects of the material match. Table 3 (U) NUMERICAL SCORE CONVERSION TABLE (U) S P x R x Q Score Normalized Score - 0 0.00 0 1 1 0.35 2 2 0.71 3 3 1.07 4 4 1.43 5 5 1.79 6 6 2.14 8 7 2.50 9 8 2.86 10 9 3.21 12 10 3.57 15 11 3.93 16 12 4.29 20 13 4.64 25 14 5.00 8 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 Approved For Release 2 C JU- 0788ROO1800080001-5 OefffetAgrita) (U) used to eliminate the nonuniformly-distributed gaps in scoring numbers that occur if one simply uses the product S'. Thus, the final score for each concept ranges from 0 to a maximum of 14. This conversion table is provided as part of the RV Assessment Form. (U) If an assessment of an individual concept is required, the final score for each concept/target-element match can be related to the quality assessment scale by using the conversions shown in the third column of Table 3 and on the assessment sheet. It should be noted, however, that the integer scores are used to simplify the remaining calculations. M To determine a final evaluation of the complete transcript assessed against a given target, a weighted average of concept scores is computed. To assist in the calculation of the weighted average, a tally box score is provided at the bottom of the RV Assessment Form. For each of the possible scores, 0 through 14, the number of concepts that attained that particular score are counted. For example, if 3 concepts were evaluated with a score of 12, a 3 is entered in the box below the 12 score. If the frequency of occurrence of score S i is f,, then the final weighted average is computed by I - - Ak = Z f i'/fiX Si/Z fi I/T Ak = 0.357 A' k (U) The normalized, weighted average score., Ak, is entered in the weighted average box on the assessment sheet. The weighted average score has been normalized to be within the range 0 < Ak -~ 5.0 To aid in the interpretation of the result, the quality assessment scale (Table 2) can be used to assess quality of the match between the whole RV response and the given target site. (U) At this point in the evaluation protocol, the following options are available, depending on the task requirement: 9 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 Approved For Release IWA -r-mM-00788ROO1800080001-5 (U) An n X n rank ordering on the basis of the weighted averages. A simple selection of the best match. A statistical evaluation on a concept-by-concept basis. 10 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 Approved For Release 2000t~08' CIA-RDP96 00788RO01 800080001-5 77~ IV CONCLUSIONS (U) A protocol has been developed to address the relative evaluation portion of the overall RV transcript assessment problem. As a demonstra- tion of the technique, we provide in Appendix A an analysis of a series of four remote viewings that were performed as calibrations , N In this series the remote viewing products were of relatively high quality, but nonetheless require a sensitive technique to differentiate because of the similarity of the targets and, hence, of the descriptions. (The series was chosen primarily for that reason.) Application of the assessment technique resulted in the correct, blind matching (highest scoring in matches versus cross matches) of three of the four. (U) Appendix B is a one page, step-by-step procedure for the application of this evaluation technique. (U) The material in this document thus constitutes an instruction 1W manual or protocol for application of a step-by-step procedure for quan- titative assessment of the relative target/transcript correlations of a series of transcripts matched into a series of targets. r MW r C Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 Approved For Release 200 Ufft LWSVKIW 88R001800080001-5 Appendix A EVALUATION OF FOUR COORDINATE REMOTE VIEWINGS (U) 12 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 Approved For Release 2000/0 A-RDP~~96-O 788ROO1800080001-5 i-C I )on 14 December 1981, four coordinate remote viewings were con- ducted as calibrationA These four calibrations were chosen as a test bed for the evaluation procedure for the following Obp I #Ce_-h*OYi reasons, (1) they were conducted in ani, setting, and (2) the targets had many similar features, and would thus provide a sensitive test of the protocol. Figures A-1 through A-4 are the transcripts that were presented to the analyst. They are exactly as they were when collected, except that the coordi'nates have been removed. Figures A-5 through A-8 are the National Ge6graphic magazine targets that were used during the calibration sessions. Finally, the task coordinator provided Tables A-1 through A-4 as target element relevance scales for the four targets in Figures A-5 through A-8. This completes the information that was given to the analyst, and thus the analysis was carried out blind as to the matching target/ transcript pairs. (U) Table A-5 is a compilation of the completed work sheets that were used by the analyst in this evaluation. They are shown in groups by session number, and alphabetized on the four targets. (The task coordinator first randomized the transcript order then assigned the session number used above.) For each of the transcripts, the analyst simply included all phrases and all drawings as concepts. For example, seven concepts were found during Session 2. (U) All concepts were then analyzed as described in the text. The matching target element, its relevance rating, and the computed score are shown for all possible combinations of transcript/target pairs in Table A-5. The score distributions and their resulting weighted averages are also shown in Table A-5. 13 C Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 Approved For ReleaseUNUASSME06-00788RO01 800080001-5 Iq &(TTI 0 00A T V" 5 ? I *~ . q 0 Pam ef FIGURE A-1 TRANSCRIPT 1 (U) 14 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 Approved For Release 00788ROO1800080001-5 T.2ec I&/ 001D. u 3.23 FIGURE A-2 TRANSCRIPT 2 (U) 15 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 Approved For Release VN4MqkSSMfiB6-00788P0q4O00OO1-5 _n t c tj FIGURE A-3 TRANSCRIPT 3 (U) 16 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788 ROO 1800080001-5 Approved For Release Uhl)"Wr -00788R001 800080001-5 Qt~ yte - FIGURE A-4(a) TRANSCRIPT 4(U) 17 UNCLASSIFIED cr Vj a Approved For Release 2000/08/08 CIA-RDP96-00788ROO1800080001-5 /vc;ik 0 Approved For Release2UfMAS!SIFVE%00788ROO1800080001-5 3 13 CIS 5 FIGURE A-40) TRANSCRIPT 4 (concluded) (U) 18 UNCLASSIFIED Approved For Release 2000/08/08 CIA-RDP96-00788ROO1800080001-5 Cp,~RgWecl For Release UF41ffAfm-f?fb-00788ROO1800080001-5 Lu D 0 (3 z T- LLJ c/) Lij LO Lli LL UNCLASSIFIED Approved For Release 2000/08/08 CIA-RDP96-00788ROO1800080001-5 Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 CPYRGHT UNCLASSIFIED "!ZQ Lu LL LLJ uj u- LL -Ot u z D 20 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 CPYRGHT UNCLASSIFIED ui z LLJ LIJ 0 01, 11P ri 11L LL . C A ca r- M ~A UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 Now- Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 I CPYRGHT UNCLASSIFIED UNCLASSIFIED FIGURE A-8 TARGET SITE PUNKAHARJU tU) a UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 Approved For Release 1JNG6A-WJR~-00788 ROO 1800080001-5 Table A-1 (U) TARGET ELEMENT RELEVANCE RATINGS FOR HANW (U) Target ElementRelevance Town 5 Cold 4 Peninsula 5 Rocky 3 Vegetation 2 Bay 3 Table A-2 (U) TARGk ELEMENT RELEVANCE RATINGS FOR INVERNESS (U) Target ElementRelevance City 5 River 4 Bridge 3 River banks 2 Vegetation 1 23 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 Approved For Release 201JjNQ-A&5WfiDO788 ROO 1800080001-5 Table A-3 (U) TARGET ELEMENT RELEVANCE RATINGS FOR FERNANDO DE NORONHA (U) Target Element Relevance Island 5 Surf 3 Hills 4 Uninhabited 3 Mountain peak 3 Temperate climate2 Vegetation 2 Ocean 4 Table A-4 (U) TARGET ELEMENT RELEVANCE RATINGS FOR PUNIKAHARJU (U) Target ElementRelevance Connect lakes 5 Town 4 Bridges 2 Cold 4 Vegetation 1 Islands 4 24 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 Begin Time: 12:40 End Time: Length of Session: Target: Inverness Method of Targeting: Coordinates Session: I Viewer: #002 Date: 14 December 1981 Class: a concept VAN:*Quality RumberTranscript Concept D F Element of Target $car* 1 Picture 1 1 Twin church towers1 1 .1 2 Picture 2 1 1 3 Up and down I Buildings 5 5 0 4 Rocky 0 5 Land/water interface 1 River 4 4 12 1 6 Picture 3 1 Twin church towers1 1 ~ 1 7 Picture 4 1 River 4 4 12 12 $. 8 Uprising I Buildings 5 1 5 5 9 Cliff 0 0 10 Fjords 1 River 5 1 5 11 Coastal city I I City 5 4 20 ' 12 13 14 is PAXQ goov* 0 0 1 1 2 2 3 3 4 4 1 5 5 6 6 8 7 9 8 10 9 12 10 25 11 26 22 20 13 Score 0 1 2 3 4 5 6 7- 8 9 10 11 12 11 141 of Concepts Weighted Average 2 5 3 2 1 J TABLE A-5 RV ASSESSMENT FORMS FOR CALIBRATION OF REMOTE VIEWING (U) > -a 0 (D CL -n 0 (D (D A) (1) (D Q 00 00 X C) C) -L 00 Q Q Q 00 Q Q 73, 0 (D CL -n 0 (D (D M C) C) C) C) 00 C) 00 0 > 6 C) 4 00 00 -A 00 CD 00 CD CD CD 73, C z > N3 M Begin Time: 12:40 End Time: Length of Session: 8 04 1-4 r3 Target: liangZ; Method of Targeting: Coordinates Session: 1 Viewer: #002 Date: 14 December 1981 Class: B Role- cowept D P vane*Qualityftore Nu"erTranscript Concept lement of Target I (R) (Q) 1 Picture 1 1 Rocks 3 3 -8 Picture 2 1 2 3 Up and down 1 Buildings 5 1 5 1 4 Rocky 1 Rocks 3 5 11 - 5 Land/water interfaces 1 Coast 5 4 13 1 Church 2 1 2 6 Picture 3 7 Picture 4 1 Bay 3 2 6 8 Uprising I Slope of land 3 1 3 1 9 Cliff 1 Coast 5 2 9 1 10 Fjords 1 Bay 3 1 3 11 Coastal city 5 5 14 1 City 12 t 13 14 15 Score of Concepts P)(2xQ Bea" 0 0 1 1 2 2 3 3 - 4 4 5 5 6 8 7 9 a 10 9 12 10 15 11 16 12 20 13 0 1 2 3 4 5 6 7 .8 9 10 11 12 11 141 UTi-ghted Average F 2.52 4 2 1 2 1 1 1 1 1 1i 0 (D -n 0 (D (D ~z t^ C) C) -4 00 00 -A 00 CD CD 00 CD 73, CD C" Table A-5 (continued) ~j Begin Time. 12:40 End Time: Length of Session: ca Em 1 Target. Fernando de Noronha Method of Targeting: Coordi rixtes Session: Role- concept D P vane*QualityBeare Numberranscript Concept lement of Target (Q) I Picture I I Hills 4 4 12 2 Picture 2 3 Up and down 1 Peak 3 1 3 4 Rocky I Coast line 1 5 5 5 Land/water interface 1 Island 5 4 13 6 Picture 3 1 Peak 3 2 ~ 6 7 Picture 4 1 Island 5 1 5 a Upwising 1 Hills 4 2 7 7 9 Cliff I Peak 5 3 11 10 Fjords I Inlets 3 2 6 11 0 Coastal city 0 12 13 14 Score 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 of Concepts 1 2 3 1 2 1 1 1 1 Viewer: #002 > Date: 14 December 1981 Class: n 0 .< PXRXQSao" (D CL -n 0 0 0 (D (D 2 2 3 3 4 4 5 3 6 6 8 7 1 9 8 109 1210 C) C) 1511 -4 - 00 00 1622 X C) 2013 C) -L 00 CD CD CD 00 CD Weighted Average CD 2 -& 36 (Y1 Table A-5 (Continued) WM 0 (D CL -n 0% (D W (D C) C) CD 00 Q 00 0 > 6 C) 4 00 00 X C) C) -L 00 C) C) C) 00 Q CD CD 73, z ti > (A M NJ M Begin Time: 12:40 End Time: Length of Session: Target: Punkaharju Method of Targeting: Coordinates Session: I Viewer: #002 Date: 14 December 1981 Class: B cou"Pt Role- D P vancoQualitygrore ranscript Concept lement of Target (Q) Picture 1 0 0~ 2 Picture 2 1 0 3 Up and down 1 Town 4 1 4 4 Rocky 0 0 5 Land/water interface I Connected lakes 5 4 13 6 Picture 3 0 0 - 7 Picture 4 1 Connected lakes 5 2 9 8 Uprising 0 0 9 Cliff 0 0 ~ 10 Fjords I I Connected lakes 5 1 5 I 11 Coastal city I Town 4 3 10 4 12 13 14 15 PXRXQ Bear* 0 0 1 1 2 2 3 3 4 4 5 5 6 8 7 9 a I 10 9 22 10 15 11 16 12 20 13 Score 0 1 2 3 4 6 7 18 9 13 14 10 Weighted Average 0.9o of Concepts 5 2 1 1 1 > 0 (D CL -n 0 (D U A) CO (D C at 100 C) C) -4 00 00 X C) C) -L 00 CD CD Q 00 CD CD CD Table A-5 (continued) 0 -n 0 (D (D M Q Q Q Q 00 Q 00 0 > 0 6 Q 4 00 00 ;U Q Q -L 00 Q Q Q 00 Q CD CD 73, z E. r) 4 rmme > E- M Begin Time: 15:23 End Time: Length of Session: Target: Inverness Method of Targeting: Coordinate 2 Session: Viewer: #002 Date: 14 December iAgi Class: B Rele- coweptTranscript Concept D P Element of Target Tar"Qua2ltyScore Number (R) (Q) or* 1 Picture 1 1 River bank 4 3 10 10 ~ 2 Straight angles 1 Buildings 5 1 5 11 3 Picture 2 1 River 4 3 10 11 4 River 1 River 5 5 14 5 Buildings I Buildings 5 5 14 6 Man-made 1 City 5 4 13 7 London I City 5 4 13 9 10 12 13 14 15 Score 0 3 4 5 6 7 8* 9 10 11 12 13 14 Weighted Average 4.13 of Concepts 2 2 2 2 2 2 0 < (D CL -n 0 (D (D Q -4 00 00 ;U Q Q -L 00 Q Q Q 00 CD CD CD 73, cyl TAble A-5 (continued) __ __- a I z > (A WOMEN Begin Time: 15:23 End Time: Length of Session: U Target: HangS Method of Targeting: Coordinate Session: 2 Viewer: #002 Date: 14 December 1981 Class: B concept vane*Quality NumberTranscript Concept D P Element of Target Score (R) (Q) 1 Picture 1 1 Buildings 5 1 5 2 Straight angles 1 Buildings 5 1 5 .1 3 Picture 2 1 Peninsula 5 1 5 River 1 4 Bay 3 2 6 5 Buildings I Buildings 5 5 14 11 6 Man-made 1 Buildings 5 4 13 7 London I Town 5 3 11 9 10 11 12 13 H 1 14 15 PxRxQ Score 0 0 1 1 2 2 3 3 4 4 1 5 6 6 8 7 9 a 10 9 12 10 1 15 11 16 12 20 13 ~l a41 25 14 Score 0 1 2 3 4 5 6 7 8 9 10 11+12 13 14 Weighted Average 2.72 of Concepts > 0 < (D CL -n 0 (D (D A) (1) (D 0) 6 Q 4 00 00 ;U Q Q _L 00 Q Q Q 00 CD CS 73, (YI Table A-5 (continued) > ,a 0 (D CL -n 0 ;U (D (D M C) C) Q Q 00 Q 00 0 > 6 CD -4 00 00 ;U C) C) -A 00 CD C1 CD 00 CD CD Rele- concept D P TalmoQualityScore ranscript Concept lement of Target (a) (Q) Picture 1 0 0 2 Straight angles 0 0 3 Picture 2 1 Island 5 1 5 4 River I Ocean 3 1 3 5 - 0 0 Buildings 6 Man-made 0 2- 7 London 0 0 8 9 10 11 JL2 23 14 2.5 Score of Concepts 0 2 3 4 5 6 7 8 9 10 11 12 13 14 1 - - 5 2 Table A-5 (Continued) > 0 < CL -n 0 ;U (D (D A) (1) (DI I 9 a >-n 10 9 TV 12 10 to 15 11 6 CD 4 16 12 00 00 20 13 Q CD -A 00 CD CD CD 00 --1 CD Weighted 0. CD Average 22 F CD 7L Cn Begin Time: 15:23 Target: Fernando de Noronha 'Viewer: #002 End Time: Method of Targeting: Coordinate Date: 14 December 1981 Length of Session: Session: 2 Class: B Begin Time: 15:23 End Time: Length of Session: Target: Punkaharju Method of Targeting: Coordinate Session: 2 Viewer: #002 Date: 14 December iQR1 Class: B comept NumberTranscript Concept D P Element of Target scor* R) Q) 1 Picture I I Bridge 2 1 2 2 Straight angles I Bridge 2 1 2 , 3 Picture 2 1 Bridge 2 1 2 4 River 1 Lakes 5 2 9 5 Buildings 1 Town 4 5 13 6 Man-made 1 1 Town 4 4 12 1 7 London I Twn 4 3 10 8 9 10 12 13 14 15 i 7 11 4 Score of Concepts 1 2 3 4 5 6 7 8 9 10 11 12 13 14 I I I Weighted Average 2.11 > 0 < (D CL -n 0 ;U (D rM 6 Q 4 00 00 ;U Q Q 00 Q Q Q 00 Q S Table A-5 (continued) Begin Time End Time: 13:13 LO Length of Session: Target: Inverness Method of Targeting: Coordinates Session: 3 Re2e- CD-.*Pt D P Element of Target VenceQualityScore Numberranscript Concept (R) (Q) 1 Picture 1 1 Buildings 5 1 5. 2 Land/water interface 1 River 4 4 12 _ 3 Ridges 0 0 4 Small ups and dawns 1 Buildings 5 1 5 5 Cold 1 Location 3 1 3 6 Picture 2 1 Church 2 1 2 7 Rocky 0 0 1 Buildings 5 1 5 8 Picture 3 9 Picture 4 0 0 10 Frozen 0 0 ling of town 1 City 5 4 13 11 Fee 12 Cliff on water 1 River bank 1 1 1 13 14 15 Score of Concepts 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 4 6 1. 1 2 3 1 1 Viewer: #002 Date: 14 December 1981 Class: --- - B Weighted Average > 0 < (D CL -n 0 (D (D a) I Q Q 4 00 00 ;U Q Q 00 CD CD CD 00 CD CD CD 73, Table A-5 (continued) z I'mmm > (A MEMO! on Immom M Begin Time: 13:13 End Time: Length of Session: E. Target: Han6 Method of Targeting: Coordinates Session: 3 Viewer: #002 Date: 14 December 1981 Class: B Concept Rele- D P VanceQua.21ty Numberranscript Concept lement of Target (R) (Q) core I Picture 1 1 Rocks 3 2 6. 2 Land/water interface I Peninsula 5 4 13 3 Ridges 1 Rocks 3 2 6 4 Small ups and downs I Rocks 3 2 6 5 Cold 1 Location 4 1 12 1 4 6 Picture 2 1 Church 2 1 2 2 7 Rocky 1 Rocks 3 5 11 F 8 Picture 3 1 Sloping rocks 3 2 6 9 Picture 4 0 0 10 Frozen I Location 4 4 12 11 Feeling of town I Town' 5 5 14 12 Cliff on water I Sloping rocks 3 2 6 13 14 15 Score of Concepts 0 1 2 3 4 5 6 7 8 9 -fO11 12 13 14 1 1 5 5 PxRxQ scare 0 0 ~O' 0' 1 1 2 2 3 3 4 4 5 5 6 6 8 7 9 8 10 9 12 10 15 11 16 12 20 13 Weighted Average 2.65-1 > -a 0 (D CL -n 0 1 (D 6 Q -4 00 00 ;U Q Q -L 00 Q Q Q 00 Q Q Q 'fable A-b ~continuect) > 0 < (D CL -n 0 (D (D M Q Q Q Q 00 Q 00 Z ress' > OWNER M W Begin Time: 13:13 End Time: Length of Session: 6 r4 r3 Go Target:_ Fernando de Noronha Viewer: #002 Method of Targeting: Coordinates Date: Session: 3 Class: 14 December 1981 B Rel*- Concept D P van"Quality lumberranscript Concept lement of Target (R) (Q) ore Picture 1 1 Hills 4 2 7 2 Land/water interface 1 nd 5 13 Isla 4 3 Ridges 1 Hills 4 5 13 4 Small ups and downs 1 Hills 4 3 12 5 Cold 0 0 6 Picture 2 1 Peak 3 3 8 7 Rocky 1 Shoreline 3 2 6 8 Picture 3 1 Hills 3 10 9 Pic;ure 4 1 Surf 3 1 1 3 10 Frozen 0 0 0 11 Feeling of town 0 1 12 Cliff on water 1 Hills by sea 4 5 13 13 14 is Score 0 1 2 3 4 5 6 7 8 10 11 12 13 14 of Concepts 3 1 2 3 1 2 3 Table A-5 (continued) Sears PXRXQ 0 0 1 1 2 2 3 3 4 4 5 5 6 a a 7 9 8 10 9 12 10 15 11 16 12 20 13 Average > a 0 < (D 0 (D W (D C M 7 Q Q Q Q 00 Q 00 0 ;U M 0 Q Q -4 00 00 ;U Q Q -L 00 Q Q Q Ei~ Q 73, 01 0 < (D CL -n 0 (D W Q Q Q 00 00 0 U Q 4 00 00 ;U Q Q -L 00 Q Q Q 00 Q Q Q Z mn M W Begin Time: 13:13 Toxget: Punkharaja End Time: Method of Targeting: Coordinates Length of Session: Session: 3 Viewer: #002 Date: 14 December 1981 Class: B Role- Concept v P Numberranscript Concept lement of Target van" nelltycore JR) (Q) I Picture 1 1 Scattered lakes 5 1 S. 2 Land/water interface I Scattered lakes 5 4 13 1 3 Ridges I Islands 4 2 7 4 Small ups and downs I Islands 4- 4 12 Cold Location 4 4 12 6 Picture 2 1 Islands 4 1 4 7 Rocky 0 a 8 Picture 3 1 Islands 4 1 4 9 Picture 4 0 0 10 Frozen I Location 4 4 12 11 Feeling of town 1 To-, 4 5 13 12 Cliff on water 0 0 13 14 15 Score 0 1 2 3 4 5 6 7 8 9-1 10 11 12 13 14 weighted Averag of Concepts 3 4 1 1 3 Table A-5 (continued) 0 (D CL -n 0 X (D (D (1) (D M CD -4 00 00 X 00 CD CD Q 00 CD Q CD > 0 0 X Q Q Q 00 Q o0 0 > ~U 0 6 Q 4 00 00 Q Q -L 00 Q Q Q 00 Q Q Q "I 7 cyl C z em) > (^ -j Begin Time: 12:46 End Time: Length of Session: Tvxget: Inverness Me thod of Targeting. Coordinates Session: 4 Viewer: #002 Date: 14 December 1981 Class: B Role- COMW D P Element of Target va4ftQUO11tySea" ftaborranscript COncePt M 0 Down jagged 0 2 picture 1 1 Sloping bank 2 2 3 vlat I Area 4 3 10 4 Water I River 4 4 12 5 Green 1 ! Grass 1 3 3 1 6 Picture 2 1 River banks 2 1 2 0 7 0 own/up 8 Deep valley 0 0 9 Pictur" e 3 2 10 1 Ran a 2 1 2 Down 11 Land/water interfaces 1 River 4 4 12 Descending I Banks 2 1 2 12 13 Trees I Trees 2 5 9 3 14 Winding river I River 4 5 1 2 1 2 15 Jungle I Trees Score of Concepts 0 1 2 3 4 5 6 7 8 9, 10 13 14 it !~ 3 5 2 2 2 2 1 1 2 1 Table A-5 (continued) PXRXQ score 0 0 2 2 3 3 4 4 5 5 8 7 9 a 10 9 12 10 15 13. 16 12 20 13 25 14 --- Weighted Averagg 1.211 > 0 -n 0 ct rq 3* =woo- 6 Q 4 00 00 ;U Q Q -L 00 Q Q Q 00 Q Q Q Begin Time: 12:46 End Time: Length of Session: 00 Target: HangZ; method of Targeting:, Coordinates Session: 4 Viewer: #002 Date: 14 December 1981 Class: B Rele- concept D VanceQualityScore ranscript Concept lement of Target Number 1 Down jagged 1 Sloping rocks 3 2 6. 2 Picture I I Sloping rocks 3 2 6 3 Flat I Bay 1 3 4 Water 1 Bay 3 4 10 5 Green 0 0. 6 Picture 2 1 Sloping rocks 3 7 Down/uP 0 0 0 a Deep valley 0 9 Picture"3 2 10 Down 1 sloping rocks 3 1 3 11 Land/water interfaces 1 Peninsula 5 4 13 12 Descending I Sloping rocks 3 1 3 13 Trees I Trees 2 4 7 14 Winding river 1 Bay 3 1 3 15 Jungle I Trees 2 3 6 Score of Concepts Table A-5 (continued) Weighted Average > 0 -n 0 IP 6 C) 4 00 00 ;U C) C) 00 CD CD CD 00 CD CD CD -L zn I 0 < (D CL -n 0 (D (D M Q Q Q Q 00 Q 00 0 > 6 Q 4 00 00 ;U Q Q -L 00 Q Q C) 00 Q Q Q 73, C z > WOMEN WOMEN M Begin Time: 12:46 End Time: Length of Session: 6 IH 4 0 Target: Fernando de Noronha Method of Targeting: Coordinates Session: 4 Rel&- conceptTranscript Concept D P Element of Tax%et VenceQua2ltyScore Number (R) (Q) - 1 Down jagged 1 Hills 4 3 20 2 Picture 1 1 Hills 4 4 12 3 Flat 0 0 4 Water I Ocean 4 4 12 5 Green 1 Vegetation 2 3 6 1 6 Picture 2 1 Hills 4 1 4 7 Down/up I Hills 4 1 4 1 8 Deep valley 0 0 9 Picture 3 2 10 Down I Sloping hills 4 1 4 11 Land/water interfaces I Island 5 4 13 1 12 Descending 4 1 Sloping hills 1 4 13 Trees I Vegetation 2 5 9 7 14 Winding river I ocean 4 2 15 Jungle 1 Vegetation 2 5 9 Score 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 I of Concepts 2 4 2 1 1 2 1 1 2 1 Viewer: #002 Date: 14 December 1981 > Class: B PxRxQSao"0 < - - =_ (D 0 0 CL -n 0 1 1 ;U 2 2 (D (D A) 3 3 0 (D C K 4 4 ) 1 Q 2 C:1 5 5 Q 00 6 6 Q 00 a 7 0 9 8 10 9 OOV 0 12 10 15 11 6 Q -4 16 12 00 00 ;U 20 23 Q Q -L 00 C) Q Q 00 IWeighted Average %-P 73, Table A-5 (continued) 0 Begin Time: 12:46 End Time: Length of Session: M Target: Punkaharju Method of Targeting: Coordinates____ Session: 4 'Viewer: #002 Date: 14 December 1981 Class: B Re2e-Q"21ty Concept D P Element of Target valme score NuMorranscript Concept (Q) 1 Down jagged 0 0 Picture 1 1 Islands 4 3 10 2 Flat Area 4 3 10 3 4 Water Lakes 5 4 13 5 Green 0 6 Picture 2 1 Lake bottoms 5 1 5 7 Down/up I Lake bottoms 5 1 5 Lake bottoms 5 1 5 Deep valley 2 9 Picturd'3 10 Down I Lake bottoms 5 1 5 I Lakes 5 4 13 11 Land/water interfaces Descending I Islands 4 1 4 12 13 Trees Trees 2 4 7 14 Winding river Connected lakes 1 2 9 - 15 Jungle 1 Trees 2 1 2 Score of Concepts 0 1 2 3 4 5 6 .7 8 9 10 11 12 13 14 - 2 6 1 2 3 4 1 1 2 2 Weighted Averag I > -a -a 0 -n 0 6 Q 4 00 00 ;U Q Q -L 00 Q Q Q 00 Q Q Q Table A-5 (concluded) Relea-UN00"WA Approved For 9&96-00788R001 800080001 -5 (u) Two types of overall assessment were chosen to emphasize the versatility of the evaluation procedure, (1) a simple rank ordering based on weighted average scores, and (2) a concept-by-concept, non-parametric, statistical technique. Table A-6 shows the results of the first method, the rank ordering. For convenience, the correct matches are underlined. Table A-6 (U) A RANK ORDERING OF WEIGHTED AVERAGES (U) Fernando Session/TargetInverness Hang6 de NoronhaPunkaharju 2 4.13 2.72 0.22 2.11 3 1.12 2.65 2.49 2.42 4 1.21 1.38 2.22 2.21 1 1.53 2.52 2.36 0.90 Scores computed with non-uniform target relevance factors. M From Table A-6, we see that there were 3 first-place matches and 1 fourth-place match. The probability of obtaining 3 of 4 possible first-place matches,,from chance fluctuations alone are less than 0.051. The point spread between the best match (Inverness) and the worst match (Punkaharju) are in qualitative agreement with a subjective "first look" at the quality of the transcripts as well. (U) The second analysis determines the significance of the difference between the correct concept/target matches and a control set of matches. All concept/target matches that are not the correct matches act as an internal control set. To avoid any invalid assumptions as to the correct parent distribution, a non-parametric statistical test, the Mann-Whitney U-Test, was chosen for the analysis. 4 41 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788RO01 800080001 -5 Approved For .ReleWNAW flf-PDP96-00788RO01 800080001 -5 (U) It is beyond the scope of this report to review the details of the Mann-WitneY U-Test; thus, only the results are quoted here. The probability that the set of correct concept/target matches is statistically indistinguishable from the control concept/target matches is less than 0.071. (U) There are a number of additional statistical procedures that could be used to analyze the results of this evaluation technique. The two cited above, however, represent a spread in complexity that demonstrates the internal consistency of the basic evaluation procedure. With only four similar RV sessions, the evaluation technique nearly reached the 0.05 level of statistical significance with each of the two statistical procedures, a result indicating a successful outcome with regard to the overall assessment procedure. 42 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788RO01 800080001 -5 Approved For ReleasANAM W-W16-00788 ROO 1800080001-5 Appendix B (U) SUMMARY OF EVALUATION TECHNIQUE (U) Step Action I Task coordinator defines the evaluation goal. He/she identifies target elements and assigns target element relevance factors as appropriate. 2 Analyst conceptualizes responses and prepares an RV assessment sheet for each response. 3 Repeated concepts are noted in the I'D" column. 4 Copies of the sheets from Item 2 are made; one for each possible target used in the analysis. FOR EACH POSSIBLE RESPONSE/TARGET COMBINATION: 5 Identify a target element for each concept not marked in the I'D" column; mark a I in the Ilp" column and write the target element and its relevance factor from Step I in the appropriate columns. (Write 0 and blanks if no element can be found.) 6 Using Table 3, assign a quality rating for all present (p = 1) concept/element combinations. 7 Compute the score as follows: a. Calculate relevance (R) X quality (Q) b. Conyert R X Q to an integer between 0 and 14 using the conversion table provided. 8 Enter the number of concepts that obtained each possible score in the space provided. 9 Calculate the weighted average using: Ak = 0.357 1Z f j /fiSi/Z fi ff where: S. is the score and f. Is the number of concepts dat obtained score J j = 0, 1, 2, ..., 14 10 For each response, rank order the weighted averages. 43 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5 Approved For ReleasLWCIAiS.MIRIED96-00788RO01 800080001-5 REFERENCES (U) 1. H. E. Puthoff and R. Targ, "A Perceptual Channel for Information Transfer over Kilometer Distances: Historical Perspective and Recent Research," Proc. IEEE, Vol. 64, No. 3, pp. 329-354 (March 1976). 11 2. C. Scott, On the Evaluation of Verbal Material in Parapsychology: A Discussion of Dr. Pratt's Monograph," J. Soc. Psych. Res., Vol. 46, No. 752, pp. 79-90 (June 1972). 3. R. Targ, H. E. Puthoff, and E. C. May, "State of the Art in Remote Viewing Studies at SRI," Technical Session on Research in Psycho- energetics, Proc. 1977 IEEE International Conf. on Cybernetics and Society, Washington, D.C. (September 20, 1977). 4. S. Siegel, Nonparametric Statistics, pp. 116-120 (McGraww-Hill, 1956). 44 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800080001-5