SErl" Aaawmak! For Release 2000108108: C IA-RDP96-00788 ROO 1800050001-8 Final Report December 1982 REVISED JULY 1983 do. 0 [01 A REMOTE VIEWING EVALUATION PROTOCOL (U) By: EDWIN C. MAY Prepared for. DEFENSE INTELLIGENCE AGENCY WASHINGTON, D.C. 20301 Allenlion: SG1J DT-5A CONTRACT MDA908-82-C-0034 SPECIAL ACCESS PROGRAM FOR GRILL FLAME. RESTRICT DISSEMINATION TO ONLY INDIVIDUALS WITH VERIFIED ACCESS. Copy No . ......I....... This document consists of 48 pages. CLASSIFIED BY: DT-5A REVIEW ON: 31 December 2002 SRI/GF-0247 333 Ravenswood Ave. - Menlo Park, California 94025 NOT RELEASABLE TO International (415) 326-6200 - 0 TWX: 910-373-1246 NATIONALS ed For Release 2000/08 .0CW-RDP96-OO788ROO18oodM Approved For Release 20tWL"qjfq0D0788R001800050001-8 CONTENTS (U) LIST OF ILLUSTRATIONS (U) . . . . . . . . . . . . . . . . . . . . iii LIST OF TABLES (U) . . . . . . . . . . . . . . . . . . . . . . . iv I OBJECTIVE (U) . . . . . . . . . . . . . . . . . . . . . . . 1 II INTRODUCTION (U) . . . . . . . . . . . . . . . . . . . . . . 2 III ANALYSIS PROTOCOL (U) . . . . . . . . . . . . . . . . . . . 4 IV CONCLUSIONS (U) . . . . . . . . . . . . . . . . . . . . . . 11 Appendix A--EVALUATION OF FOUR COORDINATE REMOTE VIEWINGS (U). 12 Appendix B-SUMMARY OF EVALUATION TECHNIQUE (U) . . . . . . . . . 43 REFERENCES (U . . . . . . . . . . . . . . . . . . . . . . . . . . 44 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 Approved For Release 20UPAUAMISM0788 ROO 1800050001-8 FIGURES (U) 1 Sample RV Assessment Form (U) . . . . . . . . . . 5 . . . . . . A-1 Transcript I (U) . . . . . . . . . . . . . . . . . 1-4 . . . . . A-2 Transcript 2 (U) . . . . . . . . . . . . . o . . . 15 . . . . . A-3 Transcript 3 (U) . . . . . . . . . . . o . . . . . 16 . . . . . A-4 Transcript 4 (U) . . . . . o . . . . . . . . . . . 17 . . . o . dw A-5 Target Site Hang6 (U) . . . . . . . . . . . . . . 19 . . . . . . A-6 Target Site Fernando de Noronha (U) . . . . o 20 . o . . . . MW A-7 Target Site Inverness (U) . . . . . o o . . . . . 21 . o . . . . A-8 Target Site Punkaharju (U) . . . . . . . . . . . . 22 . . . o . aw no =w Mw aw UNCLASSIFIED Approved For Release 2000/08/08 CIA-RDP96-00788ROO1800050001-8 Approved For Release 20oW(Ng;LAM61EQ788ROO1800050001-8 TABLES (U) 1 Task-Defined Relevance Scale Target for Elements (U) . . . . 6 2 Quality Assessment Scale (U) . . . . . . . . . . . . . . . . 8 3 Numerical Score Conversion (U). . . . . . . . . . . . Table 8 A-1 Target Element Relevance RatingsforHang6 (U) . . . . . . . 23 A-2 Target Element Relevance RatingsforInverness (U) . . . . . 23 A-3 Target Element Relevance RatingsforFernando de Noronha (U) . . . . . . . . . . . . . . . . . . . . . . . . . 24 A-4 Target Element Relevance RatingsforPunkaharju (U) . . . . . 24 A-5 RV Assessment Forms for Calibration of Remote Viewing (U). . 25 A-6 A Rank Ordering of Weighted (U) . . . . . . . . . . Averages 41 dMi ON MW W iv UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 Approved For Release 20AJjNQA&$WfiQ0788R001 800050001-8 no I OBJECTIVE (U) (U) The-objective of this task was to develop an evaluation procedure to assess the relative quality of a set of different remote viewing (RV) responses. NNW UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 Approved For Release 2&IJAQA%kEt6DbO788 ROO 1800050001-8 aw an III ANALYSIS PROTOCOL (U) (U) To quantify the analysis procedure, we have divided the task into four separate areas: subject response, target/task definition, quality assessment, and numerical analysis. Figure 1 is a sample RV Assessment Form that has been designed to emphasize the separation of the analysis tasks. Each of the parts of the form are described below. W The subject's any knowledge of either this method of response rambling response to a we have developed a set procedure. response should be prepared for analysis without the target site or the overall task. The aim of preparation is to reduce a possibly redundant, coherent set of concepts. To meet this requirement of initial guidelines to the conceptualization (U) A concept is defined as a paraphrase of a single idea that has been expressed in the RV verbal or drawing response. That coherent idea should not be fragmented into component parts. For example, a response might be of the form, "I see a large, textured, gray building." The single concept that expresses this idea should be "large, textured, gray building," rather than four separate concepts--one for each word in the phrase. Each concept should be entered under the Transcript Concept" column in the RV Assessment Form. M For this initial evaluation technique, a particular concept should be used only once in the analysis. (Some weighting factor propor- tional to concept frequency could be utilized, but, for the initial attempt, only unique concepts are used.) If in the construction of the transcript concept list a concept later in the transcript is a duplicate of an earlier one, it should be so noted by placing the concept number of the original concept in the "D" column. 4 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 t t 1. t t t > 0 < (D CL -n 0 (D (D C) c) C C) CD Z E- 00 Q 00 0 > > rA on mamma M 0) 6 CD 4 00 00 C) C) -L 00 C) C) C) C" C) CD CD 73, 00 Begin Time: Target: End Time: Method of Targeting: Length of Session: Session: Viewer: Date- Class: PXR)QScoreNormalized Score 0 0 0.00 1 1 1-16 1 2 2 0.71 3 3 1.07 4 4 1.43 5 5 1.79 1 6 6 2.14 8 7 2.50 9 2.86 10 9 3.21 12 10 3.57 15 11 3.93 16 12 4.29 20 13 25 14 Concept Rele- NumberTranscript Concept D P Element of Target vanceQualityScore (R) 2 1 3 4 6 1 7 8 9 10 12 13 14 15 Score of Concepts 0 1 2 3 4 5 6 7 8 9 10 11 1T2l3T;] d Average 0 < (D CL -n 0 (D (D (D C) 00 00 X C) (D 00 C) C) C) C" C) CD CD 73, 00 FIGURE 1 SAMPLE RV ASSESSMENT FORM (U) Approved For Release 20cWhU.A$$LFAQ0788ROO1800050001-8 (U) To utilize an analysis procedure that is capable of quantitative assessment, it is necessary to define, in advance, what the goals of the assessment are. In the RV Assessment Form, columns "Element of Target" and "Relevance" are provided to clearly define the goal of the analysis. In the ideal situation, an RV target should be completely specified in advance. A target typically consists of a number of target elements, each of which may have varying relevance with regard to the overall RV task. For any given target, an independent list of target elements should be prepared. The selection of what constitutes a target element is left completely to the discretion of the task coordinator. The target element must be selected with little regard to task relevance (target element relevance is accounted for later). Because an RV target consists, in principle, of an essentially infinite number of possible elements, discre- tion needs to be exercised in the selection process. (U) For each target element identified for the site, the task coordinator must define a relevance rating. This rating allows the coordinator to tailor the analysis to the task requirements. Table 1 shows the scale that is used for the target element relevance rating. Table 1 .old W TASK-DEFINED RELEVANCE SCALE FOR TARGET ELEMENTS (U) Rating Relevance Scale 1 A targetelementof trivial interest 2 A targetelementof minor interest 3 A targetelementof intermediate interest 4 A targetelementof major interest 5 A targetelementof key interest 6 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 Approved For Release 20oW(WQAfx$MEQ788RO01 800050001-8 (U) For each transcript concept on the RV Assessment Form (Figure 1), the analyst should attempt to find the element on the list of target elements that he/she considers to be the best match. The analyst should be quite liberal in the concept/element matching (i.e., the quality of the match should be considered at this point in the analysis). If he/she is able to identify a target element that might be considered a match to the given concept, a 1 is placed in the "p" (present) column on the assessment form. If no element can be identified, a 0 is placed in the "p" column. After making a target element identification, the selected target element, and its corresponding overall relevance rating should be entered in the appropriate columns on the assessment form. (U) Having identified a corresponding target element for each con- MW cept, it is now appropriate to assess the quality of the match. The quality assessment is done on the basis of how well the single concept OW in question matches the selected target element. The judgement is to be made without regard to any other issues, such as importance of the concept Md to the transcript, or importance (relevance) of the target element to the target. Table 2 shows the quality assessment scale that is used for this Nei part of the analysis. The appropriate quality score from Table 2 is entered in the "Quality" column on the RV Assessment Form for each concept for which a matching target element has been identified. (U) An intermediate numerical score is computed for each concept from the relevance and quality (Tables I and 2) evaluation as follows: mom S, = P x R x Q P where P is the value in the "p" column (0 or 1); R is the relevance aw evaluation; and 0 is the quality assessment. S' can assume values ranging between 0 and 25. Table 3 demonstrates how to determine the final score, S, for a given value of S' for each concept. The conversion table is 7 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 Approved For Release 20cUMMAS-3 &788ROO1800050001-8 II INTRODUCTION (U) (U) In addressing the remote viewing (RV) evaluation question, we consider two separate requirements: 0 Absolute evaluation of a single response for a single unknown target. a Relative evaluation of a set of responses for a series of known targets. The first of these is of the most interest in an operational setting. As part of the RV enhancement task, we have considered this problem in two ways. First, by conducting an operational RV session between two calibra- tion RV sessions, a tentative a priori assessment of operational efficacy can be determined. The evaluation is made on the basis of performance during the calibration sessions, and on the basis of adherence to a pre- determined session structure. (U) A second technique for an a priori evaluation was explored as part of the Fiscal Year 1982 program in an audio-linguistic task. This task provided indications that careful linguistic analysis, when coupled with technical audio analysis, could yield an assessment in the absence of knowledge about the target. (U) Various techniques have been used in the past 2* in an attempt to solve the relative evaluation problem. The most common of these was the simple rank ordering of all responses, as assessed against all possible targets used in an experimental series. In this procedure, a judge is presented with n RV transcripts and n target sites. His task is to arrange .WW (U) References are listed at the end of this report. 2 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 Approved For Release 2000/M6F*Kf&96-OO788 ROO 1800050001-8 (U) the transcripts in order of the best to least match for each of the n targets. A simple numerical counting procedure is then used to estimate the likelihood that the Judge's transcript/target matches are by chance alone.2 This early technique contained little systematic structure for determining the final order of matches. (S) The first step toward systematizing the rank order judging too procedure was to preprocess the raw data in the transcript by "concep- tualizing" both the verbal and the pictorial responses. Conceptualizing .MW a transcript requires an analyst to paraphrase the transcript into a list of coherent statements. This concept list is then compared and scored concept-by-concept to each of the targets in the experiment. The resulting scores are averaged for each response, and all responses are rank-ordered on the basis of these scores.a This improved analysis procedure was applied to a number of experiments within the Technology Transfer Task for INSCOM in FY 1980. (U) The problem with the above technique is that there are no guide- lines as to how the analyst should paraphrase the transcript; furthermore, the method in which the concepts are to be assessed against the targets remains undefined. The purpose of the Evaluation Task in FY 1982 was to identify a procedure that corrected these deficiencies. 3 CMIR!" Approved For Release 2000/08/08 CIA-RDP96-00788ROO1800050001-8 Approved For Release 2"UGLA&MM00788RO01800050001-8 not dod uni MW now Table 2 (U) QUALITY ASSESSMENT SCALE (U) Rating Discrimination Scale 1 Poor description; only one or two aspects of the material match. 2 Fair description; a few aspects of the material match, but a large ambiguity exists. 3 Reasonable description; many aspects of the material match, but there remains some ambiguity. 4 Good description; a large number of aspects of the material matches, but it is possible to conceive of material that would be a better match. 5 Excellent description; all or nearly all aspects of the material match. Table 3 (U) NUMERICAL SCORE CONVERSION TABLE (U) MW S P x R x Q Score Normalized Score 0 0 0.00 1 1 0.35 2 2 0.71 3 3 1.07 4 4 1.43 5 5 1.79 6 6 2.14 8 7 2.50 9 8 2.86 10 9 3.21 12 10 3.57 15 11 3.93 16 12 4.29 20 13 4.64 25 14 5.00 8 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 Approved For Release 20UbICLA&MMD0788ROO1800050001-8 am M .Mi used to eliminate the nonuniformly-distributed gaps in scoring numbers that occur if one simply uses the product S'. Thus, the final score for each concept ranges from 0 to a maximum of 14. This conversion table is AW provided as part of the RV Assessment Form. (U) If an assessment of an individual concept is required, the final score for each concept/target-element match can be related to the quality assessment scale by using the conversions shown in the third column of Table 3 and on the assessment sheet. It should be noted, however, that the integer scores are used to simplify the remaining calculations. (U) To determine a final evaluation of the complete transcript assessed against a given target, a weighted average of concept scores is computed. To assist in the calculation of the weighted average, a tally box score is provided at the bottom of the RV Assessment Form. For each of the possible scores, 0 through 14, the number of concepts that attained that particular score are counted. For example, if 3 concepts were evaluated with a score of 12, a 3 is entered in the box below the 12 score. If the frequency of occurrence of score S j is fi, then the final weighted average is computed by Ak = Z f i'If X S i/Z fi -~f A k = 0.357 A~ (U) The normalized, weighted average score, Ak, is entered in the weighted average box on the assessment sheet. The weighted average score has been normalized to be within the range 0 < Ak < 5.0 To aid in the interpretation of the result, the quality assessment scale (Table 2) can be used to assess quality of the match between the whole RV response and the given target site. (U) At this point in the evaluation protocol, the following options are available, depending on the task requirement: 9 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 Approved For Release 2tPNQ8AS&ff4WOO788 ROO 1800050001-8 (U) * An n X n rank ordering on the basis of the weighted averages. * A simple selection of the best match. * A statistical evaluation on a concept-by-concept basis. MW MW MW 10 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 Approved For Release 2000/015B'~ 5A-hDP96-00788 ROO 1800050001-8 IV CONCLUSIONS (U) (S) A protocol has been developed to address the relative evaluation portion of the overall RV transcript assessment problem. As a demonstra- tion of the technique, we provide in Appendix A an analysis of a series of four remote viewings that were performed as calibrations during operational sessions at DIA in December 1981. In this series the remote viewing products were of relatively high quality, but nonetheless require WON a sensitive technique to differentiate because of the similarity of the targets and, hence, of the descriptions. (The series was chosen primarily ON for that reason.) Application of the assessment technique resulted in the correct blind matching (highest scoring in matches versus cross MW matches) of three of the four. (U) Appendix B is a one page, step-by-step procedure for the application of this evaluation technique. (U) The material in this document thus constitutes an instruction manual or protocol for application of a step-by-step procedure for quan- titative assessment of the relative target/transcript correlations of a series of transcripts matched into a series of targets. aw MW MW Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 Approved For Release 2MCI fifMPtEGb0788 ROO 1800050001-8 .NW ow Appendix A EVALUATION OF FOUR COORDINATE REMOTE VIEWINGS (U) now uni m1w mili 12 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 ^ n I MON Approved For Release 2000 IV rDP96-00788ROO1800050001-8 Now (S) on 14 December 1981, four coordinate remote viewings were con- ducted as calibrations during an operational remote viewing session tar- geted on client-designated sites of interest. The calibrations and the operational RV session were conducted at the client's organization and were witnessed by client representatives. These four calibrations were chosen as a test bed for the evaluation procedure for the following reasons, (1) they were conducted in an operational setting, and (2) the targets had many similar features, and would thus provide a sensitive test of the protocol. (S) Figures A-1 through A-4 are the transcripts that were presented to the analyst. They are exactly as they were when collected, except that the coordinates have been removed. Figures A-5 through A-8 are the National Geographic magazine targets that were used during the calibration sessions. Finally, the task coordinator provided Tables A-1 through A-4 as target element relevance scales for the four targets in Figures A-5 through A-8. This completes the information that was given to the analyst, and thus the analysis was carried out blind as to the matching target/ transcript pairs. MW (U) Table A-5 is a compilation of the completed work sheets that were used by the analyst in this evaluation. They are shown in groups MW by session number, and alphabetized on the four targets. (The task coordinator first randomized the transcript order then assigned the session number used above.) For each of the transcripts, the analyst simply included all phrases and all drawings as concepts. For example, seven concepts were found during Session 2. (U) All concepts were then analyzed as described in the text. The matching target element, its relevance rating, and the computed score are shown for all possible combinations of transcript/target pairs in Table sow A-5. The score distributions and their resulting weighted averages are also shown in Table A-5. 13 Approved For Release 2000/08/08 CIA-RDP96-00788ROO1800050001-8 Pam Approved For Release 20UNCLASM&WO788 ROO 1800050001-8 Iq &(VT1 0 00A T VA 5 Q.(40 nz FIGURE A-1 TRANSCRIPT I (U) 14 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 mw Approved For Release 21JNCJA6iWdWOO788 ROO 1800050001-8 Q 1.2ec 9 / 0 0 *;L u 5.23 L4,4-1 n 7 0 FIGURE A-2 TRANSCRIPT 2 (U) 15 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 I &-f P" q- a -71=N 00:~ Approved For Release 2191-l"M&WPOO788 RON q40" 1 -8 1.13 CL-Ail I? 0 dw cyVV UP dog 14) -rc WNI WfA FIGURE A-3 TRANSCRIPT 3 (U) 16 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 Approved For Release J"LADWRE110788RO01 dog Now Vjat cbt~/ tA-O, 6(q lvc~ik FIGURE A-4(a) TRANSCRIPT 4 (U) 17 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 800050001-8 0 0 WEI ..a Ow dmi dew Approved For Release 2cUWoLA9MFjJQ0788RO01 800050001-8 13 FIGURE A-4(b) TRANSCRIPT 4 (concluded) (U) 18 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 Approved For Release 2&W98A Slg"tP0788RO01 800050001-8 film mow Nei dew Ito ir LLJ It, to LO Lu & No Al 4 ;f f:. ,ji U! LL 19 U,NCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 low Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 UNCLASSIFIED CPYRGHT . ., M Lu u- Lu w Wit -j z 20 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 Ow saw Approved For ReleaseUfiVt/qk S(qf FWD-00788RO01 800050001-8 CPYRGHT 21 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO18 -1 En Lu z Lu LLJ Lu 0 Lu 0 LL ui u- cf) z Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 22 UNC&A Approved For Release 2000/ / ~WAR6-00788RO01 800050001-8 CPYRGHT UNCLASSIFIED L1Nr,L9-R%F199788R001 800050001-8 Approved For Release 200ammu. Table A-1 (U) TARGET ELEMENT RELEVANCE RATINGS FOR HANGO (U) Now Target ElementRelevance Town 5 Cold 4 Peninsula 5 Rocky 3 Vegetation 2 Bay 3 Table A-2 (U) TARGET ELEMENT RELEVANCE RATINGS FOR INVERNESS (U) Target ElementRelevance City 5 River 4 Bridge 3 mow River banks 2 Vegetation 1 23 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 Approved For Release 20OWK L0A1Wfj[-[)788 ROO 1800050001-8 Table A-3 (U) TARGET ELEMENT RELEVANCE RATINGS FOR FERNANDO DE NORONHA (U) MW MW Target Element Relevance Island 5 Surf 3 Hills 4 Uninhabited 3 Mountain peak 3 Temperate climate2 Vegetation 2 Ocean 4 Table A-4 MW (U) TARGET ELEMENT RELEVANCE RATINGS FOR PUNKAHARJU (U) ow MW MW Target ElementRelevance Connect lakes 5 Town 4 Bridges 2 Cold 4 Vegetation 1 Islands 4 24 now UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 I t 1. 1 k I t t I I I t t Begin Target: iriewer: #002 > Time: Inverness 12:40 End Method vg: Date: 14 Time: of Coordinates December Targeti 1981 0 0 Length Session: Class: B < < of Session: (D (D CL Van" L -pt D P 'rium".3, - ranscript lement Concept of Target n Number (1)(Q) 0 0 1 Picture 1 Twin 1 1 0 0 1 church towers (D (D 2 Picture 1 1 1 2 I Buildings 5 11 2 2 (D (D 3 Up 1 5 and 1 down M M 0 3 3 C) C) 4 Rocky 0 C) C) C) C) C 1 River 4 4 12 4 4 Q C) 5 Land/water 00 00 z interface 5 5 C) Q E~ 6 Picture 1 Twin 1 00 00 ) 3 church towers eme 1 , r 7 Picture 1 River 4 4 12 6 6 0 > 0 > 4 1 - Ul 8 7 > 8 Uprising I Buildings 5 1 5 tA 0 9 8 9 Cliff 0 E. 1 River 5 1 5 10 9 a) 10 Fjords 6 6 C) C) 11 Coastal I City 5 4 20 12 10 -4 4 city 00 00 15 11 00 00 12 X X C) C) 16 12 C) C) 13 -L -L 00 00 20 13 C) C) 4 Q Q Q Q 25 14 C" C" Q Q Q Q Q Q S 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 1 73, core - -eighted Average 00 . w 3 1 1 of Concepts 2 5 3 3 2 1 TABLE A-5 RV ASSESSMENT FORMS FOR CALIBRATION OF REMOTE VIEWING (U) t k t t L > 0 (D OL -n 0 X W C z 00 (1 0 00 > X 0 MM INIONE M 4 00 00 X 00 00 Begin Time., 12:40 Target: Hangz; End Time: Method of Targeting: Coordinates Length of Session: Session: I Q E~ coucept Rele- NumberTranscript Concept D P Element of Target yauc*Qualityscore (R) (Q) 1 Picture I I Rocks 3 3 2 Picture 2 1 3 Up and down I Buildings 5 1 5 4 Rocky 1 Rocks 3 5 11 5 Land/water interfaces I Coast 5 4 13 6 Picture 3 1 Church 2 1 2 7 Picture 4 Bay 3 2 6 8 Uprising 1 Slope of land 9 Cliff I Coast 5 - 1 9 2 10 - - Bay 1 Fjords 1 3 1 3 11 Coastal city 5 5 14 1 City 12 13 14 15 Score of Concepts 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 4 2 1 1 2 1 1 1 1 1 Viewer: #002 Date: 14 December 1981 0 Class: B PXRAscore -n 0 0 0 X 0 W 2 2 3 3 C) 4 4 5 5 6 6 8 7 9 a (DOWN, 10 9 CM 12 10 -4 00 00 15 11 X a 16 12 C) 00 20 13 CD L" C) F~~Tig-hted Average Table A-5 (continued) I t t t t I t 1. 1 1 1 1 1 t t t > -a 0 -n 0 ;U (D 0 (D C 00 z 00 rmm~ 0 > -j > ;b M 6 -4 00 00 ;0 CD 00 CD 60 Begin Time: 12:40 Target: Fernando de Noronha End Time: Method of Targeting: Coordivi*.tes Length of Session: Session: Rele- concept D P vanceQuality Numberranscript Concept lement of Target (a) (Q) care I Picture I I Hills 4 4 12 2 Picture 2 1 3 Up and down 1 Peak 3 1 3 I Coast line 1 5 5 4 Rocky 13 5 Land/water interface I Island 5 4 Peak 3 2 6 Picture 3 ~ 6 7 Picture 4 1 1 1 Island 5 1 5 1 5 8 Uprising 1 Hills 4 2 7 7 9 Cliff I Peak 5 3 11 10 Fjords 1 Inlets 3 2 6 11 Coastal city 0 1 0 1 12 13 14 15 a: 0 Score of Concepts 1 2 3 4 5 6 7 8 9 10 11 12 13 14 2 3 1 2 1 1 1 1 Viewer: #002 > Date- 14 December 1981- Class: R 0 < (D PxRxQ scare CL -n 0 0 0 X (D (D 2 A) 2 CO) (D 3 3 ~3 4 4 4 5 6 6 8 7 I 10 9 12 10 Q 15 11 00 00 16 12 X Q 20 13 1 00 Q 25 14 Weighted Tveragej 2 Table A-5 (continued) t t t t I t 1 1. t I t t > 0 < (D CL -n 0 X (D M (D M Q C) C) Q 00 z C) 4 00 0 > > CO 0 T I MENEM E- (D a) M 6 C) 4 00 00 X Q C) -L 00 C) C) C) C" C) C) D Begin Time: 12:40 Target: Punkaharju Viewer: #002 Method of Targeting: Coordinates Date: 14 December 1981 End Time: Length of Session. Session: I Class: B 0 Concept Rele- NumberTranscript Concept D P Element of Target vanteQualityScare (R) (Q) 1 Picture 1 0 0 2 Picture 2 1 0 Up and down I Town 4 1 4 - 4 Rocky 0 0 5 Land/water interface 1 Connected lakes 5 4 13 6 Picture 3 0 0 7 Picture 4 1 Connected lakes 5 2 9 ~9 8 Uprising 0 0 0 9 Cliff 0 1 0 10 Fjords I Connected lakes 5 1 5 1 11 Coastal city Town 4 3 10 12 I I I 13 1 - - 1 1 14 -- -- - - 15 Scol 00 of Concepi 0 1 2 4 5 6 7 8 9 10 13 14 5 2 1 1 1 Table A-5 (continued) < PxRxQ Score (D -n 0 0 0 1 1 CD CD 2 2 M (1) CD 3 3 4 4 5 5 6 6 8 7 9 a "M 0 10 9 12 10 C) Q -j is 11 00 00 16 12 M Q Q 20 13 -A 00 C) C) C) C" CD CD CD Weighted Average t I t > ,a ,a 0 (D C1 -n 0 (D (D Q Q Q Q 00 Q 4 0 > > W 0 E4 M 6 a -4 00 CO ;U Q Q 00 Q Q Q C" Q Q Q Begin Time: 15:23 Target: Inverness End Time: Method of Targeting: Coordinate Length of Session: Session: 2 Role- concept D P Numberranscript Concept lement of Target Va- ualltycore R) (Q) Picture I River bank 4 -3 10 2 Straight angles 1 Buildings 5 1 5 3 Picture 2 1 River 4 3 10 4 River 1 River 5 5 14 5 Buildings 1 Buildings 5 5 14 6 Man-made I City 5 4 13 7 London 1 City 5 4 13- 8 9 10 12 13 14 15 L Score 00 of Concepts 1 2 3 4 5 6 7 8 9 10 11 12 13 14 2 2 2 Viewer: #002 Date: 14 December i9gi Class: B VXRXQ s,.r* (D CL -n 0 0 0 1 1 (D 2 2 (D 3 3 4 4 5 5 6 6 8 7 9 8 =mm 10 9 5- F21 d 12 10 .0 CD CD 15 11 -4 00 00 16 12 Q 20 13 -L 00 Q 5 2 14 Q Q CD CD Weighted Average C) I I 7L 00 Table A-5 (continued) t I I I I I I I t I t I I t 0 (D CL -n 0 (D (D M Q Q Q Q z 00 n 4 Q 00 rmme > W o C) M Q 4 00 00 ;U Q Q 00 Q Q C" CD CD CD 73, 00 Begin Time: 15:23 End Time Length of Session: Target: HangZ; Method of Targeting: Coordinate 2 Session: e- Concept D P Re Numberranscript Concept lement of Target Vanceualitycore (R) (Q) I Picture 1 1 Buildings 5 1 5 2 Straight angles 1 Buildings 5 1 5 1 3 Picture 2 ---- I I Peninsula 5 1 5 1 - 4 River 1 Bay 3 2 6 5 Buildings I Buildings 5 5 14 1 6 Man-made 1 Buildings 5 4 13 7 London I T. 5 11 8 9 10 12 13 14 15 Viewer:#002 Date: 14 December 1981 Class:B 0 (D PxRxQ ScoreCL -n 0 0 0 ;U 1 1 1 (D (D A) 2 2 (1) (D 3 3 M Q 4 4 5 5 6 6 8 7 9 8 10 9 13 12 10 Q 4 is 11 00 00 16 12 Q Q -L 20 13 00 Q Q 25 14 Q C" CD CD Score 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 CD -- - Weighted Average 7L of Concepts 3 1 1 1] 1 1 1 2.72 6o Table A-5 (continUed) I I I t I I I I I t t > 0 < (D CL -n 0 (D K3 Q Q z 00 Q 00 rme' LJ 0 > "4 0 U W a) M 6 Q 4 00 00 Q Q -L 00 Q Q Q CA CD CD D 15:23 Fernando de Noronha Begin Time: Target: End Time: Method of Targeting: Coordinate Length of Session: Session: 2 - itole- oncept D P Vanceualltyscom ranscript Concept lement of Target (R) (Q) Picture 1 0 0 2 Straight angles 0 0 3 Picture 2 1 Island 5 1 5 4 River I Ocean 3 1 3 5 Buildings 0 0 6 Man-made 0 0 7 London 0 0 9 10 121 13 14 15 Score 00 of Concepts 0 1 2 3 4 5 6 7 8 9 10 11 12 13 141 5 2 1 Viewer: > #002 Date: 14 December 1981 0 Class: < B (D CL PXR)Q800" -n 0 0 0 (D 1 1 2 2 (D M 3 3 Q L- 5 5 8 7 9 a rn 10 9 12 10 -4 00 15 11 00 ;U Q 16 12 Q 00 20 13 Q Q Q 25 14 cyl 1 t 1 CD CD CD Teighted Average 60 F, T O.22 Table A-5 (continued) t > 0 < (D CL -n 0 (D M Q Q Q Q z 00 Q n 00 rmm' 0 > M Q 4 00 00 X C) Q -L 00 Q Q CD CD CD 73, 00 t t I t t t I t I I I Begin Time: 15-23 Target: Punkaharju End Time: Method of Targeting: Coordinate Length of Session: Session: 2 Rele- conceptTranscript Concept D P Element of Target VILE"Qualltysco Number (R) (Q) re 1 Picture 1 1 Bridge 2 1 2 2 Straight angles I Bridge 2 1 2 3 Picture 2 1 Bridge 2 1 2 4 River I Lakes 5 2 9 5 Buildings 1 Town 4 5 13 6 Man-made I Town 4 4 3.2 7 London 1 Town 4 3 10 8 9 10 12 13 14 15 Score of Concepts 1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 1 1 3 1 1 1 1 1 1 1 I I Viewer: #002 > a Date: -0 14 i December i9g 0 Class: B < (D CL PXRXQSoo" -n 0 -3 0 0 X (D 1 1 2 2 (D M 3 3 Q 4 4 -rem 6 6 > 8 7 9 8 10 9 12 10 -4 00 15 11 00 X Q 16 12 C) 00 20 13 Q Q CD C" Q CD CD Weighted Average F 2.11 Table A-5 (continued) I I I I I I I I I t t t 0 < (D CL -n 0 X (D (D K3 C) C) C) 00 Z C) I. 00 0 > > LJ t CA ~4 00 C) C) 4 00 00 X C) C) -L 00 C) C) C) CA CD CD CD Begin Time: 13:13 Target: Inverness viewer: #002 > End Time: Method of Targeting: Coordinates Date: 14 December 1981 0 < Length of Session: Session: 3 Class: B (D Role- ConceptTranscript Concept D P Element of Target Vance Q-21ty S..re Number (R) (Q) 1 Picture I I Buildings 5 1 5, 2 Land/water interface I River 4 4 12 1 1 0 3 Ridges 0 Small ups and downs 1 5 1 5 4 Buildings I Location 3 1 3 5 Cold 6 Picture 2 1 Church 2 1 2 7 Rocky 0 0 8 Picture 3 1 Buildings 5 1 Picture 4 0 0 0 0 10 Frozen 11 Feeling of town 1 City 5 4 13 12 Cliff on water i River bank 1 1 1 - 13 14 Score 00 of Concepts 11 141 0 1 2 3 4 5 6 7 8 9 10 Ill12 6 1 1 2 3 CL PXRXQ score -n 0 -% 0 0 X (D F 1 0) 2 2 (D C) 3 3 C) 4 4 8 7 9 8 1 10 9 12 10 4 00 00 is 11 1 X Q 16 12 Q -L 00 20 13 Q Q CD 25 14 C" CD CD CD Weighted Average 00 TTiil Table A-5 (continUed) t t I I t I t t > 0 < (D CL -n 0 CD CD C) C) C) C) Z 00 C) 00 0 > W - 4 0 E- 6 C) 4 00 00 ;U C) CD -L 00 Q C) C) C" C) C) > Begin 13:13 Target: Viewer:#002 -0 Time: Hang6 End Method Date:14 Time: of December Targeting: 1981 Coordinates 0 Length ession. Session: 3 Class:B < of S (D Rele- CL concept Transcript D P Element vancoqull'tyScore P)(R)(Qscoro Concept of -n Target Number (R)(Q) 0 oll I Picture Rocks 3 2 6 0 0 1 i X CD 2 Land/water 1 Peninsula 5 4 13 1 1 interface 3 Ridges I Rocks 3 2 6 2 2 CD 4 Small 1 Rocks 3 2 6 3 3 C) ups C) and downs 5 Cold I Location 4 4 12 4 4 6 Picture 5 5 2 I Church 2 1 2 Lill, 7 Rocky I Rocks 3 5 11 6 6 - 8 Picture 1 Sloping 3 2 a 7 3 rocks 9 Picture 0 9 a 4 0 on lion 10 Frozen 1 Location 4 4 12 10 9 Fn 11 Feeling 1 Town' 5 5 14 12 10 4 of town 00 12 Cliff I Sloping 3 2 6 15 11 00 on rocks X water Q 13 16 12 Q -L 00 14 20 3 Q ~ CD Q 25 14 Q CD 0 1 2 3 5 6 7 8 9 10 11 12 13 14 CD Score -;L r-W-leighted Average 60 00 of Concepts 2 5 1 2 2.65 Table A-b kcontinuea) I t t t t I t t t t t t t > -a 0 (D CL -n 0 (D (D M Q CD CD Q 00 Z 6 Q 00 0 > LWO > rw 0 men E- a) M 6 Q -4 00 00 ;U Q Q -L 00 Q Q Q Q CD 73, 00 Begin Time: 13:13 Target: Fernando de Noronha End Time: Method of Targeting: Coordinates Length of Session: Session: 3 c-ept Rele- NumberTranscript Concept D P Element of Target VanceQualityscore (R) Picture 1 1 Hills 4 2 2 Land/water interface 1 Island 5 4 13 3 Ridges 1 Hills 4 5 13 - 4 I Hills 4 3 mall ups and downs 12 5 Cold 0 0 - 6 Picture 2 1 Peak 3 3 1 8 7 Rocky I Shoreline 3 2 6 - 8 Picture 3 1 Hills 4 3 10 0 9 Picture 4 1 Surf 3 1 3 3 10 Frozen 0 0 1 11 Feeling of town 0 0 0 12 Cliff on water 1 Hills by sea 4 5 13 ~13 13 14 15 Scol of Concepi 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 3 1 2 3 1 2 1 1 1 2 3 Table A-5 (continued) Viewer: #002 > -a Date: 14 December 1981 Class: B 0 (D CL PXRXQscore -n 0 0 0 (D 1 1 1 2 2 (D M 3 3 Q CD 4 4 1 5 5 6 6 a 7 9 8 10 9 jFn 12 10 4 00 15 11 00 ;U Q 16 12 Q -L 00 20 13 Q Q Q Q CD CD Weighted 2.49J Average t t t I t I t I t t I t t > 0 (D CL -n 0 (D Q Q Q Q Z 00 n 1~ Q 00 reme 1 0 > LI) ON 0 E- M 6 Q 4 00 00 ;U Q Q _L 00 Q Q Q C" CD CD Begin Time: 13:13 Target: Punkharaja Viewer: #002 > End Method of Date: 14 Time; Targeting: December Coordinates 1981 LengthSession: Session: 3 Class:B 0 of Role- (D ConceptTranscript Concept D P Element of TargetvanceQu&"tyS.. PxRxQ scoreCL Number (R) (Q) -n 0 I Picture 1 1 Scattered lakes 5 1 5 0 0 2 Land/water interface1 Scattered lakes 5 4 13 1 (D (D 3 Ridges 1 2 7 2 2 Islands 4 (1) (D 3 3 4 Small ups and downs1 Islands 4 4 12 Q 5 Cold I Location 4 4 12 1 6 Picture 2 1 Islands 4 1 4 71 Rocky 0 0 - 8 -- 1 Islands 4 1 4 Picture 3 9 Picture 4 0 0 10 Frozen I Location 4 4 12 11 Feeling of town I Town 4 5 13 12 Cliff on water 0 0 13 14 15 CD Scol of Concepi 0 1 2 3 4 5 6 7 8 9 10 12 13 14 3 3 1 4 1 1 +3 2 Table A-5 (continUed) 4 4 1 5 5 6 6 8 7 9 8 10 9 12 10 15 11 00 00 16 12 Q Q 20 13 _L 00 Q Q Q C" CD CD Weighted Average CD 1 2.42 00 t I I I t t t I t t t t I t > 0 < (D CL -n 0 "1 X (D A) (1) a C C) C) z C) 00 .4 in C) rm~ 00 > 0 CA "J > Cn M 6 C) 4 00 00 X C) C) -L 00 C) C) C) CD Begin Time: 12-46 Target: Inverness End Time: Method Of Targeting: Coordinates Length of Session: Session: 4 Rele- ConceptTranscript Concept P Element of Target vanceQualitys..,.. NuWwr (Q) 1 Down jagged 0 0 2 picture 1 1 . Sloping bank 2 1 2 3 Flat I Area 4 3 10 4 Water I River 4 4 12 5 Green I Grass 1 3 3 6 Picture 2 1 River banks 2 1 2 7 Down/up 0 0 8 Deep valley 0 0 9 Picture 3 2 1 10 Dow 1 RqnkS 2 1 2 1 11 Land/water interfaces River 4 4 12 1 12 Descending Banks 2 1 2 1 1 13 Trees Trees 2 5 9 14 Winding river River 4 5 13 is Jungle 1-11 Trees 2 1 1 1 2 CD Score CD 71 00 of Concepts 0 1 2 3 4 5 6 7 8 9 10 12 13 14 3 5 2 2 2 2 1 1 2 1 Viewer: #002 > Date: 14 December 1981 Class: B 0 < CD PxRxQscare OL -n 0 -1 0 0 X 1 1 CD 2 2 (1) (D 3 3 4 4 5 5 6 6 8 7 9 8 10 9 12 10 Q -4 15 11 00 00 X 16 12 Q Q -& 20 13 0" Q :14:] Q 25 Q 0 Q CD CD Weighted Averag ;& 00 Table A-5 (continued) t I I I t I t I t t t t t t I > 0 < (D OL -n 0 % X (D (D Q Q z ra Q 00 Q Immm' 00 > LO ~q ru > Ln 6 ~U FA 0 U CID M 0) a 6 Q 4 00 00 X Q Q 00 Q Q Q Q Begin Time: 12:46 Target: HangZ; Method of Targeting: Coordinates End Time: 4 Length of Session: Session: ReI*- couceptTranscript Concept D P Element of Target VenceQualityScare Number (R) (Q) 1 Down jagged I Sloping rocks 3 2 6 2 Picture I Sloping rocks 3 2 6 3 Flat I - Ba 3 1 3 y 4 Water I Day 3 4 10 5 Green 0 0 6 Picture 2 1 Sloping rocks 3 1 3 7 Down/up 0 . 0 8 Deep valley 0 0 - 9 Picture 3 2 101 Dow I Sloping rocks 3 1 3 11 Land/water interfaces I Peninsula 5 4 13 12 Descending 1 Sloping rocks 3 1 3 13 Trees I t I Trees 2 4 7 river I Bay 3 1 14 Winding 3 is Jungle I Trees 2 3 U Q Scol CD 731 00 of Concepi 0 1 3 4 6 7 8 9 10 11 12 13 141 - 3 5 2 1 3 1 1 1 Table A-5 (continued) inewer: #002 > a Date: 14 December 1981 Class: B 0 < PxRxQ Score -n 0 0 0 ;U (D 2 2 (D 3 3 4 4 5 5 6 6 8 7 9 8 10 9 12 10 -4 00 15 11 00 ;U 16 12 Q 20 13 00 Q age 73, Weighted Aver T1.381 00 I t I t 1. t t I I t I t t I I t > 0 -n 0 (D (D M C) C) C) C) 00 C) 00 0 > M 6 10 C) -4 00 00 Q C) -L 00 C) C) C) 01 C) C) Begin Time: 12:46 End Time: Length of Session: Target - Fernando de Noronha Method of Targeting: Coordinates Session: 4 Concept Rele- D P VanceQualityscore Nu"o ranscript Concept lement of Target (R) (Q) I Down jagged 1 Hills 4 3 10 2 1 Picture 1 1 Hills 4 4 12 3 Flat 0 0 4 Water 1 Ocean 4 4 12 5 Green 1 Vegetation 2 3 6 6 Picture 2 1 Hills 4 1 4 7 Dcpwn/up I Hills 4 1 4 8 Deep valley 0 0 9 Picture 3 2 10 Down 1 Sloping hills 4 1 4 I 11 Land/water interfaces I Island 5 4 13 12 Descending I Sloping hills 4 1 4 1 _ 13 Trees I Vegetation 2 5 9 14 Winding river 1 4 2 7 15 Jungle Vegetation 2 5 9 i 1 1 1 1 CD SCO] 3, 7 00 of Concepi 0 1 2 3 4 5 6 7 8 1 9 10 11 12 13 14 2 4 1 2 1 1 1 2 1 Table A-5 (continued) Viewer: #002 > Date: 14 December 1981 Class: B 0 (D PxRxQscore CL -n 0 0 0 1 1 (D (D 2 2 (1) (D M 3 3 C) 4 4 5 5 P 6 6 8 7 1 9 8 10 9 E 12 10 I P -4 00 5 1 00 16 12 C) C) -L 20 13 00 C) C) C) 01 C) CD CD 22 FWeighted ~2- 73, Aver;7gq 00 t 1. 1. 1. 1 t I I t t I t t I > 0 -n 0 C) C) C C) C) z 6 00 E~ C) 00 0 > 0 E~ 6 C) 4 00 00 C) C) -L 00 C) C) CD C" Begin Time: 12:46 End Time: Lengtb of Session: Target: Punkaharju Viewer: #002 Metbod of Targeting: Coordinates Date: 14 December 1981 Session: 4 Class: B Role- conceptTranscript Concept D P Element of Target VanceQualityscore Number (R) (Q) I Down jagged 0 0 2 Picture I I Islands 4 3 10 1 3 Flat 1 Area 4 3 10 4 Water 1 Lakes 5 4 13 5 Green 0 0 6 Picture 2 1 Lake bottoms 5 1 5 7 Down/up I Lake bottoms 5 1 5 8 Deep valley 1 Lake bottoms 5 1 5 9 Picture 3 2 10 Down 1 Lake bottoms 5 1 5 11 Land/water Interfaces I Lake. 5 4 13 4 12 Descending 1 islands 4 1 13 Trees 1 Trees 2 4 7 14 Winding river 1 Connected lakes 5 2 9 15 Jungle Trees 2 1 2 1 CD CD CD Score 73, 00 of Concepts 0 1 2 3 4 5 6 7 8 9 10 11 12 13 141 2 6 1 2 3 4 1 1 2 2 FxRxQ Score 0 0 1 1 2 2 3 3 4 4 5 5 6 6 8 7 1 9 8 10 9 12 10 > 0 -n 0 C) 15 11 00 00 16 12 C) C) -L 00 C) C) C) C" CD CD CD Weighted AverageT-2.21] Table A-5 (concluded) Approved For Release 20UbJ6LA&NMD0788R001 800050001-8 (U) Two types of overall assessment were chosen to emphasize the versatility of the evaluation procedure, (1) a simple rank ordering based on weighted average scores, and (2) a concept-by-concept, non-parametric, statistical technique. Table A-6 shows the results of the first method, the rank ordering. For convenience, the correct matches are underlined. Table A-6 (U) A RANK ORDERING OF WEIGHTED AVERAGES (U) Fernando Session/TargetInverness Hangd de Noronha-Punkaharju 2 4.13 2.72 0.22 2.11 3 1.12 2.65 2.49 2.42 4 1.21 1.38 2.22 2.21 1 1.53 2.52 2.36 0.90 1 Scores computed with non-uniform target relevance factors. (U) From Table A-6, we see that there were 3 first-place matches and 1 fourth-place match. The probability of obtaining 3 of 4 possible first-place matches from chance fluctuations alone are less than 0.051. The point spread between the best match (Inverness) and the worst match .Ni (Punkaharju) are in qualitative agreement with a subjective "first look" at the quality of the transcripts as well. mw (U) The second analysis determines the significance of the difference between the correct concept/target matches and a control set of matches. All concept/target matches that are not the correct matches act as an internal control set. To avoid any invalid assumptions as to the correct parent distribution, a non-parametric statistical test, the Mann-Whitney U-Test, was chosen for the analysis. 4 41 MW UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 Approved For Release 20UNQA&59JfiQO788 ROO 1800050001-8 (U) It is beyond the scope of this report to review the details of the Mann-Witney U-Test; thus, only the results are quoted here. The probability that the set of correct concept/target matches is statistically indistinguishable from the control concept/target matches is less than 0.071. (U) There are a number of additional statistical procedures that could be used to analyze the results of this evaluation technique. The two cited above, however, represent a spread in complexity that demonstrates the internal consistency of the basic evaluation procedure. With only four similar RV sessions, the evaluation technique nearly reached the 0.05 level of statistical significance with each of the two statistical procedures, a result indicating a successful outcome with regard to the overall assessment procedure. ONO 42 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 Approved For Release 20RISGAM-M&R-0788 ROO 1800050001-8 ROW Appendix B (U) SUMMARY OF EVALUATION TECHNIQUE (U) Step Action 1 Task coordinator defines the evaluation goal. He/she identifies target elements and assigns target element relevance factors as appropriate. 2 Analyst conceptualizes responses and prepares an RV assessment sheet for each response. 3 Repeated concepts are noted in the I'D" column. 4 Copies of the sheets from Item 2 are made; one for each possible target used in the analysis. FOR EACH POSSIBLE RESPONSE/TARGET COMBINATION: 5 Identify a target element for each concept not marked in the I'D" column; mark a 1 in the Ilp" column and write the target element and its relevance factor from Step 1 in the appropriate columns. (Write 0 and blanks if no element can be found.) 6 Using Table 3, assign a quality rating for all present (p = 1) concept/element combinations. 7 Compute the score as follows: a. Calculate relevance (R) X quality (Q) b. Convert R X Q to an integer between 0 and 14 using the conversion table provided. 8 Enter the number of concepts that obtained each possible score in the space provided. 9 Calculate the weighted average using: Ak = 0 .357 1Z f /f Si/Z fi ff where: S. is the score and f. is the number of concepts tAat obtained score J j = 0, 1, 2, ..., 14 10 For each response, rank order the weighted averages. 43 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8 Approved For Release 20oUNGLASSWI6.11788RO01 800050001-8 .10 god MW REFERENCES (U) 1. H. E. Puthoff and R. Targ, "A Perceptual Channel for Information Transfer over Kilometer Distances: Historical Perspective and Recent Research," Proc. IEEE, Vol. 64, No. 3, pp. 329-354 (March 1976). 2. C. Scott' "On the Evaluation of Verbal Material in Parapsychology: A Discussion of Dr. Pratt's Monograph," J. Soc. Psych. Res., Vol. 46, No. 752, pp. 79-90 (June 1972). 3. R. Targ, H. E. Puthoff, and E. C. May, "State of the Art in Remote Viewing Studies at SRI," Technical Session on Research in Psycho- energetics, Proc. 1977 IEEE International Conf. on Cybernetics and Society, Washington, D.C. (September 20, 1977). 4. S. Siegel, Nonparametric Statistics, pp. 116-120 (McGraww-Hill, 1956). no 44 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788ROO1800050001-8