Approved For Release 2000/08/10 CIA-RDP96-00789ROO3800020001-8 B. (S/NF/SS-2) EVALUATION PROCEDURES: 1. (S/NF/SS-2) Operational Phase: (S/NF/SS-2) The Project Manager will assess the value of the operational phase of the SUN STREAK program by requiring all data consumers (i.e., operational task originators) to furnish him with data accuracy and data utility assessments. Table 1 illustrates the general format and assessment scale (0-3) for this evalUation. Specific data categories may vary depending on the nature of the task or on specific objectives; however, the overall approach will remain the same. (S/NF/SS-2) The Project Manager will enter results of the accuracy and utility assessment into the project data base. He will make periodic reviews (quarterly) of this data to assess overall effectiveness. (S/NF/SS-2) The Project Manager, in coordination with the task originators and the Intelligence Community Task Coordinating Group, will develop an additional measure of program value. This measurement (overall program value) is designed t'o measure the net worth or total contribution of the task in relation to other tasks and to the overall intelligence mission. Data accuracy and utility may be high for a particular consumer, but the results may not have a significant overall impact. This program value scale is similar to the utility scale; however it is based on specific measures of over-all benefit. This value scale is as follows: Program Value Scale 0 - Little or no over-all program value. 1 - Some program value (e.g., helped refine estimates). 2 - Moderate value (e.g., identified new data, narrowed down possibilities). 3 - High value (e.g., led to significant cost savings, identified critical high priority S&T or operational activities). 4 - Exceptional value (e.g., predicted major events/activities, located lost/missing resources or personnel). The value evaluation will be used in conjunction with other program evaluation parameters (accuracy, utility) for determining program accom- plishments and for defining future program directions. NOT RELEASABLE TO FOREIGN NATIONALS 3 Approved For Release 2000/08/10 : CIA-RDP96-00789ROO3800020001-8 'U)For the summary evaluation, check the followingboxes the uracy of the erial. please as to acc submitted mat ACCURACY Site Contact, Little with Not > Correspondence Mixed Results Good Excellent Unknown Applicable > 0 1 2 3 0 0 S)Gfpgraphical locale descrip- tP)n (terrntn, wnter, etc.) n -n S)Lfi)rge-scale martmade elements 0 - 1 1 (*ties,, buildings, silos, X (D docks, railroad lines, agrfields, etc.) S)90(noll-scale manmade elements (D (bhtennas, computers, tanks, M iles F-1 CD offices etc mp ) Ws CD . .* CD CD S) Gaeral target ambience (re- CD Agrch, productton, adminis- 00 t AtionP storage, troop move- -A is CD D -1 D ments, naval activity, air aElivity, weapons testing, 0 etc.) S)R evant specific activities Qyclear testing, missile 0 fl7ingp CBW storage., ELINT _U F-1 to mWitoring., - etc.) 1 6 S) Pjanonality information CD (1gygical descriptions, 4 00 atftion.q responsibilities to P I ED X p9n9J9 etc.) CD - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - CD - - - - - - - - - - - - - - - - - - - - W - - - 0 1 2 3 00 Cannot de- 00 be CD ) O~rall utility None ID arginal seful ery sefulFl at this time CD terminedID Q- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - CD - - - - - - - - - - - - - - - - - CD CD CD U)Rfinitions for the accuracy CD scale: 0.1- Little correspondenc( Self explanntory. . . . . . . FO- Site contact with Mixture of correctand incorrect nts, but enoughof the eleme former to mixed results indicate source probably the target has accessed site, 2 - Good 0 . 0 0 Good correspondencewith several t some correct information. elements in matching, bu 3 - Excellent . . . . . . . Good correspondencewith unambiguous elementsand relatively . . . . unique matchable little incnrrect formntton. in TABLE I Approved For Release 2000/08/10: CIA-RDP96-00789ROO3800020001-8 2. (S/NF/SS-2) Trainingand Operational Qualifying Phase: (C) There are two other aspects of this program that will require evalu- ation by the Project Manager or by his representative; the basic training phase and the operational qualifying (or operational training) phase. (S/NF/SS-2) The basic training phase consists of a large variety of easy-to-verify training tasks that are designed for various stages of proficiency. Overall evaluations will'be based on the accuracy scale used for operational projects and on an additional quantitative procedure. The accuracy scale will permit assessment of an individual's progress through the six training stages. The quantitative procedure (i.e., concept/element analysis) will allow assessments of degree of accuracy and will provide the basis for determining what general classes of operational projects a viewer can best perform. (S/NF/SS-2) The operational qualifying phase follows satisfactory completion of the training phase. This phase consists of tasks that simulate operational projects but differ in that ground truth can be readily determined, and there are no operational consumers. The main purpose of this phase is to systematically quantify the operational readiness of each viewer for a wide variety of potential operational projects. Data from this phase will also help identify the best types of operational tasks for remote viewing (RV) applications. (S/NF/SS-2) Each RV source will perform several hundred operational qualifying tasks per year. These tasks will include representative examples from DIA's intelligence production codes and from other potential operational requirements. Evaluation will be performed by the Project Manager or his representative and will be based on the accuracy scale used for operational projects. In addition, accurate estimates of operational reliability and utility will be made using quantitative methods (i.e., concept/element analysis, pattern evaluation). These evaluation procedures will require extensive use of a dedicated data base management system for storing large amounts of information and for performing appropriate analyses. (S/NF/SS-2) It is anticipated that a limited operational capability will be achieved by mid-1987. Some operational projects will be initiated in FY 1986; however, the type of task accepted will depend on the nature of the proposed task and on capability levels of available RV sources. NOT RELEASABLE TO FOREIGN NATIONALS 5 Approved For Release 2000/08/10: CIA-RDP96-00789ROO3800020001-8 Approved For Release 1!==-RDP96-00789ROO3800020001 6. Operational Activity (U): a. Tasking (U): N (S/NF/WN) All operational tasks will be reviewed and prioritized by DIA/DT. These tasks can originate from DIA/DT and from elsewhere within the intelligence community or from other potential users. Although the *&Lnism for generating/receiving potential tasks from users has not yet been finalized, it is anticipated that the Intelligence Community Task Coordinating Group may serve this role. These potential tasks would then be transmitted to DIA/DT for disposition. (S/NF) There is an important condition for accepting potential operational tasks: It is that feedback data must be provided to DIA/DT so that adequate evaluation of results can be performed. For sensitive projects, special caveats and controls will be established. This procedure will assure the user cowunity that details of the iensitive task will only be divulged to those who have the need to know. b. Performin2 Operational Tasks (U): (1) Task Management (U): (S/NF) To facilitate optinum accomplishment of all routine and quick reaction operational tasks, DIA/DT will establish a detailed procedure to insure that all operational needs can be immediately resolved and implemented. Activities/functions addressed would include: - 24 hr. availability of management approval authority Approved For Relebd6t2etWolpIts)CIA-RDP96-00789ROO3800020001-8 Q Q 1Q"W-'k -W-'% r-. , LJ1!A j [V P J I Approved For Release 2 . gyjEjw,&%tm%rmut-vio-uut'89ROO3800020001-8 24 hr. availability of people who can perform operational tasks (viewers, interviewers); Resource allocation authority (people required, specific assignments, travel); - Procedures for interface with the user community (for intelligence and outside); - Procedures for interface with others (i.e., VO, HUMINT Committee); - riethods/procedures for transmItting RV data to users (i.e., special courier); - Procedures for preliminary evaluation of data and decision-making for follow-on actions; - Procedures for maintaining security controls, and safeguarding viewer identity; These and others'would be constantly refined to improve over-all responsiveness and effectiveness of operational activities. Approved For Release 2000/QiMiGk!1=6-00789ROO3800020001-8 Ljj-uk:)x V177- Approved For Release 2000_/64PP_1ZM 9-6-00789ROO3800020001-8 (2) (S/NF/WN) Methodology: A consistent and scientifically-valid approach to methods and procedures for active conduct of operational sessions will be developed. These will be established as fi rm project requirements, and any exception Me., due to error,.unforseen circumstances) will be clearly noted. The user community personnel involved in operational tasks (i.e., as observers, or for data analysis) will-be informed of all methods/procedures in advance of the conduct of any operational task. This is necessary for operational consistency, for data evaluation, and for making decisions on how best to apply the RV data. Specific methods/ procedures will be developed-for each operational task in accordance with the nature of the task (i.e., QRT, long-term), type of targeting methods required, and the type of training program used for developing the viewers' capability. (3) Records M: (S/NF) Detailed records will be maintained for all RY operational sessions. Specific data will include people involved, chronology of events, data provided to the viewer, targeting methods used, degree of knowledge of target information by those involved in the session, or any other issue pertinent to performing the session and evaluating the results. All these factors will be identified and clear documentation procedures will be established prior to initiation of any operational project. The actual RV data will also be carefully recorded and controlled, and will be entered into the appropriate data base. Data format will include verbal records (tape recordings) as well as raw data (written description, sketches). These records will be stored in such a way as to protect identify of the user and the viewers involved, and will be subject to all other security controls estaAbWWeVft& available to any authorized individual or project review group as necessar4.,,,._ Approved For Release 20 6-00789ROO3800020001-8 0144=- 9 c. Evaluation of Data (U.): (S/NF/WN) A data package with pertinent raw RV data, and a summarized version performed by project personnel, will be returned to the task originator and the IC Task Coordinating Group via DIA/DT. All session methods and conditions will be included with this data package. The task originator wili provide an evaluati;n of the data according to pre-established criteria. The originator will return this evaluation to DIA/DT along with sufficient background and feed-back data so that the basis of the evaluation can be clearly' understood. Any difference of opinion will be discussed with the task originator and noted in the project records. 1n cases where evaluation is only partial (i.e., for tasks that can't be immediately evaluated) the file ,*III be kept active until such a time when sufficient data becomes available. The task originator will also indicate how the data could be (or was) utilized, and will describe the impact such data had on resolution of the specific problem of interest. Additional,data analysis will be performed on the raw data in the growing data base to evaluate trends, patterns, statistical implications, operational constraints, reliability/utility, and other aspects pertinent to data utlization. d. Rep2rting of Results M: (S/NF) In addition to data reports provided to the users, over-all project reports will also be prepared on a periodic basis. They will contain general information on types of tasks, over-all results, summaries of task originator evaluations, statisitics/trends, and other relevant data as Approved For Release 2000/.08/10 : CIA-RnP96-00789ROO3800020001-8 -,.%a - - -Miff. 1!Unb-JL__ Approved For Re lease 2~008~~~~-00789RO03800020001-8 required by DIA/DT. These reNrts would not contain sensitive data; however such data could be provided separately on a strict need-to-know basis pending appropriate data release approval. 7. Process Flow Chart (U): (C/NF) All project procedures and operational activity identified in the section for trai-ning-end for operational projects will be developed in detail in a Project Operations Manual. A process fl ow chart will also be prepared. QU IQ;W Approved For Release 2UOUi%BAIDZALmwLDP96-00789ROO3800020001-8