Approved For Release 2QQA?3Jfftb@M- P&RA,00788RO01 500170028-9 Memo To P. McNelis Date 12-Oct-1988 From E. C. May Location G-201 SG1J Subject : Response to McNelis' Comments on Analytics Cc: My response to your comments (18 August 1988) with regard to the then proposed addendum to the Enhanced Human Performance Investigations FY 1988 Technical Mid-year report (now considered the final report to Objective E, Task 4) will be confined to your specific comments 1-13 (see Attachment). 1. The reference to the FY 1986 is accurate. The work described on page 23 of the 87 Mid-year Technical Report is a description of work performed under Object E, Task 2-Search in FY 1986. (Note page B-9 of the FY 1987 SOW, Objective F Task 10, "Building upon the results of the FY 1986 program explore the , limits of quality of remote viewing of analytical information. ") During FY 1986 there was no specific tasking for the analytics program, but I felt that, if successful, it would be applied,in the binary search arena, and therefore was a justifiable portion of that task. Since "analytics" was not part of the formal tasking, the reporting of it appears in the Mid-year Technical Final report for FY 1987 ahead of the required time of the end of 3rd quarter, FY 1987 (see SOW). 2. The techniques referred to in (1) and (2) are not necessarily the same. The description is confusing as written, so I have clarified that point in the report, 3&4. As stated, the objectives were confusing. I have reworded them to more accurately reflect the SOW and the intent of the project. In my opinion only a part of science is of the form of hypothesis formulation, test, and reformulation. The creative part of science manifests itself in determining the appropriate hypotheses. As part of the FY 1986, SRI initiated this ,concept as part of the search task because we felt it was a good area for ,exploration, given the literature and given the application potential, and that it met the spirit of Phase I of the contract-knowledge building. In FY 1986, we did not send a protocol to the SOC in that to do so within a completely unknown area that is being explored would be inefficient. The protocols change from moment to moment in that type of environment, and to wait until an SOC response is inappropriate in this case I will present this whole effort as a specific agenda item during this year's SOC meeting. The protocols that were used in the formal studies as well as in the exploratory studies will be reviewed, and the SOC will determine if the results and conclusion presented in the final report for Objective E, Task 4 are justified, Noe Approved For Release 2001/03/07 : CIA-RDP96-00788ROO1500170028-9 Approved For Release 2001/03/07: CIA-RDP96-00788ROO1500170028-9 5. Yes, this bullet is accurate. I have double checked both our formal reports vow, to you with the "raw" data log and find the following. For the three years, there has been a total of 2224 binary trials (529-FY 1986, 1168-FY 1987, and 527-FY 1988). During that time there were only @ total of 382 non-binary trials (200-FY 1986, O-FY 1987, and 182-FY 1988). 6. The differential comparison point has been added to the report in the appropriate place. 7. A minor error was made in the FY 87 Mid-Year. The p-value quoted there (0.024) used a normal approximation to the binomial. The value in for final report is computed from the exact binomial and is 0.033. The difference is of no significance. Thanks for catching this technical error. The ideal of sensitivity to putative decline effect was intended to mean that the very high baseline (or as rewritten-fiducial point) is not as sensitive to increases in hitting rate as it is to decreases, given that a decline effect might be true. In any case, this section has been completely rewritten. 8. As it turned out, you did have valuable input into the final test-auto-typing of Ready. This section has been rewritten to reflect the final report nature of the document, and all the final trials were completed withing the agreed upon deadline. 9. The wording has been changed to "The hitting rate for these 18 trials was 78%, p ::E@ 1.69 x 10-2, and the effect size was 0.50." 10. In the final report, all binary trials are displayed, so the confusion in the 1qW earlier report is no longer present. 11. The section regarding specific and different internal strategies has been expanded for the final report. 12. What was meant was that since no decline effect was observed, some other hypothesis must account for the data. All the conclusions have been rewritten to reflect the complete data set. 13. Along with yours, Jim's comments have been addressed in the final report. N"011 Approved For Release 2001/03/07 : CIA-RDP96-00788ROO1500170028-9 Approved For Release 2001/03/07 : CIA-RDP96-00788ROO1500170028-9 PROJECT 1291 CLIENT REPORT APPROVAL FORM Transcript of McNells' Comments Dated August 18, 1988 (Re. Addendum to FY 1988 Mid-Year Technical Report) There are a number of serious concerns I have regarding this document and the actual conduct of this experiment which, until clarified and corrected, preclude my giving approval, My primary concern is with the lack'of a rigorous design for the work you are doing with this subject. In FY 86 he did 50 formal trials on a binary task; follow-up in FY 87 was frustrated when, after a large number of pilot efforts, the subject declined formal testing (it is noted that the results of the pilot tests were not encouraging). In FY 88 the subject continued to do vaguely defined tasks which, in my opinion, will make it impossible to attain the primary objective of improving single-bit hitting rates in binary tasks since (a) base line data are not being collected (or, if FY 86 data is being relied on, it is questionable whether 50 trials is sufficient) and (b) we will not have the time to run an adequate number of formal trials in FY 88 to demonstrate improvement. Specific comments follow: 1. Page 3, first paragraph - reference to FY 86 work in Objective E, Task 2 is not accurate. Mention is made to the FY 86 effort for the first time in the FY 87 Mid-Year Technical Report (page 23) under Objective F, Task 10. 2. Page 3, Section A, (1) and (2) - are the "techniques" referred to in (1) the same as those in (2) or are they external versus internal. Please clarify and specify. 3. Section A, general - multiple objectives are confusing and there is no way one could design an experiment at this point regarding the two secondary -objectives since (a) the techniques are not defined, and (b) formal base line data has not been gathered. These secondary objectives should have been eliminated or more scientifically addressed. 4. The SOC should have reviewed the "protocol" for this "experiment," 5. Page 4, Section C, first bullet - implies a binary tasks is the usual protocol, is this accurate since much/most of the data being collected appears to involve multiple choices. 6. Page 4, Section C, last paragraph - alludes to "formal tests." If single bit hitting rates in a binary task are to show improvement, they have to demonstrate improvement compared to like base line data and/or include sufficient trials over time to demonstrate this. The clarification of this is critical. Approved For Release 2001/03/07 : CIA-RDP96-00788ROO1500170028-9 0 Approved For Release 2001/03/07 : CIA-RDP96-00788ROO1500170028-9 7. Page 5, Section D - indicates a p value of ::E@ 0.033, while the FY 87 Mid-Year Tech Report states the p value for the same data'is :E@ 0.024. Also, this paragraph indicates that the FY 86 data was "sensitive to a putative decline effect" - what does this phrase mean an_d where is the data to support it? 8. Page 5, Section E, second paragraph - states "SRI and sponsor's COTR will determine structure of formal test." This "arrangement" is not in the contract and is first mentioned by you in this document, presented to me in mid-August, 1988. 1 will be happy to input, but see my comment number 4 above. Also, it may well prove to be too late to do a formal test (see my introductory comments and concerns). 9. Page 7, Section F, second sentence - unclear. Perhaps this should read "The task was to determine whether ... etc. " 10. Page 7, Section F, last sentence - when you say the "remaining 63 trials" are you referring to 63 of the 81 original binary trials? If the first 18 were thrown out, then it would seem that your total binary trials are just 63 (especially since the first 18 did not fit your "usual protocol" regarding type of targets). 11. Page 8, first paragraph, first sentence - what are the "specific and different internal strategies" mentioned. This is critical information and, if only Ingo knows the answer (assuming he knows), it's not meaningful to us. 12, Page 8, first paragraph, last sentence - what are the "other two hypotheses" you are referring to; where are they so stated? 13. 1 have also attached the comments made by regarding this SG1J document. they are also critical of this work and indicate his contention that the work is seriously flawed. N"W Approved For Release 2001/03/07 : CIA-RDP96-00788ROO1500170028-9