C) C) C) 00 C) C) Iq Q Q C) W CN CD r- C) Q 120 Papers assigned to the target is converted into a standardized average rating score for the target (SAR score). The distribution of the sum of ratings for the controls can be considered as the distribution of ratings associated with that condition. Reduced to the 16el of individual trials we assume this distribution to be t, condition and express an ratings . 'I for he ~P,c 'of average ratings. Thus, all ratings are con- in this distribution e verted into standard normal s res by computing its distance from the mean of average ratings r the controls of the trials and divid- fir, ing it by the standard devia ' n observed for these average ratings. Th - for each trial a r o e a c ne"I Is the differe e between this ' e t n th e b we s and the aver e standard nc ve r e ta ndard nc Since the SAR ores are ba SAR ores are b. which means scor obtained ans ,-r obtained can be considered mal to( :'n.id,red n_al t_~ for controls and targe is z _1s and targe isz e-J: \h Jg top in samples we might comp in calculating a product-mom I ~gapdct_.., I nditln. the two conditions. ~R score for the target is defined as :andard normal score for the target -mal score for target and controls. ed on true standard normal scores, from a normal distribution, SAR scores For each trial the sum of SAR scores ro. Therefore, in the case of related ividual achievement over conditions by correlation between the SAR scores of Although the ra omi4tiN test described above seems sta- u;d f, her s t tistically sound we uaiN Tits tproperties, especially regard ing its sensitivity to detect SP 3 his and we conducted a com 10,0 ..p puter simulation of m ept or2each combination of two S., t~ f z variables, Each e tent Ic,,) n,,, of , trials and 5 pictures per trial and was =a d an g'Jenerating 20 rows of 5 0m) 'I c numbers between ati v4es 0 an in , uive. The two vari d ables involved weir ng 3 -o e s ,e ubj rating b, aN rand amount of ESP. For rating behavior we ma4ipulated the p- ~abllity of selecting rat- ing values of zero h 't of ESP was operationalized. as the oun' as ope T number of subject,,; assHig the highest rati g value to the target ,ati g 1 in addition to what could e expected by chan h \a, From the data obts ditions the sensitivity of that when, for instance, in extreme cases of ratir scores become more sen in the case of 5 ESP hi pected, the binomial yi whereas the SSR score ated one-tailed i)robabil ~ed it can be concluded\hat in most con- he SSR scores is rat' her w , d less than w i simple binomial test Was p7hE d. Only behavior and amount of ~F do the SSR ive than the binomial test. For instance, when in total 5 + 15/5 = 8 hits can be ex- s an exact one-tailed probability of p = .01 olds on average a Z of 1. 7 with an associ- of .045. In the same sim0ation studies Stanford Z-scores were com- puted. We know that the distributions for these Z-scores are non- normal but leaving this aside we found that in most cases the sen- sitivity of t-test evaluations based on Stanford Z-scores is compar- able to that of evaluations based on SSR scores. However, SSR Statistical Issues and Methods 121 scores appear moYvsensitive than S~Anford Z-scores in cases of strong ESP and extih~pe rating belAvior. From these find 9voTmye ractical conclusions can be drawn. ~g In general we must as he ESP influence on the NT data is e t IL ' relatively little. Hence, un there is reason to expect a strong en c un Q th ESP influence in the exTpe J ent 1e binomial test can be assumed to e e er ent ~ be more sensitive than Q ban a e a b sed C v luat t L a evaluati based on the rating values The same applies for e C) 00 f r oe h riment. in eriments in hich no extreme rating be havior can be expecte ,for instance, an experimentT_ in which .pe.t. r i!I -e an 0 -t- 1\ Q oa. to t hd an atomistic approac to the judging is Wed. In that case we g is Q q nzro rating. expect in general nzero ratings assigned o all pictures, and our asigned findings Q in tcr tha a ~e t, show th in that case the SSR scores, as well as Stan- _ CD ford's z scores, e rather insensitive. Q erath risen, W N 0) A METHODOLOGY FOR THE DEVELOPMENT OF A C) CD KNOWLEDGE-BASED JUDGING SYSTEM FOR FREE-RESPONSEa to MATERIALS Dick J. Bierman (Dept. of Psychology, University of Amsterdam) It has been found that certain judges perform consist-ILly better than others when matching targets to a target set. It seems unlikely that this is purely because of the judge's psi, since psi generally does not display consistent behavior. Therefore, it might be hypothesized that it is the (intuitive) knowledge of the specific 00 judge that accounts for his better performance on this task. It C) has been proposed (Morris, EJP, 1986, 137-149) that the use of C) expert systems might help psi researchers in tasks where they lack Q expertise, such as in the detection of fraud. Morris argues that C) the expertise of magicians could be formalized in such a system and CN made available to each individual researcher. Similarly, the exper- a) U) tise of the best judges of free-response material could become avail- W able through implementation of a knowledge-based free-response (1) judging system. This use of techniques from the faeld of artificial Z intelligence (AI) to represent scarce knowledge should not be con- fused with the use of Al techniques for the representation of free- response material (Maren, RIP 1986, 97-99). According to Maren, 0 the free-response material and the protocols should be represented LL in the form of trees in which the nodes are perceivable "objects , a) like "flames," and the links represent relations, like "adjacent to.?' > We expect that focusing our attention on the (knowledge used in 0 the) human matching process might reveal more fundamental informa V. tion about the role of the meaning of the material. It is striking CL that in Maren's proposed representation of complex target material < only visual features are present. Actually, the type of visual matching that Maren proposes to be done by a machine can be bet- ter performed by any sighted human. 122 Papers It should be remarked that the crucial element in the develop- ment of expert systems nowadays is not the implementation of the system but the elicitation of the knowledge that has to be entered into the system. In the case of knowledge about trickery, for in- stance, it is doubtful that one can find experts who are willing to transfer their knowledge. Apart from that, the detection of trick- ery is largely driven by visual information. The proper represen- tation of this visual knowledge might also 00 be a major problem in 1his T_ domain of expertise. in the case of free-response judging one can expect cooperation from the expert judges. Although the material is also visual there are strong indications that simple key words are able to represent these pictures satisfactorily. This conclusion can be drawn from the analytical judging procedures developed by Jahn et al. (Jahn et al., JP, 1980, 207-231). Analytical judging versus knowledge-based judging. It has been found that simple (linear) regression formulas make predictions comparable to or better than human experts in the domain of psy- chodiagnostics. Thus, it is not surprising that the analytical judg- (7) ing procedure very similar to an approach by linear regression also yields satisfactory results. However, it should be noted that al- though its average performance is adequate, this approach fails in pathological cases. It appears that this is because of the failure to take into account any interaction between the predictor variables. In the analytical judging procedure, for instance, the simultaneous occurrence of two elements is counted as the sum of the scores for the cases when they occur alone. Thus, if two elements together have a symbolic meaning that is not contained in either element separately, this meaning is missed in the 00 analytical judging Pro- cedure. A knowledge-based judging system is capable of repre- senting and using this type of knowledge. Matching as classification task. Most problem-solving tasks N can be seen as classification tasks. In the case of matching free- response material from psi experiments, however, there is a special U) cu problem. Since the categories "correct match" and "incorrect match" in psi research are determined by chance, these categories do not have objective attributes. Thusq the task cannot be modeled as a direct classification task. Therefore, we propose to model the matching process as a double classification process. The judge is 0 thought to begin with a classification of the protocol in one of his LL internalized categories. Secondly, this procedure is repeated for each of the members of the target set. Finally, the results of > these classifications are evaluated using overlap measures. If no clear-cut match can be made a secondary evaluation 2 is done which CL takes into account (subtle) interactions among attributes. CL < Knowledge-elicitation methods. The elicitation of knowledge needed to drive expert systems is a "bottleneck problem." This was one of the reasons to simulate the research in machine-learning methods as a means of explicating knowledge. Very oft ,en the rather Statistical Issues and Methods 123 unstructured interview approach is accompanied by so-called rapid prototyping. This means that the system is implemented while the knowledge base is essentially of low quality and incomplete. This might result in poor final systems, like most rule-based systems to date. If this is already the case for q. rather well-understood areas of human expertise it seems unwise to 1 use an unstructured elicita- T- tion procedure for the expertise of free-response judging. In more a structured approaches emphasis is givena to the necessity of a well- cao specified framework for interpretation of the verbal material, be it interviews with, or thinking aloud protocolsr*- produced by, the expe In the present paper it is proposed to combine the structured knowlJO Q edge -elicitation procedure with the use of learning systems. Proposed procedure. The proposed methodology differs from accepted methodologies by using information already present in the data base of classified cases. The elicitationC,4 procedure consists of three major parts: (1) Learn, (2) Pathology(D detection, (3) Can- frontation. In the first phase the expert judge will be interviewed on the set of attributes that are used to describe(D a target picture. Also, the primary set of classes is formulated. After that, a training set 0. of old cases is selected to be presented to a learning system. Each case consists of a series of attribute values together with the clas- sification by the expert judge. After the training the systems are able to classify other cases from the old data base and to compare classifications of the target set with the classification of the proto- cal. The trained system has become a (first-order) model of the expert judge. 00 In the second phase the remainder of the old data base is presented to the "trained" system for judging. If the judging by the system differs from that made in ,C) the past by the human expert we call this a "patholdgical case." 04 In the third phase the human expert (1) is confronted with the set of pathologies. The knowledge engineer might directly ask the CU expert why he or she deviated from the a) model or give him or her the cases to solve again while thinking7~ aloud. Analysis of the thinking-aloud protocol should occur in terms of deviations from them model and thus produce additions to the knowledge base. 0 The automated concept learner. PreviousLL work that tried to apply learning systems to the process of knowledge acquisition used13 systems like Automated Concept Learninga) System (ACLS), which > construct a decision tree from examples. However it was found thato L_ although the resulting decision trees were able to classify new casesd- properly, these trees, which represent r the knowledge of the human .L expert, very often were hardly recognized< by the same expert. This decision-tree representation offered therefore not a fruitful framework for the knowledge engineer to base his or her further > 0 L_ CL CL 124 Papers interviews. This situation is not very different from a representa- tion by linear regression models wl-dch have shown to have consid- erable predictive power. However, the linear regression formula does not make a lot of sense to the human expert. Therefore, we have proposed elsewhere not only to use an ACLS type of learning system but also to use a learning system that is supposed to create a psychologically valid representation of the human expert's knowl- edge. jjj~~~. The "prototype" model has been de- veloped by Rosch. In contrast with linear regression models, the "Prototype" model allows for nonmonotonic relations between the values of the attributes and the class determination. So, apart from an implementation of a decision-tree building system ~ la ACLS, a system has been implemented that is capable of learning categor- ies as proposed in the Rosch model. During the learn phase a training set of old cases, consisting of the values of the attributes and the resulting classification, are offered to the system. The system learns which attributes contribute to which degree to the final classification decision. After the learning phase new cases can be offered to the system which will calculate an overlap score of the new instance with the "prototype" of a class. Concluding remarks. Current work by the present author using a similar knowledge-elicitation approach in the domain of psychodiagnostics is promising. It appears that "intuitive" knowl- edge can be elicited with the proposed approach and implemented as a moderator of a primarily pattern - m atchin g- based classification. NEW INTERPRETATIONS OF ESP LITERATURE* Q Q Q 00 T_ Q Q A CRITICAL REVIEW OF THE DISPLACEMENT EFFECT It Q Julie Milton (Dept. of Psychology, University of Edinburgh, Q Q 7 George Square, Edinburgh EH8 9JZ, Scotland)** W 04 The lldisplaced~ent effect" in ESP research refers to a situa- Q tion in which the perNpient, ins ad of describing the intended Q target for a particular~rila de cribes some other experimental ma- ,~t r terial. Despite the fact~that o e 100 papers have dealt with some (D aspect of the displaceme \ef t since the effect caught the general interest of parapsycholo N 940 , no exhaustive review of the 1 0 displacement literature has a eared. It was felt that such a re- view would be timely for a n er of reasons, partly because the < attitude of researchers these day to the apparent occurrence of displacement is generally on of i ' ation, whereas earlier research- ers reacted with a more pos tive (an hence possibly more produc- tive) interest; partly beca e recentl some researchers have sug- gested that in the context f finding 'mits for psi, the clrcumstances~_ 00 n' under which displacemen~t ccurs and e extent to which displace- Q ment is a "deliberate" err r or a genui e error on the part of the Q percipient may have som, theoretical im ortance. Another reason Q for a review would be examine the ch racteristics of displacement Q t' 04 as a phenomenon of inte st in itself. (D U) In the past, rese rchers have explo d two main lines of re- M search with respect to isplacement; the fi t has involved the pos- sibility of I t hi between scoring on targets of different displacements, and the second, the possibility of a relationship be- tween displaced scorin and psychological and situational variables. 0 LL Concerning the possibility of a relationship between scoring on targets of differe t displacements, there are a couple of poten- tially important st ' tical artifacts that apply to forced-choice > studies which can give rise to the appearance of displacement 0 CL CL *Chaired by Erlendur Haraldsson. **I am grateful to the Perrott-Warrick Studentship in Psychical Research for financial support during the writing of this paper.