Memoy & Cognition 2001, 29 (1), 165-175 Popositional easoning: The diffeential contibution of ules to the difficulty of complex easoning poblems FRANK RIJMEN and PAUL DE BOECK Univesity of Leuven, Leuven, Belgium In Expeiment 1, complex popositional easoning poblems wee constucted as a combination of seveal types of logical infeences: modus ponens, modus tollens, disjunctive modus ponens, disjunctive syllogism, and conjunction. Rule theoies of popositional easoning can account fo how one combines these infeences, but the difficulty of the poblems can be accounted fo only if a diffeential psychological cost is allowed fo diffeent basic ules. Expeiment 2 uled out some altenative explanations fo these diffeences that did not efe to the intinsic difficulty of the basic ules. It was also found that pat of the esults could be accounted fo by the notion of epesentational cost, as it is used in the mental model theoy of popositional easoning. Howeve, the numbe of models as a measue of epesentational cost seems to be too coasely defined to captue all of the obseved effects. Deductive easoning is the ability to daw logically valid conclusions fom given infomation (the pemises of a easoning poblem). One fom of deductive easoning is popositional easoning: deductive easoning with connectives, such as if/then, and, o, o not. Fo example, when the following popositions ae given: John is in Bussels. If John is in Bussels, then Mike is in Dublin. one can easily deduce Mike is in Dublin. Fank Rijmen was suppoted by the Fund fo Scientific Reseach, Flandes (FWO). We ae also gateful to Walte Schaeken and Lance Rips fo thei helpful comments on an ealie daft of the manuscipt. Coespondence concening this aticle should be addessed to F. Rijmen, Depatment of Psychology, Tiensestaat 102, B-3000 Leuven, Belgium (e-mail: fank.ijmen@psy.kuleuven.ac.be). Accoding to ule theoies of deductive easoning (e.g., Baine, 1978; Baine & O Bien, 1991; Baine, Reise, & Rumain, 1984; Macnamaa, 1986; Osheson, 1974, 1975; Rips, 1983, 1994; Rips & Conad, 1983), human beings solve a logical poblem in thee stages. Fist, in the encoding stage, the logical stuctue of the poblem is uncoveed and epesented in woking memoy. Second, in the easoning stage, one o moe basic easoning ules ae applied to this language-like stuctue. Finally, in the decoding stage, the esult of the easoning pocess is tanslated back into the fom demanded by the task fo example, natual language. Rule theoies concentate on the second stage and on the basic ules used in that stage. The set of basic ules is assumed to be a epetoy of syntactic, logically valid infeences. Thoughout the text, the tem basic ule efes to a psychologically elementay easoning step, such as those poposed by ule theoies. The tem (logical) infeence, on the othe hand, is used as a moe geneal tem in two espects: Thee ae logical infeences fo which thee is no basic ule (e.g., modus tollens), and the tem logical infeence is not confined to a paticula psychological theoy. Each of the basic ules specifies how to eason with a paticula sentence connective, such as if, o, and, o not. Fo example, the basic ule fo modus ponens states that: Given the popositions of the fom IF p, THEN q and p ; conclude q. In a common notation: if p, then q p ------------------ q Accoding to ule theoies, solving complex easoning poblems consists of solving consecutive simple easoning poblems fo which basic ules ae available. Rule theoies assume that people will conside a conclusion to be tue if they succeed in foming a chain of infeences linking the (tue) pemises with the conclusion to be evaluated. In ule theoies, poblem difficulty is accounted fo in tems of the easoning opeations: The length of the mental poof and the difficulty of the basic ules ae assumed to detemine the difficulty of a poblem. The moe basic ules thee ae that must be applied and the moe difficult they ae, the moe difficult the poblem. Hence, accoding to ule theoies, poblem difficulty is a function of the numbe and the difficulty of the constituent basic ules. Both Rips (1983) and Baine et al. (1984) tested this pediction, although a somewhat diffeent method was used in each study. Rips (1983) tested a model 165 Copyight 2001 Psychonomic Society, Inc.
166 RIJMEN AND DE BOECK fo complex easoning poblems that was based on availability paametes fo the basic ules. Given that a basic ule is applied coectly if available, the difficulty of the basic ule lies in its availability. The paametes wee estimated fom the popotions of coect answes on the complex easoning poblems. The coelation between pedicted and obseved popotions was.93, cooboating the ule theoy of Rips (1983). Baine et al. (1984) have tested thei theoy by egessing the peceived difficulty of complex poblems on the assumed constituent basic ules, coded as dummy vaiables. The esulting weights could be intepeted as the difficulty of the basic ules, since the peceived difficulty was modeled as a sum of basic ule weights. They obtained a multiple coelation of.92. When only the numbe of basic ules was used as a pedicto, the coelation was only.79, which is significantly lowe. When actual difficulty (an eo index) was used as a dependent vaiable, the coesponding coelations wee.73 and.57, espectively, the diffeence not eaching significance. These high coelations do cooboate the ule theoies of popositional easoning, but the use of diffeent paametes fo each of the basic ules is a stong indication that the ules diffe as to thei difficulty. If thee wee diffeences, poviding a basis fo the diffeences would not only enhance the compehensibility of ule theoies, but would also ende them moe pasimonious by educing the numbe of fee paametes (fom one fo each basic ule to, ideally, one undelying paamete). In this espect, the ealy wok of Osheson (1974), the fist poponent of the ule theoies, offes some inteesting ideas. Osheson (1974) poposed diffeent pefomance models (egession equations) to account fo success ate in a boad ange of easoning poblems. All that these models have in common is that the weight of the basic ules in the egession equation is conceptualized as the amount of mental pocessing space o computing aea equied by the application of the basic ule. On the one hand, because Osheson (1974) povides no account of how to detemine the amount of computing space fo the basic ules, we ae left with no moe than an indication of the poblem. On the othe hand, teatment of the diffeential contibution of basic ules to the difficulty of a complex easoning poblem as a diffeential amount of equied computing aea is elated to the notion of epesentational cost. This is the way in which difficulty is conceived in the mental model theoy, to which we now tun. The mental model theoy (Johnson-Laid, 1983; Johnson-Laid & Byne, 1991; Johnson-Laid, Byne, & Schaeken, 1992) is anothe main appoach to popositional easoning. In mental model theoy also, easoning consists of thee stages. In a fist stage, called compehension, a mental model is constucted fo the poblem, which is a mental epesentation of a situation consistent with the pemises. This fist stage can be consideed as an encoding stage. Second, in the desciption stage, a putative conclusion is geneated on the basis of this model. This putative conclusion contains infomation not yet explicitly asseted by the pemises. If no putative conclusion can be geneated, the answe is that nothing follows fom the pemises. The thid stage, the only eally deductive stage (Johnson-Laid & Byne, 1991, p. 36), is one of validation, since paticipants ae assumed to seach fo altenative models of the pemises that falsify the putative conclusion. If a falsifying model is found, it is necessay to etun to the second stage, to seach fo anothe putative conclusion that holds in the models constucted thus fa. If no falsifying model is found, the conclusion is consideed tue. In the mental model theoy, moe attention is paid than in ule theoies to the fist stage, the encoding o compehension stage, in which the poblem is epesented. Given the conditional if p, then q, one can constuct fou possible situations (coesponding to the ows of a tuth table), of which only the situation p not q is inconsistent with the conditional. Howeve, given that one wants to minimize the epesentational cost because of the limited capacity of woking memoy, human beings ae assumed to epesent initially as much infomation as possible in an implicit way, instead of epesenting all thee possible models explicitly ight fom the beginning. The assumed initial epesentation consists of one explicit model, in which it is indicated that p is exhausted with espect to q ( p within squae backets) that is, that p cannot occu with not q and of an implicit model indicating that thee might be cases in which p is not pesent (the thee dots): [p] q... If p is given as a second pemise, the fist model is selected, and the implicit model eliminated, with the esult of [p] q This leads to the conclusion q, which cannot be falsified. If not q is given as a second pemise, the explicit model is eliminated, and all thee models that ae consistent with the conditional have to be made explicit (o fleshed out ): p not p not p q q not q Only the thid model is consistent with the pemises, esulting in the conclusion not p. Poblems diffe as to the numbe of explicit models needed in the compehension (encoding) stage to each a subsequent conclusion, and this is what detemines the epesentational cost of a poblem. Fo example, the empiical obsevation that modus ponens is easie than modus tollens can be accounted fo by the mental model theoy because the fist equies only one explicit model (the initial one) and the latte equies thee (all thee that ae consistent with the conditional). Johnson-Laid et al. (1992) epot seveal expeiments that confim the cen-
RULES AND DIFFICULTY 167 tal pediction of the mental model theoy that a poblem is hade the geate the numbe of explicit models that have to be constucted and epesented in woking memoy. In contast with ule theoies, the mental model theoy does pedict diffeences in difficulty between logical infeences that coespond to single basic ules accoding to ule theoies, since, depending on the basic ule, a diffeent numbe of explicit models is involved. As has aleady been illustated, modus ponens equies the constuction of only one explicit model. A disjunctive syllogism, howeve, equies the constuction of two explicit models. The disjunction p OR q has the following initial epesentation (following Johnson-Laid et al., 1992): p q The categoical pemise NOT p eliminates the fist and completes the second: 2 p q The conclusion q cannot be falsified. Also, a disjunctive modus ponens equies the constuction of two explicit models. Following Johnson-Laid et al. (1992, p. 428), the pemise IF p OR q, THEN is initially epesented as [p]... [q] The categoical pemise p eliminates the second model and the implicit model, which yields. Again, this conclusion cannot be falsified. Thus, the mental model theoy pedicts that modus ponens is easie than both disjunctive modus ponens and disjunctive syllogism. In an expeiment (Johnson-Laid et al., 1992) in which people had to geneate conclusions (as opposed to the conclusion-evaluation paadigm of the ule theoies), a modus ponens infeence was found to be simple (made moe often) than a disjunctive syllogism infeence with an exclusive disjunction. In conclusion, mental model theoy does offe an explanation fo diffeences in difficulty between basic ules in tems of woking memoy load. It is ou aim to test futhe whethe it is necessay to assume that basic ules diffe in thei contibution to the difficulty of complex easoning poblems. If so, ule theoies will have to be complemented with a pinciple that can account fo these diffeences and, hence, fo the difficulty of complex easoning poblems. Insofa as diffeences between basic ules ae found, we will compae these diffeences with the pedictions of the mental model theoy. Independent of the outcome, we will also test the mental model theoy on the same data. If the diffeences ae as pedicted, this will cetainly stengthen the claim of mental model theoists that the difficulty of easoning poblems is pimaily affected by epesentational cost (the numbe of explicit models equied to solve the poblem). EXPERIMENT 1 Analogous to Baine et al. (1984) and Rips (1983), we constucted complex easoning poblems by combining the basic ules, and we used a conclusion-evaluation paadigm (as opposed to the conclusion-geneation paadigm used by Johnson-Laid et al., 1992). In contast with the studies just mentioned, we used only a limited numbe of basic ules. This allowed us to combine the basic ules factoially, while keeping the total numbe of poblems to be solved easonable. By combining the basic ules factoially, coelations and tadeoffs between paametes wee avoided as much as possible. The following basic ules wee focused on in this study: conjunction (p, q \ p AND q), modus ponens (IF p, THEN q; p \ q), disjunctive modus ponens (IF p OR q, THEN ; p \ ), disjunctive syllogism (p OR q; NOT p \q), and solving a contadiction (p, not p \ incompatible). We used poblems with a easonable eo ate to avoid a ceiling effect, and we woked with a lage numbe of paticipants than in the pevious studies, in ode to obtain eliable success popotions pe poblem. Apat fom the basic ules, fou othe factos wee manipulated: content, ode of pesentation of the pemises, tuth2 value of the conclusion to be evaluated, and diectness of the easoning poblems. The last facto equies some explanation. Baine et al. (1984) made a distinction between poblems that can be solved by applying one o moe basic ules in a quasi-automatic way (diect easoning poblems), which ae supposed to be univesally available, and poblems that equie the application of some stategy, which is consideed not to be univesally available and, hence, not a pat of the epetoy of basic ules (indiect easoning poblems). One such stategy is eductio ad absudum, which, accoding to ule theoies, is used to solve a modus tollens (a logically valid infeence fo which thee is no basic ule assumed to account fo its geate difficulty): given: (1) IF p, THEN q (2) NOT q easoning: (3) suppose p supposition (4) q modus ponens 1+3 (5) incompatible contadiction: 2 and 4 cannot both be tue (6) not p eductio ad absudum, a supposition leading to a contadiction must be false. In half of the poblems, the modus ponens, which is a diect easoning poblem (thee is even a basic ule fo it), will be eplaced with a modus tollens, which equies an indiect way of easoning. Method Paticipants. Two hunded fouteen high school students between 16 and 19 yeas of age paticipated in the expeiment.
168 RIJMEN AND DE BOECK Mateial. Each paticipant eceived a booklet, with 2 pages of instuctions and 10 pages with poblems. On each of the latte, 3 poblems wee pesented, fo a total of 30 poblems. Unde each poblem, fou esponse altenatives wee pesented: necessaily tue necessaily not tue undecidable I don t know It was explained in the instuctions that the paticipants should choose necessaily tue if they believed that, given tue pemises, the conclusion must be tue; necessaily false if they believed that the conclusion must be false; undecidable if they believed that, given the pemises, the conclusion could be tue, but also could be false (hence, that not enough infomation was given); and I don t know if they could not wok it out. It was explained to the paticipants that undecidable is a genuine conclusion that could be coect fo some of the poblems, wheeas I don t know means that one cannot choose among the othe thee esponse altenatives o has given up solving the poblem. I don t know was offeed as an altenative esponse in ode to avoid guessing behavio. Fou andom sequences wee geneated fo the poblems, equally divided ove the paticipants. Design. Each paticipant had to evaluate the conclusion of 30 poblems. Fo 12 poblems, the coect conclusion was necessaily tue ; fo 12 othes, the coect conclusion was necessaily not tue ; fo the othe 6, the coect conclusion was undecidable. The 24 poblems with eithe a tue o a false conclusion wee constucted by combining pemises as illustated in Table 1, esulting in six poblem types. A lette epesents an elementay poposition, possibly containing a negation. It was neve necessay to pefom a double negation (e.g., fo Poblem Type 3 [see Table 1], q neve epesented a poposition with a negation). Modus ponens vesus modus tollens was the fist facto in the design. Fo 12 easoning poblems, one infeence was a modus ponens (left column; diect easoning poblems). Fo the othe 12, this infeence was a modus tollens (indiect easoning poblems). The second facto of the design, which efes to the infeences that ae combined with modus ponens o modus tollens, had thee levels: modus ponens + conjunction, disjunctive syllogism, and disjunctive modus ponens. This facto is othogonal with the fist and makes a distinction between thee sets of eight poblems. Fo the fist set of eight easoning poblems, poblems of level modus ponens + conjunction (fist ow of Table 1), the second infeence was a modus ponens (equiing the application of one basic ule). This modus ponens infeence could be made independently fom the modus ponens o modus tollens infeence aleady discussed. The esults of both conditional infeences subsequently had to be combined with a conjunction, which was then the thid infeence (again equiing one basic ule). Fo the second set of eight poblems, poblems of level disjunctive syllogism (second ow of Table 1), the second (and last) infeence to be made was a disjunctive syllogism (equiing one basic ule). The categoical poposition of the disjunctive syllogism was the esult of the modus ponens o modus tollens infeence fom the fist step. The thid set of eight poblems, poblems of level disjunctive modus ponens (thid ow of Table 1), was simila to the second set, except that the second infeence was now a disjunctive modus ponens (also equiing one basic ule). A thid facto, othogonal to the fist two, was the tuth value of the conclusion to be evaluated: tue o false. Fo 12 poblems, the conclusion to be evaluated was tue; fo the othe 12, the conclusion to be evaluated was false. Fo the latte, the conclusion to be evaluated contadicted the conclusion that followed fom the pemises. Solving this contadiction was an additional infeence (equiing one basic ule) fo these poblems. To avoid the possibility that the pesence of a negation would act as a cue fo the coect answe, some popositions of the pemises contained a negation, although no poblem equied an infeence involving a double negation. The fouth facto, manipulated othogonally, was the content of the poblems. Half of the poblems wee about people in cities o counties, fo example John is in Pais. The othe half wee about a functioning factoy fo example The geen light flashes. The othogonal manipulation of these factos esulted in 2 * 3 * 2 * 2 5 24 cells in the design, each coesponding to exactly one poblem. A fifth facto, not othogonally manipulated because this would ende the expeiment too long, was the ode of pesentation of the pemises: gouped vesus ungouped. Fo each type of content, the pemises of a given infeence wee pesented in a goup in half of the poblems (as pesented in Table 1) and sepaated by anothe pemise in the emaining half of the poblems (e.g., fo Type 1: p became the fist pemise and IF p, THEN q the thid, with a pemise [ IF, THEN s ] of the othe modus ponens in between). Fo the content of a factoy, these poblems wee of Types 2, 3, and 6, and fo the content of people in cities o counties, these poblems wee of Types 1, 4, and 5. Apat fom the 24 poblems just descibed, 6 easoning poblems with undecidable conclusions wee added as fille items. These poblems wee constucted by eplacing the modus ponens o modus tollens infeence with an infeence involving an affimation of the con- Table 1 The Six Poblem Types of Expeiment 1 Modus Ponens Modus Tollens Modus ponens + 1. IF p, THEN q 2. IF p, THEN q conjunction p p IF, THEN s IF, THEN s NOT s -------------------- ----------------------- q AND s q AND NOT Disjunctive syllogism 3. p OR q 4. p OR q IF, THEN NOT q IF q, THEN NOT --------------------------- ----------------------- p p Disjunctive modus ponens 5. IF p OR q, THEN 6. IF p OR NOT q, THEN IF s, THEN q IF q, THEN s s NOT s -------------------------- -----------------------
RULES AND DIFFICULTY 169 sequent o a denial of the antecedent, espectively (neithe is valid in classical logic). In this way, we ensued that the coect conclusion was sometimes something othe than necessaily tue o necessaily not tue. Pocedue. The expeiment was conducted in goups of 20 paticipants, moe o less. At the beginning of the expeiment, the instuctions wee biefly explained. The paticipants wee asked not to guess but to choose the esponse altenative I don t know in cases in which they had not made one of the thee evaluations: necessaily tue, necessaily not tue, o undecidable. It was stessed that they would not be allowed to wite anything down while solving the poblems. The paticipants had to complete the task within 50 min, and they all succeeded in doing so. Results We will concentate now on the analysis of the difficulty of the 24 easoning poblems. The six fille poblems wee not included in the analysis. The ange of the difficulty of the 24 poblems, expessed in popotion of coect answes, goes fom.44 to.98, with a mean of.72. The popotions of coect answes to the 24 poblems ae given in Table 2. These popotions wee conveted into thei logit: logit(p) 5 ln [p/(1-p)], whee p 5 popotion of coect answes. Woking with the logit avoids the poblem of a esticted ange between 0 and 1 and compession in the neighbohood of these limits. The logit was the dependent vaiable in the subsequent egession analyses, following the egession equations of Baine et al. (1984). Since the length of the poblem (expessed in numbe of wods) did not coelate significantly with the objective difficulty of the poblem, as expessed in the logit measue ( 5.30, n.s.), it was not included in the pedicto set. Given that the logit of the popotion coect answes is an invese measue of difficulty, it is clealy not the case that poblems with a longe woding ae moe difficult. In a fist egession equation, the pedicto was the numbe of basic ules necessay to solve the poblem. Hence, each basic ule was consideed to be of equal difficulty. Appendix A descibes how the value on this pedicto was computed fo the diffeent poblem types. The numbe of basic ules explained only 16% of the vaiance, which is not significant [F(1,22) 5 4.14, n.s.]. Accoding to Baine et al. (1984), a modus tollens infeence equies fou easoning steps (see the intoduction). It is unclea, howeve, whethe it is also appopiate to count fou basic ules, as we did. Fist, a poblem involving a modus tollens infeence is an indiect easoning poblem (Baine et al., 1984). It cannot be solved using the epetoy of univesally available basic ules (see the intoduction). Second, Rips (1994, p. 116) conceptualizes thee of the easoning steps involved as one, although cognitively complex, ule. Thid, when the conditional is intepeted as a biconditional, a simple solution exists fo a modus tollens infeence (Baine et al., 1984, p. 343). Fo all these easons, it seems woth tying out an altenative that leaves the numbe of ules fo a modus tollens infeence unspecified. In ode to assess the pedictive value of the numbe of ules independently fom the numbe of basic ules assumed fo a modus tollens infeence, a second egession analysis was conducted with two pedictos: a binay vaiable, which coded fo modus ponens (value of one) vesus modus tollens (value of zeo; see below fo the way the factos of the design wee coded into binay pedictos), and the numbe of basic ules minus the numbe of basic ules fo modus ponens/modus tollens. The two pedictos explained 28% of the vaiance [F(2,21) 5 4.01, p <.05]. The educed numbe of basic ules had no significant effect in the egession analysis (t 5 0.722, n.s.). Hence, the numbe of basic ules does not seem to have much pedictive value beyond the diffeence between modus ponens and modus tollens, at least fo the set of poblems unde consideation hee. To investigate whethe the elation between the numbe of basic ules and the logit of the popotion of coect answes (an invese measue of difficulty) is monotonic (without having to make assumptions about the paticula fom, as in the linea egession analysis), we computed the Speaman ank coelation. The coelation was 2.42, which although not vey high, is significantly smalle than zeo ( p <.05). Nevetheless, this is not much bette than the Peason coelation between numbe of basic ules and difficulty (in logit; 5 2.40), indicating that the monotonic elation between the numbe of basic ules and the logit of the popotion of coect answes does not depat stongly fom lineaity. The next step in the analysis was to allow fo a diffeential weighting of the basic ules, as in Baine et al. (1984). Theefoe, the thee factos of the design efeing to basic ules wee coded into fou binay vaiables as follows: P1: 1/0 if fist infeence modus ponens/modus tollens. P2: 1/0 if second and thid infeence modus ponens + conjunction/othe. P3: 1/0 if second infeence disjunctive syllogism/othe. Table 2 Popotion of Coect Answes fo the 24 Poblems of Expeiment 1 People in Cities o Counties Functioning Factoy Poblem Type Tue Conclusion False Conclusion Tue Conclusion False Conclusion 1.95.87.98.92 2.77.82.86.68 3.59.53.79.51 4.50.49.78.47 5.88.83.95.84 6.73.70.44.51
170 RIJMEN AND DE BOECK P4: 1/0 if conclusion false/tue. Poblems with a value of zeo on both P2 and P3 wee poblems with a disjunctive modus ponens as the second infeence. By pefoming a multiple egession analysis with P1 to P4 as pedictos, diffeent weights ae allowed fo the basic ules. The fou pedictos, P1 P4, explained 74% of the vaiance [F(4,19) 5 13.82, p <.001]. It was possible to pefom a diect compaison between the model incopoating P1 P4 as pedictos and the one with only the numbe of basic ules as a pedicto, since the latte is nested within the fome. The impovement of the pediction was significant [F(3,19) 5 14.50, p <.001]. The standadized egession weights fo P1 P4, in descending ode of absolute value, wee.51 (P1), 2.38 (P3),.35 (P2), and 2.30 (P4; all p <.05). Positive coefficients indicate that poblems with a coding of 1 ae easie. These esults imply that the contibutions of the basic ules to the difficulty of the total poblem ae not equal but depend on the kind of basic ule. Futhemoe, a easoning poblem with a modus tollens is moe difficult than a poblem with a modus ponens, and a easoning poblem with a disjunctive syllogism o a disjunctive modus ponens is moe difficult than a poblem with a modus ponens + conjunction; a easoning poblem with a disjunctive syllogism is moe difficult than a easoning poblem with a modus ponens + conjunction o a disjunctive modus ponens, and finally, a poblem with a false conclusion is moe difficult than a poblem with a tue conclusion. In addition, to assess diectly the diffeence between a poblem with a disjunctive syllogism and a poblem with a disjunctive modus ponens, P3 (disjunctive syllogism vs. not) was eplaced with P3, which was coded as follows: P3 5 0 if a modus ponens plus conjunction wee the second and thid infeences. 5 1 if the second infeence was a disjunctive syllogism. 5 2 1 if the second infeence was a disjunctive modus ponens. The standadized egession weight of P3 in the egession analysis with P1, P2, P3, and P4 as pedictos was 2.33 ( p <.05). Hence, a easoning poblem with a disjunctive syllogism is moe difficult than a easoning poblem with a disjunctive modus ponens. In ode to test the effects of the othe factos of the design (content and pesentation ode of the pemises), these factos wee also coded into binay vaiables, as follows: P5: 1/0 if content of functioning factoy/content of people in cities o counties. P6: 1/0 if pemises of the same infeence not pesented togethe/pesented togethe. When eithe P6 o P5 was added to the pedicto set, these factos wee not significant. Note that P6 is not othogonal to the othe Ps. We also tested each of the paiwise inteactions fo its additional contibution. Among these inteactions, only one was significant ( p <.01), P1 * P3, with a standadized egession weight of 2.46. This indicates that the diffeence between modus ponens and modus tollens becomes smalle when a disjunctive syllogism is also pat of the poblem. Discussion The objective of Expeiment 1 was to investigate whethe diffeent basic ules have diffeent effects on the difficulty of a complex easoning poblem. The answe is yes. The numbe of basic ules as such explains only 16% of the vaiance. One can explain 74% of the vaiance in the difficulty of the easoning poblems if one allows a diffeential weighting of the basic ules. Hence, it appeas that basic ules diffe as to thei difficulty. Howeve, ule theoies offe no explanation as to why thee ae diffeences between basic ules. Only two of the main effects, found in the analysis with P1 to P4 as pedictos, can be explained by ule theoies without allowing fo a diffeential difficulty in the basic ules: A modus tollens is hade than a modus ponens (fou basic ules 1 vs. one), and a poblem with a false conclusion is hade than a poblem with a tue conclusion (detecting a contadiction equies the application of an additional basic ule). The othe main effects cannot be accounted fo by ule theoies if no diffeential difficulty is allowed fo the basic ules: A modus ponens plus conjunction is easie than a poblem with a disjunctive modus ponens o a disjunctive syllogism (despite the fist poblems equiing two basic ules, vs. one fo the latte poblems), and a disjunctive syllogism is moe difficult than a disjunctive modus ponens (despite the fact that both equie the application of one basic ule). Befoe we intepet these esults as intinsic to the basic ules themselves (such as the epesentational cost of a basic ule), we conside the possibility that they ae meely a eflection of the paticula way the diffeent infeences ae combined in the poblems of Expeiment 1 (i.e., that they ae extinsic to the basic ules). If the latte wee the case, the ule theoies would still have to be supplemented to account fo ou esults, but an explanation would then moe likely be found in a specification of pinciples concening the way people combine basic ules than in a pinciple concening the intinsic difficulty of basic ules. Fo each of (1) the diffeence between a disjunctive modus ponens and a disjunctive syllogism and (2) the diffeence between these two and a modus ponens plus conjunction, an altenative explanation can be povided in tems of the paticula way the infeences of Expeiment 1 wee combined. We shall call these possible effects combination effects. Fist, the diffeence between disjunctive modus ponens and disjunctive syllogism could be a consequence of a spontaneous pefeence fo stating with a paticula infeence. Fo the easoning poblems with dependent infeences, a modus ponens o a modus tollens infeence had to be made fist, befoe a disjunctive syllogism o a disjunctive modus ponens could be applied. Dekeyse, Schoyens, Schaeken, Spitaels, and d Ydewalle (2000)
RULES AND DIFFICULTY 171 obseved that some people have a pefeence fo stating with the disjunction when a poblem contains a disjunction. If a majoity of the paticipants in Expeiment 1 had such a pefeence, this would have misled them in Poblem Types 3 and 4 (see Table 1), because the disjunctive syllogism can be applied only afte dawing the conditional infeence (modus ponens o modus tollens). Hence, a stategy of stating with the disjunction would ende these poblems moe difficult. Second, the diffeence in difficulty between poblems of Types 1 and 2 (see Table 1: modus ponens plus conjunction) and the fou othe poblem types (with a disjunctive syllogism o a disjunctive modus ponens) could be caused by the fact that, fo the latte, the second infeence equies the esult of the fist, wheeas fo the fome the infeences ae independent (see Table 1). Hence, an altenative explanation fo the obsevation that the poblems of Types 1 and 2 ae the easiest poblems is that thei infeences ae independent. Note, howeve, that in ode to ule out the existence of intinsic diffeences between basic ules, the combination effect infeences dependent vesus independent would have to be substantial: If thee ae no intinsic diffeences between basic ules, the finding in Expeiment 1 that poblems with a disjunctive syllogism o a disjunctive modus ponens ae moe difficult than poblems with a modus ponens plus conjunction (although the latte equies an additional basic ule) can only be accounted fo if the combination effect on the difficulty is lage than the effect of a basic ule. A second expeiment was devised to test fo these altenative explanations and to eplicate the findings of Expeiment 1 with a slightly diffeent set of poblems. Compaison of the esults of Expeiment 1 against the pedictions of the mental model theoy will be defeed until the altenative explanations have been tested. EXPERIMENT 2 To test the possibility that the obtained difficulty effects ae combination effects athe than the eflection of intinsic diffeences between basic ules, fo each of the eight poblems of Expeiment 1 with a disjunctive syllogism o a disjunctive modus ponens (Poblem Types 3 6) and with a factoy-elated content, two vaiants wee constucted. In one, the infeences emained dependent, but thei ode was evesed; in the othe, the infeences wee independent (hence, thei ode did not matte). This esulted in 3 * 8 poblems: the eight poblems of Expeiment 1 plus two vaiants of each. We etained only eight poblems fom Expeiment 1 because we wanted to keep the numbe of poblems manageable fo the paticipants and equal acoss expeiments. Since the content type had no effect in Expeiment 1, we etained only the poblems with a factoy-elated content. In addition, since the effect of a pefeence fo stating with a disjunction can be tested only when a disjunction is pat of the poblem and the effect of dependence between infeences can also be tested with such poblems, we excluded the poblems involving modus ponens plus conjunction. Fo the eight poblems identical to those in Expeiment 1 with dependent infeences, modus ponens o modus tollens was the fist infeence. In Vaiant 1, we evesed the ode of infeences; moe specifically, disjunctive modus ponens o disjunctive syllogism was the fist infeence to be pefomed (independent of the pesentation ode of the pemises). In Vaiant 2, the infeences wee independent (in which case, the ode of infeences did not matte). A poblem fom Expeiment 1, with its two vaiants, is pesented in Table 3. Othe factos, such as pesentation ode of the pemises, negations, and content, wee, as fa as possible, held constant within each tiplet of poblems. Method Paticipants. Two hunded and twenty-nine high school students between 16 and 19 yeas of age paticipated in the expeiment. Design and Pocedue. Each paticipant had to solve 30 easoning poblems. Twenty-fou poblems wee constucted using the following design. The fist facto was modus ponens vesus modus tollens (12 poblems of each type). The second facto was disjunctive modus ponens vesus disjunctive syllogism (12 poblems of each type). The thid facto was the tuth-value of the conclusion to be evaluated: tue vesus false (12 poblems of each type). The fouth facto was the dependence and ode of infeences: Eight poblems wee dependent, as in Expeiment 1, eight poblems wee dependent but with the evese infeence ode, and eight poblems wee independent. The fou factos wee manipulated othogonally. The othogonal manipulation esults in 2 * 2 * 2 * 3 epeated measues. As in Expeiment 1, a fifth facto was the pesentation ode of the pemises elated to one infeence. Pemises of the same infeence wee pesented eithe togethe o sepaately (12 poblems of each type). Gouped pemise pesentation was used fo poblems with disjunctive syllogism and modus tollens and fo poblems with disjunctive modus ponens and modus ponens. Sepaate pemise pesentation was used fo the othe poblems. This means that the Table 3 Poblem Fom Expeiment 1 and Two Vaiants Expeiment 1 Vaiant 1 Vaiant 2 (dependent, evesed ode) (independent) p NOT p p q OR IF q, THEN OR s IF p, THEN NOT q p OR q IF p, THEN NOT q --------------------------- -------------- NOT --------------------------- NOT q AND s
172 RIJMEN AND DE BOECK fifth facto is the inteaction between modus ponens vesus modus tollens and disjunctive syllogism vesus disjunctive modus ponens. The position and numbe of negations wee held constant acoss vaiants as fa as possible, with the estiction that the poblems wee always such as to avoid the necessity of pefoming double negations (accoding to ule theoies). As in Expeiment 1, thee wee also six poblems with an undecidable conclusion. The pocedue of this second expeiment was the same as that of Expeiment 1. Results and Discussion The altenative explanations fo the two effects obseved in Expeiment 1 wee tested by compaing the poblems of Expeiment 1 with the coesponding Vaiant 1 and Vaiant 2 poblems. We used Wilcoxon tests fo lage samples. The altenative account fo the geate difficulty of disjunctive syllogism, in tems of the ode of the dependent infeences, was tested by compaing the numbe of coect answes on the fou poblems that had a disjunctive syllogism etained fom Expeiment 1 (disjunctive syllogism as second infeence) with the numbe of coect answes on the fou coesponding Vaiant 1 (disjunctive syllogism as fist infeence) poblems. The mean numbe of coect answes was 2.7 vesus 2.2. This diffeence is in the diection opposite to that pedicted by the altenative account and was significant (z 5 2 5.23, p <.001, two-tailed). The altenative account fo difficulty diffeences, in tems of whethe o not the infeences ae dependent, was tested by compaing the numbe of coect answes on the eight dependent poblems etained fom Expeiment 1 with the numbe of coect answes on the eight poblems of Vaiant 2 (independent infeences). The means wee 5.5 and 5.7, espectively (z 5 1.82, p <.05, one-tailed). Although the effect was significant and in the diection pedicted by the altenative account, it is too small to explain the substantial diffeence found in Expeiment 1 between modus ponens plus conjunction (when the mean fo the eight poblems was 6.84), on the one hand, and disjunctive syllogism and disjunctive modus ponens, on the othe hand (means wee, espectively, 4.66 and 5.87), as a combination effect. Hence, the elevant esults of Expeiment 1 cannot be explained by the effects found in Expeiment 2, which makes an intepetation in tems of intinsic diffeences in the difficulty of basic ules moe plausible. Nevetheless, ou esults indicate that combination factos do play a ole in the difficulty of moe complex poblems: Thee is a athe substantial effect of the ode of infeences, although this is in the diection opposite to ou pediction, and a smalle effect of the infeences being dependent o not. As in Expeiment 1, we also tested the effect of the numbe of basic ules. The ange in popotions of coect answes fo the 24 poblems of Expeiment 2 was lage: fom.38 to.97, with a mean of.68 (see Table 4). Again, the logit of the popotion of coect answes was used in a futhe analysis. The logit did not coelate with the numbe of wods in the poblem ( 5 2.027, n.s.). The numbe of basic ules explained 36% of the vaiance in the logit [F(1,22) 5 12.18, p <.01; see Appendix A fo the constuction of the pedicto]. The Speaman ank coelation between numbe of basic ules and the logit of the popotion of coect answes was computed to test fo a monotonic (not necessaily linea) elation. The Speaman coelation was 2.50 ( p <.01, one-tailed). The 36% explained vaiance in this expeiment is substantially highe than the 16% of Expeiment 1, which is pobably due to the paticula basic ules involved. In Expeiment 1, modus ponens + conjunction (two basic ules) was easie than both disjunctive syllogism and disjunctive modus ponens (one basic ule). This contadicts the pediction that the numbe of basic ules is positively elated to difficulty. In Expeiment 2, no modus ponens + conjunction poblems wee included, so that the numbe of basic ules was a bette pedicto. Hence, the bette fit follows fom the paticula choice of basic ules. The cucial diffeence between basic ules that both is unpedicted by the numbe of basic ules and can be tested in Expeiment 2 is the diffeence between disjunctive syllogism and disjunctive modus ponens. This was tested in two ways. Fist, when a binay vaiable (1 fo a disjunctive syllogism, 2 1 fo a disjunctive modus ponens) was added as a pedicto to the numbe of basic ules, the two pedictos explained 44% of the vaiance. The standadized egession weight of the additional vaiable was 2.30, which is significantly negative [t(23) 5 2 1.81, p <.05, one-tailed], confiming the esults of Ex- Table 4 Popotion of Coect Answes fo 24 Poblems of Expeiment 2 Poblems of Poblem Type Expeiment 1 Vaiant 1 Vaiant 2 Tue Conclusion Disjunctive syllogism modus ponens.92.55.71 modus tollens.77.66.72 Disjunctive modus ponens modus ponens.97.97.89 modus tollens.53.74.67 False Conclusion Disjunctive syllogism modus ponens.48.62.60 modus tollens.50.38.78 Disjunctive modus ponens modus ponens.90.91.71 modus tollens.40.41.58
RULES AND DIFFICULTY 173 peiment 1. A second way to compae disjunctive syllogism with disjunctive modus ponens is to use Wilcoxon tests, as in the assessment of combination effects. The mean scoe fo the 12 poblems with a disjunctive syllogism was 7.69; fo the 12 poblems with a disjunctive modus ponens, the mean scoe was 8.68. The latte is significantly lage (z 5 6.22, p <.001, one-tailed), again confiming ou hypothesis. The esults of Expeiment 2 cooboate the findings and conclusions of Expeiment 1 concening the existence of intinsic diffeences in difficulty between basic ules. We have seen that ule theoies offe no account fo such diffeences. Howeve, one may wonde how much of the vaiance in the difficulties of the poblems can be explained on the basis of the mental model theoy. This theoy may be able to explain the intinsic diffeences in difficulty between basic ules. THE CONTRIBUTION OF THE MENTAL MODEL THEORY In mental model theoy, each explicit model epesents a possible situation. Hence, the numbe of explicit models needed (to be fleshed out ) to solve an infeence can be consideed to be the epesentational cost of the infeence. This notion of epesentational cost can be used to account fo seveal esults, including some of the intinsic diffeences between basic ules infeed fom ou expeiments. The mental model theoy can account fo the fact that modus ponens (one model) is easie than modus tollens (thee models), a diffeence that is not due to an intinsic diffeence between basic ules. Futhemoe, it pedicts that modus ponens + conjunction (one model; the conjunction to be made in this expeiment does not equie the constuction of a model, but is meely the eading of a model containing two elementay popositions) is easie than disjunctive syllogism and disjunctive modus ponens (two models). The esults of Expeiment 2 suggest that the latte is an intinsic diffeence between basic ules. Howeve, the mental model theoy offes no clea pediction concening the elative difficulty of disjunctive syllogism vesus disjunctive modus ponens. Both equie the constuction of two models (accoding to Johnson- Laid et al., 1992). One might ague that the mental model theoy pedicts the diffeence in difficulty that we found between disjunctive syllogism and disjunctive modus ponens, because in a disjunctive syllogism, an inconsistency has to be detected (between one of the disjuncts and the categoical pemise), wheeas this is not needed in a disjunctive modus ponens. On the othe hand, the two explicit models of the initial epesentation of the disjunction p OR q ae smalle than those of the conditional if p OR q, THEN (see the intoduction fo the initial epesentation). Hence, one could ague equally well that the mental model theoy pedicts the evese of the obtained effect. We will now ty to complement the ule theoies with the notion of epesentational cost, as conceived in the mental model theoy. Of couse, fom the pevious evaluation, it is clea that this will not esult in a pefect account of the data. Nevetheless, it is useful to see how much we can impove upon ou pediction of the difficulty and, also, to see whee we fail when we take the notion of epesentational cost into account. A possible way of complementing the ule theoies with the notion of epesentational cost is to weight the diffeent infeences by thei epesentational cost, as pedicted by the mental model theoy (the numbe of explicit models). The value of a poblem on this new pedicto of poblem difficulty is then the sum of the epesentational costs, ove the infeences to be made. Appendix B descibes in detail how the value on this pedicto fo the diffeent poblems of Expeiments 1 and 2 was counted. The pedicto explains 56% of the vaiance in Expeiment 1 [F(1,22) 5 28.5, p <.001] and 41% of the vaiance in Expeiment 2 [F(1,22) 5 15.08, p <.001]. The new pedicto can be compaed with the pedicto numbe of basic ules by testing whethe the coelation of the fome with difficulty diffes fom the coelation of the latte with difficulty. Fo Expeiment 1, the coelation with the logit of the popotion of coect answes is significantly moe negative fo the pedicto infeences weighted by the numbe of explicit models than fo the pedicto numbe of basic ules [the coelations ae 2.75 and 2.40, espectively; t(21) 5 3.908, p <.001]. Fo Expeiment 2, no significant diffeence was obtained [the coelations ae 2.64 and 2.60, espectively; t(21) 5 0.76]. The Speaman ank coelations of the pedicto infeences weighted by the numbe of explicit models with the logit of the popotion of coect answes ae 2.76 ( p <.001) and 2.58 ( p <.01) fo Expeiments 1 and 2. In Expeiment 1, weighting the infeences with the numbe of explicit models needed to pefom them esults in a consideable impovement ove the numbe of basic ules pedicto. Nevetheless, in Expeiment 1, a lage pat of the vaiance emains unexplained (56% vs. 74% when a diffeential weight is allowed fo diffeent basic ules). This is not vey supising, because, as we have aleady pointed out, the mental model theoy does not pedict the diffeence between disjunctive modus ponens and disjunctive syllogism. If P3 (second infeence is disjunctive syllogism o not) is added as a pedicto to the mental model pedicto, 74% of the vaiance is explained [F(1,21) 5 13.58, p <.01], eaching the same level of pediction as the fou factos of the design. When in addition, we add the only inteaction effect that was found, P1 * P3 (the inteaction between modus ponens vesus modus tollens and disjunctive syllogism o not), the explained vaiance amounts to 83% [F(1,20) 5 10.68, p <.01]. With espect to Expeiment 2, when the facto coding fo disjunctive syllogism vesus disjunctive modus ponens is added, 49% of the vaiance is explained, which is slightly moe than the pecentage explained by
174 RIJMEN AND DE BOECK the mental model theoy pedicto [F(1,21) 5 3.61, p <.10]. A consideable impovement of the latte pediction is obtained when the same inteaction as that found in Expeiment 1 (modus ponens vesus modus tollens and disjunctive syllogism vesus disjunctive modus ponens) is added as a pedicto. The thee pedictos explain 71% of the vaiance, which is moe than the pecentage explained by the pevious two pedictos [F(1,20) 5 14.63, p <.01]. The effect of the inteaction is in the same diection as that in Expeiment 1 (beta 5 2.46). Although we have no explanation fo this inteaction effect, it is clealy a combination effect: It esults fom the paticula way the infeences ae combined in the design of ou expeiments. In conclusion, although the incopoation of the numbe of models in the egession equations esults in a bette pediction, a substantial amount of the vaiance emains unexplained. The numbe of models seems to be too coase a notion of epesentational cost. The notion pobably needs efining in ode to account fo the unexplained intinsic diffeence between disjunctive syllogism and disjunctive modus ponens. Futhemoe, an account of the combination effect that could be deived fom the inteaction between disjunctive syllogism vesus disjunctive modus ponens and modus ponens vesus modus tollens, would also add to the explanatoy powe. Finally, the othe combination effects we have found in Expeiment 2 also necessitate a futhe elaboation of the theoy. A moe detailed examination of the diffeent pocesses enteing into a easoning task would pobably add to the undestanding of these effects. It might be that the diffeences (intinsic as well as combination effects) can be captued in an elegant way by a moe fine-gained notion of epesentational cost that takes into account what must be epesented at a paticula moment in time and how long it has to emain epesented. Moe paticulaly, a potocol analysis of complex easoning poblems, such as those of Expeiment 2, may esult in a bette undestanding of the diffeent infeential steps people take. CONCLUDING REMARKS Rule theoies cannot account fo the difficulty of the complex easoning poblems used in ou two studies, unless they allow a diffeential weight fo diffeent basic ules. The diffeences in the weights themselves emain unaccounted fo. On the basis of the pinciples of the mental model theoy, we hypothesized that the notion of epesentational cost offes an explanation fo at least pat of these diffeences. Weighting the infeences to be made in a athe complex easoning poblem by the numbe of models equied to epesent them esults in a bette fit, but this measue seems too coasely defined to captue all of the vaiance. Nevetheless, elaboating the notion of epesentational cost seems to be a pomising appoach. By examining the pocesses undelying easoning in moe detail, it should be possible to quantify the epesentational cost of a poblem moe pecisely and to aive at a bette account of the data. REFERENCES Baine, M. D. S. (1978). On the elation between the natual logic of easoning and standad logic. Psychological Review, 85, 1-21. Baine, M. D. S., & O Bien, D. P. (1991). A theoy of if: A lexical enty, easoning pogam, and pagmatic pinciples. Psychological Review, 98, 182-203. Baine, M. D. S., Reise, B. J., & Rumain, B. (1984). Some empiical justification fo a theoy of natual popositional logic. In G. H. Bowe (Ed.), The psychology of leaning and motivation (Vol. 18, pp. 313-371). New Yok: Academic Pess. Dekeyse, M., Schoyens, W., Schaeken, W., Spitaels, O., & d Ydewalle, G. (2000). Pefeed pemise ode in popositional easoning: Semantic infomativeness and co-efeence. In A. Vandieendonck, G. d Ydewalle, G. De Vooght, & W. Schaeken (Eds.), Deductive easoning and stategies (pp. 73-95). Mahwah, NJ: Elbaum. Johnson-Laid, P. N. (1983). Mental models: Towads a cognitive science of language, infeence, and consciousness. Cambidge: Cambidge Univesity Pess. Johnson-Laid, P. N., & Byne, R. M. J. (1991). Deduction. Hillsdale, NJ: Elbaum. Johnson-Laid, P. N., Byne, R. M. J., & Schaeken, W. (1992). Popositional easoning by model. Psychological Review, 99, 418-439. Johnson-Laid, P. N., Byne, R. M. J., & Schaeken, W. (1994). Why models athe than ules give a bette account of pepositional easoning: A eply to Bonatti and to O Bien, Baine, and Yang. Psychological Review, 101, 734-739. Macnamaa, J. (1986). A bode dispute: The place of logic in psychology. Cambidge, MA: MIT Pess, Badfod Books. Osheson, D. N. (1974). Logical abilities in childen: Vol. 2. Logical infeence: Undelying opeations. Hillsdale, NJ: Elbaum. Osheson, D. N. (1975). Logical abilities in childen: Vol. 3. Reasoning in adolescence: Deductive infeence. Hillsdale, NJ: Elbaum. Rips, L. J. (1983). Cognitive pocesses in popositional easoning. Psychological Review, 90, 38-71. Rips, L. J. (1994). The psychology of poof. Cambidge, MA: Routledge. Rips, L. J., & Conad, F. G. (1983). Individual diffeences in deduction. Cognition & Bain Theoy, 6, 259-289. NOTE 1. As was explained ealie, the fou easoning steps to solve a modus tollens ae, stictly speaking, not basic ules, because they do not belong to the diect easoning pogam. Equating the easoning steps with basic ules, as we do fo pagmatic easons, does not change the pediction that modus tollens is hade than modus ponens, howeve. This pediction does not change if a biconditional intepetation is assumed, eithe. APPENDIX A The Value of the Poblems on the Pedicto Numbe of Basic Rules Fo the poblems of Expeiment 1 with a tue conclusion, the pedicto was defined as follows fo the six poblem types (see Table 1): Poblem Type 1: 1 (modus ponens) + 1 (modus ponens) + 1 (conjunction to combine the two intemediate conclusions of the conditional infeences) 5 3. Poblem Type 2: 4 (modus tollens) + 1 (modus ponens) + 1 (conjunction to combine the two intemediate conclusions of the conditional infeences) 5 6. Poblem Type 3: 1 (modus ponens) + 1 (disjunctive syllogism) 5 2.