Wednesday, July 11, 2012

The Linda Hypothetical and Testing

Most readers are familiar with the Kahneman and Tversky  classic Linda hypothetical which involves this fact pattern and the follow-up question: "Linda is thirty-one years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in antinuclear demonstrations.”

Which alternative is more probable?
 (1) Linda is a bank teller.
 (2) Linda is a bank teller and is active in the feminist movement.

Most people choose (2) over one although it is illogical to do so since it includes (1) and involves a joint and, thus, lower probability.

I've never been a believer in the Linda hypothetical. First, I am not sure it tells us as much about logic as it does interpretation. Having listed feminist as a possibility in choice (2), "not a feminist" may be inferred to be included in (1). If so,both are examples of joint probability. I am also not sure if many people know what "more probable" means. Suppose one reads that as "best" answer meaning the one that captures Linda in a more precise fashion. Finally, how about telling subjects this is a test on logic?  In short, is the test a valid test of reasoning?

For law professors, especially those using multiple choice machine graded exams the same questions are relevant.  There are many reasons for choosing a wrong answer and sometimes the answers reveal more about the teaching and testing than the students. On the typical essay exam or multiple choice with explanation test  the teacher can assess the quality of the question by examining why people missed it. Flaws in the questions are revealed. On a machine graded exam the process of "testing the test" needs to happen before the test is used. I wonder how many machine graders either copy the questions from another source and assume the questions are pretested or actually do test the questions by having a sample of students answer the questions and then debrief those giving the wrong answer. I am betting not many. Ironically, when the of issue machine graded testing comes up many of the defenses are also illogical.

4 Comments:

Anonymous Nate said...

In my experience with multiple choice exams in law school, the questions are either too easy or do not provide a correct answer. In the too easy category, you have questions like “What are the elements of negligence.” Almost everyone is going to be able to answer that correctly, and if every question were like that, there wouldn’t be much of a grade distribution.

The questions that are unanswerable are the ones that are asking about “gray” areas of the law. If the question were posed on an essay exam, I would provide reasons that support different conclusions but probably not actually reach a conclusion. The multiple choice exam, however, forces the test taker to reach a conclusion when there is no objective correct conclusion to the issue. So I always ended up trying to psychoanalyze how the professor would view the issue to determine what he would think is the correct conclusion.

7/11/2012 4:00 PM  
Blogger Jeffrey Harrison said...

Only one of the reasons I do not give them. Others: I want to know how a student gets to the answer; I can better assess the fairness of the question; I do not want to test memorization.

7/11/2012 5:57 PM  
Blogger Jim Chen said...

Jeff, how do you feel about the use of multiple choice questions on the multistate bar exam?

7/11/2012 10:42 PM  
Blogger Jeffrey Harrison said...

Jim, It's a tough one. I am told those questions are written, pretested and vetted by experts. That does not describe your basic law professor. Plus, the happenstance of grading thousands of essay exams by hundreds of graders who have different standards could lead to some odd outcomes. So, I guess I would say it's necessary -- an unfortunate consequence.

7/12/2012 12:26 AM  

Post a Comment

Links to this post:

Create a Link

<< Home