More on Numbers
In recent posts, Sam and Jim raise the issue of the important of empirical evidence as the basis for all sorts of decisions law schools, law students and law professors make. The unwillingness to think in terms of actual data seems pervasive and I can think of specific examples where use of empirical studies would be useful. My impression it is that is rarely used in at least 4 instances:
1. Bar passage: Many schools, but not all, are concerned with bar passage rates but I wonder how many have undertaken a study to see if bar passage is related to LSAT, entering GPA, first year GPA, bar courses taken, pass/fail courses taken, transfer/non transfer status and a host of other variables with an eye to diagnosis? I know there are data like this around but how many schools have actually analyzed there own graduates to see where there might be improvement.
2. Teaching evaluations: I recently participated in University grievance process in which part of the reason for a tenure denial was poor student evaluations. I wonder if any law school has made an effort to determine whether their students’ evaluations of professors are correlated with teaching effectiveness?
3. Hiring: This is the time of year when the top schools go after the graduates at the top of the class at elite schools. Mid level schools go after people at the bottom of the top third of the same schools and largely ignore the top ranked graduates of non elite schools. (or so it seem so me.) I am not sure I have seen any empirical evidence that shows that the second tier students from elite schools out perform the top tier from non elite schools. In fact, I have seen none that shows that even the top students from elite schools out perform top students from non elite schools. Instead of attempting to discover who the best teachers and scholars will be, the practice of self-referential hiring persists.
4. Scholarship. The dearth of empirical scholarship is well known but that does not seem to stop anyone. One case that comes to mind in the current attack on “predatory” lending. I am not a fan of advantage takers and would probably be on this bandwagon myself. But then I ran across a discussion of a study of by Dean Karlan and Jonathan Zinman – reported in the August 4th 2007 issue of The Economist. The gist is that so-called victims of this type of lending are better off than they would have been had their loans been denied. If one assumes that those who borrow from predatory lenders do so because it is the only alternative, then the empirical evidence suggests we may be too quick to be paternalistic in this area. I choose only this one because I had just read the Economist article but my hunch is that it is representative of a great deal of legal “scholarship.”
1. Bar passage: Many schools, but not all, are concerned with bar passage rates but I wonder how many have undertaken a study to see if bar passage is related to LSAT, entering GPA, first year GPA, bar courses taken, pass/fail courses taken, transfer/non transfer status and a host of other variables with an eye to diagnosis? I know there are data like this around but how many schools have actually analyzed there own graduates to see where there might be improvement.
2. Teaching evaluations: I recently participated in University grievance process in which part of the reason for a tenure denial was poor student evaluations. I wonder if any law school has made an effort to determine whether their students’ evaluations of professors are correlated with teaching effectiveness?
3. Hiring: This is the time of year when the top schools go after the graduates at the top of the class at elite schools. Mid level schools go after people at the bottom of the top third of the same schools and largely ignore the top ranked graduates of non elite schools. (or so it seem so me.) I am not sure I have seen any empirical evidence that shows that the second tier students from elite schools out perform the top tier from non elite schools. In fact, I have seen none that shows that even the top students from elite schools out perform top students from non elite schools. Instead of attempting to discover who the best teachers and scholars will be, the practice of self-referential hiring persists.
4. Scholarship. The dearth of empirical scholarship is well known but that does not seem to stop anyone. One case that comes to mind in the current attack on “predatory” lending. I am not a fan of advantage takers and would probably be on this bandwagon myself. But then I ran across a discussion of a study of by Dean Karlan and Jonathan Zinman – reported in the August 4th 2007 issue of The Economist. The gist is that so-called victims of this type of lending are better off than they would have been had their loans been denied. If one assumes that those who borrow from predatory lenders do so because it is the only alternative, then the empirical evidence suggests we may be too quick to be paternalistic in this area. I choose only this one because I had just read the Economist article but my hunch is that it is representative of a great deal of legal “scholarship.”
posted by Jeffrey Harrison at 10:31 AM
2 Comments:
Jeff --
I'm glad that you bring up teaching evaluations. To me, these are the one of the great embarrassments of what we do. I can't get anyone at my school to tell me where our evals come from, how they were created, etc. Even getting any kind of normalizing information about the evals -- what is the overall mean and median, what is the mean and median for untenured faculty, for this course across faculty, etc. -- seems almost wholly impossible to come by. Given that careers can rise and fall based on these forms, you'd think they'd garner more attention.
By contrast, I think lots of schools do a good job analyzing their bar pass rate empirically. As you know, I have been looking at the question at my own school for some time, and am aware of good studies being done at a number of schools.
Thanks, Sam, we too are in the midst of a bar passage study. As it turns out the number crunching has fallen to me.
Post a Comment
<< Home