The Perils of Empirical Research

Jim's column,the Michigan loss, and the persistence of hiring committees to go by rankings rather than performance inspired me to attempt an empirical study comparing the scholarly productivity of graduates of elite schools with that of graduates of non elite schools. I chose to do this by examining the productivity of faculty at four schools ranked at the bottom of the top 50.Why that group? If follows from my view, perhaps misguided, that it is at that level a critical distinction must be made. Those schools are at a disadvantage when it come to hiring top ranked graduates from elite schools. So they choose between second tier grads of elite schools and first tier grads of non elite schools. (As I write this I do recall hearing stories about top grads from elite schools having trouble. Whether true or not, this is often attributed to their lack of diversity. But, let's leave that aside for now.)

I am continuing the effort and will report it even if it does not support my own biases. Still there are decisions to be made that make any result suspect. Here are some issues:

1.What is an elite school? It could be an expensive school but not all expensive ones are ranked highly. If I only go with highly ranked ones, expensive or not, I do not fully capture my own interest in class-based effects.

2. Where do you draw the line between elite and non elite?

3. What is an elite education? Some people have J.D.s from elite schools. Some have LLMs. Some have both. There are at least 7 combinations. I am starting with seven levels of elite from no elite school exposure to the triple elite -- undergrad, J.D., post J.D. all from elite schools. (Note this raises the separate issue of ranking undergraduate schools.)

4. How to assess scholarship. This is the old problem of quality and quality. It is also the problem of number of pages or number of articles. I settled on number of articles in Westlaw, reasoning that it would be representative of overall productivity. But then, if you have been down this road, you know that an "article" often means an "Introduction" to something, or something coauthored, a book review (could be comparable to an article but may not be). In a couple of instances I found huge numbers of publications and they invariably included many of the above.

Maybe it all evens out, but I sincerely hope medical reasearchers are able to avoid these types of issues.