Voter Fraud in U.S. News Surveys?
In ranking law schools, U.S. News and World Report weights peer reputation more heavily than any other measure of quality. A school's reputation among its peers counts for 25% of its overall score in the rankings (the next-most important measure, in contrast, counts for only 15%). How does USN&WR calculate a school's peer reputation? It says that it relies on surveys sent to "law school deans, deans of academic affairs, the chair of faculty appointments, and the most recently tenured faculty members" of each of the ABA-accredited law schools that it ranks. In truth, however, other people also get the chance to vote on USN&WR's reputation surveys.
Two people recently and independently told me that they had received USN&WR reputational surveys even though they do not fit any of the criteria—law school dean, dean of academic affairs, etc.—that USN&WR has published. Both people work at law schools. One of my informants told me that he/she got the forms both at his/her present employer and at a law school she worked at earlier. My other informant told me that he/she knows of a similarly situated person who likewise got an apparently unauthorized USN&WR reputation survey. Both informants asked that I not identify them—hence my coy phrasing—but their claims strike me as completely credible.
Those few anecdotes do not, of course, establish how often USN&WR sends reputation surveys to people other than those it (says it) intends to poll. Notably, however, the reports I've received came to me unbidden, simply because I have a reputation as a rankings geek. Query how many more such cases a comprehensive investigation would uncover; a lot, I'd guess.
Query, too, whether USN&WR really means to send surveys to people such as those who contacted me. Perhaps it has a "secret list" of reputation survey recipients, people whose opinions it holds in high regard but whom it wants to safeguard from the taint of law school public relations campaigns designed to influence USN&WR voters. Yet another caveat: Perhaps USN&WR manages to screen out reputation surveys that get filled out and returned by unqualified parties.
We thus have, as yet, no solid proof that voter fraud materially affects the way that USN&WR ranks law schools. We do, however, have reason to wonder whether the most important part of USN&WR's rankings really works as advertised.
[Crossposted at Agoraphilia, MoneyLaw, and College Life O.C.]
Two people recently and independently told me that they had received USN&WR reputational surveys even though they do not fit any of the criteria—law school dean, dean of academic affairs, etc.—that USN&WR has published. Both people work at law schools. One of my informants told me that he/she got the forms both at his/her present employer and at a law school she worked at earlier. My other informant told me that he/she knows of a similarly situated person who likewise got an apparently unauthorized USN&WR reputation survey. Both informants asked that I not identify them—hence my coy phrasing—but their claims strike me as completely credible.
Those few anecdotes do not, of course, establish how often USN&WR sends reputation surveys to people other than those it (says it) intends to poll. Notably, however, the reports I've received came to me unbidden, simply because I have a reputation as a rankings geek. Query how many more such cases a comprehensive investigation would uncover; a lot, I'd guess.
Query, too, whether USN&WR really means to send surveys to people such as those who contacted me. Perhaps it has a "secret list" of reputation survey recipients, people whose opinions it holds in high regard but whom it wants to safeguard from the taint of law school public relations campaigns designed to influence USN&WR voters. Yet another caveat: Perhaps USN&WR manages to screen out reputation surveys that get filled out and returned by unqualified parties.
We thus have, as yet, no solid proof that voter fraud materially affects the way that USN&WR ranks law schools. We do, however, have reason to wonder whether the most important part of USN&WR's rankings really works as advertised.
[Crossposted at Agoraphilia, MoneyLaw, and College Life O.C.]
Labels: U.S. News
posted by Tom W. Bell at 7:01 PM
4 Comments:
Interesting. Also query what such recipients should do with the surveys -- I'd be inclined to say: strike a blow for educational quality, give high marks to schools with high bar passage rates, student satisfaction, innovative curricula, and submit!
Also, why call them "reputation" surveys? I know it's a quibble, but they actually ask for "quality assessment." On why it matters, see http://money-law.blogspot.com/2008/10/us-news-survey-vote-quality-not.html
Tom: are you serious in even speculating that there could be a "secret list"? Why in God's name would there be? So the rankings could more expertly not change?
Jason: are you serious about submitting? I can maybe see submitting it with a note that says "I believe this was sent to me by mistake, since it is inconsistent with the stated terms of your survey, but if that's wrong please accept my input and change your claim." Even then, given that it would either be disallowed or the request for clarification ignored, it seems purposeless. And presumably you'd acknowledge that sending it in without such a caveat, and with an increased likelihood that it would be counted, is accentuating the fraud on the reading public.
"There has never been any baseline information on population size," said Katherine Kendall, the lead researcher, who said the results speak for themselves. "There has been huge investments of time and money to recover [the grizzly bear] but they don't know whether their actions have been successful."
_________________________
nickysam
SMM
Jason: I believe that none of my informants sent in the surveys. But, then again, they are a self-selected group; someone who wrongly got and sent in a survey is not very likely to fess up to having done so!
Ani: I offered one reason--to protect the reputation score from manipulation. USN&WR might have others, I suppose. Granted, I don't think it very likely that it has a "secret list," but I wanted to give USN&WR the benefit of every doubt.
Post a Comment
<< Home