U.S. News: Less Transparency = More Fairness
Robert Morse today announced that, in response to evidence that law schools had been gaming its rankings, U.S. News would change the way it estimates the "Employment at 9 Months" measure for schools that decline to report that figure. Paul Caron offers some background here. Said Morse: "U.S. News is planning to significantly change its estimate for the at-graduation rate employment for nonresponding schools in order to create an incentive for more law schools to report their actual at-graduation employment rate data. This new estimating procedure will not be released publicly before we publish the rankings."
I understand that U.S. News generated the formula it formerly used to estimate the Emp9 figure for non-reporting schools by running a regression comparing the Emp0 and Emp9 data from reporting schools. It used to puzzle me that U.S. News did not evidently re-run the regression each year, but rather stuck with the original estimate. In retrospect, though, I see that sticking to the same formula might have partially helped U.S. News offset the gaming it so dislikes. After all, as more and more schools with low numbers refused to report Emp9 data, opting to rely instead on the publicized formula, the correlation between Emp0 and Emp9 scores would change so as to favor non-reporting schools. Better to stick with the old formula, dated though it might be, than to increase the incentive to opt out of reporting.
U.S. News thus avoided a vicious cycle, but only at the cost of signaling to schools exactly when hiding Emp9 data would help their rankings. Will its new reticence work? Schools can now only guess at how U.S. News will turn Emp0 numbers into Emp9 estimates, and will rightly worry that they might misjudge the new cutoff. Even if big-E ethics does not counsel reporting Emp9 numbers, therefore, small-c conservatism will. Granted, a school might reason, "U.S. News will still try to find a reasonably accurate way to turn Emp0 data into Emp9 estimates, and it has always helped us to not report in the past, so it remains a gamble worth taking." But such schools should also rightly worry that U.S. News might throw a punitive little kick into its new formula, to encourage schools to worry more about accuracy than about rankings.
[Crossposted at Agoraphilia and MoneyLaw.]
I understand that U.S. News generated the formula it formerly used to estimate the Emp9 figure for non-reporting schools by running a regression comparing the Emp0 and Emp9 data from reporting schools. It used to puzzle me that U.S. News did not evidently re-run the regression each year, but rather stuck with the original estimate. In retrospect, though, I see that sticking to the same formula might have partially helped U.S. News offset the gaming it so dislikes. After all, as more and more schools with low numbers refused to report Emp9 data, opting to rely instead on the publicized formula, the correlation between Emp0 and Emp9 scores would change so as to favor non-reporting schools. Better to stick with the old formula, dated though it might be, than to increase the incentive to opt out of reporting.
U.S. News thus avoided a vicious cycle, but only at the cost of signaling to schools exactly when hiding Emp9 data would help their rankings. Will its new reticence work? Schools can now only guess at how U.S. News will turn Emp0 numbers into Emp9 estimates, and will rightly worry that they might misjudge the new cutoff. Even if big-E ethics does not counsel reporting Emp9 numbers, therefore, small-c conservatism will. Granted, a school might reason, "U.S. News will still try to find a reasonably accurate way to turn Emp0 data into Emp9 estimates, and it has always helped us to not report in the past, so it remains a gamble worth taking." But such schools should also rightly worry that U.S. News might throw a punitive little kick into its new formula, to encourage schools to worry more about accuracy than about rankings.
[Crossposted at Agoraphilia and MoneyLaw.]
Labels: law school rankings, U.S. News
posted by Tom W. Bell at 5:28 PM
4 Comments:
Is this really a big problem compared to schools hiring their own grads and then reporting?
It depends on whom you ask, I suppose. The non-reporting represents a problem for USN&WR because it leaves gaps in the data they present to their customers. The manipulated reporting looks facially better, but arguably misleads would-be law students far worse than mere silence.
To me, the whole issue is whether schools are setting the right example when they manipulate data to look as if they're performing better (assuming we count rankings as "performing") than they are. For an analogy, see my post here: http://nancyrapoport.blogspot.com/2010/05/weve-learned-nothing-from-enron.html.
All employment data should be eliminated entirely from the US News rankings. It is unreliable and always will be, because there is no universally accepted definition of what to count. That, combined with self-reporting, and the importance of this data in the competition between schools, and the fact that prospective students may look at the data without context (of the lack of a definition) is an obvious recipe for disaster.
Post a Comment
<< Home