SSRN v. USN&WR v. Truth
Chapman University School of Law did very well in the most recent Social Sciences Research Network (SSRN) rankings. Chapman topped all other law schools in the number of new papers posted in the last 12 months, and ranked 71 out of the 353 schools in the total number of downloads over that same period. So much, so good. Chapman apparently ruffled some feathers, however, by publicly acknowledging its success.
Joseph A. Hodnicki, Associate Director for Library Operations at the University of Cincinnati Law Library, characterized Chapman's announcement as "an amazing display of hubris . . ." Prof. Dan Markel, of FSU Law School, agreed, praising Hodnicki's "super post puncturing Chapman School of Law's inflated claim to outperform Yale, Harvard, etc. . . ." And, yet, Chapman plainly did outperform its counterparts in terms of what the SSRN rankings measure. What's the problem, then?
The SSRN's "# of New Papers" measure does not discriminate between old and new publications. Hodnicki complains that Chapman aced that measure because some of its faculty members recently posted on SSRN papers they published years ago. (Hodnicki seems more comfortable with Chapman's top-100 debut in the SSRN's "total downloads" measure.) I agree with Hodnicki insofar as he thinks the SSRN should do a better job of gauging law schools' scholarly performance. Indeed, I'll go farther: I'd bet that the SSRN defines "# of New Papers" as it does in order to encourage the posting of old papers, which helps to increase the size and value of the SSRN database.
The SSRN's "# of New Papers" measure thus merits criticism—of SSRN. Chapman perhaps committed a gaucherie, granted, in trumpeting its success. And perhaps, as a member of Chapman's faculty, I lack a sufficiently objective view of the matter. It seems to me, though, that Chapman caught flack for truthfully reporting that it did a good job at sharing its scholars' work with the world. Why criticize Chapman for that?
Hodnicki tries to discredit Chapman's achievement by citing the school's low reputation score in the U.S. News & World Report rankings. As I've explained elsewhere, though, Chapman's (presently) low reputation score in the USN&WR rankings says more about law school ranking games than it does the school's actual qualities. The most USN&WR recent rankings credited Chapman with:
I'm happy to join Hodnicki and anyone else in casting a skeptical eye on the SSRN rankings. Let's aim our fire at the SSRN, though, rather than on the schools it ranks. And let's not assume that USN&WR's reputation scores offer a more accurate measure of law school quality.
Update: See this more recent post for an analysis of the validity of SSRN's "# of New Papers" measure.
[Crossposted to Agoraphilia.]
Joseph A. Hodnicki, Associate Director for Library Operations at the University of Cincinnati Law Library, characterized Chapman's announcement as "an amazing display of hubris . . ." Prof. Dan Markel, of FSU Law School, agreed, praising Hodnicki's "super post puncturing Chapman School of Law's inflated claim to outperform Yale, Harvard, etc. . . ." And, yet, Chapman plainly did outperform its counterparts in terms of what the SSRN rankings measure. What's the problem, then?
The SSRN's "# of New Papers" measure does not discriminate between old and new publications. Hodnicki complains that Chapman aced that measure because some of its faculty members recently posted on SSRN papers they published years ago. (Hodnicki seems more comfortable with Chapman's top-100 debut in the SSRN's "total downloads" measure.) I agree with Hodnicki insofar as he thinks the SSRN should do a better job of gauging law schools' scholarly performance. Indeed, I'll go farther: I'd bet that the SSRN defines "# of New Papers" as it does in order to encourage the posting of old papers, which helps to increase the size and value of the SSRN database.
The SSRN's "# of New Papers" measure thus merits criticism—of SSRN. Chapman perhaps committed a gaucherie, granted, in trumpeting its success. And perhaps, as a member of Chapman's faculty, I lack a sufficiently objective view of the matter. It seems to me, though, that Chapman caught flack for truthfully reporting that it did a good job at sharing its scholars' work with the world. Why criticize Chapman for that?
Hodnicki tries to discredit Chapman's achievement by citing the school's low reputation score in the U.S. News & World Report rankings. As I've explained elsewhere, though, Chapman's (presently) low reputation score in the USN&WR rankings says more about law school ranking games than it does the school's actual qualities. The most USN&WR recent rankings credited Chapman with:
- a median LSAT (156) above that of USN&WR's third tier (155);
- an average GPA (3.31) closer to the third tier's (3.39) than the fourth's (3.22);
- an employment at nine months figure equal to the third tier's (91%);
- an acceptance rate (30.2%) better than the third tier's median (35.0%); and
- a Bar pass/jurisdiction figure (109.3%) better than the second tier median (103.9%).
I'm happy to join Hodnicki and anyone else in casting a skeptical eye on the SSRN rankings. Let's aim our fire at the SSRN, though, rather than on the schools it ranks. And let's not assume that USN&WR's reputation scores offer a more accurate measure of law school quality.
Update: See this more recent post for an analysis of the validity of SSRN's "# of New Papers" measure.
[Crossposted to Agoraphilia.]
posted by Tom W. Bell at 7:58 PM
6 Comments:
I think the problem is highlighting this as a "key" measure of scholarly output. I hardly think that simply posting papers to SSRN, standing alone, means very much. It certainly shows effort, I suppose, but what else?
Anonymous: I guess that single word--"key"--offers the most justifiable reason for the teeth-gnashing. But for that, I think we'd have to chalk the criticism of Chapman up to snobbery about it having "acted above its station." But I don't think Chapman erred in describing the "# of New Papers" measure as a "Key Scholarly Output Ranking."
Bernie Black & Paul Caron concluded that both number of downloads *and number of posted papers* are useful and “play a valuable role in the rankings tapestry.” See, “Ranking Law Schools: Using SSRN to Measure Scholarly Performance," at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=784764.
The authors moreover devote a significant portion of their conclusion to the “biggest rankings disparities” between USN&WR rankings and SSRN rankings. Consider this quote: “The SSRN measures have important field and other biases. Still, they offer up-and-coming schools a way to ‘show their stuff,’ long before the US News rankings respond to the school’s improvement. That alone is an important contribution.”
But, Tom, when Bernie Black and Paul Caron talk about number of papers posted, I think everyone but whoever drafted the Chapman press release understands that they're talking about papers written this decade.
The fact that Chapman ranked #71 in downloads is laudable, but makes the value of its #1 ranking in uploads even more dubious. With apologies to trees and forests everywhere, if a paper posts in cyberspace and nobody reads it, does it "make a sound" -- that is, does it have any scholarly impact?
Perhaps next year Chapman can release "Special Edition" serializations of its professors' past articles, dividing each article into five separate papers for SSRN posting.
Anon2: I have not found in Black & Caron's article any such qualification. Nor, as I mentioned, does the SSRN's definition of "# of New Papers" limit it to only recently-published ones.
I have discovered, though, that Chapman is hardly alone in posting old papers. Consider what scholars from some other schools have posted:
Harvard: Lucian Bebchuck, http://papers.ssrn.com/sol3/cf_dev/AbsByAuth.cfm?per_id=17037, with 134 uploads of papers or abstracts published as far back as 1985.
Pennsylvania: Paul Robinson, http://papers.ssrn.com/sol3/cf_dev/AbsByAuth.cfm?per_id=47441, with 77 uploads dating back to 1972.
Georgetown: J. Gregory Sidack, http://papers.ssrn.com/sol3/cf_dev/AbsByAuth.cfm?per_id=206474, with 68 uploads dating back to 1980.
UCLA: Stephen Bainbridge, http://papers.ssrn.com/sol3/cf_dev/AbsByAuth.cfm?per_id=16596, with 55 uploads dating back to 1985.
So I don't think Chapman deserves to be singled out for having some old papers in its "# of New Papers" count. Especially since, as a new school with many faculty who only just started posting to SSRN, our recent uploads of necessity include a mix of old and new papers. Harvard, Penn, Georgetown, etc., posted their old papers years ago. Soon, like them, Chapman will have only newer papers to post.
Anon3: Would that be sort of like the working papers that inflate other schools' "# of New Papers" count? Or would it be more like the derivative papers that so many established scholars spin off their earlier works.
All kidding aside, thanks for the suggestion, but I think that what you propose (as opposed to what Chapman actually did) really would constitute "corrupt" behavior.
Here it might be true that the messenger conveys more than the message. The utter delight shown in making sure that a law school knows its place is troubling.
Post a Comment
<< Home