ABA v. USN&WR on "Employment at 9 Months" Data
I earlier discussed U.S. News and World Report's plan to change the way it measures "Employment at 9 Months" for its law school rankings, a change that will make it harder for schools to game their Emp9 numbers. An anonymous commentator on that post asked, "Anyone want to recompute the 2008 rankings using this 'more accurate' methodology based on the figures conveniently made available by the ABA in excel format?" I had to admit that the project would generate interesting results. And since I've created just such a model of the 2008 rankings, I figured I was the guy for the (tedious and nonpaying) job. That labor of love generated interesting results sooner than I'd expected, however.
USN&WR's published rankings include each law school's Emp9 score. The magazine calculates those scores using data from questionnaires it sends to the law schools it ranks—questionnaires that ask each school to repeat what it earlier said in reply to the American Bar Association's annual questionnaire. Happily for ranking geeks, the ABA recently began publishing that data in a downloadable Excel file. But when I plugged that ABA data into USN&WR's Emp9 formula, I discovered that it did not always generate the Emp9 scores that USN&WR used in its 2008 rankings. Herewith the problematic cases:
I see four explanations for these divergences: 1) Errors in the ABA data; 2) Errors in the USN&WR data; 3) Errors in my calculations; or 4) Differences between what a law school told the ABA and USN&WR. The first two explanations seem most likely to me, but I of course cannot rule out the third. As for the last, it bears noting that a school's Emp9 data comes from the prior February—seven or eight months before a school fills out its ABA and USN&WR questionnaires. It's thus hard to imagine how a school could dig up new data, after so long a remove, in the relatively short span between when it fills out the former questionnaire and the latter.
With luck, I'll have firmer answers, soon. I've emailed administrators at the four law schools that stake out the extremes on the above list, asking if they could please help me understand this phenomenon. That out of the way, I plan soon to return to my original goal: recalculating the 2008 USN&WR rankings using the Emp9 formula that will officially take effect next year.
[Crossposted to Agoraphilia.]
Earlier posts about the 2008 USN&WR law school rankings:
Earlier posts about Emp9 measure:
USN&WR's published rankings include each law school's Emp9 score. The magazine calculates those scores using data from questionnaires it sends to the law schools it ranks—questionnaires that ask each school to repeat what it earlier said in reply to the American Bar Association's annual questionnaire. Happily for ranking geeks, the ABA recently began publishing that data in a downloadable Excel file. But when I plugged that ABA data into USN&WR's Emp9 formula, I discovered that it did not always generate the Emp9 scores that USN&WR used in its 2008 rankings. Herewith the problematic cases:
I see four explanations for these divergences: 1) Errors in the ABA data; 2) Errors in the USN&WR data; 3) Errors in my calculations; or 4) Differences between what a law school told the ABA and USN&WR. The first two explanations seem most likely to me, but I of course cannot rule out the third. As for the last, it bears noting that a school's Emp9 data comes from the prior February—seven or eight months before a school fills out its ABA and USN&WR questionnaires. It's thus hard to imagine how a school could dig up new data, after so long a remove, in the relatively short span between when it fills out the former questionnaire and the latter.
With luck, I'll have firmer answers, soon. I've emailed administrators at the four law schools that stake out the extremes on the above list, asking if they could please help me understand this phenomenon. That out of the way, I plan soon to return to my original goal: recalculating the 2008 USN&WR rankings using the Emp9 formula that will officially take effect next year.
[Crossposted to Agoraphilia.]
Earlier posts about the 2008 USN&WR law school rankings:
Earlier posts about Emp9 measure:
posted by Tom W. Bell at 5:44 AM
16 Comments:
Nice work, Tom. Thanks.
You're very welcome, Jeff. Wait 'til you see step two, though; *that* should *really* be interesting.
Could I propose that this analysis distracts from the real problem?
The national employment rate is around 95%. Anyone with a highschool education, really with a pulse, can get *a job* in this country. Who cares if some school manipulated their emp@9 number from 91% to 94%?
What you guys need to look at is the quality of the jobs. At my law school it's the pits. It's awful. I am in utter shock at the terrible market value of a top 25% Loyola Law School degree. Most of my class work at small firms for $15/hour or so over the Summer, and then about $25/hour if they are lucky enough to get an offer.
Below are three recent entries on our job board. These are not cherry picked jobs either, they are the market.
Loyola 2L
-----------
Employer Name:
Contact Name: xx
Address: xxx
City: Woodland Hills, CA
Telephone: 818-xx
Facsimile: 818-xx
E-Mail: xx
Description: HOURS: Part-time. DURATION: Contract. SALARY: $11.00/hour. STUDENT LEVEL: 1L, 2L JOB DESCRIPTION: File Clerk - help organize files, organize storage boxes, some legal research and writing. EMPLOYER PRACTICE/DESCRIPTION: Commercial litigation, real estate, personal injury. FIRM SIZE: 3 attorneys. NOTES: Temporary position, but could possible become permanent. HOW TO APPLY: Submit resume and cover letter via email to xx
Date Entered: 07/25/07
Job ID: 413784
-----------------------------
Contact Name: xx
Address: xx Wilshire Boulevard
City: Beverly Hills, CA xx
Telephone: xx
Facsimile: xx
E-Mail: xx
Description: HOURS: Part-time. DURATION: Permanent. SALARY: $20/hr negotiable. BAR ADMISSION: Required. WILLING TO CONSIDER: Recent Grads Awaiting Results, New Admits. YEARS OF EXPERIENCE: 1 QUALIFICATIONS: Some prior criminal defense and complex civil litigation. NOTES: Great opportunity to expand with firm. HOW TO APPLY: Submit resume and cover letter via email to xx
Date Entered: 07/25/07
Job ID: 413788
----------------------------
Employer Name:
Contact Name: xx
Address: xx
City: xx, CA xxx
Telephone: 310-xx
Facsimile: 626-xx
E-Mail: xxx
Description: HOURS: Part-time. DURATION: Contract. SALARY: $12/hour STUDENT LEVEL: 1L, 2L, 3L, grads awaiting results, grads admitted to bar. QUALIFICATIONS: Attentive, able to go to courthouse to check on files, not afraid to use the phone. JOB DESCRIPTION: Administrative/paralegal help needed to close out about 10 different types of cases at all stages of settlement and litigation. Possible legal research involved. EMPLOYER PRACTICE/DESCRIPTION: Personal injury, civil litigation. FIRM SIZE: one attorney NOTES: Closing law practice after 30 years and need administrative help closing out last 10 cases. Good learning experience from an experienced attorney. HOW TO APPLY: Submit resume via email to xx
Date Entered: 07/25/07
Job ID: 413810
-----------------------------
This is very interesting. I think the same sort of thing is going on with international university rankings, especially THES-QS. See rankingwatch.blogspot.com/2007/05/more-about-student-faculty-ratios-i.htm
Can I suggest you try this kind of analysis on these rankings.
Richard Holmes
Dear Loyola 2L:
What baffles me is that you think it's any better at schools outside the top 20 or so. I went to Emory instead of taking a scholarship at Loyola (largely based on ranking), spent 3 years in the rank, filthy south, finished in the top third or so, and struggled like HELL to find meaningful legal employment in California.
Alas, I secured a job at a small firm doing crap litigation making 70k a year after mass mailing. At least you are in California during school.
Bottom line: outside the top 10-15 schools, grades matter (whether its top 15 or 25 to get Biglaw) and something tells me you never had a chance at a top 10, so, like everyone at schools outside the top 15, you have to scramble. The only difference is kids at UC Hastings and Ohio State and Georgia aren't bitching incessantly on the internet.
Wow. Thank you for sharing your experience Emory grad.
Still, I wonder if you struggled because you wanted to work in LA? Is it really that hard for an Emory grad in your local city?
Loyola 2L
Emory,
Your post lead me to look up Emory's median starting salary figure. In 2005 it was $90,000. What's interesting is that the University of Georgia's median salary figure was $100,000, even though they are ranked almost 20 spots under Emory.
This is the point. If UGA students do better on the job market than Emory students, then Emory's US News rankings should be dinged for this. Unless you dig, you would never know that UGA students, on average, make more than Emory students, and that Emory has the worst job placement of any top 20 law school.
Loyola 2L
Anon: Please see my reply to your like comment on Agoraphilia.
Richard Holmes: Well, you can certainly *suggest*! But I hope you'll excuse me if I decide I have my hands plenty full with just law school chicanery.
Mr. Anon:I feel your pain but I fear what you are experiencing is the market at work. I am not sure law schools can change the nature of the jobs but they could do a better job of warning students about their likely prospects. But that would be against their own interests wouldn't it?
A subscriber to this blog brought this article to my attention and I applaud Professor Bell's efforts. There a few reasons for the discrepancy in the data submitted by the Florida International University College of Law.
First, the data submitted to USNWR is the official employment data as of February 15th, 2006 as required by NALP. However, the data submitted to the ABA was as of February 1st, 2006. While on the surface 2 weeks may not appear to make a difference, in reality, virtually all law school placement offices use the weeks prior to the February 15th deadline to locate as many of their recent graduates as possible.
Second, the College of Law is somewhat unique in that is extremely small. The Class of 2005 consisted of 51 graduates. On February 1st, we knew the employment status of 49 of the 51 graduates. On February 15th, we had managed to locate all 51 graduates. As such, the addition of these two graduates to the total number (or denominator) of students when computing the employment rate drastically changes the statistics.
Finally, many students find the NALP employment survey somehwat confusing which has lead some students to erroneously report their employment status. For example, a student may report that he/she is "unemployed and not seeking employment" when in actuality he/she plans on attending graduate school, will be seeking work at a later date, or is not sure what he/she wants to do. Consequently, in many instances upon reviewing the employement survey, we must clarify the student's employment status.
Maggie D. Austin
Assistant Dean for Career Planning and Placement
A subscriber has brought this article to my attention. I applaud Professor Bell’s efforts in bringing uniformity to the employment statistics. I believe there are a number of factors that have lead to the discrepancy between the statistics the FIU College of Law provided to USN&WR and those provided to the ABA.
First, the data provided to USN&WR was the employment data as of February 15th, 2006, the official NALP 9-month date. The data provided to the ABA was as of February 1st, 2006. While on the surface it may seem that two weeks should not amount to any significant difference, in actuality it does. The majority of law school placement offices use the final weeks prior to the NALP deadline to locate recent grads and ascertain their employment status.
Second, the College of Law is somewhat unique in that its student body is extremely small. On February 1st, 2006, we were aware of the employment status of 49 of the 51 members of the Class of 2005. By February 15th, we managed to locate the additional two graduates and were able to include them in the statistics submitted to NALP. The addition of two students when the class size is as small as ours can lead to a drastic difference in the data.
Finally, many students find the NALP survey somewhat confusing and inaccurately report their official employment status. For example, a student may report that he/she is “unemployed and not seeking employment” and in actuality he/she plans on attending graduate school, studying for the bar or simply unemployed . The additional time allows placement office to clarify the employment status of their students.
Again, I applaud Professor Bell’s efforts and look forward to the time when the data used to compute employment statistics is standardized.
Maggie D. Austin
Assistant Dean for Career Planning and Placement
Florida International University College of Law
I can see what you are implying for the schools with higher US News data than ABA data, but what are you implying when the ABA data is higher? That the schools are purposely underreporting to US News?
The point is that the market needs to adjust through consumer awareness sparked by honesty and transparency by ALL schools to their future consumers. (thanks to Professor Bell for leading the charge to keep schools accountable).
Students will be empowered to make educated cost/benefit assessments of attendance on a school by school basis and will not make the same mistake i did, i.e., going to on overpriced private school where i did not intended to practice thinking the ranking would translate into a degree that travels outside Atlanta.
Alas, I bill 2000 hours a yr. for 70k and have a six-figure high interest debt load. I regret my decision-and the irony, Loyola 2L, is that I wish every day that I had just taken the money and gone to Loyola...
Emory
This comment has been removed by the author.
Anon of 7/30/2007 4:47 PM:
I've heard from only one such school. It told me that it committed an error, and that it has asked USN&WR to correct its Emp9 numbers.
Tom
Ms. Austin,
Thanks for your encouraging words, and for your careful reply. I'd emailed you, to no apparent effect, and so am especially glad that you managed to find my blog post. Please allow me to clarify one thing, though. The ABA's 2006 questionnaire requested, "The percentage of graduates who are employed (as a percentage of those whose employment status is known) as of February 15 as reported to NALP." I thus don't undestand why your school would have reported employment data as of Feb. 1.
Tom
Post a Comment
<< Home