Hacking the New "Employment at Nine Months" Formula
The people have spoken—or, rather, 30 readers have voted. I earlier polled you for advice about what to do with a rankings hack I've thought up. Specifically, I have in mind a way for law schools to game the new "employment at 9 months" formula that U.S. News & World Report will start using in next year's law school rankings. "What should I do with the new Emp9 hack?" I asked. Most of you replied, "Tell everybody, to level the playing field." (And nobody ponied up the money to make, "Don't be a chump! Sell it to the highest bidder," an attractive option.) I'm happy to oblige.
In brief, some law schools might improve their Emp9 scores next year by remaining willfully ignorant of whether alums likely to be studying full time for the Bar nine months after graduation have found paying work. That strategy would allow a school to classify those graduates as "Employment Status Unknown." Why do that? Because USN&WR counts each such grad as 25% employed.
Consider an example. If we take the average of all law schools covered by the most recent ABA-LSAC Official Guide [Excel format] we get these numbers:
Plug those numbers into the old Emp9 formula, and you get:
[194 + (8*.25) + 5] / (222 – 5) = 92.6%.
If our hypothetical average school were willing to play the classification game I described earlier (classifying graduates studying full-time for the Bar as "Unemployed, Not Seeking Employment,") its Emp9 calculation would run thusly:
[194 + (8*.25) + 5] / (222 – 5 - 5) = 94.8%.
The new Emp9 formula, because its denominator does not subtract out grads "Unemployed, Not Seeking Employment," will no longer respond to that classification trick. Under the new formula, our hypothetical school's Emp9 score will suffer accordingly:
[194 + (8*.25) + 5] / 222 = 90.5%.
A law school could still game even that new and improved Emp9 formula, however. How? By classifying as "Employment Status Unknown" grads who would otherwise qualify as "Unemployed, Studying Full-Time for the Bar." Such grads would then count as 25% employed in the Emp9 formula. Consider the result for our hypothetical law school:
[194 + ([8 + 5] *.25) + 5] / 222 = 91.1%.
That's not a huge effect, granted, but a law school with the right numbers could gain a lot of grounds from the hack. Suppose that 10 of our hypothetical law school's grads had failed the Bar the first time around, for instance. It might then trick the Emp9 formula into spitting out "92.2%."
How could a law school implement the hack? An outright lie would suffice, of course. Importantly, however, law school administrators need not resort to anything so crude as that. They might instead remain deliberately ignorant of whether alums who failed the summer Bar exam had found work nine months after graduation. That strategy would, however, put at risk the opportunity to claim as "Employed" grads who'd found jobs that don't require Bar membership. An especially daring and clever administrator might thus devise mechanisms to carefully control how much she learns about graduates who failed the summer Bar exam, keeping close tabs on those who find work while ignoring those who don't. Suffice it to say that a subtle mind could find that "Unknown" includes many shades.
For all I know, experienced law school administrators might regard these observations as old hat. Judging from the ABA data, however, I don't think that many schools have exploited the hack I've described. (For evidence of that, I calculated for each school the ratio, "number of first-time takers to fail the summer Bar exam/number of alums with 'Employment Status Unknown' nine months after graduation." The median ratio for all schools was 17%; the average 32%. Only 11 schools had ratios above 100%, 17 above 75%, and 28 above 50%.) Until now, after all, schools eager to pump up their Emp9 numbers have typically found other strategies more fruitful. It will prove interesting to review the ABA data next year, to see if this new rankings hack attracts new rankings hackers.
[Crossposted to Agoraphilia.]
Earlier posts about Emp9 measure:
In brief, some law schools might improve their Emp9 scores next year by remaining willfully ignorant of whether alums likely to be studying full time for the Bar nine months after graduation have found paying work. That strategy would allow a school to classify those graduates as "Employment Status Unknown." Why do that? Because USN&WR counts each such grad as 25% employed.
Consider an example. If we take the average of all law schools covered by the most recent ABA-LSAC Official Guide [Excel format] we get these numbers:
- # of Graduates: 222
- Employment Status Known: 214
- Employment Status Unknown: 8
- Employed: 194
- Pursuing Grad Degree: 5
- Unemployed, Not Seeking Employment: 5
- Unemployed, Studying Full-Time for Bar: 5
Plug those numbers into the old Emp9 formula, and you get:
[194 + (8*.25) + 5] / (222 – 5) = 92.6%.
If our hypothetical average school were willing to play the classification game I described earlier (classifying graduates studying full-time for the Bar as "Unemployed, Not Seeking Employment,") its Emp9 calculation would run thusly:
[194 + (8*.25) + 5] / (222 – 5 - 5) = 94.8%.
The new Emp9 formula, because its denominator does not subtract out grads "Unemployed, Not Seeking Employment," will no longer respond to that classification trick. Under the new formula, our hypothetical school's Emp9 score will suffer accordingly:
[194 + (8*.25) + 5] / 222 = 90.5%.
A law school could still game even that new and improved Emp9 formula, however. How? By classifying as "Employment Status Unknown" grads who would otherwise qualify as "Unemployed, Studying Full-Time for the Bar." Such grads would then count as 25% employed in the Emp9 formula. Consider the result for our hypothetical law school:
[194 + ([8 + 5] *.25) + 5] / 222 = 91.1%.
That's not a huge effect, granted, but a law school with the right numbers could gain a lot of grounds from the hack. Suppose that 10 of our hypothetical law school's grads had failed the Bar the first time around, for instance. It might then trick the Emp9 formula into spitting out "92.2%."
How could a law school implement the hack? An outright lie would suffice, of course. Importantly, however, law school administrators need not resort to anything so crude as that. They might instead remain deliberately ignorant of whether alums who failed the summer Bar exam had found work nine months after graduation. That strategy would, however, put at risk the opportunity to claim as "Employed" grads who'd found jobs that don't require Bar membership. An especially daring and clever administrator might thus devise mechanisms to carefully control how much she learns about graduates who failed the summer Bar exam, keeping close tabs on those who find work while ignoring those who don't. Suffice it to say that a subtle mind could find that "Unknown" includes many shades.
For all I know, experienced law school administrators might regard these observations as old hat. Judging from the ABA data, however, I don't think that many schools have exploited the hack I've described. (For evidence of that, I calculated for each school the ratio, "number of first-time takers to fail the summer Bar exam/number of alums with 'Employment Status Unknown' nine months after graduation." The median ratio for all schools was 17%; the average 32%. Only 11 schools had ratios above 100%, 17 above 75%, and 28 above 50%.) Until now, after all, schools eager to pump up their Emp9 numbers have typically found other strategies more fruitful. It will prove interesting to review the ABA data next year, to see if this new rankings hack attracts new rankings hackers.
[Crossposted to Agoraphilia.]
Earlier posts about Emp9 measure:
- Change to U.S. News Law School Rankings Methodology
- How U.S. News Calculates "Employment at 9 Months"
- The How, Who, and Why of Strategic Emp9 Reporting
- USN&WR to Change Employment Measure
- ABA v. USN&WR on "Employment at 9 Months" Data
- The Ethics of Strategic Emp9 Reporting
- 2008 USN&WR Law School Rankings Under New Emp9 Formula
- The Emp9 Puzzle: Answers and Questions
- Rankings Hack Hot Potato
posted by Tom W. Bell at 7:23 PM
0 Comments:
Post a Comment
<< Home