Now, the hard question: how do we tell the difference between schools that are experimenting (something we want) and schools that are, um, engaged mostly in self-indulgence?I am grateful for this question. The truth is that I am not confident I know "the" answer. I do know that the hard questions about existing programs are very rarely asked.
That being said, the vanity press analogy is close to perfect and helps answer the question. There are at least three ways to make the distinction but I would modify the question and say it is not so much a school by school analysis but a program by program analysis.
1. Was there was a demand for the program before it was instituted. Or is it a case in which the supply came first? Even this might be OK if there was a vague unfocused demand that the supply helps focus. On the other hand, do there have to be huge promotional efforts every year -- for example, in the case of a foreign study program -- just to make sure students sign up. This would be a sign that the program, like a bad book or movie, does not have "legs" and that the experiment has failed.
2. Are there two or three faculty members responsible for the existence of the program who resist serious, objective, periodic review? If there are two or three who are the "but for" reasons for its existence, this leans toward it being a bad idea. An added element of this one is whether they make emotional and vague appeals to keep the program. Do they habitually understate the costs of the program? This ties to the civility issue. Do they send the message that questions about the program are somehow "personal" or "inappropriate?"
3. If the program had never existed, is it likely that a critical mass of the faculty would support starting it next year? This goes to the inertia issue. Suppose I had been going to France every year to teach American students American law. In the course of things I realized I like France and have friends plus it is really cool for people (outside of legal education) to know I spend my summers in France. I eventually come to honestly believe this is a good thing for the School. It is very hard to let go of something like this -- especially when someone else is paying the tab.
But in reality I think we all know which ones are right and which ones are vanity programs. In a log-rolling, captured faculty there is just no incentive to ask hard questions.