I can see the beauty of mathematics…You can hold infinity in your hands

This is a rant and probably isn’t terribly interesting but involves this ranking list in Forbes magazine. In the interest of being honest I need to preface this one with several disclosures.

I am a former student of Carnegie Mellon University. I felt we got maligned by the list.

Much of what I’m writing was brought to my attention through class I took at CMU last year. A good part of this post is retelling things from that class and hence many of the points are not entirely from my own original thought.

I think all of the schools on the list are perfectly great places to get a quality education my gripe is with the method by which Forbes calculated their rankings.

With that out of the way I applaud Forbes for their goal of looking past the established names and trying to come up with some kind of objective list. Unfortunately their methods were sloppy at best, deliberately rigged to be inflammatory at worst. Certainly their ratings system doesn’t hold up to what one would expect from a news organization of their size and potential impact on people’s college decision.

Forbes calculations of rank, from their own site:

1. Listings of Alumni in the 2008 edition of Who’s Who in America (12.5%)

2. Salaries of Alumni from PayScale.com (12.5%)

3. Student Evaluations from Ratemyprofessors.com (25%)

4. Four-Year Graduation Rates (16.66%)

5. Students Receiving Nationally Competitive Awards (8.33%)

6. Faculty Receiving Awards for Scholarship and Creative Pursuits (5%)

7. Four-year Debt Load for Typical Student Borrowers (20%)

Taking them in order. The Who’s Who list is a debatable point. Forbes defends the inclusion by stating that many notable figures have appeared in the list. It is important to note though that Who’s Who is still a for profit service. It’s fairly questionable how relevant this service is the age of the internet. How many truly notable alumni are going to be willing to pay for, and go through the effort of applying to what is essentially a vanity book? If making a list of notable Alumni is an important factor in the rankings perhaps Forbes should consider counting the number of Alumni with Wikipedia articles, at least Wikipedia has a published set of standards for what makes up individual and academic notability.

Salary data from payscale.com is the newest entry to Forbes criteria. In my personal opinion it’s one of the worst. First and foremost payscale.com is a forprofit salary gossip site. Regardless of how many total records the site may index how many people per College/University truly participate in this program? Secondly the data is self reported and users are required to enter their own salary information to get the comparison information (i.e., the information they want). There is no real incentive to be honest in presenting your own information, particularly when that information could be perceived as being found and used against you by an employer. This is especially true of alumni working at smaller companies. This method of taking career information also discounts undergraduates joining or founding startups which is a fairly important factor in a schools with strong business programs. Finally this completely discounts undergraduate alumni who go on to further education. Forbes list seems to have neglected undergraduate programs which place students into top flight graduate, professional and medical programs, where starting salary data is completely meaningless.

Forbes actually addressed my primary complaint with RateMyProfessor.com on their explanation of the rankings. They mentioned that Dartmouth complained strongly that it had an internal professor tracking system. Forbes defended their decision citing the large number (4 millions rankings). Now I’m not a statistician but something is really fishy about this. Carnegie Mellon also has an internal professor ratings system. It makes much more sense for students to give honest feedback on that system where their feedback actually gets examined and studied by the various departments. Forbes cites that 86% of schools have an internal ranking system, but they do no analysis of how significantly that system is used, or encouraged by the university. If I were an enterprising CMU or Dartmouth student I might think of getting a bunch of people together and gaming the rankings since no one at CMU will use those rankings for anything worthwhile anyway. Even more bothersome is the fact that ratemyprofessor.com isn’t using any kind of CAPTCHA system to prevent a script from voting lots of times automatically….hmmmmmm this sounds like a job for the computer science department.

Four year graduation rate is probably a pretty valid metric to evaluate a college. There are some possible exceptions though. I know that at Rochester Institute of Technology (RIT) for example many, if not most students do a semester or yearly co-op. This means that the average student is finishing their education in five years by choice, not as a result of anything the school is doing. Schools with large concentration of architecture majors are also heavily penalized here.

Students receiving Nationally competitive awards is as valid a metric as you’re going to get. I don’t really have too much complaint about this one.

The faculty awards section is also a reasonable choice for evaluation but Forbes heavily weighted it in the favor of some programs and not others. For example, notably missing from the list are: Fields Medalist (Math), Turing Award Winners (Computer Science), ACM Fellows (Computer Science), Tony Award Winners (Drama), Pulitzer Prize (Journalism), etc, etc. Forbes also noted that they weighted the Nobel Prize more heavily than others by a factor of three owing to it’s competitive nature . Given that they only included Nobel Prize winners from 1997-2008 this represents a pretty small possible set of faculty members when considering all Colleges and Universities giving recent winners disproportionally high rankings, compared with older Nobel winners (who probably have more teaching experience to boot). Similarly their inclusion of faculty who won their prizes recently downgrades programs which are able to consistently attract prize winning faculty to their academic staff.

The debt ratio upon graduation is probably my favorite factor Forbes included. Again, kudos to them for including this metric and helping to point out the ridiculously high price of education. I don’t have much to complain about.

I recognize that rankings systems for colleges are always going to be contentious and are essentially entirely arbitrary. Forbes, however, has a duty to their readers to ensure that they are using the best information available and doing everything in their power to remove biasing factors. Their current rankings system does a really poor job of holistically approaching the various programs and giving each a fair shake. It’s particularly egregious because Forbes buries the methodology information within the article, and on a second page far from where most readers will end up. Their inability to consider the students not immediately entering the work force is also a big mark against their list.

Still, congratulations to all the West Point folk. West Point has been getting unfairly punished on the US News and World Report list for years so good for them for topping Forbes’s.