I can see the beauty of mathematics…You can hold infinity in your hands

This is a rant and probably isn’t terribly interesting but involves this ranking list in Forbes magazine. In the interest of being honest I need to preface this one with several disclosures.

I am a former student of Carnegie Mellon University. I felt we got maligned by the list.

Much of what I’m writing was brought to my attention through class I took at CMU last year. A good part of this post is retelling things from that class and hence many of the points are not entirely from my own original thought.

I think all of the schools on the list are perfectly great places to get a quality education my gripe is with the method by which Forbes calculated their rankings.

With that out of the way I applaud Forbes for their goal of looking past the established names and trying to come up with some kind of objective list. Unfortunately their methods were sloppy at best, deliberately rigged to be inflammatory at worst. Certainly their ratings system doesn’t hold up to what one would expect from a news organization of their size and potential impact on people’s college decision.

Forbes calculations of rank, from their own site:

1. Listings of Alumni in the 2008 edition of Who’s Who in America (12.5%)

2. Salaries of Alumni from PayScale.com (12.5%)

3. Student Evaluations from Ratemyprofessors.com (25%)

4. Four-Year Graduation Rates (16.66%)

5. Students Receiving Nationally Competitive Awards (8.33%)

6. Faculty Receiving Awards for Scholarship and Creative Pursuits (5%)

7. Four-year Debt Load for Typical Student Borrowers (20%)

Taking them in order. The Who’s Who list is a debatable point. Forbes defends the inclusion by stating that many notable figures have appeared in the list. It is important to note though that Who’s Who is still a for profit service. It’s fairly questionable how relevant this service is the age of the internet. How many truly notable alumni are going to be willing to pay for, and go through the effort of applying to what is essentially a vanity book? If making a list of notable Alumni is an important factor in the rankings perhaps Forbes should consider counting the number of Alumni with Wikipedia articles, at least Wikipedia has a published set of standards for what makes up individual and academic notability.

Salary data from payscale.com is the newest entry to Forbes criteria. In my personal opinion it’s one of the worst. First and foremost payscale.com is a forprofit salary gossip site. Regardless of how many total records the site may index how many people per College/University truly participate in this program? Secondly the data is self reported and users are required to enter their own salary information to get the comparison information (i.e., the information they want). There is no real incentive to be honest in presenting your own information, particularly when that information could be perceived as being found and used against you by an employer. This is especially true of alumni working at smaller companies. This method of taking career information also discounts undergraduates joining or founding startups which is a fairly important factor in a schools with strong business programs. Finally this completely discounts undergraduate alumni who go on to further education. Forbes list seems to have neglected undergraduate programs which place students into top flight graduate, professional and medical programs, where starting salary data is completely meaningless.

Forbes actually addressed my primary complaint with RateMyProfessor.com on their explanation of the rankings. They mentioned that Dartmouth complained strongly that it had an internal professor tracking system. Forbes defended their decision citing the large number (4 millions rankings). Now I’m not a statistician but something is really fishy about this. Carnegie Mellon also has an internal professor ratings system. It makes much more sense for students to give honest feedback on that system where their feedback actually gets examined and studied by the various departments. Forbes cites that 86% of schools have an internal ranking system, but they do no analysis of how significantly that system is used, or encouraged by the university. If I were an enterprising CMU or Dartmouth student I might think of getting a bunch of people together and gaming the rankings since no one at CMU will use those rankings for anything worthwhile anyway. Even more bothersome is the fact that ratemyprofessor.com isn’t using any kind of CAPTCHA system to prevent a script from voting lots of times automatically….hmmmmmm this sounds like a job for the computer science department.

Four year graduation rate is probably a pretty valid metric to evaluate a college. There are some possible exceptions though. I know that at Rochester Institute of Technology (RIT) for example many, if not most students do a semester or yearly co-op. This means that the average student is finishing their education in five years by choice, not as a result of anything the school is doing. Schools with large concentration of architecture majors are also heavily penalized here.

Students receiving Nationally competitive awards is as valid a metric as you’re going to get. I don’t really have too much complaint about this one.

The faculty awards section is also a reasonable choice for evaluation but Forbes heavily weighted it in the favor of some programs and not others. For example, notably missing from the list are: Fields Medalist (Math), Turing Award Winners (Computer Science), ACM Fellows (Computer Science), Tony Award Winners (Drama), Pulitzer Prize (Journalism), etc, etc. Forbes also noted that they weighted the Nobel Prize more heavily than others by a factor of three owing to it’s competitive nature . Given that they only included Nobel Prize winners from 1997-2008 this represents a pretty small possible set of faculty members when considering all Colleges and Universities giving recent winners disproportionally high rankings, compared with older Nobel winners (who probably have more teaching experience to boot). Similarly their inclusion of faculty who won their prizes recently downgrades programs which are able to consistently attract prize winning faculty to their academic staff.

The debt ratio upon graduation is probably my favorite factor Forbes included. Again, kudos to them for including this metric and helping to point out the ridiculously high price of education. I don’t have much to complain about.

I recognize that rankings systems for colleges are always going to be contentious and are essentially entirely arbitrary. Forbes, however, has a duty to their readers to ensure that they are using the best information available and doing everything in their power to remove biasing factors. Their current rankings system does a really poor job of holistically approaching the various programs and giving each a fair shake. It’s particularly egregious because Forbes buries the methodology information within the article, and on a second page far from where most readers will end up. Their inability to consider the students not immediately entering the work force is also a big mark against their list.

Still, congratulations to all the West Point folk. West Point has been getting unfairly punished on the US News and World Report list for years so good for them for topping Forbes’s.

Please leave comments and criticism here

I spent most of the last couple of weeks dealing with car issues.   Fortunately at this point they are mostly resolved. At the onset of my troubles a good bit of help from dad and some thinking, listening, and researching I got a pretty good diagnosis before I took it in to the shop which was a win for everyone. I could give the shop a much more detailed description of the symptoms which let them diagnosis the problem faster. I knew ahead of time a suspected cause, which helped me make sure they were looking at the right thing and let me expect how much it was all going to cost.

The whole process got me thinking about how much I enjoy/enjoyed debugging code. Yes, this makes me a huge nerd, but it’s also probably the skill I most highly valued in my computer science education. Debugging code, particularly tricky, frustrating, impossible bugs is a skill that I feel translates well outside of programming.   Sometimes bugs just refuse to manifest themselves when you have the debugger on, much like that noise your car makes right before you take it in to the shop for repairs.  Sometimes the tools you use to check for the problem makes the problem disappear.  Sometimes the bug (or car problem) only shows up on the second Tuesday of July at 12:01 pm.

Learning to debug frustrating bugs taught helped to teach me to logically examine a problem,  eliminating possible causes until I reached the root cause.   It also taught me that sometimes you have to examine all the assumptions you bring to the table before you can solve a problem.  Case and Point:

Probably the best debugging story I wrote at CMU happened during 15-410 Operating Systems (a great class!).  The details are not particularly important but my partners and my OS was crashing when running a certain test.  We worked on debugging it and eventually using all the techniques referenced above narrowed it down to one line of code that was supposed to be reading an instruction from a particular memory address, but instead of an instruction is saw all 0’s.  I smugly thought “gotcha” and I told the debugging tool to flag the address and tell me anytime my program read, wrote, or executed anything in that particular address.  I stopped right before the offending line of code and double checked that the address had a valid instruction…good to go.  Then one more step and BOOM…the program blew up.  Needless to say, I was perturbed.  No flag, no message, but suddenly, where my data had been…all 0’s

Frustrated I retraced everything I’d done thus far.  All my data structures looked fine,  everything was up to date.  Clearly I’d found a bug in the simulator/compiler, I told myself (hint, 99.999% of the time, the bug is in your code, not in the compiler or OS).  My partner and I trooped down to our professors office to explain our bug.  He patted us on the head and told us to get back to it, that we’d probably find the issue in an hour or two.   Gently chastened, I got back to work.  1:20 later, my partner and I came to the stunning conclusion.  We had accidentally been using an address that should have been reserved privately for the Operating System in our code.  When the OS decided to use that address for it’s own valid purposes, it simply zeroed it out and our tool never saw the change.   The failure hadn’t been in my debugging technique but rather in my assumption that I was the only one who could validly access that memory address.

That long story had really did have a point.  The kind of thought processes required to be proficient at debugging code are really useful in resolving most problems in life.   It makes me think that maybe this kind of problem solving should be introduced earlier in school curriculum, even down at the elementary school levels.  Probably not something as complex as writing an Operating System but I feel like some type of more open ended problem solving strategy would have been a very valuable part of my education.  It would certainly have been more relevant to my everyday life, even outside my job, than calculus ever will be.

Also, if you happen to live in or near Charleston, SC and happen to need any mechanical work done, I’d recommend Burke Motor. The garage staff was thorough and knowledgeable. I’m sure there are cheaper places around but they seemed to really know their stuff.