« Eight rail stations in Phase 1 of digital sign rollout; ridership up 5.4% in 2008 | Main | I don't play games; I report and analyze facts »

Performance metrics tables - the meat behind the percentage increases and decreases

Over the last few days, our pal Rusty (and others) have questioned the worth of the Ron Huberman's President's Report, calling it a "self-written job review," "self-generated PR" and "spin." Rusty says, ". . . let's get the numbers parsed and crunched in a way that comes up with conclusions that have some real meaning beyond an annual self-review."

Fair enough. So have at it Rusty. The numbers come from the monthly Performance Metrics tables that the CTA has published since 2004. However, when Huberman came on board, he didn't like the format and wanted to make sure they were measuring the right things, so he changed results they were measuring and released a dashboard report, where you could see at a glance whether the CTA was hitting its target (Green), coming within 10% of the target (Yellow), or missing the target by 10% or more (Red). And the target/goal is clearly defined.

In retrospect, I realize I should have posted this yesterday. Hindsight is 20-20.The most recent table (pdf) on the site shows the latest November figures, while the board report has December numbers. But they are chock-full of informational nuggets, such as:

  • In November 2008, there were 66 rail delays of 10 miutes or more.
  • In January 2008, there were 132 such rail delays.
  • The target is 78 such delays or fewer.

Below is my own edited, mash-up of the table to fit into this tight space.

Rail delays2

But you should go ahead and download the PDF to get your head spinning from all those figures. A quick glance shows that the CTA never met its goal in 2008 of having less than "2% of weekday bus intervals (time between two buses at a bus stop) that are 60 seconds or less divided by the total number weekday bus intervals traveled during the month."

And Rusty, let us know what you find when you are done parsing and crunching.

PS: Congrats to Barack Obama on his inauguration. Go here to order the Obama Commemorative SmarTrip Card to use on DC's Metro. (Thanks to Martha and others for the tip.)



You know what Mark Twain said about statistics...

"There are lies, damned lies and statistics.."



The naysayers are quiet so far in this thread...

I wish CTA would issue a special Chicago Card to commemorate Blago's impeachment.

HA, Martha! That's the funniest thing I've heard all morning. I would totally buy one...or ten!

Yeah, and the card can have some 19th century British poetry he is so fond of.

The detractors seem almost too quite...are they planning some big presentation later in this thread...maybe a in power point form?

I said a couple of threads ago that the numbers weren't up, and wait to see how many red squares there were. Now, the November ones are.

While I granted that Ron uses Daley's PR apparatus, a report that would be based on self-congratulation would use standards that resulted in no red squares. It may help to know if these are generally accepted standards in the transit industry, but at least there are some standards, not like in Kruesi's days.

Actually, those numbers are already parsed and crunched. And the metrics shown are hand chosen.

Think of it this way: What you're looking at online is comperable to your credit card statement. You may have a line for each transaction, but you still don't know the raw numbers behind that transaction. For that you need to have access to the cash register reciepts.

The raw data isn't going to be released on the website. You probably could get it with a FOIA request, and a hefty check to cover the reasonable costs of actually getting you the reams and reams of data. And then you could gather a team to re-key it into an electroic format that would be useful for a team of auditors to re-parse.

You'd also have to decide what metrics are actually relevant, and how the raw data can provide you with that kind of information. That should be a team of impartial auditors, and since my ability to be impartial has already been sharply contested, I don't believe I qualify.

As I said yesterday, if you want an independent opinion, you need outside auditors who will have access to the raw data. The pre-parsed, pre-packaged stuff you're pointing to on the website doesn't even come close to being adequate for any kind of relevant independent review.

So there are four ways to approch this:

1. Use the pre-parsed, pre-packaged data that Kevin is mistaking for raw data, and attempt to use them to come to valid conclusions. This, even though the metrics have admittedly been changed to prevent historic comparisons.

2. Make a FOIA request for the raw data, and re-crunch the numbers manually.

3. Get the CTA to give an outside auditor access to the raw data, and let the outside auditor decide what metrics are important, and how those metrics should be measured.

4. Decide that it's too costly to bother getting an independent review of Ron and the CTA's performance, and either view the President's Report with a critical eye, or accept it as the word of God.

But let's not pretend that the pre-parsed, pre-crunched numbers that have "changed format" since Ron's rise to power are actually raw data that can be used for independent review. That's just naive.

Now if you really want me to crunch the raw data instead of giving it to independent auditors, make the FOIA request, pay to get the raw data, and give me $25k, and I'll give it a shot just for you. But don't insult us by pretending that the crap on the website is anything but pre-parsed and pre-crunched numbers.

If you want to play games, then your rhetoric is nice. But if you seriously want a real examination of CTA's performance, the game being played here is bush league.

I vote for option 4, with my view being somewhere in between critical eye and word of god, with just a dash of indifference about these numbers in comparison to my actual experience.

At least the credit card statement showed that Circuit City tried to charge me twice, and Microsoft charged me for a service I did not order, and claimed it was my responsibility to cancel. That's better than no information.

The Auditor General once went through the RTA and child agencies' operations, with little effect. If you want that again, I'm sure that's an option. But there seems to be more wasted in this state that needs to be attacked before getting into essentially a quibble over the methodology of performance reports. I'd like to start with asphalt contracts and the operation of Cook County Hospital.

And, as Martha points out, there is the urine smell test.


I got a better idea. How about: "I got this card and it's [bleep]ing golden."

I got a better idea. How about: "I got this card and it's [bleep]ing golden."

Probably only those receiving free fares would be eligible. :-)

Or maybe issue it to them once free fares are repealed (highly unlikely, though).

Achicagoan, that made me laugh really hard! Nice!

Yeah, all seniors, military, etc would get the bleeping golden ticket.

It would be a physical personification of what Rod gave them.

Shouldn't there also be a card that looks like the giant shitpile Blago gave the rest of us?

I've been part of a lot of baffling online exchanges, but this whole controversy about the performance statistics may well be my favorite.

[I've been part of a lot of baffling online exchanges, but this whole controversy about the performance statistics may well be my favorite.]

Agreed. It's fun watching Rusty complain endlessly without having an actual complaint.

The problem with Rusty's trolling is that it precludes any discussion of what we actually learn from the stats.

For instance, the Yellow line increase, (which is real - sigh, yes, you have to say that, so the troll can't say it's not).

What's interesting is to figure out exactly how and why it's increased.

As of September, there were about 1,900 new riders/week because of weekend service, and about 1,000 new riders/week because of higher weekday ridership (200/day). It'll be interesting to see where those number go, and how they compare to the Blue/Pink after weekend service returned. Are there people who become more habitual riders, rather than just commuters, when the service is available most of their waking hours?

If we weren't so focused on trolls, we could laugh at the CTA because, in their ridership report, they remembered to take out the percentage increases for Yellow weekend service (since you can't have a percentage increase with a denominator of 0.) But somebody was goofing around first, and then forgot to correct his mistake, so one of the totals reads 2007 - 0 riders, 2008 - 876 riders, percent increase - 2580.0%.

If we weren't so focused on trolls, we might actually discuss the substance of that CTA critique that someone linked to, which is quite good. It doesn't say that Huberman fudged any numbers, though you'd be forgiven if you thought it did based on the gloss given by one of the trolls.

Instead, it says the big problem is pension decisions made years ago. One of the big problems (and most of us could have predicted this) is that offering people early retirement turns out to destroy your finances, not help them, because then you have to pay employees, and pay extra retirees. Cook County did this too. This was long before Huberman, but it's one of the regular tricks of local government under the machine - because it gets costs off the current books.

But instead, we have to listen to Rusty babble about how it's "meaningless" that the Yellow Line is up by 21%. That's why I think he's a frickin' destructive troll, and I really wish he'd reform his act or go away.

To offer a genuine critique of the numbers, from the standpoint that they're real, but we need to put them in perspective:

1) The runs held stats are very impressive. I'd guess that's where most of the improvement in gapping is coming from. Some of it is the natural effect of having more new buses, but a huge part of it seems to be the absentee initiative. I'd like to hear more about that. I look in at Chicagobus.org sometimes, and a lot of drivers seem to post over there. I see no particular griping. I would have thought a major absentee initiative would have met push-back from employees. Just look at the horrendous police morale evidenced on the secondcitycop blog. I'd like to hear more about how this was approached and how it worked.

2) the rail on-time improvements are small; rather than touting them, it might have been more useful to explain them - how much of the change is related to 3-tracking. Half of 07 was 3-track. Most of 08. Was there a dramatic decline in on-time performance at the beginning of 3-track, followed by improvement as operators figured it out? Was the schedule changed after it was seen how 3-track was working? It would be useful to see the change over a 2 or 3-year period instead of just 07 v. 08.

3) In fact, too many of the graphs are month-by-month graphs starting either Jan. 08 or Sept. 07. It would be more useful to have a longer time horizon, even if it meant condensing things to quarterly rather than monthly. Some of the effects in the graph are merely seasonal issues - there are always more riders in September, for instance. In some cases, the gloss on the graph seems to suggest a year-over-year improvement, but it would be nice to see that documented.

Even if we got everything Rusty asked for, he could complain that someone changed the data between the machine and the raw data. You have to actually plug yourself directly into the machine (and be a cyborg) to get the REAL RAW DATA! Anything less is meaningless!!!

The comments to this entry are closed.


TrackBack URL for this entry:

Listed below are links to weblogs that reference Performance metrics tables - the meat behind the percentage increases and decreases:

Share news tips