Memorylessness at the Bus Stop: Using R on transit data to test a hypothesis

I'm a data blogger and a statistics student who likes to run experiments on his hometown of New York City. This post introduces the mathematical concept of memorylessness and then runs some R code on transit data published by New York's MTA to try finding that concept in the wild.

In the realm of probability theory, there’s a spooky concept called memorylessness that characterizes certain scenarios. Anytime the chances associated with an outcome are unchanged as time or trials go by, your situation can be described as memoryless. In other words, whatever has happened does not affect what will — the past is always “forgotten.”

This idea is a lot easier to understand with an example, and for that we’ll call upon probability’s poster child: flipping coins. If you flip a coin a thousand times, the probability of getting heads on any one flip is 50%, irrespective of what has already occurred. So even if you miraculously flipped a thousand straight heads, the chances of you flipping one again is still 50%, and hence the scenario is memoryless.

This might seem obvious, but an entire phenomenon has been named after humankind’s inability to grasp this truth in certain environments. Indeed the Gambler’s Fallacy, which describes the mistaken belief that past events must be balanced out somehow in subsequent developments (like a gambler thinking he’s due for a windfall), is a common pitfall for everyone from teenagers unsettled by a string of consecutive lettered answers on a test to investors clinging to a declining stock.

The continuous version of memorylessness, whereby events are not discrete happenings like flips of coins but can occur at any time, is a little trippier. Continuous, memoryless processes describe situations where probabilities conditional on time passing are the same as if that time had not passed. These usually describe “waiting” for something that has no guarantee of happening in a certain timeframe.

One of statisticians’ favorite examples is waiting for a bus. A memoryless wait for a bus would mean that the probability that a bus arrived in the next minute is the same whether you just got to the station or if you’ve been sitting there for twenty minutes already.

Another way to think of this is that the probability of waiting any amount of time for the bus to come is equivalent to waiting that time in addition to a pre-established duration: the chances of a bus getting there in 2 minutes are the same as it getting there in 7 minutes given that 5 have already passed, or getting there in 102 minutes given that 100 have. In that sense, you can’t earn better odds just by sitting there.

As it turns out, only the exponential distribution can describe such a setup in the continuous case. Graphically, the memoryless property can be demonstrated by focusing solely on the portion of a density curve to the right of a certain point — i.e. how long you’ve already waited.

You might profess that the curve is lower there, so the odds of waiting longer are smaller if you have already waited some. But our new sample space is also smaller, since several minutes have already passed, requiring us to expand the curve upward, or normalize it so that the underlying area is one. When we do so, we realize it has assumed its original shape, just shifted over:

The bus example is memorable (no pun intended) because it so effectively captures the futility of waiting. When you’re sitting idly with no knowledge of when the next bus might come, you start to wonder if the chances it comes have actually improved since you got there. The exponential distribution provides the sad answer to that question: no!

But does the arrival of actual buses follow that same memoryless distribution?

We can test this proposal using data published by New York City’s transit authority, which tracks buses as they complete their routes throughout the day. I'm going to pick the stop at 96th Street and 5th Avenue served by the crosstown M96 bus, since it's a personal favorite.

First we must combine all the daily files into one giant 2016 data frame in R:

# upload first day (Jan 1 2016). Station ID number is 404087
bus.all <- read.csv("bus_time_20160101.csv")
bus.all<- bus.all[bus.all$next_stop_id==404087,]
bus.all$timestamp <- as.POSIXct(strptime(bus.all$timestamp, "%Y-%m-%dT%H:%M:%SZ")

# create vector of all other dates (Dec 29 missing for some reason)
dates <- c(20160102:20160131, 20160201:20160229, 20160301:20160331,
           20160401:20160430, 20160501:20160531, 20160601:20160630,
           20160701:20160731, 20160801:20160831, 20160901:20160930,
           20161001:20161031, 20161101:20161130, 20161201:20161228,
           20161230:20161231)

# cycle through all of them, reading the csv and binding it to bus.all
for (i in dates){
  file <- paste ("bus_time_", as.character(i), ".csv", sep="")
  bus_trans <- read.csv(file)
  bus_trans <- bus_trans[bus_trans$next_stop_id==404087,]
  bus_trans$timestamp <- as.POSIXct(strptime(bus_trans$timestamp, "%Y-%m-%dT%H:%M:%SZ"))
  bus.all <- rbind(bus.all, bus_trans)
}

Now we have logs of every bus that was headed to the examined stop on 96th Street, since we selected only rows with the required next_stop_id.

There's one twist though: the MTA doesn’t record exactly when buses arrive at stops but rather logs what each bus’s next stop is based on periodic pings. That means multiple rows with different timestamps will exist for the same instance of a bus approaching our stop. So in lieu of an exact arrival time, we'll have to settle for the final ping that a bus sent before it's next stop updated to the subsequent station:

# order buses by vehicle id (already ordered by time) 
bus.all <- bus.all[order(bus.all$vehicle_id),]
# add column showing time difference btwn pings
bus.all$diff <- c(601, diff(bus.all$timestamp))

# find index for last ping before arrival
# by subtracting 1 from index of first ping in string of pings going to stop
index <- which(bus.all$diff<0 | bus.all$diff>600)
index <- index - 1
bus.all.real <- bus.all[index,]

# make a new column for the wait between buses in minutes
bus.all.real <- bus.all.real[order(bus.all.real$timestamp),]
bus.all.real$wait <- c(0, diff(bus.all.real$timestamp)/60)

Now that we've created a value that closely estimates the wait between buses, we can graph its distribution. In total, over 60,000 buses came and went from the stop in 2016, or about one every nine minutes:

Pretty close to our model! The graph has two major discrepancies from the ideal one from earlier, but both have plausible explanations. First off, in a truly memoryless distribution, the shortest time (our “less than a minute” bucket) would have the highest probability. But in the above distribution, such waits are actually less frequent than other short-time buckets like two or three minutes.

The reason behind this may be that bus drivers try to give the buses in front of them some more time and space if they realize they are too close. So if drivers are intentionally slowing down to avoid tailgating their predecessors, some of the waits that would have naturally taken less than a minute might take slightly longer.

The other divergence from the exponential model occurs much further down the graph near the 40-minute mark, where a modest but noticeable bump occurs. This might indicate that at a certain point (probably in the dead of night), the MTA falls back on a schedule that is more reliable albeit much less frequent. In that sense, the overall shape is probably actually a combination of two: an exponential distribution during waking hours and a modest normal distribution during the graveyard shift.

Given that you’re probably not about to catch the 3AM M96 across town, an exponential or memoryless distribution still describes the waiting process pretty accurately. So next time you’re headed for the bus, remember this blogpost as a guide. Or, keep with the theme, and forget all about it…