Exploratory Data Analysis: Pandas Framework on a Real Dataset

Today we’ll leverage Python’s Pandas framework for Data Analysis, and Seaborn for Data Visualization.

Sometimes when facing a Data problem, we must first dive into the Dataset and learn about it. Its properties, its variables’ distributions — we need to immerse in the domain.

As a geeky programmer with poor sense of aesthetics, I’ve found Seaborn to be an awesome visualization tool whenever I need to get a point accross.

It uses Matplotlib under the hood, but sets graphics up with default style values that make them look a lot prettier than I could ever manage to make them. We’ll take a look at a Dataset, and I’ll try to give you an intuition of how to look at different features. Who knows, maybe we’ll actually gain some insights from this!

Can’t Make an Omelette with no Eggs: The Dataset.

For this analysis I’m using the 120 years of olympics Dataset, which you can download or read more about by clicking the link.

I downloaded it for free from Kaggle, an awesome site if you need to get a Dataset to try out some new Machine Learning algorithm, brush up on some framework’s API, or just have a bit of fun.

I’ll only be using the ‘athlete_events’ CSV file, which has a row for every competitor on every Olympics game since 1900, with their country of birth, whether they obtained a medal, etc.

As an interesting fact, the _medals _column is 85% empty, so on average only about 15% of Omlympics Athletes actually get a medal. Add to that that some Athletes get more than one, and it shows how an even narrower quantity of the already few Olympic level athletes actually earn a medal. All the more credit to them!

First Steps in Exploratory Data Analysis: What’s the Dataset like?

First of all, before I try to glean any insights, I’d like to get a better intuition on what the Dataset’s like. How many of it’s data are missing? How many different columns are there? Those are the questions I like to start off from.

I’m using a Jupyter Notebook for this analysis, but I’ll be adding code snippets for every relevant line I run so you can follow along.

I made the Notebook available in this repository nonetheless, in case you want to take a peek yourself, and want a place to start from.

What I’ll do first is load the data with Pandas, and check its size.

In this case, the Dataset has 15 different columns, and a whole 271116 rows! That’s over 270k athletes! I wonder how many different athletes there actually are, though. Also, how many of them actually won medals?

In order to check this, I’ll first list the Dataset’s rows calling the ‘list’ function on the Dataset. I see we have many interesting features.

A few of the things I can think of here are, we could look at average height and weight for Olympics Athletes or divide them by sports. We could also see the distribution of both variables depending on Sex. We may even see how many Medals each country got as a time Series, and see the rise and fall of civilizations throughout the twentieth century.

The possibilities are endless! But first let’s tackle the difficult issues: how complete is our Dataset?

Using Pandas’ count method on a series, I can get the quantity of non-empty rows. However looking at the shape property, I can actually see the total quantity of rows, regardless of whether they’re empty or not.

After that it’s just a matter of substracting and dividing. We see only four columns are incomplete at all: Height, Weight, Age and Medal.

Medal is incomplete any time an athlete doesn’t actually win a medal, so it is to be expected for it to not be too full. However on the Weight, Height and Age fronts we’re pretty challenged.

I tried filtering the rows by different years, but the incompleteness seems to be consistent over time, which makes me think there may be a few countries that do not supply those data about their Athletes.

Starting our Actual Data Analysis: What’s the Deal with Medals?

The first question we asked about this domain was, how many different people actually won a medal since 1900? The following snippet answers that question:

As you see, almost 135 thousand different people have competed on the Olympic games in the last 120 years, but only a little over 28 thousand have earned at least one medal.

That’s about one in five, which is not that bad. It’s not that optimistic once you consider many are actually competing in more than one category though.

Now that we’re at it, how many medals have actually been earned throughout these 120 years?

Unsurprisingly, the Medal distribution is almost uniform: Almost the same quantity of gold, silver and bronze medals have been earned.

However, a total of almost 39 thousand medals have been awarded, which means if you belong to that top 20% of athletes that earn any medals, you’re expected to earn more than one on average.

What about distribution by countries? To obtain this, we’ll run the following snippets:

Using that function, we could get the amount of medals of each type a certain country got, whereas by fetching the head of the Pandas DataFrame we may see the countries with the most medals.

Interestingly, the second country to ever earn more medals is still the Soviet Union, even though it hasn’t been around for almost 20 years.

First spot is for the USA -in all categories-, and third spot is for Germany. I also looked into Argentina and Croatia, my two nations, and was surprised to see Croatia has already earned 58 gold medals, even though it has only been around since 1991 (that’s 1992 in Olympics’ years).

Finding a script to get the different years on which a country participated of the Olympics will be left as an exercise for the reader — I bet you can do this!

Female Participation Over Time

Another interesting thing I thought of was, what was female representation like in the Olympics throughout the century? This snippet answered the question:

I was surprised to see women participated in the games even since as early as 1900. However, historically there have been three men in the Olympics for every one woman. Surprised by women’s participation in the games in 1900, I decided to check their numbers throughout time. I finally used Seaborn!

We can see female participation has been ramping up very fast these past few decades, starting from almost zero and reaching the thousands. However, are they actually growing in participation faster than men? Or is this just a matter of world population? To solve that, I made this second graph:

This time we can clearly see how a pattern emerges: women are actually approaching men in sheer numbers! Another funny thing emerges here: see those little dots below, to the right? I think those are the Winter Olympics! Anyway, the picture looks pretty optimistic for female representation, even though there hasn’t been a single year with more female participants than male yet.

Miscellaneae Data Analysis: Height and Weight

I spent a long while looking at height and weight graphs, but didn’t glean any interesting conclusions.

• Both are clearly normally distributed for most sports
• Men are consistently heavier and taller than women in all sports I checked
• the only interesting variation seems to be how far apart a gender falls from another depending on the Sport.

If you have any interesting ideas about things you could analyse using Weight and Height, please let me know! I haven’t gone deep enough in grouping by each sport, so there might be some insights to be gained there. That was all for today, I hope you found this analysis interesting, or at least you learned a bit about Pandas or Data Analysis.

I made the notebook available on GitHub so you can fork the project, do your own analysis and then do a pull request.

Of course you can take all the credit! Hopefully you’ll be better than I with graphs and visuals.

Part 2 on extracting insights about sports_ is already available._

If you wish to go deeper into Data Analysis with Python, I highly recommend this O’Reilly book.

If you haven’t already, please follow me on Medium and Twitter. And if you really liked this article, share it with a friend!

The post Exploratory Data Analysis: Pandas Framework on a Real Dataset appeared first on Data Stuff.