Welcome to another tutorial of Python. In this blog we learn about
- What is Pandas
- Installation of Pandas
- What is DataFrame?
- How to make DataFrames?
- Operations and Manipulations on DataFrame?
This topic lays the foundation of the Data Analysis with Python.
What is Pandas?
Pandas is the Python module builds on top of NumPy. It is one of the primary pillars of Data Analytics. It has high-level data structures.
DataFrames are the Data Structures provided by Pandas, accompanying with Panel and Series. Numpy arrays are homogeneous, to overcome the shortcoming, DataFrames are included in Pandas and are heterogeneous.
Installation of Pandas
The installation of Pandas is easy using pip command. Assuming you already have pip installed on your system.
pip install pandas install it on your local machine.
What is DataFrame?
- Two-Dimensional Data Structure with the different or same type of columns.
- DataFrame in Python consists of:
- DataFrames may contain:
- NumPy Arrays
- Pandas Series
- CSV file
- Dictionaries, lists.
How to Make DataFrame?
Besides various data to create a DataFrame, we will start with the NumPy Library and then discuss distinct inputs.
import numpy as np import pandas as pd array_1 = np.array([np.arange(10, 15), np.arange(15, 20)]) dataframe_1 = pd.DataFrame(array_1) print("Value of Array_1: \n", array_1) print("\nValue of DataFrame_1: \n", dataframe_1)
Execute the program.
You will see the following output on your screen.
Value of Array_1: [[10 11 12 13 14] [15 16 17 18 19]] Value of DataFrame_1: 0 1 2 3 4 0 10 11 12 13 14 1 15 16 17 18 19
When we create a DataFrame using the NumPy arrays, Pandas automatically indexed the DataFrame rows and columns. Please refer to the below snapshot. The first row and the first column are indexed as 0 and then incremented by 1 till the end of the row or column.
We also say that Pandas DataFrames are closely correlated with Microsoft excel like structure, Where we index rows and columns with values in it.
What we learned so far is DF is dynamically index by Pandas. In practical scenarios, it hardly happens. Instead of dynamic indexing, we want user-defined indexes on DataFrames.
Let’s do that.
import numpy as np import pandas as pd index = ['Row1', 'Row2', 'Row3'] columns = ['Col1', 'Col2', 'Col3', 'Col4'] array_1 = np.array([ (np.arange(11, 15)), (np.arange(15, 19)), (np.arange(19, 23))]) print("Value of Array_1: \n", array_1) dataframe_1 = pd.DataFrame(data=array_1, index=index, columns=columns) print("\nValue of DataFrame_1 is: \n", dataframe_1)
Here is the output, when you execute the above program.
Value of Array_1: [[11 12 13 14] [15 16 17 18] [19 20 21 22]] Value of DataFrame_1 is: Col1 Col2 Col3 Col4 Row1 11 12 13 14 Row2 15 16 17 18 Row3 19 20 21 22
In the output, you can see the Row and Columns name in the DataFrame output.
Give attention to the line pd.DataFrame(data=array_1, index=index, columns=columns).
In the above line,
data -- contains the value of array_1
index -- has Row Names.
Columns -- has Column Names.
Index and column are optional parameters to DataFrame, if not provided then indexing starts from ‘0’.
You have created the DataFrame, if you have to find the number of rows and columns of the DataFrame.
In the above program, add print(dataframe_1.shape). And print(len(dataframe_1.index)), to get the number of elements of DataFrame.
In the previous example, we create DataFrames from the numpy arrays. However, in the real world, we rarely create DataFrame using numpy arrays directly. The CSV or Excel files serve as input to DataFrames.
Our next task is to create the DataFrames from the CSV file. We already have a sample CSV file uploaded to GitHub. You can download it from here.
Make sure you download the file, where your python script resides.
Customer.csv file contains the Customer information
- Customer Name
- Customer id
- Date of birth
from pandas.io.parsers import read_csv customer_csv_file = 'Customer.csv' customer_dataframe = read_csv(customer_csv_file, delimiter=';') print(customer_dataframe)
We can also read the CSV from 'import csv' module, but we will use pandas read_csv because it magically converts CSV into the DataFrame. 1st-row imports read_csv from pandas.
This CSV file has around 5147 rows, hence we are not printing the output here. But make sure you executes the program before proceeding.
Right now, we are printing all the elements of DataFrame on Screen, But what if we need to print a single element or index of a DataFrame.
For e.g., if we want to print the 6th row of the DataFrame, write print(customer_dataframe.iloc), to the previous example.
It displays the 6th row on the screen.
But, print(customer_dataframe.iloc) will display 0th Column index of 6th row i.e. 'bikad' in our case.
Operations and Manipulations on DataFrame
In this section, we will extend the previous example and learn few more manipulations on DataFrames.
from pandas.io.parsers import read_csv customer_csv_file = 'DA_Customer.csv' customer_dataframe = read_csv(customer_csv_file, delimiter=';') # Print 6th Row print(customer_dataframe.iloc) # Print 6th Row and 0th Index of Column print(customer_dataframe.iloc) # Print Top 5 rows print("\n Top 5 Rows are") print(customer_dataframe.head(5)) # Print Last 5 Rows print("\n Last 5 rows are") print(customer_dataframe.tail(5))
Take a look head() and tail() functions. Head() displays the top 'n' results from the DataFrame whereas tail() displays bottom 'n' results of the DataFrame.
If you are already aware of UNIX, in that case, you are reasonably familiar with head and tail commands. Head() and tail() functions in Pandas are their analogy.
At present, we have carried out very elementary operations on DataFrames. Let’s move towards a grouping of Data.
Our CSV file contains Customer Name, Customer ID, Date of Birth, Gender, City.
We want to identify the customer base in each of the cities. It is remarkably meaningful to understand our customer demographics.
By regrouping the data according to City, then we know:
Which city has the highest Customer base?
Which city possesses the least number of Customers?
These outcomes are essential to identify because it provides insights to us on the number of Customers distributed across cities. Eventually, it points where we want to expand our market share.
Let’s include the code to our python script.
# Group the data by City city_group = customer_dataframe.groupby('City') count = 0 for city_name, group in city_group: count = count + 1 print("City", count, city_name) print(group)
We create an object and customer DataFrame is grouped (see groupby) according to the "City".
NOTE:- The city is one of the Columns in the CSV file.
Execute the program, the results are too big, hence only a portion of the output is copied here.
City 1 Bangalore Customer Name Customer_ID DOB Gender City 3 rowap 419472292 1987-06-14 M Bangalore 32 zewov 493686428 1986-05-10 F Bangalore 48 sehob 174535964 1964-02-14 F Bangalore 75 xohiv 751472877 1997-12-03 F Bangalore …. ….. [379 rows x 5 columns]
We have already covered a lot about DataFrames on Pandas. However still there is lot to cover. That's enough for Part-1.
Stay tuned for the Part-2 of the Tutorial.
Test Your Knowledge
- Find the shape of the DataFrame
- Find the length of the DataFrame