🗓️ Week 05
Exploratory Data Analysis
DS101 – Fundamentals of Data Science
28 Oct 2024
RCTs: how do they work?
You have a group of people
- Half of them get a pill
- The other half gets a placebo (sugar pill)
You split them into two groups at random
A/B testing
Causal Inference
📗 Book recommendation:
Pearl, Judea, and Dana Mackenzie. 2018. The Book of Why: The New Science of Cause and Effect. London: Allen Lane.
EDA: A story of diamonds
What if I want to have a look at the distribution of the carat variable?
Click here for the code to produce the results from this slide
from plotnine import ggplot, geom_histogram, geom_bar, aes, scale_fill_discrete, labs, element_text, scale_fill_manual
from plotnine.themes import theme, theme_bw
from plotnine.data import diamonds
(ggplot(diamonds, aes(x="carat")) +
theme_bw() +
geom_histogram(binwidth=0.5, fill='lightseagreen'))Why would I look at this distribution plot? Here are some questions I could ask after seeing this plot.
- What is the range of the variable
carat? - Which values are the most common? Why?
- Which values are rare? Why? Does that match expectations?
- Can you spot any unusual patterns? What might explain them?
Here, we see that most of the carat values are below 3: we should probably zoom in the range of carats below 3 to discover more interesting patterns.
Click here for the code to produce the results from this slide
from plotnine import ggplot, geom_histogram, geom_bar, aes, scale_fill_discrete, labs, element_text, scale_fill_manual
from plotnine.themes import theme, theme_bw
from plotnine.data import diamonds
smaller_diamonds = diamonds.query("carat < 3").copy()
(ggplot(smaller_diamonds, aes(x="carat")) + geom_histogram(binwidth=0.01,fill='rebeccapurple'))This histogram suggests some interesting questions:
- Why are there more diamonds at whole carats and common fractions of carats?
- Why are there more diamonds slightly to the right of each peak than there are slightly to the left of each peak?
EDA: A story of diamonds
- Outliers are sometimes difficult to see in a histogram when you have lots of data
- If we look at the distribution of the
yvariable, the only indication of possible outliers is an unusually wide x-axis.
Click here for the code to produce the results from this slide
(ggplot(diamonds, aes(x="y")) + geom_histogram(binwidth=0.5,fill='slateblue'))
Click here for the code to produce the results from this slide
from plotnine import coord_cartesian
(
ggplot(diamonds, aes(x="y"))
+ geom_histogram(binwidth=0.5,fill='slateblue')
+ coord_cartesian(ylim=[0, 50])
)EDA: Flying from New York
One thing you might want to do is compare the scheduled departure times for cancelled and non-cancelled times.
Click here for the code to produce the results from this slide
from plotnine import geom_freqpoly
import pandas as pd
flights2 = flights.assign(
cancelled=lambda x: pd.isna(x["dep_time"]),
sched_hour=lambda x: x["sched_dep_time"] // 100,
sched_min=lambda x: x["sched_dep_time"] % 100,
sched_dep_time=lambda x: x["sched_hour"] + x["sched_min"] / 60,
)
(
ggplot(flights2, aes(x="sched_dep_time"))
+ geom_freqpoly(aes(color="cancelled"), binwidth=1 / 4)Not exactly the best plot given that there are many more flights that were not cancelled than flights were (thankfully)!
EDA: Step-by-step
We have total 9 these floating columns:
1) Birth Rate
2) Calling Code
3) Fertility Rate
4) Infant mortality
5) Life expectancy
6) Maternal mortality ratio
7) Physicians per thousand
8) Latitude
9) Longitude
Click here for the code to produce the results from this slide
import seaborn as sns
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
plt.style.use('dark_background')
import plotly.express as px
import warnings
warnings.filterwarnings('ignore')
import seaborn as sns
# numerical columns
numerical_column = df.select_dtypes(include=['float']).columns
print(f'We have total {len(numerical_column)} these floating columns:')
for count,column_name in enumerate(df.select_dtypes(include=['float']).columns,1):
print(f'\t\t\t\t{count}) {column_name}')
df.describe().T.style.bar(
subset=['mean'],
color='purple', # text color
).background_gradient(subset=['std'],cmap=plt.cm.coolwarm).background_gradient(subset='50%',cmap='viridis')EDA: Step-by-step example
Click here for the code to produce the results from this slide
import missingno as msno
msno.matrix(df)EDA: Step-by-step example
Click here for the code to produce the results from this slide
msno.bar(df)EDA: Step-by-step example
Summary metrics of numeric variables
Click here for the code that produced the above figure
# Calculate summary statistics for numerical columns
numerical_columns = df.select_dtypes([float,int]).columns
describe_numerical = df[numerical_columns].describe()
print('Summary statistics is describe below: ')
describe_numerical.T.style.bar(
subset=['mean'],
color='purple',
).background_gradient(subset=['std'],cmap=plt.cm.coolwarm).background_gradient(subset='50%',cmap='viridis').background_gradient(subset='max',cmap='viridis')EDA: Step-by-step example
And get some facts about GDP per capita:
Country with highest GDP per capita is Vatican City
Country with lowest GDP per capita is Burundi
Click here for the code that produced the above results
gdppercapita_country = df.groupby('Country')['GDP per capita'].mean().reset_index()
# using loc we will do label indexing and find max and min gdp per capita countries
max_gdppercapita_country = gdppercapita_country.loc[gdppercapita_country['GDP per capita'].idxmax(),'Country']
min_gdppercapita_country = gdppercapita_country.loc[gdppercapita_country['GDP per capita'].idxmin(), 'Country']
# print these statements
print('Country with highest GDP per capita is {}\n' .format(max_gdppercapita_country))
print('Country with lowest GDP per capita is {}'.format(min_gdppercapita_country))
top_5_gdppercapita_country_countries= gdppercapita_country.sort_values(by='GDP per capita',ascending=False)
top_5_gdppercapita_country_countries = top_5_gdppercapita_country_countries.head()
fig1 = px.bar(data_frame=top_5_gdppercapita_country_countries,x=top_5_gdppercapita_country_countries['Country'],y=top_5_gdppercapita_country_countries['GDP per capita'],color='Country',template='plotly_dark')
fig1.update_layout(title = 'Top 5 GDP per capita Countries',title_x = 0.5,title_font = dict(size=29))# update title
fig1.show()# show our plotEDA: Step-by-step example
What does the data tell us so far?
- The 5 countries with highest GDP per capita all have small populations, are located in Western Europe, are known for their very low tax rates and for being tax havens (to some extent).
- The 5 countries with lowest GDP per capita are African countries and third world countries.
What kind of economic output does the GDP per capita really measure? What is its relationship with population size? tax rates? education? governance (not in this dataset)?
Click here for the code that produced the above results
bottom_5_gdppercapita_country_countries= gdppercapita_country.sort_values(by='GDP per capita',ascending=True)
bottom_5_gdppercapita_country_countries = bottom_5_gdppercapita_country_countries.head()
fig1 = px.bar(data_frame=bottom_5_gdppercapita_country_countries,x=bottom_5_gdppercapita_country_countries['Country'],y=bottom_5_gdppercapita_country_countries['GDP per capita'],color='Country',template='plotly_dark')
# update title
fig1.update_layout(title = 'Bottom 5 GDP per capita Countries',title_x = 0.5,title_font = dict(size=29))
# show our plot
fig1.show()References
Enders, Craig K. 2022. Applied Missing Data Analysis. Guilford Publications.
Pearl, Judea, and Dana Mackenzie. 2018. The Book of Why: The New Science of Cause and Effect. London: Allen Lane.
Scheffer, Judi. 2002. “Dealing with Missing Data.”
