pandas-eda-workflow

name	pandas-eda-workflow
description	Pandas EDA skill - Covers data loading, cleaning, transformation, aggregation, and visualization workflows used in real-world data analysis.

Overview

Pandas is a powerful data analysis library in Python used for handling structured data.

This skill walks through a practical Exploratory Data Analysis (EDA) workflow using Pandas, including inspecting data, cleaning it, transforming features, and generating basic visual insights.

Use this when you're working on:

• Data analysis projects
• Machine learning preprocessing
• CSV / tabular datasets
• Understanding and preparing features

---

Setup

pip install pandas matplotlib seaborn

---

Getting Started

import pandas as pd

df = pd.read_csv("resources/dataset.csv")

df.head()
df.info()
df.describe()

---

Understanding the Dataset

Data Structure Basics

• Series → single column
• DataFrame → table of data

df.shape
df.columns
df.dtypes

---

Exploring the Data

df.head()
df.tail()
df.sample(5)

df.info()
df.describe()

---

Dealing with Missing Data

df.isnull().sum()

Removing missing values

df = df.dropna()

Filling missing values

df.fillna(df.mean(), inplace=True)

---

Cleaning and Preparing Data

Renaming columns

df.rename(columns={"old_name": "new_name"}, inplace=True)

Removing duplicates

df.drop_duplicates(inplace=True)

Fixing data types

df["column"] = df["column"].astype(int)

---

Filtering and Selecting Data

df[df["age"] > 25]

df[(df["age"] > 25) & (df["salary"] > 50000)]

---

Summarizing Data

df.groupby("department")["salary"].mean()

df.groupby("department").agg({
    "salary": "mean",
    "age": "max"
})

---

Ordering Data

df.sort_values(by="salary", ascending=False)

---

Creating New Features

df["bonus"] = df["salary"] * 0.1

---

Visualizing Insights

Using Matplotlib and Seaborn:

Distribution plot

df["age"].hist()

Salary spread

import seaborn as sns

sns.boxplot(x=df["salary"])

Feature relationships

sns.heatmap(df.corr(), annot=True)

---

Handy Operations

Selecting specific columns

df[["name", "salary"]]

Applying transformations

df["salary"] = df["salary"].apply(lambda x: x * 1.1)

---

Best Practices

• Prefer vectorized operations over loops
• Use .loc and .iloc correctly
• Avoid unnecessary modifications
• Work on copies when needed

---

Tool Selection Guide

• Pandas → tabular data handling
• NumPy → numerical operations
• Seaborn → statistical plots
• Matplotlib → custom visualizations

---

References

• https://pandas.pydata.org/docs/
• https://seaborn.pydata.org/
• https://matplotlib.org/