Data Wrangling in Python

Data wrangling is the process of cleaning, transforming, and organizing raw data into a structured format suitable for analysis. Real-world datasets often contain missing values, duplicates, inconsistent formats, incorrect data types, and other quality issues, so wrangling helps resolve these problems and make the data more reliable.

• Handles missing, duplicate, and invalid records.
• Standardizes inconsistent values and data formats.
• Converts data into suitable types and structures.
• Makes datasets easier to analyze and interpret.
• Improves the reliability of statistical analysis and machine learning models.
• Pandas provides functions to perform most data-wrangling operations efficiently in Python.

Loading the Dataset

First step in data wrangling is to load the raw data into a DataFrame. Since our dataset is stored in an Excel workbook, Pandas can read the worksheet directly using read_excel().

• Load the original dataset without modifying it.
• Keep the raw structure intact so that its problems can be inspected before cleaning.
• Same DataFrame will be used throughout the wrangling process.

Sample dataset can be downloaded from here.

import pandas as pd

df = pd.read_excel(
    "messy_ecommerce_sales_1000(1).xlsx",
    sheet_name="Messy_Sales_Data"
)

df.head()

Output

Explanation:

• pd.read_excel() loads the Excel worksheet into a Pandas DataFrame.
• sheet_name selects the worksheet containing the sales records.
• df.head() displays the first five rows of the raw dataset.
• No cleaning or transformation has been performed yet.

Exploring the Dataset

Once data is loaded, the next step is to examine its structure and identify potential problems. Initial exploration helps determine which cleaning and transformation steps are required.

• Check the number of rows and columns.
• Inspect column names and data types.
• Check the number of non-null values.
• Examine basic statistics.
• Use the findings to decide the next wrangling operations.

"Shape:", df.shape

print("\nData Types:")
df.dtypes


print("\nDataset Information:")
df.info()


print("\nStatistical Summary:")
df.describe(include="all")

Output

Explanation:

• df.shape returns the number of rows and columns.
• df.dtypes shows the current data type of each column.
• df.info() displays non-null counts and other structural information.
• df.describe(include="all") provides summary statistics for numerical and categorical columns.
• Inspection provides the information needed to determine which data-quality issues should be handled next.

Identifying Data Types

A column may contain values that look numeric or date-like but are actually stored as text. Before performing calculations such as finding an average or median, these columns must have an appropriate data type.

• A numeric-looking column may still have the object data type.
• Mathematical operations on such columns can produce errors.
• Data types should be checked before applying numerical transformations.

df.dtypes

Output

Explanation:

• Quantity, Customer_Age, and Rating are already stored as numerical columns.
• Unit_Price is stored as object because its values are represented as text.
• Values in Unit_Price need to be converted before numerical operations can be performed.
• Issue was identified during the exploration and missing-value analysis, so it should be corrected before imputation.

Converting Data Types

Converting columns to appropriate data types makes them suitable for calculations and further processing. In this dataset, Unit_Price contains numeric values stored as strings, so it needs to be converted to a numerical type.

• Use numeric conversion when a column represents numbers but is stored as text.
• Use errors="coerce" when invalid values should be converted to NaN so they can be handled later.
• Do not convert a column to numeric if its values represent categories or descriptive text.

df["Unit_Price"] = pd.to_numeric(
    df["Unit_Price"],
    errors="coerce"
)

print(df["Unit_Price"].dtype)

Output

float64

Explanation:

• pd.to_numeric() converts numeric-looking text into numbers.
• errors="coerce" converts values that cannot be interpreted as numbers into NaN.
• Unit_Price can now be used in calculations such as mean, median, and total sales.
• Any values that become NaN can be handled during the missing-value step.

Identifying Missing Values

Missing values are common in real-world datasets and can affect calculations, filtering, and machine learning models. Before deciding how to handle them, we first need to identify which columns contain missing values and how many records are affected.

• Missing values can occur in numerical as well as categorical columns.
• Not every missing value should be handled in the same way.
• The appropriate method depends on the meaning and data type of the column.

missing_values = df.isnull().sum()

missing_values

Output

Explanation:

• isnull() identifies cells containing missing values.
• sum() counts the missing values in each column.
• Columns with a count of 0 do not contain missing values.
• Customer_Name, Quantity, Unit_Price, Discount, Customer_Age, City, and Rating require attention.
• Identifying the missing values first helps us choose an appropriate replacement strategy.

Handling Missing Values

After identifying the missing values, we need to decide how they should be treated. The method depends on whether the column contains numerical, categorical, or descriptive information.

• Numerical columns can often be filled using a measure such as the median when the missing records should retain the overall distribution.
• Categorical columns can be filled with a suitable label such as "Unknown".
• Deleting rows should be considered only when the missing records cannot be reasonably retained.
• Goal is to preserve useful records while avoiding assumptions that could distort the dataset.

df["Quantity"] = df["Quantity"].fillna(df["Quantity"].median())
df["Unit_Price"] = df["Unit_Price"].fillna(df["Unit_Price"].median())
df["Customer_Age"] = df["Customer_Age"].fillna(df["Customer_Age"].median())
df["Rating"] = df["Rating"].fillna(df["Rating"].median())


df["Customer_Name"] = df["Customer_Name"].fillna("Unknown")
df["City"] = df["City"].fillna("Unknown")
df["Discount"] = df["Discount"].fillna("0%")

df.isnull().sum()

Output

Explanation:

• Median values are used for numerical columns because they are less affected by extreme values than the mean.
• Customer_Name and City are filled with "Unknown" because assigning an existing name or city would introduce unsupported information.
• Missing Discount values are treated as 0% so that the field can be used consistently in later calculations.
• Final isnull().sum() check confirms that the dataset no longer contains missing values.
• Cleaned DataFrame can now be passed to the next wrangling step without carrying forward the missing-value problem.

Identifying Duplicate Records

Duplicate records can cause the same transaction to be counted more than once during analysis. Before removing them, we should determine what makes a record unique in the dataset.

• Checking the complete row with duplicated() finds only exact copies.
• This dataset, there are no completely identical rows.
• Order_ID should uniquely identify an order, so repeated Order_ID values need further investigation.
• Do not automatically remove duplicate rows because two records can have the same value in one column while containing different information.

print("Complete duplicate rows:", df.duplicated().sum())


print("Duplicate Order_ID values:", df["Order_ID"].duplicated().sum())

duplicate_orders = df[df["Order_ID"].duplicated(keep=False)]

print("\nDuplicate Order_ID records:")
duplicate_orders.sort_values("Order_ID")

Output

Explanation:

• df.duplicated() checks whether an entire row is repeated.
• Since the result is 0, there are no exact duplicate rows.
• Checking Order_ID separately reveals 29 repeated order identifiers.
• These records should not be deleted immediately because their other fields may contain different information.
• Next step is to inspect these repeated identifiers and decide which record should represent each order.

Standardizing Categorical Values

Categorical data often represents the same value in multiple forms because of differences in capitalization, spacing, or naming conventions. These variations can cause incorrect counts and grouping results.

• Convert text values to a consistent case.
• Remove unnecessary leading and trailing spaces.
• Replace alternate representations with a single standard value.
• Do not standardize columns where capitalization has a meaningful difference.

In our dataset, Product, Category, City, Payment_Method, and Order_Status contain inconsistent representations.

df["Product"] = df["Product"].str.strip().str.title()

df["Category"] = df["Category"].str.strip().str.title()

df["City"] = df["City"].str.strip().str.title()

payment_map = {
    "cash_on_delivery": "Cash on Delivery",
    "credit_card": "Credit Card",
    "debit_card": "Debit Card",
    "upi": "UPI",
    "net_banking": "Net Banking"
}

df["Payment_Method"] = (
    df["Payment_Method"]
    .str.strip()
    .replace(payment_map)
)


df["Order_Status"] = (
    df["Order_Status"]
    .str.strip()
    .str.title()
)

print("Products:")
df["Product"].unique()

print("\nCategories:")
df["Category"].unique()

print("\nCities:")
df["City"].dropna().unique()

print("\nPayment Methods:")
df["Payment_Method"].unique()

print("\nOrder Status:")
df["Order_Status"].unique()

Output

Explanation:

• str.strip() removes unnecessary spaces from the beginning and end of text values.
• str.title() converts variations such as electronics and ELECTRONICS into Electronics.
• Payment methods such as credit_card and Credit Card are mapped to the same value.
• Standardizing these fields prevents the same category from being counted as multiple categories.
• Cleaned values can now be safely used for grouping, filtering, and aggregation.

Cleaning and Standardizing Date Values

Dates may contain both date and time information even when only the date is required for analysis. Converting the column to a proper datetime format makes it possible to sort records chronologically and perform operations such as extracting the month, year, or day.

• Date columns should use a datetime data type.
• Invalid date values should not silently remain as strings.
• Converting the column before extracting date components prevents errors in later analysis.

df["Order_Date"] = pd.to_datetime(
    df["Order_Date"],
    errors="coerce"
)

print("Data Type:", df["Order_Date"].dtype)

print("\nSample Dates:")
print(df["Order_Date"].head())

Output

Explanation:

• pd.to_datetime() converts the Order_Date column into Pandas datetime values.
• errors="coerce" converts values that cannot be interpreted as dates into missing values instead of stopping execution.
• Datetime type allows Pandas to perform date-based operations correctly.
• Column can now be used to sort records and extract components such as year and month.
• Any values converted to missing values can be checked and handled separately rather than causing an error later.

Converting and Standardizing Discount Values

Discount column contains percentage values stored as text, along with inconsistent representations such as "ten" and "5 percent". Before using discounts in calculations, these values should be converted into a consistent numerical format.

• Percentage symbols and text representations need to be handled before conversion.
• A numerical discount makes calculations such as discounted price easier.
• Values should be converted to decimal form, where 10% becomes 0.10.

df["Discount"] = (
    df["Discount"]
    .astype(str)
    .str.strip()
    .str.lower()
    .replace({
        "ten": "10%",
        "5 percent": "5%"
    })
)

df["Discount"] = (
    df["Discount"]
    .str.replace("%", "", regex=False)
    .astype(float)
    / 100
)

print(df["Discount"].unique())
print("\nData Type:", df["Discount"].dtype)

Output

Explanation:

• astype(str) ensures that the values can be processed consistently.
• str.strip() removes unnecessary spaces.
• replace() converts text representations such as "ten" into a standard percentage.
• str.replace() removes the % symbol.
• Dividing by 100 converts percentages into decimal values.
• Column can now be used directly in mathematical calculations.

Validating Numerical Values

Having a numerical data type does not guarantee that all values are valid. A column can contain numbers that are technically valid but do not make sense for the business context, such as a negative quantity or an unusually large quantity.

• Quantity should be greater than 0 for a completed sales record.
• Zero or negative quantities should be investigated before analysis.
• Extremely large values may indicate data-entry errors or invalid records.
• Validation rules should be based on the meaning of the column rather than only its data type.

print("Quantity values:")
print(sorted(df["Quantity"].dropna().unique()))

invalid_quantity = df[df["Quantity"] <= 0]

print("\nInvalid quantity records:")
print(invalid_quantity[["Order_ID", "Quantity"]])

large_quantity = df[df["Quantity"] > 100]

print("\nUnusually large quantity records:")
print(large_quantity[["Order_ID", "Quantity"]])

Output

Explanation:

• First statement shows the range of quantity values present in the dataset.
• Condition df["Quantity"] <=0 identified values that cannot represent a normal positive sales quantity.
• Second condition identifies unusually large quantities that may require investigation.
• These records should not be removed automatically because an unusually large quantity could represent a genuine bulk order.
• Validation helps distinguish between values that are structurally correct and values that make sense in the context of the dataset.

Handling Invalid Values

Data validation may reveal values that are technically numeric but do not make sense for the intended analysis. In the sales dataset, Quantity contains 0 and -2, while 500 is unusually large compared with the normal range.

• A sales quantity should normally be a positive number.
• Zero and negative quantities can be treated as invalid values.
• An unusually large value should be investigated rather than automatically deleted because it may represent a genuine bulk order.
• Invalid values should be converted to missing values before applying an appropriate replacement strategy.

df.loc[df["Quantity"] <= 0, "Quantity"] = pd.NA

print("Invalid quantities:", (df["Quantity"] <= 0).sum())
print("Missing quantities:", df["Quantity"].isna().sum())

print("\nLarge quantity values:")
print(df.loc[df["Quantity"] > 100, ["Order_ID", "Quantity"]])

Output

Explanation:

• df.loc[...] selects the records that violate the quantity rule.
• Values less than or equal to 0 are replaced with pd.NA.
• 500 value is not removed because an unusually large order may still be valid.
• Converting invalid values to missing values allows them to be handled using the same missing-value strategy used earlier.

Detecting and Handling Outliers

Outliers are observations that are unusually far from the typical values in a dataset. They may represent genuine extreme cases, but they can also indicate data-entry or measurement errors.

• Detect outliers before removing them.
• Statistical methods such as the Interquartile Range (IQR) can be used for numerical data.
• Do not remove an outlier simply because it is large or small.
• Consider the context of the data before deciding whether it should be retained.

Q1 = df["Unit_Price"].quantile(0.25)
Q3 = df["Unit_Price"].quantile(0.75)

IQR = Q3 - Q1

lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

outliers = df[
    (df["Unit_Price"] < lower_bound) |
    (df["Unit_Price"] > upper_bound)
]

print("Lower Bound:", lower_bound)
print("Upper Bound:", upper_bound)
print("\nNumber of Outliers:", len(outliers))

print("\nOutlier Records:")
print(outliers[["Order_ID", "Product", "Unit_Price"]])

Output

Explanation:

• Q1 represents the 25th percentile.
• Q3 represents the 75th percentile.
• IQR is calculated as Q3 minus Q1.
• Values below the lower bound or above the upper bound are flagged as potential outliers.
• Records are inspected rather than automatically deleted.
• An outlier should only be removed or replaced when there is sufficient evidence that it is invalid.

We should not delete all the detected outliers. The dataset contains genuine price variation, so the IQR result should be treated as a detection step.

Creating and Transforming Columns

Data wrangling often involves creating new columns from existing information. Derived columns can combine multiple fields to produce values that are more useful for analysis.

• New columns should be based on existing, meaningful fields.
• Keep the original columns when they may still be useful later.
• Make sure the source columns have already been cleaned and converted to appropriate data types.
• Derived values should follow a clearly defined formula.

df["Discount"] = original_df["Discount"].fillna("0%").astype(str).str.strip().str.lower()


df["Discount"] = df["Discount"].replace({"ten": "10%", "5 percent": "5%"})

df["Discount"] = df["Discount"].str.replace("%", "", regex=False).astype(float)


df["Discount_Amount"] = df["Unit_Price"] * (df["Discount"] / 100)
df["Discounted_Unit_Price"] = df["Unit_Price"] - df["Discount_Amount"]
df["Total_Amount"] = df["Discounted_Unit_Price"] * df["Quantity"]


df[["Order_ID", "Quantity", "Unit_Price", "Discount", "Discount_Amount", "Discounted_Unit_Price", "Total_Amount"]].head()

Output

Explanation:

• Quantity represents the number of units purchased.
• Unit_Price represents the price of one unit.
• Discount is stored as a decimal value, so it can be directly used in the calculation.
• Formula first calculates the gross amount and then subtracts the discount.
• Records with missing input values may produce a missing Total_Amount, which preserves the uncertainty instead of creating an incorrect value.
• Creating derived columns makes the dataset more useful for later analysis.

Cleaning and Standardizing Data

Different records may represent the same value using different formats. Cleaning these values removes unnecessary characters and makes them consistent before further transformation.

• Remove unnecessary spaces and symbols.
• Standardize alternative representations of the same value.
• Keep the cleaned values in a form that is easy to understand.
• Perform numerical calculations only after the values have been converted appropriately.

df = df.map(
    lambda x: x.strip() if isinstance(x, str) else x
)

df["Discount"] = (
    df["Discount"]
    .astype("string")
    .str.strip()
    .str.lower()
    .replace({
        "ten": "10%",
        "5 percent": "5%"
    })
    .str.replace("%", "", regex=False)
)

df["Discount"] = pd.to_numeric(
    df["Discount"],
    errors="coerce"
)

df["Discount"].value_counts(dropna=False).sort_index()

Output

Explanation:

• "ten" is standardized to 10%.
• "5 percent" is standardized to 5%.
• The % symbol is removed so the column can be converted to numeric values.
• Resulting values represent percentages directly, such as 5 for 5%.
• Missing values remain as NaN and will be handled separately.

Sorting Data

Arranges records according to the values in one or more columns. It is useful when we want to examine the highest or lowest values, arrange dates chronologically, or organize records before further analysis.

• Use sorting when the order of records is important.
• ascending=True places smaller values first.
• ascending=False places larger values first.
• Sorting does not change the actual values in the dataset.

df_sorted = df.sort_values(
    by="Total_Amount",
    ascending=False
)


df_sorted[
    ["Order_ID", "Product", "Quantity", "Total_Amount"]
].head()

Output

Explanation:

• sort_values() arranges the rows according to the selected column.
• by="Total_Amount" specifies the column used for sorting.
• ascending=False places the largest values at the top.
• Sorted result is stored in df_sorted, so the original DataFrame is not modified.
• Multiple columns can also be supplied when sorting requires additional ordering.

Filtering Data

Selects only the records that satisfy a specified condition. It allows us to work with a smaller, relevant subset of the dataset without permanently removing the remaining records.

• Use filtering when a particular subset is required for analysis.
• Conditions can be combined using operators such as & and |.
• Filtering is useful for examining specific ranges, categories, or business conditions.
• Original DataFrame remains unchanged unless the filtered result is assigned back to it.

high_value_orders = df[
    df["Total_Amount"] > 50000
]


high_value_orders = high_value_orders.sort_values(
    by="Total_Amount",
    ascending=False
)


high_value_orders[
    ["Order_ID", "Product", "Category", "Total_Amount"]
].head()

Output

Explanation:

• df["Total_Amount"] > 50000 creates a Boolean condition.
• Condition selects only the rows that satisfy the requirement.
• Filtered result is stored in high_value_orders.
• sort_values() arranges the selected records from highest to lowest amount.
• Original df still contains all records.

Grouping and Aggregating Data

Combines records that share a common value and then calculates summary statistics for each group. It is useful when we want to understand patterns across categories, cities, products, or other segments.

• Use grouping to summarize large datasets into meaningful categories.
• Common aggregation functions include sum(), mean(), count(), min(), and max().
• Multiple statistics can be calculated at the same time.
• Grouping produces a new summary result without changing the original DataFrame.

category_summary = (
    df.groupby("Category")
      .agg(
          Total_Sales=("Total_Amount", "sum"),
          Average_Price=("Unit_Price", "mean"),
          Total_Quantity=("Quantity", "sum"),
          Order_Count=("Order_ID", "count")
    )
    .reset_index()
)

category_summary

Output

Explanation:

• groupby("Category") groups records with the same category.
• sum() calculates the total value within each group.
• mean() calculates the average unit price.
• count() counts the available order records.
• reset_index() converts the grouped index back into a regular column.
• Original DataFrame remains unchanged.

Reshaping Data

Changes the structure of a dataset without changing the underlying information. It is useful when data needs to be reorganized into a format that is easier to analyze, compare, or visualize.

• Use reshaping when rows and columns need to be reorganized.
• Wide-format data stores different categories in separate columns.
• Long-format data stores observations as rows.
• Pandas provides functions such as melt(), pivot(), and pivot_table() for reshaping.

sales_summary = (
    df.pivot_table(
        index="Category",
        columns="Order_Status",
        values="Total_Amount",
        aggfunc="sum",
        fill_value=0
    )
)

sales_summary

Output

Explanation:

• pivot_table() creates a summarized and reshaped DataFrame.
• index="Category" keeps categories as rows.
• columns="Order_Status" creates separate columns for each order status.
• values="Total_Amount" specifies the data being summarized.
• aggfunc="sum" calculates total sales for each combination.
• fill_value=0 replaces combinations with no recorded value by 0.
• Structure is useful for comparing multiple groups at once.

Merging DataFrames

Merging combines related DataFrames using a common column. It is useful when information about the same records is stored in separate tables.

• Use merge() when two datasets share a common key.
• Key should identify how records from both datasets are related.
• An inner merge keeps only matching records, while a left merge preserves all records from the left DataFrame.
• Check the result after merging because duplicate or unmatched keys can change the number of rows.

product_lookup = pd.read_excel(
    "/content/drive/MyDrive/Data Wrangling/e_commerce_retail_dataset.xlsx",
    sheet_name="Product_Lookup"
)

product_lookup["Product"] = (
    product_lookup["Product"]
    .astype("string")
    .str.strip()
    .str.title()
)

df["Product"] = (
    df["Product"]
    .astype("string")
    .str.strip()
    .str.title()
)

if "Product_Category" in df.columns:
    df = df.drop(columns=["Product_Category"])


df = df.merge(
    product_lookup,
    on="Product",
    how="left"
)


display(df[["Order_ID", "Product", "Category", "Product_Category"]].head())

Output

Explanation:

• product_lookup loads the second worksheet containing product-level information.
• Product is used as the common key between the two DataFrames.
• how="left" keeps every record from the sales DataFrame.
• Product_Category is added from the lookup table.
• Keeping both Category and Product_Category allows us to compare the existing classification with the lookup information before deciding whether one should replace the other.

Concatenating DataFrames

Concatenation combines DataFrames by stacking them along rows or columns. Unlike merge(), which matches related records using a key, concat() simply combines data according to its position and structure.

• Use concat() when datasets have the same type of records and compatible columns.
• Use row-wise concatenation to append additional records.
• Use column-wise concatenation when corresponding rows contain related information.
• Check column names and data types before concatenating.

df_part1 = df.iloc[:500]
df_part2 = df.iloc[500:]

df_combined = pd.concat(
    [df_part1, df_part2],
    axis=0,
    ignore_index=True
)

print("Original rows:", len(df))
print("Combined rows:", len(df_combined))

print("\nFirst five rows:")

Output

Explanation:

• iloc[] separates the DataFrame into two row-based sections.
• concat() combines the sections vertically.
• axis=0 specifies row-wise concatenation.
• ignore_index=True creates a fresh sequential index.
• No new records are created; the two existing parts are simply combined.
• In real-world workflows, this technique is commonly used to combine data from multiple files or time periods with the same structure.

Validating the Final Dataset

After cleaning and transforming the data, the final DataFrame should be checked to make sure it satisfies the expected conditions. Validation helps confirm that the previous wrangling steps did not introduce new problems.

• Check the shape of the DataFrame.
• Check for remaining missing values.
• Check for duplicate identifiers.
• Verify important data types.
• Check whether numerical values fall within expected ranges.

print("Shape:", df.shape)


print("\nMissing Values:")
print(df.isnull().sum())


print("\nDuplicate Order IDs:", df["Order_ID"].duplicated().sum())

print("\nData Types:")
print(df[["Order_Date", "Quantity", "Unit_Price",
          "Discount", "Total_Amount"]].dtypes)


print("\nQuantity Range:")
print(df["Quantity"].min(), "to", df["Quantity"].max())

print("\nDiscount Range:")
print(df["Discount"].min(), "to", df["Discount"].max())

Output

Explanation:

• shape confirms the current size of the DataFrame.
• isnull().sum() identifies any unresolved missing values.
• duplicated() checks whether Order_ID remains unique.
• dtypes confirms that numerical and date columns have appropriate types.
• Range checks help identify values that may still require investigation.
• Validation should be performed before exporting the final dataset.

Resetting the Index

Cleaning operations such as filtering, removing records, and concatenating DataFrames can leave gaps in the index. Resetting the index creates a new continuous sequence for the final DataFrame.

• Reset the index after major row-level changes when the old index is no longer meaningful.
• Use drop=True to prevent the old index from becoming an additional column.
• Resetting the index does not change the actual data values.

df = df.reset_index(drop=True)

print(df.index[:5])

Output

RangeIndex(start=0, stop=5, step=1)

Explanation:

• reset_index() creates a new sequential index.
• drop=True removes the previous index instead of storing it as a column.
• A clean index makes the final DataFrame easier to work with.
• Step is especially useful after filtering or removing records.

Removing Unnecessary Columns

A dataset may contain columns that are no longer required for the intended analysis. Removing irrelevant fields reduces clutter and makes the final dataset easier to understand.

• Remove only columns that are genuinely unnecessary.
• Keep fields that may be useful for future analysis or validation.
• Check the column names before dropping them.
• Avoid removing the original variables before all required transformations are complete.

df = df.drop(
    columns=["Discount_Amount", "Discounted_Unit_Price"],
    errors="ignore"
)

Output

Explanation:

• dDiscount_Amount and Discounted_Unit_Price were used to calculate Total_Amount.
• Total_Amount is retained because it contains the final derived value.
• errors="ignore" prevents an error if an intermediate column is not present.

Comparing the Dataset Before and After Wrangling

Comparing the raw and cleaned datasets provides a simple way to understand the effect of data wrangling. It shows how the structure and quality of the data changed throughout the workflow.

• Compare the number of rows and columns.
• Compare missing values before and after cleaning.
• Check whether duplicate records were removed.
• Compare data types and important derived fields.
• Comparison helps verify that the cleaning process achieved its intended goal.

raw_df = pd.read_excel(
    "/content/drive/MyDrive/Data Wrangling/e_commerce_retail_dataset.xlsx",
    sheet_name="Messy_Sales_Data"
)

print("Raw Dataset Shape:", raw_df.shape)
print("Cleaned Dataset Shape:", df.shape)

print("\nMissing Values in Raw Dataset:")
print(raw_df.isnull().sum().sum())

print("\nMissing Values in Cleaned Dataset:")
print(df.isnull().sum().sum())

print("\nDuplicate Order IDs in Raw Dataset:",
      raw_df["Order_ID"].duplicated().sum())

print("Duplicate Order IDs in Cleaned Dataset:",
      df["Order_ID"].duplicated().sum())

Output

Explanation:

• raw_df represents the original dataset.
• df represents the cleaned and transformed dataset.
• Comparing their shapes shows whether records were removed or added.
• Comparing missing-value counts shows the effect of the cleaning process.
• Comparing duplicate identifiers confirms whether duplicate records were addressed.
• Provides a final overview of the data-quality improvements.