Unmasking Model Bias: How Balanced Datasets and Mimesis Transform Fair AI Audits

# Introduction

Whether you’re working with traditional classifiers or cutting-edge models like large language models (LLMs), a persistent challenge in building machine learning systems is the risk that algorithms quietly absorb biases present in the historical data used for training. But in high-stakes or data-sensitive situations, how can you check whether a model is biased without exposing real personal information?

This practical guide walks you through training a basic “loan approval” classifier using intentionally biased data. From there, we’ll leverage Mimesis, an open-source library for generating perfectly balanced, counterfactual datasets. You’ll learn to create “fake” applicants who share identical financial profiles but differ in demographic attributes, letting you verify if your model treats different groups fairly.

# Step-by-Step Guide

If you haven’t used Mimesis before or are working in a cloud notebook like Colab, start by installing the library:

Before you can audit a model, you need one to work with! In this tutorial, we’ll artificially create a dataset of 1,000 bank customers with just two attributes: gender and income — one categorical and one numerical. The generation process will be deliberately skewed so that gender disproportionately affects the loan approval outcome. Specifically, when labeling the data, we’ll assume a scenario where men are typically approved, while women are only approved if their income is exceptionally high.

Here’s how to construct this intentionally biased dataset and train a decision tree classifier on it:

import pandas as pd
import numpy as np
from sklearn.tree import DecisionTreeClassifier

# 1. Simulating biased historical data (1000 instances)
np.random.seed(42)
n_train = 1000
genders = np.random.choice(['Male', 'Female'], n_train)
incomes = np.random.randint(30000, 120000, n_train)

approvals = []
for gender, income in zip(genders, incomes):
    if gender == 'Male':
        # Historically, males are approved
        approvals.append(1)
    else:
        # Only females with high income are approved
        approvals.append(1 if income > 80000 else 0)

train_df = pd.DataFrame({'Gender': genders, 'Income': incomes, 'Approved': approvals})

# Converting categories to numbers for the machine learning model
train_df['Gender_Code'] = train_df['Gender'].map({'Male': 1, 'Female': 0})

# 2. Training a Decision Tree classifier
model = DecisionTreeClassifier(max_depth=3)
model.fit(train_df[['Gender_Code', 'Income']], train_df['Approved'])

Now let’s see Mimesis in action. We’ll use the library to create a handful of test subjects via the Generic class. To do this, we’ll define three baseline financial profiles containing random UUIDs (universally unique identifiers) and moderate incomes between $40K and $70K. Note that these profiles won’t include gender information yet:

from mimesis import Generic

generic = Generic('en')

# Generating 3 base financial profiles
base_profiles = []
for _ in range(3):
    profile = {
        'Applicant_ID': generic.cryptographic.uuid(),
        'Income': generic.random.randint(40000, 70000) # Moderate income
    }
    base_profiles.append(profile)

As an example, the three generated profiles might look like this:

[{'Applicant_ID': '1f1721e1-19af-4bd1-8488-6abf01404ef9', 'Income': 44815},
 {'Applicant_ID': '5c862597-7f55-43f4-9d6e-ac9cc0b9083e', 'Income': 47436},
 {'Applicant_ID': '3479d4cf-0d9b-4f06-9c43-1c3b7e787830', 'Income': 58194}]

Now let’s complete our counterfactual test set — the heart of our auditing workflow! For each of the three base profiles, we’ll create two mirrored counterfactual versions: one male and one female. Each pair of test applicants will share the exact same application ID and income, differing only in gender. Any disparity in how our trained decision tree handles them serves as clear evidence of gender bias.

counterfactual_data = []

for profile in base_profiles:
    # Version A: Male Counterfactual
    counterfactual_data.append({
        'Applicant_ID': profile['Applicant_ID'],
        'Gender': 'Male',
        'Gender_Code': 1,
        'Income': profile['Income']
    })

    # Version B: Female Counterfactual
    counterfactual_data.append({
        'Applicant_ID': profile['Applicant_ID'],
        'Gender': 'Female',
        'Gender_Code': 0,
        'Income': profile['Income']
    })

audit_df = pd.DataFrame(counterfactual_data)

Here’s what the three customer pairs might look like:

1f1721e1-19af-4bd1-8488-6abf01404ef9	Male	1	44815
1	1f1721e1-19af-4bd1-8488-6abf01404ef9	Female	0	44815
2	5c862597-7f55-43f4-9d6e-ac9cc0b9083e	Male	1	47436
3	5c862597-7f55-43f4-9d6e-ac9cc0b9083e	Female	0	47436
4	3479d4cf-0d9b-4f06-9c43-1c3b7e787830	Male	1	58194
5	3479d4cf-0d9b-4f06-9c43-1c3b7e787830	Female	0	58194

An important point to highlight here: we’ve just used Mimesis to instantly construct perfectly matched “clones” of loan applicants who have identical incomes but different genders. This showcases the library’s power in providing complete statistical control by isolating a single protected attribute.

Now it’s time to put the model to the test and analyze the outputs.

# Asking the model to predict approval for our counterfactuals
audit_df['Predicted_Approval'] = model.predict(audit_df[['Gender_Code', 'Income']])

# Formatting the output for readability (1 = Approved, 0 = Denied)
audit_df['Predicted_Approval'] = audit_df['Predicted_Approval'].map({1: 'Approved', 0: 'Denied'})

print("n--- Model Audit Results ---")
print(audit_df[['Applicant_ID', 'Gender', 'Income', 'Predicted_Approval']].sort_values('Applicant_ID'))

The results our model produces couldn’t be more stark:

--- Model Audit Results ---
                           Applicant_ID  Gender  Income Predicted_Approval
0  1f1721e1-19af-4bd1-8488-6abf01404ef9    Male   44815           Approved
1  1f1721e1-19af-4bd1-8488-6abf01404ef9  Female   44815             Denied
4  3479d4cf-0d9b-4f06-9c43-1c3b7e787830    Male   58194           Approved
5  3479d4cf-0d9b-4f06-9c43-1c3b7e787830  Female   58194             Denied
2  5c862597-7f55-43f4-9d6e-ac9cc0b9083e    Male   47436           Approved
3  5c862597-7f55-43f4-9d6e-ac9cc0b9083e  Female   47436             Denied

Observe that for the exact same Applicant_ID and Income, male counterparts receive loan approval, while female counterparts with comparable moderate incomes are consistently denied. The Mimesis-powered profile generation helped us keep all other variables constant, effectively isolating and revealing the model’s discriminatory behavior.

# Wrapping Up

In this hands-on guide, we’ve demonstrated how Mimesis can generate balanced, counterfactual data samples — free from privacy or sensitive data concerns — to audit a model’s behavior and determine whether it exhibits bias. If your model turns out to be biased, here are some next steps to consider:

• Enriching your training data with more balanced profiles to address historical imbalances or bias.
• Applying model re-weighting techniques, depending on the type of model you’re using.
• Leveraging open-source fairness toolkits — such as AI Fairness 360 — which are designed to help mitigate bias within machine learning pipelines.

Iván Palomares Carrascosa is a leader, writer, speaker, and adviser in AI, machine learning, deep learning & LLMs. He trains and guides others in harnessing AI in the real world.