# Building a Fraud Detection Pipeline for Banking Transactions: Lessons From a Final-Year Project
## The Model on Deployment Wasn’t the Best One
There is a version of this story that sounds impressive: I trained six machine learning models on a fraud detection dataset, logged every run, and confidently shipped the winner. The reality is less clean. The model currently serving live predictions is not the one that scored highest. It isn’t even the one I would choose if I made the decision today.
This is the account of how an academic project for detecting fraudulent banking transactions turned into something more layered than expected, and why the system you end up deploying is rarely the one the spreadsheets tell you to pick.
—
## Bringing Two Datasets Into Alignment
The training data came from two public sources. PaySim is a simulated stream of mobile money transactions, with fields that track how much money sits in an account before and after a transfer — fields like `oldbalanceOrg` and `newbalanceOrig`. The IEEE-CIS fraud detection dataset, by contrast, covers card-based transactions and contains no balance information at all, relying instead on features like `ProductCD`, `card1`, `card2`, and dozens of other anonymized signals.
These two sources do not share a schema, so before any merging could happen, each one had to pass its own validation step. I defined separate schema models for each dataset using structured validation, ensuring that a malformed row would fail loudly rather than silently degrade the training set. Every field was checked for type correctness, value ranges, and expected presence.
Once both datasets were clean and validated, I mapped their disparate fields into a single shared structure. The transaction type from PaySim was translated into a channel label, and the product code from IEEE-CIS was mapped through a lookup table to a human-readable channel name.
What I came to understand slowly was that a unified schema constrains which features are actually usable. The PaySim balance fields couldn’t carry over because IEEE-CIS has no equivalent. The three features that ended up powering the deployed model were the transaction amount, a one-hot encoding of the channel, and a one-hot encoding indicating which source dataset the row originated from. This limitation matters — it shapes everything that follows.
—
## Working With Data Where Fraud Is the Rare Event
Fraud occurrences are scarce in both sources. A classifier trained on such data can reach misleadingly high accuracy simply by predicting “not fraud” for every transaction, since the overwhelming majority of transactions are legitimate.
To address this, I applied SMOTE (Synthetic Minority Oversampling Technique) to the training split only. This generates synthetic examples of the minority fraud class by interpolating between existing fraud cases in feature space, giving the model more positive examples to learn from without inflating the test set.
I also built a hand-written interpolation fallback as a safeguard. If the `imbalanced-learn` library was unavailable in a deployment environment, the fallback would compute synthetic minority samples by finding each fraud case’s nearest neighbors using Euclidean distance and creating new points along the line between them. It is a simplified version of what SMOTE does internally, but it prevents the entire pipeline from failing silently in a minimalistic setup.
A guard was included to check whether enough minority-class samples existed to perform resampling. If there weren’t, the training data was passed through unchanged with a warning rather than crashing mid-training.
—
## Six Models and an Uncomfortable Spreadsheet
I trained and compared six models: Random Forest, Logistic Regression, XGBoost (both baseline and hyperparameter-tuned versions), and LightGBM (again, both baseline and tuned). Every model ran through the same evaluation helper so that the numbers — accuracy, precision, recall, AUC-ROC, and AUC-PR — were all measured consistently and would be genuinely comparable.
Every run was logged to a JSON file with its hyperparameters and metrics. Months later, that log would become the reason I noticed something uncomfortable: the model in production was not the one at the top of the leaderboard.
Logistic Regression delivered the highest recall of any model at 0.95, meaning it caught nearly every fraud case. But its precision was only 0.14, which means the vast majority of its fraud alerts were false alarms. A bank deploying that model would be drowning in false positives, and legitimate transactions would constantly be interrupted for verification.
Tuning XGBoost produced a slight improvement in AUC-PR but worsened precision relative to the baseline version. This is a useful caution: hyperparameter optimization optimizes the metric you tell it to optimize, and that metric may not align with what actually matters once the model is making live decisions.
LightGBM’s default baseline configuration outperformed its tuned version on AUC-PR and ranked as the best performer across all six models. Despite this, the production API continued to serve XGBoost (Tuned), because that choice had been locked in before the LightGBM results were fully evaluated.
Looking back, that decision is worth revisiting.
To make any of these results meaningful to a human reviewer, I used SHAP to explain each prediction. Every flagged transaction comes with a list of features that pushed the model toward its fraud decision, because “the model said so” is not an answer a bank or a regulator should accept on its own.
—
## What My Advisor Really Wanted Me to Build
Early on, my advisor made an observation that sparked a significant shift in the project’s direction. He had seen a classmate’s anti-money laundering project and felt the two were too similar on the surface — both systems took transaction data and produced a fraud-or-not fraud decision.
He was not wrong about the surface resemblance. What he wanted me to build was not a better classifier, though. He wanted me to think about what happens after a transaction gets flagged.
Most academic AML work stops at the point of classification. His question pushed me into the territory that follows: who reviews the flag, what options do they have, and how are their decisions tracked?
This led to the creation of a Regulatory Notification Center — a review layer structured around role types modeled on Nigeria’s Central Bank, the Economic and Financial Crimes Commission (EFCC), and the National Deposit Insurance Corporation (NDIC). Each role type has its own login and its own queue of flagged transactions.
A reviewer can approve a transaction, request OTP verification, or block it outright. Every decision is recorded in an audit log with the identity of the actor and a timestamp. Building this review workflow is also what introduced LightGBM into the comparison: the system needed multiple model options to feed into the workflow, and LightGBM’s strong baseline performance made it a natural candidate for that role.
—
## How the Confidence Gate Works
The API does not treat a fraud prediction as a binary yes or no. Instead, a threshold of 0.50 sits at the base of a tiered routing system.
– **Below 0.50:** The transaction passes through with no alert triggered. No flag, no notification, no interruption.
– **Between 0.50 and 0.80:** The transaction is flagged but not blocked. It is routed to a `PENDING_OTP` state, meaning a human or a secondary verification step must review it before it can proceed.
– **At or above 0.80:** The transaction is blocked from completing. It never reaches the recipient.
– **At or above 0.85:** In addition to being blocked, an alert is dispatched immediately via SMS or email through a background thread. This notification fires independently of any request-response cycle, ensuring that critical cases surface in real time.
The system generates a unique alert ID for each flagged transaction, attaches the model’s probability score and the specific rule that was triggered, and routes the payload to the appropriate destination — whether that is an analyst queue, a regulator’s dashboard, or a notification channel.
A transaction never receives a single up-or-down answer. It is routed to whoever should be looking at it next: an analyst, a regulator, or nobody at all.
—
## What Defense Day Taught Me
The defense panel asked the standard questions first: which models were used, what hyperparameters were tuned, where the data came from, and whether the datasets were representative enough of real transaction behavior. I told them about XGBoost (Tuned) because that was what was live. What I did not mention — partly because I had not sat with the evidence myself yet — was that LightGBM’s baseline had already outperformed it on the most important metric days before I stood in front of the panel. Nobody on the panel followed up on it. I am following up on it now.
Then came the question I had not prepared for: what makes this fraud detection solution different from the many others that exist, and what gaps does it fill?
The panel also asked scenario questions. If a CEO moved a million dollars in a single transaction one afternoon, would it get flagged? What if a student attempted the same?
I answered that the system studies patterns and behavior, so the two transactions would be treated differently depending on who was making them.
That was the comfortable answer. The accurate one, looking at the code months later, is different. The model on deployment makes decisions based on thirteen features — the transaction amount, the channel, and which source dataset the row came from. There is no account identity, no transaction history, no historical behavior baseline for any individual user.
A million-dollar transaction is seen as a million-dollar transaction regardless of whose account it left. The system does not know the difference between a CEO and a student. It only knows the number, the channel, and the dataset origin.
What it does instead of using account-level reasoning is rely on the confidence gate. If the model flags that million-dollar transaction as fraud with a probability above 0.85, the transaction is blocked and a human reviewer gets it in their queue. That reviewer has access to far more context than the model does — account history, customer relationship, business purpose.
The confidence gate is doing the job that account-level behavioral features would do if they existed. It is a real and functional answer, just a more modest one than the one I gave at the panel.
—
## What I Would Do Differently
If I were to rebuild the system today, two things would change immediately.
First, I would add behavioral features per account — rolling averages of transaction amounts, deviation from a user’s historical spending patterns, and frequency-based features. These would give the model the ability to distinguish between a CEO and a student without needing the confidence gate to approximate that reasoning. The panel question revealed this gap clearly.
Second, I would switch the production model to LightGBM’s baseline configuration, which scored highest on the metric that matters most for this problem class. Locking in XGBoost (Tuned) before the full comparison was finished was a mistake I only noticed while writing this down.
More broadly, the experience reinforced that the highest score on a leaderboard is not the same as the right production decision. The model is only one component of a functioning fraud detection system. The confidence gate, the audit log, the human reading a pending alert, the regulatory review queue — these are doing as much of the actual fraud-catching as the model itself.
There are parts of the system I have not covered here: the role-based access control setup, the audit logging infrastructure, the regulatory notification pipeline. Each of those deserves its own deep dive, and I plan to revisit them in future writing.
—
## Frequently Asked Questions
**Why does the deployed model differ from the highest-scoring model?**
The production model was selected and locked in before the full comparison of all six models was complete. By the time LightGBM’s baseline outperformed the deployed XGBoost (Tuned) on AUC-PR, the system was already in production and the switch had not been made. This is a common pitfall in ML projects: the decision to deploy can happen before all alternatives have been fairly evaluated.
**What problem does SMOTE solve, and what happens if there aren’t enough fraud samples to use it?**
SMOTE generates synthetic examples of the minority fraud class by interpolating between existing fraud cases. This prevents a model from achieving high accuracy by simply predicting “not fraud” for everything. If there are too few fraud samples for SMOTE to work — fewer than two positive examples — the system skips resampling entirely and passes the training data through unchanged, logging a warning rather than crashing.
**Why are balance-related features from PaySim not used in the final model?**
The IEEE-CIS dataset has no equivalent fields to PaySim’s account balance tracking. When the two datasets are merged into a shared schema, only features that exist in both sources can be used consistently. The PaySim balance fields were excluded because they would not apply to card transactions from the IEEE-CIS dataset.
**What role does the confidence gate play if the model cannot distinguish between different types of users?**
The confidence gate acts as a proxy for user-aware reasoning. Instead of the model knowing that a CEO and a student have different risk profiles, the gate ensures that high-confidence fraud predictions are escalated to human reviewers, who bring contextual knowledge the model lacks. It is not a perfect substitute for behavioral features, but it provides a functional safeguard.
**What is the difference between PENDING_OTP and BLOCKED statuses?**
A transaction with a fraud probability between 0.50 and 0.80 is routed to PENDING_OTP, meaning it is flagged but not stopped — a human or secondary verification step must clear it. A transaction scoring 0.80 or above is BLOCKED and prevented from completing. The 0.85 threshold adds an additional layer by triggering an immediate SMS or email notification to the relevant team.
**What are the three features the deployed model actually uses?**
The model trains on the transaction amount, a one-hot encoding of the transaction channel (such as card web, card phone, or card store), and a one-hot encoding of the source dataset (PaySim or IEEE-CIS). These were the only features shared across both datasets after schema alignment.
**What is SHAP used for in this system?**
SHAP provides feature-level explanations for each fraud prediction. Rather than returning a bare classification, the system can show which features contributed to a transaction being flagged. This transparency is essential for both regulatory compliance and human reviewers who need to understand and trust the model’s output.
—
## Conclusion
Building a fraud detection system taught me that the hardest parts of machine learning are rarely about model selection or hyperparameter tuning. They are about integration, transparency, and the decisions that happen after a prediction is made. The model scores on a spreadsheet are just the beginning. The confidence gate, the review queues, the audit trails, and the willingness to revisit production decisions after the fact — these are what turn a classifier into a real system.
The project started as a straightforward classification exercise and ended as a decision-support pipeline with multiple models, a regulatory review workflow, and a healthy dose of humility about which model actually belongs in production. If there is one takeaway, it is this: the best model is not always the one that wins on the metric. Sometimes it is the one that fits into a system where human judgment, automation, and accountability work together.
—
Thank you for reading
