Fraud Detection ML Pipeline (Undergraduate Thesis)

Context

This is my undergraduate thesis at Institut Teknologi Sepuluh Nopember (ITS): fraud detection on the IEEE-CIS Fraud Detection dataset (Kaggle, Vesta Corporation) — 590,540 e-commerce transactions, 3.5% fraud (20,663 fraud vs. 569,877 legitimate). The proposed method combines an Autoencoder with LightGBM under Bayesian Optimization; the interesting part is that the Autoencoder’s role in that pipeline changed once I diagnosed why the original design wasn’t working.

Problem

At 3.5% fraud, a classifier optimized for accuracy is close to useless — predicting “not fraud” every time already scores 96.5%. The real work is in the representation of the minority class, a fair evaluation protocol under stratified 60/20/20 splitting, and a tuning budget that can’t afford grid search.

Approach

• Diagnosed the original design first. The initial plan used the Autoencoder as a feature extractor: train it on legitimate transactions, append reconstruction error and/or latent codes as extra features for LightGBM. Across every variant I tried — global and grouped reconstruction error, 32-dim latent concatenation, and a score-level ensemble — none beat the plain LightGBM baseline (deltas −0.005 to −0.024 AP, all p≈1.0 under paired bootstrap); a score ensemble gave the AE zero weight. The root cause: the AE was reconstructing the same V-block of engineered features LightGBM’s trees already split on, so it added noise, not signal.
• Pivoted to latent-space oversampling. Instead of feeding the AE’s output to the classifier, I use it to synthesize fraud: train an undercomplete Autoencoder (128→64→64 bottleneck, MSE loss, Adam) on fraud-only training rows, embed those rows into latent space, run SMOTE-style k=5 interpolation between latent fraud points (Chawla et al., 2002, applied in z-space rather than raw feature space — a tabular adaptation of DeepSMOTE/nDeepSMOTEc), then decode the interpolated points back into synthetic fraud rows appended only to the training split.
• Locked the sensitivity parameters on validation, not by guesswork. latent_dim and target synthetic fraud rate were chosen from a grid sweep (latent_dim of 8, 16, 32, or 64, crossed with a target rate of 10%, 15%, 20%, 30%, or 50%) scored by validation PR-AUC — locked at latent_dim=64, rate=10%.
• Controlled comparison. The AE-oversampled scenario is compared against a plain LightGBM baseline and a SMOTE-NC control at the same target fraud rate, on the same frequency-encoded + z-scored feature representation, so the win can’t be attributed to oversampling in general.
• Bayesian Optimization over LightGBM’s hyperparameters — Optuna with a TPE sampler, 10 trials over a 7-dimensional space (learning rate, num_leaves, min_child_samples, subsample, colsample_bytree, reg_alpha, reg_lambda), objective = validation PR-AUC, on a 2000-tree LightGBM with 100-round early stopping.

Evidence

All four figures below are taken directly from the thesis paper (POMITS submission); axis labels and annotations are in Indonesian because that is the language of the original document.

Key Decisions

Why latent-space oversampling instead of feature extraction. This wasn’t the original plan — it’s the result of a diagnosis. Reconstruction-error and latent-code features never beat the baseline because the AE was redundant with what LightGBM’s trees already extracted from the same engineered features. Rather than force a feature-extractor narrative to fit disappointing numbers, I re-read where the literature (DeepSMOTE, nDeepSMOTEc) uses autoencoders for imbalanced tabular data — as a generator, not an encoder for the classifier — and that reframing is what actually worked.

Why SMOTE-NC as an explicit control, not just a second baseline. Any oversampling method reduces the effect of class imbalance; the question that matters is whether the autoencoder specifically is doing something SMOTE-NC alone can’t. Matching the target fraud rate between the two isolates that one variable, so the AP gap between them is attributable to the AE’s on-manifold interpolation, not to “more fraud examples” in general.

Why Bayesian Optimization over random or grid search. Grid search doesn’t scale past a couple of hyperparameters at this dataset size, and I wanted evidence rather than an assumption for choosing Optuna’s TPE sampler over plain random search — so I re-ran the same 7-D space, same 10-trial budget, with random search as a control: TPE reached 0.8589 validation PR-AUC vs. 0.8548 for random search, in essentially the same wall-clock time.

Result

Scenario (test PR-AUC)	Untuned	Bayesian-tuned
Plain LightGBM	0.8084	0.8497
SMOTE-NC control	0.8157	0.8500
AE latent-SMOTE	0.8267	0.8547

The tuned AE latent-oversampling pipeline reaches 0.8547 test PR-AUC (Average Precision), vs. 0.8497 for a tuned plain-LightGBM baseline and 0.8500 for tuned SMOTE-NC. Untuned, the gap is wider (0.8267 vs. 0.8084 baseline, +0.0183). At the validation-selected operating threshold, the AE model recalls 74.71% of fraud at 89.04% precision — 41 more fraud transactions caught than the baseline (3,087 vs. 3,046 true positives), at the cost of 55 more false positives (380 vs. 325) out of 113,976 legitimate test transactions.

What I’d Improve

• Run paired-bootstrap significance testing on this exact final checkpoint — an earlier iteration of the same method had a bootstrap-tested AE-vs-baseline gap (p<0.001), but I didn’t re-run that test on the final submitted numbers, and I’d rather say so than imply a stats test that wasn’t done.
• Add a temporal/chronological validation split to check for concept drift — under a naive chronological split the baseline PR-AUC collapses to ~0.5, which I currently attribute to entity leakage across train/test under the stratified split rather than a real drift signal, but that’s a limitation worth closing out properly.
• Extend the latent oversampler with the clustering-enhanced variant from nDeepSMOTEc (Fan et al., 2025) rather than plain k=5 interpolation.