Indonesian Transaction Understanding (Dataset + SLM + Evals)

Context

Indonesians record expenses the way they talk: “kemarin beli kopi 25rb di gofood pake gopay” — slang, amount shorthand (25rb, 2,5jt, sejuta), Indo-English code-switching, and relative dates. Turning that into structured data (amount, category, merchant, payment method, date) is the capture step every local finance app needs, and no public Indonesian dataset covers it. Indonesian is a high-population, low-resource language in NLP; this project builds the missing piece from scratch: schema, annotation guidelines, data, baselines, a fine-tuned model, and — before any of the modeling — an evaluation harness.

Problem

Frontier LLMs can parse this text but need a network call, which is the wrong architecture for private financial data, and their per-request cost is hard to justify for a single-purpose extraction task. A small specialized model that runs on-device is the right shape — but “right shape” has to be proven with numbers, not vibes: against a rule baseline, on data the model has never seen, with per-field metrics.

My Role

Everything: extraction schema and 12-label category taxonomy, annotation guidelines with tie-breaker rules for the genuinely ambiguous cases (is an e-wallet top-up an expense or a transfer?), synthetic data generator, rule-based baseline, evaluation harness, and the fine-tuned model.

Evidence

Every number below comes out of one script: the harness writes a versioned JSON report per model per split, and the write-up quotes those reports — per-field exact-match accuracy on the same 15 held-out probe examples for both systems.

Field (probe split, n = 15)	B0 rules	M1 hybrid
Amount	0.93	0.93
Direction	0.93	0.93
Date reference	0.87	0.87
Payment method	1.00	1.00
Category	0.73	0.93
Merchant (slot F1)	0.86	0.86
Transaction accuracy	0.47	0.67

The only row that moves is category — the one field the classifier was trained to fix — and it drags the headline transaction accuracy up with it; the deterministic extraction fields stay identical because they are the same code in both systems. The harness run, quoted as-is:

$ python scripts/run_evals.py --model m1 --split probe
model=m1 split=probe
n                     15
amount_acc            0.9333
direction_acc         0.9333
date_ref_acc          0.8667
payment_method_acc    1.0
category_acc          0.9333
merchant_precision    1.0
merchant_recall       0.75
merchant_f1           0.8571
transaction_accuracy  0.6667
examples_with_errors  5
report written: reports\m1_probe.json

Approach

• Schema and guidelines first. Seven extraction fields with a Pydantic schema that rejects invalid records at load time, and written annotation guidelines with explicit tie-breakers (treating a friend to a meal is food_drink, not social; top-ups are transfer; generic venues like “warteg” are not merchants, brands like “gofood” are).
• Evals harness before any model. A CLI (run_evals.py --model X --split Y) computes per-field exact-match accuracy, merchant slot F1, and a headline transaction accuracy (all non-null gold fields correct), and writes a versioned JSON report. Every number in this write-up comes out of that one script.
• Rule baseline (B0) as the floor. Regex + dictionaries for amounts (25rb / 2,5jt / sejuta / bare numbers), direction keywords, brand and payment-method lexicons with span-based disambiguation (“pake gopay” is a payment method; “topup gopay” makes gopay the counterparty).
• A held-out probe set that the rules never saw. B0 scores a perfect 1.00 on the 30 seed examples its rules were developed against — a number that means nothing, which is the point. On 15 held-out probe examples it drops to 0.47 transaction accuracy, failing exactly where hand-written rules run out: spelled-out numbers (“lima puluh ribu”), brand typos (“gofud”), keyword-less categories (“arisan”), and unseen date forms.
• Synthetic training data, honestly labeled. A seeded generator composes ~400 training utterances from slot banks (items, venues, payment methods, amount renderings, date phrases); every record carries source: synthetic, the probe set is verified disjoint from training by a test, and the plan’s human-written golden set stays a separate, unbuilt milestone rather than a claim.
• Hybrid M1: learn what rules can’t, keep rules where they win. Category classification is where B0 is weakest, so that field gets a fine-tuned IndoBERT (indobert-base-p1) classifier trained on the synthetic set (CPU-only, minutes), while deterministic rules keep the extraction fields. Both models run through the same harness on the same probe set.

Key Decisions

Why the harness came before the model. A model without a measurement is a demo. Building run_evals.py first meant the baseline, the probe gap, and the model comparison all use identical definitions — and a regression on the frozen split fails loudly instead of silently.

Why a hybrid instead of end-to-end. B0’s extraction fields (amount 0.93, payment method 1.00 on probe) are already strong and fully interpretable; its category accuracy (0.73) is the bottleneck. Fine-tuning a classifier for exactly the weak field is cheaper, more debuggable, and more honest than replacing working deterministic code with a model.

Why the perfect seed score is reported at all. The 1.00 → 0.47 drop between development data and held-out data is the clearest illustration of why frozen eval sets exist. Publishing that gap, rather than hiding the flattering number’s origin, is the discipline the whole project is built to demonstrate.

Result

The hybrid M1 — IndoBERT (indobert-base-p1) fine-tuned for category over ~400 synthetic examples in four CPU-only epochs, plus B0’s deterministic extractors — went through the same harness, on the same held-out probe, as the baseline. Transaction accuracy rose from 0.47 to 0.67, driven entirely by the field the model was trained to fix: category accuracy went from 0.73 to 0.93, while the rule-owned extraction fields stayed put (amount 0.93, payment method 1.00, merchant F1 0.86).

Two details matter more than the headline. First, the remaining probe failures are now extraction, not classification — spelled-out amounts (“lima puluh ribu”), a brand typo (“gofud”), an unseen relative date (“bulan depan”), and one income utterance with no income keyword — which is a concrete, prioritized worklist rather than a vague “model needs to be better.” Second, on the seed split the classifier scores below the rules (0.93 vs 1.00): it stopped memorizing the seed set’s quirks and generalizes instead, which is the trade you want and exactly what the two-split setup exists to reveal.

All current numbers are on synthetic data (AI-drafted probe, generator-built training set) — labeled as such throughout the repo. The next milestone is the human-written golden set (60 frozen examples), which converts these preliminary numbers into citable ones; after that, a HuggingFace dataset/model release and a cost-latency benchmark against a few-shot frontier-LLM baseline.

What I’d Improve

• Replace the AI-drafted probe with the human-written frozen golden set before quoting any number externally — the probe measures generalization beyond the rules, but it shares authorship with the training generator’s design.
• Add the few-shot frontier-LLM baseline (B1) for the quality-ceiling and cost-per-1,000-parses comparison the project is designed around.
• Distill or quantize the classifier (ONNX) and measure CPU latency to complete the on-device story.