Small models vs. GPT-5 on fraud detection
Can a fine-tuned open-weight model beat a frontier model at spotting deception? Across seven domains, the smaller one did.
- When
- 2025 – 2026
- Role
- Independent researcher, AP Research
- Outcome
- A fine-tuned 20B model beat GPT-5: 0.78 vs. 0.65 F1
- Built with
- GPT-OSS-20B, GPT-OSS-120B, GPT-5, Fireworks.ai, Python
Why I started
Everyone assumes the biggest model wins. For something like fraud filtering, which runs on every message, I wanted to know whether that's true, and what it costs.
What I did
I fine-tuned GPT-OSS-20B and GPT-OSS-120B for one epoch each on Fireworks.ai, then tested them against GPT-5 with no fine-tuning.
The data came from the DiFrauD benchmark: seven domains (fake news, SMS spam, phishing, Twitter rumours, fake product reviews, job scams, and false political statements) and 9,589 test examples. I scored precision, recall, F1, accuracy, and latency, then wrote it up as a full research paper.
Progress so far
The smallest model won: 0.78 mean F1, against 0.69 for the 120B model and 0.65 for GPT-5. It answered in about 425 ms, against roughly 7 seconds for GPT-5.
That contradicted my hypothesis. The gap between 20B and 120B wasn't statistically significant (p = 0.51), so the extra size bought nothing here.
GPT-5 was excellent on SMS spam (0.97 F1) but caught fewer than half of fake news articles. The paper earned a 5 on AP Research.
What's next
A small, fine-tuned open model can be cheaper, faster, and more accurate than a frontier model, but only if you test it domain by domain. I want to keep testing claims like that one.