If you have ever shipped an LLM-as-classifier, you know the specific frustration: it works annoyingly well, and it is miserable to operate. You cannot set a threshold. You cannot trade precision against recall. You cannot tell whether it read the structured data you pasted in. And when it underperforms, your only lever is to go back and fiddle with the prompt, a practice about which "advice abounds on the internet but wisdom is scarce."
A post making the rounds on September 13, 2026 proposes a reframing that fixes most of this: the LLM verdict is not your classifier. It is a feature.
TL;DR
| Problem with LLM-as-classifier | What the fix restores |
|---|---|
| Hard labels, no trustworthy confidence | Calibrated probabilities from the fitted model |
| No principled precision/recall tradeoff | A threshold you can actually tune |
| Structured data may be ignored in-prompt | Covariates enter the model directly, with fitted weights |
| Baked-in priors mismatch your population | The model fits your base rate and adapts to your data |
| Improvement means prompt roulette | Standard ML levers: more data, more features, better architecture |
Measured on SemEval 2018 irony detection:
| Setup | Brier score | F1 (test) |
|---|---|---|
| LLM hard label only | 0.259 | 0.747 |
| + logistic regression wrapper | 0.175 | 0.747 |
| + LLM sub-features | 0.131 | 0.768 |
| + rule-based features | 0.127 | 0.779 |
For reference, random guessing scores a Brier of 0.25. The raw LLM classifier was barely better than a coin flip on calibration, while being genuinely good at ranking.
Why the hard label is the problem
The core issue is that an LLM is not designed as a classifier and has no mechanism for producing a calibrated probability.
You can extract token log-probabilities, but there is no reason to believe they are calibrated. You can ask the model for its confidence, and there is no reason to believe that either. Without calibrated probabilities you cannot choose an operating point, which means you cannot say "I will accept more false positives here because a miss costs ten times more than a false alarm." That decision is the entire job in most production classification.
Two subtler failures compound it:
You do not know what it used. You can paste structured data into the prompt, but the model does not have to use it. The same applies to your prose instructions. As the post notes drily, this shows up with multimodal models that do not even look at the images.
Its priors are not your priors. The model has no idea whether your positive class is rare in the wild or enriched in your sample. You can state the base rate in the prompt, and you are back to hoping it incorporated that correctly.
The fix, in one equation's worth of idea
Take the LLM's verdict and use it as an input feature to a logistic regression, fitted on training data.
In the degenerate case this recovers the LLM classifier exactly. But with fitted parameters, the binary verdict collapses to two empirical probabilities:
| LLM verdict | Fitted P(Ironic) |
|---|---|
| Ironic | 0.687 |
| Not | 0.188 |
That single change takes Brier from 0.259 to 0.175 without altering the ranking at all, so F1 stays identical at 0.747. You have not made the model smarter. You have made its output usable, because those are now real probabilities you can threshold.
The ordering matters here. Calibration and discrimination are different properties, and the LLM had decent discrimination and terrible calibration. Most teams try to fix a calibration problem with prompt engineering, which cannot touch it.
Then it gets more interesting: more features
Once the LLM is a feature generator rather than the decision-maker, the standard ML playbook reopens.
Ask the model for sub-verdicts. Rather than "is this ironic," the improved prompt asks for the verdict plus nineteen dimensions: is it trying to be funny, does it describe a realistic situation, does it need external context, is it a genuine complaint, is there a mismatch between a negative situation and upbeat language, is it self-deprecating, does it contain explicit contrast, is it a rhetorical question, a backhanded compliment, feigned surprise, mock enthusiasm, wordplay, political, celebrity, mundane daily life, and so on.
Each becomes a boolean feature. Brier drops to 0.131, F1 rises to 0.768.
Add features the LLM cannot see. Deterministic, computable signals: is it a reply (starts with @), does it contain a URL, character length, hashtag count, exclamation count, all-caps word count, ellipsis presence, starts with a quote, emoji count. Brier 0.127, F1 0.779.
That last step matters more than its size suggests. Those features cost nothing, involve no model call, and the gain proves the LLM was leaving signal on the table that no prompt would have recovered.
How you improve now changes entirely. Instead of prompt roulette you have: collect more data, screen features empirically (if a feature should always imply positive, check that it does), generate more features from residuals, or swap logistic regression for XGBoost or a small neural net. All plug-and-play.
The results in context
| System | F1 |
|---|---|
| Random baseline | 0.373 |
| SVM + tf-idf (paper baseline) | 0.589 |
| THU_NGN (competition winner) | 0.705 |
| NTUA-SLP (post-competition SOTA) | 0.786 |
| LLM hard label | 0.747 |
| LLM + logistic regression | 0.779 |
Two things stand out. The zero-shot LLM already beat the competition winner, which is the genuinely remarkable part: no training, and it outperforms a task-specific system built by a team for a shared task. And the wrapped version reaches 0.779 with confidence intervals overlapping the post-competition state of the art, using nothing more exotic than logistic regression on LLM-extracted features.
The criticism, which is worth engaging
The top Hacker News comment was unimpressed, arguing that the improvement over straight LLM classification is relatively small and that working on the prompt, or simply telling the LLM which features to consider, would likely work as well.
Half of that is a fair empirical challenge. Putting the nineteen dimensions into the prompt as guidance, rather than extracting them as features, is a genuine alternative that deserves a head-to-head. Nobody has run it here.
The other half misses the point. Calibration is not a performance improvement, it is a different property. The first wrapper step improved Brier by 32% while leaving F1 untouched. No prompt achieves that, because the issue is not that the model's judgements are wrong, it is that its outputs are not probabilities. If your application needs a threshold, and most do, prompt quality is orthogonal to your problem.
The same commenter's framing that "something general also solves particular problems" is true and not a rebuttal: a general system that solves your particular problem without giving you a dial is still unusable in production.
The honest cost is the one the original post concedes: you need labelled training data, which forfeits the training-free appeal of LLM classification. The counterargument is sound, though. If you were never going to measure performance, you had no idea whether your classifier worked.
How this connects to the decision-model debate
This lands in the same week as TypeSafe's Jev, a model that returns typed decisions with calibrated confidence instead of text, precisely because using a chat model as a classifier is a bad fit.
Two answers to the same complaint, with different tradeoffs. Jev replaces the LLM with a purpose-built decision model, buying latency and calibration at the cost of a new dependency and no vision. The feature-engineering approach keeps your existing LLM and adds a thin statistical layer, buying calibration and interpretability at the cost of needing labelled data.
Notably, one developer is already using Jev as the classifier inside an LLM autorouter, which is the same instinct: do not ask a text generator to make a typed decision.
If you are choosing: use the wrapper when you have labels and care about calibration and interpretability, and reach for a decision model when per-call latency or cost is the binding constraint.
What to do on Monday
- Check whether your LLM classifier is calibrated. Compute a Brier score on your test set. If it is near 0.25, your probabilities are decorative.
- Keep the prompt, change the plumbing. Store the verdict as a feature rather than as the answer. Fit a logistic regression. This is an afternoon.
- Ask for sub-verdicts. Decompose the judgement into dimensions and let the model report each. One call, many features.
- Add free deterministic features. Lengths, counts, structural flags. They cost nothing and reliably help.
- Now you can tune a threshold, which is what you wanted all along.
Related on explainx.ai
- TypeSafe AI launches Jev — the other answer to the same problem
- Top Jev use cases — including classification and autorouting
- How to read AI benchmarks — why Brier and F1 tell you different things
- Why AI models hallucinate and how to catch it — verification as a systems problem
- Structured output and JSON mode — the mechanics of getting typed fields out of a model
Figures are from the linked analysis published September 13, 2026, using Gemini 3.1 Flash Lite on the SemEval 2018 Task 3 dataset (3,834 train / 784 test). Results are from a single author's replication, not an independent reproduction, and the comparison against published SemEval systems uses that author's own evaluation harness.