Important is not the same as useful

I call the series Model forensics, because that is what interpretability turns into once you push past the bar chart: the model has left evidence everywhere, and the work is reading it. This post judges the features. Part 2 stress-tests the classes. Part 3 interrogates the labels.

A confusion matrix told me two of my 77 classes trade seven mistakes between them — and nothing about why. The answer took three posts to write down: compositions on top of SHAP, the method that splits a prediction into per-feature contributions. The model is deliberately boring — TF-IDF plus LightGBM, 86% accuracy on the BANKING77 intent dataset — because boring models give exact SHAP values and features you can read. This part covers the cheapest composition: the same importance numbers normalized two ways, which together prove a feature can be important without being useful, and sort all 10,296 features into four buckets, each with its own action. The strangest thing that fell out: my model reads punctuation.

01 — 86% accurate, and stuck

The setup: BANKING77, a public dataset of 13,083 customer messages sorted into 77 banking intents. Features are TF-IDF — word counts weighted by rarity — plus four structural ones I added by hand (word count, question mark, exclamation count, uppercase ratio). 10,296 features total, fed to LightGBM, a gradient-boosted tree model. Test accuracy 86%, macro F1 0.86 — the per-class average, so small classes weigh as much as big ones. Fine-tuned transformers reach about 93% here. I wasn’t chasing the leaderboard; trees give exact SHAP and features a human can read.

The confusion matrix is where the trouble starts. Most of it is a clean diagonal. The errors cluster in neighborhoods — card classes confused with card classes, transfers with transfers.

And one pair kept trading mistakes: card_payment_wrong_exchange_rate and wrong_exchange_rate_for_cash_withdrawal. Seven errors between them. Same problem, different channel.

The matrix says these two are confused. It cannot say why. Seven misclassifications, zero mechanism. This is where a lot of projects stall: the metric points at the problem and offers nothing to act on.

02 — What the model pays attention to

SHAP assigns every prediction a per-feature breakdown. Average the absolute values per class and you get a matrix: how strongly each feature influences each class. Below, 15 features × 13 classes out of 10,296 × 77 — hand-picked rows and columns, real numbers.

Two patterns. contactless_not_working has one dark cell — tfidf__contactless at 7.69 — and almost nothing else. terminate_account leans on tfidf__delete at 2.40. Meanwhile tfidf__card is moderately dark in nearly every row: 1.82 for card_about_to_expire, 1.53 for card_not_working, 1.33 for card_acceptance.

So is “card” the model’s most valuable feature, or its least? The raw view can’t answer that.

A feature can be important without being useful.

03 — Who owns each word

To separate the two, normalize the other way: take each feature’s total SHAP mass across all 77 classes and ask what share each class owns. I got this wrong on the first attempt — normalized the wrong axis and read fingerprints that weren’t there. The chart below is computed against all 77 classes, with 13 rows shown.

Now the roster is unambiguous. Four words have a sole owner at exactly 100%: contactless, delete, salary, expires. See one of them and you know the class. That is a fingerprint — the forensic kind: one pattern, one suspect.

And the anchors collapse. tfidf__card tops out at 7% ownership — at card_linking, its best class anywhere. tfidf__my peaks at 6%, at virtual_card_not_working. Present everywhere, owned by no one.

The heatmap is also leaking an answer I’m saving for part 2. The exchange-rate pair from the confusion matrix — the one trading seven errors — co-owns its vocabulary: “rate” splits 46% / 36% between the two, 82% of everything the model knows about that word, held by exactly the two classes that keep mistaking each other. “wrong” splits 40 / 15, “exchange” 26 / 14. One scale down, “currencies” splits 47 / 22 between fiat_currency_support and exchange_via_app — which turns out to be the other pair that confuses most. Shared custody of words, shared mistakes. Part 2 makes that precise.

04 — Four quadrants, four actions

Cross the two views and every feature lands in one of four quadrants, each with its own action.

Strong fingerprint — high raw, high ownership. tfidf__contactless: 7.69, 100%. Keep it. But a class leaning this hard on one word is exposed, and part 2 measures how much.

Quiet fingerprint — low raw, high ownership. tfidf__mugged: 0.41, 100% owned by lost_or_stolen_phone. The specificity is there; the volume isn’t. Action: collect more training data with this vocabulary and its synonyms.

Domain anchor — high raw, low ownership. tfidf__card. Deleting it because it’s “generic” would be wrong: it narrows 77 classes down to the card neighborhood. It just can’t finish the job. Action: keep it, and check that each class in the neighborhood has a fingerprint of its own for the last step.

Noise — low raw, low ownership. No signal, no specificity. Candidate for removal.

The test I actually cared about: my own four engineered features, measured across all 77 classes rather than read off the heatmap. word_count peaks at 6.7% ownership and sits in the top-20 of 30 classes. has_question_mark: 6.1%, 25 classes. uppercase_ratio: 10.1%, 66 classes. Three domain anchors — broad, shallow, worth keeping, not worth expecting miracles from. Thirty seconds per feature to grade my own feature engineering. The fourth one broke the pattern.

05 — The model reads punctuation

exclamation_count concentrates 53.4% of its entire SHAP mass on one class, lost_or_stolen_card, and cracks a top-20 in exactly one of 77 classes. The model reads punctuation. People who just lost their card press the keys harder — “My card was stolen!” — and the model learned the pressure, not only the words.

A structural fingerprint I never designed on purpose. Which raises the question section 04 kept deferring: what happens to a class that hangs on a single signal — word or punctuation — on the day people stop producing it? Part 2 puts a number on that (single-feature dependency runs from 7.6% to 79.3% across the 77 classes), explains why the exchange-rate pair keeps trading errors, and runs a shortcut audit the model happens to pass.

---

Dataset: BANKING77 (Casanueva et al., 2020), CC BY 4.0. Model, code, and all figures are mine. Next: part 2 — stress-test the classes.