Topics That Have to Earn Their Name

Part 2 of the "Behind the Map" series — how thousands of free-text reviews become named, defensible topics

Abstract

Automatically grouping free text into topics sounds like a solved problem — TF-IDF and k-means have been around for decades. Applied naively, though, the method lies in three distinct ways: it turns the app's own name into a "topic," it collapses into one giant cluster on small samples, and it can be pushed out of balance by over-aggressive filtering. This article shows the three additional rules that decide whether a topic is allowed to keep its proposed name — backed by the measured before/after comparisons that led to each rule.

1. Diagnosis: where naive clustering lies

Text clustering means: reviews become vectors, similar vectors land in the same bucket, each bucket gets named after its most frequent words. That's the theory. In practice, exactly this naive approach produced three concrete, reproducible failures on real app reviews.

Failure 1 — the app name becomes the topic. If naming runs purely on frequency, the app's own name almost always wins: nearly every Instagram review mentions the word "Instagram." Real test runs produced topics named "instagram," "whatsapp," and "netflix" this way — technically well-clustered, substantively worthless. A topic that touches every review equally distinguishes nothing from anything else.

Failure 2 — the fast variant lies on small samples. There are two common variants of the k-means algorithm: the full version, which looks at every point on every round, and a "mini-batch" variant that only samples random subsets — built for datasets too large to fit in memory. A review window, though, is a few hundred short documents, not millions. Measured on the same 100 real Instagram reviews:

The "faster" variant wasn't less accurate at this scale — it was practically useless: one giant cluster plus noise.

Failure 3 — over-aggressive filtering collapses the separation. Words like "fast," "slow," "new," or "old" are poor topic names (they don't say what the review is about) but good separating features (they distinguish a complaint about speed from one about advertising). When they were experimentally removed from the computation entirely, rather than just from the naming step, Instagram's largest cluster jumped from roughly half the window to 90 % — a split that separates almost nothing anymore.

2. Existing approaches and their limits

• Fixed keyword lists (a lexicon approach) are predictable and stable, but only recognise what was written into the list beforehand — a new problem stays invisible until someone extends it.
• Naive k-means without a check produces exactly the three failures above.
• A language model for topic naming would likely avoid all three failures — but not deterministically, not free per request, and with the risk of inventing a topic that isn't actually present in the text (see part 1, section 5).

YelmeAtlas actually runs both of the first two approaches side by side — the lexicon approach as a deterministic, model-free alternative — but defaults to k-means, backed by a validation step that needs no language model at all.

3. The YelmeAtlas approach

The principle: clustering produces candidates; three rules decide whether a candidate gets to keep its name. Failing the check doesn't delete the reviews — they land in an honestly labelled catch-all bucket, "Other," visible but set apart, never in a headline.

4. The algorithm in detail

flowchart TD
    A["Review free text"] --> B["TF-IDF vectorisation<br/>rare, recurring words<br/>weigh more than filler"]
    B --> C["Full k-means<br/>not the mini-batch variant"]
    C --> D["Dictionary-free noun detection<br/>capitalised words<br/>mid-sentence"]
    D --> E{"Three-gate validation per cluster"}
    E -->|"Support ≥ 40 %<br/>Lift ≥ 1.5×<br/>Outside-share ≤ 50 %"| F["Named topic"]
    E -->|"any gate fails"| G["Catch-all bucket 'Other'<br/>dimmed, never in a headline"]

Step 1 — TF-IDF. Every word gets a weight: high if it appears often in few reviews (typical of a real topic), low if it's equally common everywhere (typical of filler). That's the standard trick that keeps "the," "and," or "app" from dominating the computation.

Step 2 — clustering with full k-means. As measured in section 1: the mini-batch variant brings no advantage at this data size, but a significant quality loss. The full variant is fast enough on a few hundred short texts that the approximation's speed advantage simply isn't needed.

Step 3 — dictionary-free noun detection. For a topic name to be a noun ("advertising," not "fast"), you'd normally need to consult a dictionary or a language model. There's a simpler path that fits the German language in particular: German capitalises essentially only nouns, and does so mid-sentence. A word that isn't at the start of a sentence and is still capitalised is, with high reliability, a noun — no dictionary, no language model, no extra dependency. (For English review text, the same signal doesn't apply as cleanly, which is why the lexicon path stays the reliable option there — but the validation gates below work independently of the language.)

Step 4 — the three-gate validation. This is the actual answer to Failure 1 above. A cluster only keeps its proposed name if it passes all three conditions:

Only a term that clears all three gates gets to name a topic. What fails any gate lands in "Other" — visible, but honestly marked as leftover.

5. A measured example

Before the three-gate validation, the same pipeline produced topics like "instagram," "whatsapp," and "netflix" on real runs — each one simply the app's own name. After introducing validation, exactly those terms fail the lift gate: they're supported (nearly every review mentions the app), but not lifted — they're equally frequent everywhere and therefore distinguish nothing. The topics that survive are the ones that actually carry content: in the Spotify example from part 1, "advertising," "free-tier limits," and "notifications" — each one genuinely separates part of the reviews from the rest.

6. Limits of the method

• Purely word-based, not meaning-based. Two reviews describing the same problem in entirely different words can land in different clusters. The method finds patterns in word choice, not deep language understanding. <br/>*(Deliberately so — see part 1, section 5: the same predictability that limits it here is what makes the result auditable elsewhere.)*
• Deterministic for a fixed random seed, but not globally stable. Across different library versions, internal cluster IDs can swap order — which is why comparisons are always made against the grouping itself, never a raw internal number.
• Needs a minimum amount of text to separate meaningfully. On very few reviews in the window, the method honestly returns one collective cluster instead of inventing false granularity.

7. Conclusion

The real progress here isn't the clustering algorithm itself — TF-IDF and k-means are textbook material. It's the recognition that clustering finds groups, but says nothing about whether the most obvious name for a group is actually a good one. The three-gate validation separates exactly these two questions: how are reviews grouped? And does this group deserve its own name at all? Without that second question, every Instagram analysis would have ended with "top topic: Instagram" — correctly grouped, but saying nothing.

The next part stays with the data itself: how reviews get into the system at all without a developer account — and why a nightly archiving run fights a problem the stores themselves cause: they forget faster than people log back in. → Part 3: Getting Data Without a Developer Account (coming next)

← Part 1: A Map for Thousands of Reviews