The Report That's Allowed to Say No

Part 5 of the "Behind the Map" series — rule-based recommendations with a built-in counter-calculation, and a threshold that's allowed to stay quiet

Abstract

Many analytics tools always deliver an answer — even when the underlying data is genuinely too thin for one. That creates false confidence. This article shows how the YelmeAtlas report does the opposite: below 30 reviews it deliberately withholds recommendations, and every recommendation it does make cites a number from the same report — including a counter-calculation that shows exactly what happens if you act on the advice.

1. Diagnosis: false confidence from always having an answer

An analytics tool that derives a "top recommendation" from twelve reviews suggests a reliability the data simply doesn't support. Twelve reviews can tip in any direction by chance — a recommendation built on that is noise with a headline. The real risk isn't the wrong recommendation itself; it's the confidence a confidently-worded recommendation creates, regardless of whether the numbers behind it can carry it.

A second, related problem: recommendations that cite no number ("users are unhappy with performance") can be neither verified nor prioritised. How unhappy? Compared to what? What changes if you fix it?

2. Existing approaches and their limits

• Always deliver a recommendation, regardless of sample size. Produces exactly the false confidence from section 1.
• Recommendations with no number attached ("could be better"). Not verifiable, not prioritisable.
• Let a language model phrase it. Often sounds more convincing than the underlying data supports — exactly the risk discussed at length in part 1 of this series.

3. The YelmeAtlas approach

Two principles carry the entire report: first, a fixed minimum number of reviews, below which the report describes but doesn't recommend. Second: every recommendation is pure arithmetic over numbers already shown in the same report — verifiable with a calculator, never taken on trust.

4. The algorithm in detail

flowchart TD
    A["Reviews in the window"] --> B{"≥ 30 reviews?"}
    B -->|no| C["Description only,<br/>no recommendation, no alerts —<br/>the report says so openly"]
    B -->|yes| D["Determine the dominant topic"]
    D --> E{"≥ 25% of complaints<br/>AND ≥ max(3, 10% of all complaints)?"}
    E -->|no| F["No dominant topic —<br/>picture judged not diffuse enough<br/>for a single recommendation"]
    E -->|yes| G["Recommendation with counter-calculation:<br/>rate would drop from X% to Y%"]

The honesty floor. Below 30 reviews in the observed window, the report deliberately withholds every recommendation and every alert — it only describes what's there, and says so openly: "With {total} reviews in the window the data is too thin for reliable recommendations. Widen the window or wait for the next nightly refresh — the report would rather not advise than advise from too little." The number 30 is a fixed, deliberately chosen line — not a statistical confidence interval, but a simple, auditable one.

The dominant topic. A topic only counts as "dominant" enough to earn its own recommendation if it satisfies two conditions at once: at least 25 % of all complaints in the window and at least max(3, 10% of all complaints) in absolute terms. The same double condition as the unusual days in part 4 — a relative share alone isn't enough if the absolute count is still tiny.

The counter-calculation. This is the actual difference from a generic recommendation. Verbatim, as the report phrases it:

"{share} of the complaints hang on this single topic. Arithmetically: if it is fully fixed, the complaint rate drops from {rate} to {rate_after} — no other investment in the window moves the number more."

The formula behind it:

rate_after = (total_complaints − dominant_topic_complaints) ÷ total_reviews

No estimate, no model prediction — a recalculation of the exact same formula, with one fewer number in the numerator. Any reader can check it themselves using the numbers already in the same report.

More rules, all following the same pattern:

Every one of these five rules is pure arithmetic over numbers already present in the same report — none of them introduces a new, unsupported claim.

5. A measured example

A window with 200 reviews and 60 complaints (rate: 30 %). One topic carries 20 of those 60 complaints — that's 33 % (above the 25 % gate) and absolutely more than max(3, 6) = 6 (above the absolute gate). The counter-calculation: (60 − 20) ÷ 200 = 20 %. The recommendation correctly reads: "fully fixing this topic drops the rate from 30 % to 20 %" — a number anyone can re-derive from the two starting figures themselves.

6. Limits of the method

• The counter-calculation is an arithmetic ceiling, not a promise: it shows what happens if a topic fully disappears — in practice, a single fix rarely eliminates every complaint tagged to a topic.
• The 30-review threshold is deliberately fixed and not adjusted per app — a very small but very active niche app gets the same floor as a large one.
• Five fixed rules in a fixed order aren't a ranking in the strict sense — with more recommendation types, a real prioritisation would become necessary, but at five, a fixed, auditable order is more honest than a score that would look more rigorous than it is.

7. Conclusion

The report is never smarter than its own numbers — and that's the point. Every recommendation traces back to a formula the same report already shows; the honesty floor would rather withhold a statement entirely than invent one from twelve reviews. That's not a feature meant to impress — it's the precondition for a report being quotable in a meeting at all, without anyone having to ask "where did that number come from?"

The last part of this first round shows how two apps get evaluated side by side — without needing a second account for the one you don't own. → Part 6: Two Apps, One Look (coming next)

← Part 4: The Complaint Rate and the Days That Stand Out