The evidence gap in product decisions

Analyst primer

The sections below walk through how we close the gap in practice: decision framing, disconfirming tests, and provenance that survives board and investor scrutiny — not another internal deck.

The problem with owned data

Most product teams sit on rich behavioral data — funnels, cohorts, feature usage, support tickets. That data is invaluable for describing what happened. It is structurally weak at explaining why a segment churned, why a price point failed, or why a new category entrant is winning share you never measured.

Owned data is biased toward people who already chose you. It under-represents rejections, near-misses, and segments you have never served. Strategic decisions — packaging, positioning, pricing, category expansion — require evidence from outside that lens.

In Indian FMCG we see this constantly: modern-trade dashboards glow while kirana repeat stalls — not because data is missing, but because the population that defines the next bet (rejecters, switchers, absent occasions) never appears in owned records. The evidence gap is geographic and channel-specific as often as it is demographic.

Before your next brief, list three decisions that would change if a competitor’s trialist explained their last three shop trips. If that list is non-empty, you are already past what CRM can safely support.

Decision framing before fieldwork

Before any interview or survey runs, the research question must be tied to a decision. What will leadership do differently if the finding is X versus Y? Without that framing, fieldwork produces interesting narratives that do not change bets.

We map each engagement to a decision tree: the primary choice, the material risks, and the disconfirming evidence that would change the recommendation. That tree drives sampling, instrument design, and the triangulation plan.

“Interesting research is cheap. Decision-grade research is expensive because it is designed to survive scrutiny.”

Decision framing is where research ROI lives or dies. A well-framed tree forces explicit disconfirming tests — ‘if kirana findability is not the issue, we will see X in shop-alongs’ — so the team cannot retreat to opinion after field.

We refuse engagements where the primary decision is ‘learn about consumers’ without a verb for leadership. Learning is unlimited; decisions are scarce. Frame the verb first, then pick instruments.

When to commission primary research

Primary research is warranted when the cost of being wrong exceeds the cost of learning — re-pricing, repositioning, entering a new segment, or defending a board narrative under diligence.

The bar is not "we lack data." The bar is "we lack valid evidence for this specific decision." Re-analysis of CRM exports rarely clears that bar for strategic moves.

Commissioning primary research is not an admission that analytics failed — it is recognition that strategic moves require valid evidence about people who are not in your database. The cost of a wrong repositioning or national SKU in India still dwarfs a disciplined four-week field plan.

When boards ask for ‘more data’, ask back: ‘data for which claim, at what confidence, with what provenance?’ That question alone surfaces whether you need field or another dashboard refresh.

Closing the gap

Closing the evidence gap means pairing behavioral signal with designed fieldwork: adaptive interviews with churned users, conjoint with category considerers, ethnography where self-report fails.

The output is not more slides — it is a recommendation with provenance, graded confidence, and an explicit link between evidence and the decision it supports.

Closing the gap is iterative. Most clients keep behavioural signal running while primary waves answer the question of the quarter — churn root cause, pack architecture, purity legibility. The two streams meet in triangulation, not in one omnibus study.

If you take one action from this note, rewrite your next brief opening as: ‘We will decide X by date Y; evidence must survive challenge Z.’ Everything else — sample, language, mode — follows.