Methodology

NoiseGate uses transparent, fixed statistical methods to evaluate crowd responses. Nothing is adjusted based on your results. Every threshold is the same for every post, every user, every time.

← Plain-language explainer

On this page

→ Signal clarity — Ratings → Signal clarity — Polls → Signal clarity — Yes / No → Confidence tiers → Attention checks → Filtered vs. all → Confidence intervals

Signal clarity — Ratings

For 1–5 rating questions, clarity is determined by the standard deviation of all responses. A low SD means respondents largely agreed. A high SD means the crowd was split.

σ = √( Σ(xᵢ - x̄)² / n )

where:
  xᵢ  = each individual rating
  x̄   = mean of all ratings
  n   = number of responses

✦ Clear signalσ < 1.0

~ Mixed1.0 ≤ σ < 1.5

≠ Contestedσ ≥ 1.5

Clarity is only shown at 5+ responses. Below that threshold, standard deviation is unreliable as a population estimate.

Signal clarity — Polls

For multiple-choice questions, clarity is measured by how strongly one option dominates relative to what random chance would predict given the number of options.

dominance = top_option_share / (1 / num_options)

where:
  top_option_share  = votes for leading option / total votes
  num_options       = number of available choices

Example: 3 options, top option gets 60% of votes
  dominance = 0.60 / (1/3) = 1.80  →  Mixed

✦ Clear signaldominance ≥ 2.5

~ Mixed1.5 ≤ dominance < 2.5

≠ Contesteddominance < 1.5

··· Scatteredspread ≤ 1 vote across ≥ 3 options

A dominance of 2.5 on a 4-option question means the top choice received 62.5% of votes — 2.5× what chance alone predicts.

Signal clarity — Yes / No

Binary questions use tighter thresholds than polls. With only two options, any deviation from 50/50 is more statistically meaningful — a 60/40 split on a yes/no question is far more significant than a 60/40 split across four options.

top_share = max(yes_count, no_count) / total_responses

✦ Clear signaltop_share ≥ 0.70

~ Mixed0.55 ≤ top_share < 0.70

≠ Contestedtop_share < 0.55

For comparison: a dominance ratio of 2.5 on a 2-option question requires 83% of the vote. We use 70% as the Clear signal threshold to reflect realistic decision-making rather than statistical extremes.

Confidence tiers

Confidence tiers answer a different question than clarity. Clarity tells you what the crowd said. Confidence tells you how much to trust it given the sample size.

The tiers are minimum sample size guidelines derived from two statistical conventions:

1.Statistical power: to detect a medium effect (Cohen's d = 0.5) at 80% power with α = 0.05 requires approximately 25–30 observations for means.

2.Margin of error: for proportions, the 95% confidence interval narrows as n increases. At n = 50, MoE is ±14%. At n = 20, MoE is ±22%.

Margin of error for proportions (95% CI):
  MoE = 1.96 × √( p(1-p) / n )

At p = 0.5 (worst case, widest interval):
  n = 5   →  MoE = ±44%   (trend only)
  n = 15  →  MoE = ±25%   (emerging confidence)
  n = 20  →  MoE = ±22%   (confident)
  n = 30  →  MoE = ±18%   (strong for ratings)
  n = 50  →  MoE = ±14%   (strong for polls)

Tier

Ratings / Rank

Polls / Yes–No

Strong

n ≥ 30n ≥ 50

Confident

n ≥ 15n ≥ 20

Trend

n ≥ 5n ≥ 5

These are practical guidelines, not hard statistical cutoffs. A result at n = 18 is not suddenly unreliable — it sits between Trend and Confident and should be interpreted accordingly.

Attention checks

Before responding to a post, users complete a verification step — a question with a known correct answer embedded naturally in the flow. Responses that fail are flagged silently. The respondent is never informed.

Checks are matched to question type: pick-one checks for pick-one posts, rating checks for rating posts, and so on. This ensures the verification feels native rather than disruptive.

NoiseGate also runs a speed trap: responses submitted in under 4 seconds from when the question rendered are flagged as likely inattentive. The 4-second floor reflects the minimum time required to read a typical question and make a deliberate choice — based on average reading speed (~200 wpm) and basic motor response time.

✦

Attention check fail: Incorrect answer on the verification question

✦

Speed trap: Response submitted in under 4 seconds

✦

Both flags: Stored separately — visible in CSV export

Verification is session-based — once passed, it is not repeated within the same browser session. Flagged responses are never deleted. Owners can always view them via the All toggle.

Filtered vs. all results

Post owners see two views of their results. The toggle works the same way it does on your results page:

✦ Filtered

All

← as it appears on your results page

✦ FilteredExcludes responses flagged by attention checks or the speed trap. All aggregations — clarity, confidence, means, percentages — recompute on this subset.

AllIncludes every response. Useful for seeing the full picture or auditing what was filtered and why.

Nothing is ever deleted. The filtered count is always shown in the results header so owners know exactly how much was excluded.

Confidence intervals (coming soon)

Confidence intervals quantify the range within which the true population value likely falls, given your sample size. NoiseGate calculates these for all quantitative question types — full display is coming in a future release.

Proportions (polls, yes/no):
  CI = p ± 1.96 × √( p(1-p) / n )

Means (ratings):
  CI = x̄ ± t* × ( σ / √n )

where:
  p   = observed proportion
  n   = number of responses
  x̄   = sample mean
  σ   = standard deviation
  t*  = critical t-value for desired confidence level
        (≈ 1.96 at 95% confidence, large n)

At 95% confidence: if you ran the same question with a different random sample of the same size, the true population value would fall within this interval 95% of the time.

Current confidence tiers are derived from the same margin-of-error formula shown above. Full CI display per result is on the roadmap.

Want the plain-language version?

No equations — just what everything means and when to use it.

How it works →