Methodology

NoiseGate uses transparent, fixed statistical methods to evaluate crowd responses. Nothing is adjusted based on your results. Every threshold is the same for every post, every user, every time.

← Plain-language explainer

On this page

→ Signal clarity — Ratings → Signal clarity — Polls → Signal clarity — Yes / No → Confidence tiers → Attention checks → Filtered vs. all → Confidence intervals

Signal clarity — Ratings

For 1–5 rating questions, clarity is determined by the standard deviation of all responses. A low SD means respondents largely agreed. A high SD means the crowd was split.

σ = √( Σ(xᵢ - x̄)² / n )

where:
  xᵢ  = each individual rating
  x̄   = mean of all ratings
  n   = number of responses

✦ Clear signalσ < 1.0

~ Mixed1.0 ≤ σ < 1.5

≠ Contestedσ ≥ 1.5

Clarity is only shown at 5+ responses. Below that threshold, standard deviation is unreliable as a population estimate.

Signal clarity — Polls

For multiple-choice questions, clarity is measured by how strongly one option dominates relative to what random chance would predict given the number of options.

dominance = top_option_share / (1 / num_options)

where:
  top_option_share  = votes for leading option / total votes
  num_options       = number of available choices

Example: 3 options, top option gets 60% of votes
  dominance = 0.60 / (1/3) = 1.80  →  Mixed

✦ Clear signaldominance ≥ 2.5

~ Mixed1.5 ≤ dominance < 2.5

≠ Contesteddominance < 1.5

··· Scatteredspread ≤ 1 vote across ≥ 3 options

A dominance of 2.5 on a 4-option question means the top choice received 62.5% of votes — 2.5× what chance alone predicts.

Signal clarity — Yes / No

Binary questions use tighter thresholds than polls. With only two options, any deviation from 50/50 is more statistically meaningful — a 60/40 split on a yes/no question is far more significant than a 60/40 split across four options.

top_share = max(yes_count, no_count) / total_responses

✦ Clear signaltop_share ≥ 0.70

~ Mixed0.55 ≤ top_share < 0.70

≠ Contestedtop_share < 0.55

For comparison: a dominance ratio of 2.5 on a 2-option question requires 83% of the vote. We use 70% as the Clear signal threshold to reflect realistic decision-making rather than statistical extremes.

Confidence tiers

Confidence tiers answer a different question than clarity. Clarity tells you what the crowd said. Confidence tells you how much to trust it given the sample size.

The tiers are minimum sample size guidelines derived from two statistical conventions:

1.Statistical power: to detect a medium effect (Cohen's d = 0.5) at 80% power with α = 0.05 requires approximately 25–30 observations for means.

2.Margin of error: for proportions, the 95% confidence interval narrows as n increases. At n = 50, MoE is ±14%. At n = 20, MoE is ±22%.

Margin of error for proportions (95% CI):
  MoE = 1.96 × √( p(1-p) / n )

At p = 0.5 (worst case, widest interval):
  n = 5   →  MoE = ±44%   (trend only)
  n = 15  →  MoE = ±25%   (emerging confidence)
  n = 20  →  MoE = ±22%   (confident)
  n = 30  →  MoE = ±18%   (strong for ratings)
  n = 50  →  MoE = ±14%   (strong for polls)

Tier

Ratings / Rank

Polls / Yes–No

Strong

n ≥ 30n ≥ 50

Confident

n ≥ 15n ≥ 20

Trend

n ≥ 5n ≥ 5

These are practical guidelines, not hard statistical cutoffs. A result at n = 18 is not suddenly unreliable — it sits between Trend and Confident and should be interpreted accordingly.

Attention checks

NoiseGate periodically inserts a short verification question — a question with a known correct answer — into the response flow. The cadence is randomized per respondent (roughly every 2–5 responses, re-rolled each time) rather than fixed or shown to everyone every time, so it can't be timed or anticipated. The check's format always matches the surrounding post — a Yes/No post gets a Yes/No-style check, an Open Response post gets a short-answer check, and so on — so it reads as part of the flow rather than an interruption.

Correct answers are never sent to the browser. Grading happens entirely on the server, comparing your submitted answer against a value we store independently — inspecting the page's code reveals nothing about what the correct answer is.

NoiseGate also runs a speed trap, using a reading-speed floor from published survey methodology research rather than an arbitrary number:

threshold = max(1500ms, reading_time + interaction_overhead)

reading_time = (words in prompt + words in options) × 300ms

interaction_overhead by question type:
  Yes/No       +500ms   (one click)
  Poll         +500ms   (one click)
  Rating       +800ms   (move a slider)
  Rank Order   +1500ms  (multiple drag actions)
  Open Response +2000ms (typing a real answer)

The 300ms-per-word reading floor is drawn from published research on detecting inattentive ("speeding") survey responses: Zhang, C. & Conrad, F.G. (2014), "Speeding in Web Surveys," Survey Research Methods, 8(2), 127–135. The interaction overhead is added on top since the research threshold covers reading time specifically, not the full click/drag/type action.

Rank Order responses have one additional check: if the submitted order exactly matches the original, untouched order, the response is flagged regardless of timing — this catches a response that was never actually interacted with, independent of how long it sat on screen.

✦

Attention check fail: Incorrect answer on the verification question

✦

Speed trap: Response submitted faster than the reading-time floor for that question

✦

Unmoved ranking: Rank Order submitted without changing the original order (rank_order posts only)

✦

Multiple flags: Combined in a single comma-separated field — visible in CSV export

A flagged response still shows a brief, generic note to the respondent — that it may not count toward their signal score — without naming which check caught it. Revealing the specific mechanism would make it easier to learn how to evade next time. Flagged responses are never deleted; owners can always view them via the All toggle.

Filtered vs. all results

Post owners see two views of their results. The toggle works the same way it does on your results page:

✦ Filtered

All

← as it appears on your results page

✦ FilteredExcludes responses flagged by attention checks or the speed trap. All aggregations — clarity, confidence, means, percentages — recompute on this subset.

AllIncludes every response. Useful for seeing the full picture or auditing what was filtered and why.

Nothing is ever deleted. The filtered count is always shown in the results header so owners know exactly how much was excluded.

Confidence intervals (coming soon)

Confidence intervals quantify the range within which the true population value likely falls, given your sample size. NoiseGate calculates these for all quantitative question types — full display is coming in a future release.

Proportions (polls, yes/no):
  CI = p ± 1.96 × √( p(1-p) / n )

Means (ratings):
  CI = x̄ ± t* × ( σ / √n )

where:
  p   = observed proportion
  n   = number of responses
  x̄   = sample mean
  σ   = standard deviation
  t*  = critical t-value for desired confidence level
        (≈ 1.96 at 95% confidence, large n)

At 95% confidence: if you ran the same question with a different random sample of the same size, the true population value would fall within this interval 95% of the time.

Current confidence tiers are derived from the same margin-of-error formula shown above. Full CI display per result is on the roadmap.

Want the plain-language version?

No equations — just what everything means and when to use it.

How it works →