Reference: The Five Bands of "I'm Not Sure"

When you say “I’m not sure” in statistics, you might mean five different things. Each has its own formula, its own philosophical baggage, and its own thing it is and is not allowed to tell you. This reference disambiguates them: standard error, confidence interval, credible interval, bootstrap confidence interval, and prediction interval. The single most-repeated confusion in the corpus --- and in the field --- is treating these as interchangeable. They are not.

Roster

Name	What it answers	One-line definition	Typical width	When to use	Corpus article
Standard error (SE)	“How jumpy is my estimate?”	The standard deviation of the estimator’s sampling distribution. For the mean: $s/\sqrt{n}$ .	Narrowest; it’s a number, not an interval.	As a building block for CIs and t-tests; reporting precision of an estimate.	The Central Limit Theorem
Confidence interval (CI)	“What range would the true parameter plausibly fall in, if I repeated this experiment?”	$\hat\theta \pm z_{\alpha/2}\,\mathrm{SE}(\hat\theta)$ . A procedure-level guarantee, not a per-interval probability.	Narrow (depends on $n$ and SE).	Estimating a population parameter (mean, regression coefficient, AUC) from a sample.	Confidence Intervals: What “95% Confident” Actually Means)
Credible interval (Bayesian)	“Given my data and prior, what range contains the parameter with 95% probability?”	The central 95% of the posterior distribution. A genuine probability statement about the parameter.	Narrow; comparable to CI but shaped by the prior.	When you want a direct probability statement, when you have real prior knowledge, or when communicating to stakeholders who read “95%” as “probability.”	Think Like a Bayesian)
Bootstrap CI	“What range for the parameter, without assuming a normal distribution?”	Resample the data with replacement $B$ times, compute the statistic on each, take the 2.5th and 97.5th percentiles.	Slightly wider than the parametric CI in finite samples; converges to it as $n\to\infty$ .	When you can’t or won’t assume normality; for complex statistics (medians, ratios, correlations) with no closed-form SE.	The Bootstrap
Prediction interval	“Where will one new observation land?”	$\hat\mu \pm t_{\alpha/2,\,n-1}\,s\sqrt{1 + 1/n}$ . Wider than the CI because it folds in the noise of a single new draw.	Much wider than the CI.	Forecasting a single future measurement, not the population mean.	Is Your Model Actually Better?)

The decision tree: which band for which question

Start from the question you are actually being asked, not from the tool you happen to know.

“How precise is my single number?”
- You want the standard error. It’s a standard deviation of an estimator, not a range. Multiply by ~2 to eyeball a 95% CI.
“Give me a range for the population parameter.”
- Is the parameter a mean or something with a known (or plausibly normal) sampling distribution?
  - Yes → confidence interval (parametric, $z$ or $t$ ).
  - No, or you don’t want to assume → bootstrap CI.
- Are you forecasting a single new observation, not the parameter?
  - Yes → prediction interval. This is wider. Stop here.
- Do you want to make a genuine probability statement about the parameter (“there’s a 95% chance the true mean is between 40 and 45”)?
  - Yes, and you have a defensible prior → credible interval.
  - No, or you want to stay frequentist → confidence interval. Be careful with wording: a 95% CI is not “95% likely to contain the true value.” It’s “this procedure produces intervals that contain the true value 95% of the time, across hypothetical repeats.”
“Is my model actually better than a baseline?”
- You want a CI around the difference in metrics. If it excludes zero, you have evidence of a real difference --- but check statistical power before celebrating.
“I have a tiny / weird dataset and no distributional assumption.”
- Bootstrap CI, but with caveats (see Edge cases). Bootstrap estimates variability, not bias.

Frequentist vs. Bayesian, restated side-by-side

	Frequentist	Bayesian
What is random?	The data. The parameter is fixed but unknown.	The parameter. The data is fixed once you’ve seen it.
What does “95%” mean?	95% of intervals produced by this procedure, across hypothetical repeated experiments, would contain the true parameter.	95% of the posterior mass lies in this interval.
Can you say “the probability that $\theta$ is in [a, b] is 95%”?	No. $\theta$ is fixed; it either is or isn’t.	Yes, with respect to your posterior.
Prior required?	No.	Yes. The result can shift with the prior.
Interval name	Confidence interval	Credible interval
Computation	Closed-form (for common cases) or bootstrap	MCMC, conjugate updates, or variational inference

The two answers usually land near each other numerically when the prior is weak and the sample is large, but they answer different questions. This is not a pedantic distinction: it changes what you are allowed to say in the methods section and what stakeholders will hear.

A worked example: all five bands on one dataset

Below is a single small dataset ( $n=30$ ) with all five bands computed. Notice the ordering of widths: SE (a number) < CI $\approx$ credible interval $\approx$ bootstrap CI < prediction interval.

(Captured with numpy==2.4.4, scipy==1.17.1. np.random.default_rng streams are stable across numpy versions for standard distributions, but the exact version is noted here to remove any ambiguity for a reader trying to reproduce these numbers.)

import numpy as np
from scipy import stats

rng = np.random.default_rng(42)
# 30 measurements; true mean ~42, true sd ~7
data = rng.normal(loc=42, scale=7, size=30)

n = len(data)
xbar = data.mean()
s = data.std(ddof=1)
print(f"n={n}, mean={xbar:.3f}, sd={s:.3f}")
# n=30, mean=42.118, sd=5.437

# 1. Standard error of the mean
se = s / np.sqrt(n)
print(f"SE = {se:.3f}")
# SE = 0.993

# 2. 95% confidence interval (t-based, honest for n=30)
t_crit = stats.t.ppf(0.975, df=n - 1)
ci = (xbar - t_crit * se, xbar + t_crit * se)
print(f"95% CI = [{ci[0]:.2f}, {ci[1]:.2f}]")
# 95% CI = [40.09, 44.15]

# 3. Bootstrap 95% CI (percentile method)
B = 10_000
boot_means = np.array([
    rng.choice(data, size=n, replace=True).mean()
    for _ in range(B)
])
boot_ci = np.percentile(boot_means, [2.5, 97.5])
print(f"95% Bootstrap CI = [{boot_ci[0]:.2f}, {boot_ci[1]:.2f}]")
# 95% Bootstrap CI = [40.16, 44.00]

# 4. 95% credible interval (flat prior, normal model)
#   With a flat prior and normal likelihood, the posterior mean
#   is the sample mean and the posterior SD is approximately se.
cred = stats.norm.ppf([0.025, 0.975], loc=xbar, scale=se)
print(f"95% Credible interval = [{cred[0]:.2f}, {cred[1]:.2f}]")
# 95% Credible interval = [40.17, 44.06]

# 5. 95% prediction interval (one new observation)
pi_sd = s * np.sqrt(1 + 1 / n)
pi = (xbar - t_crit * pi_sd, xbar + t_crit * pi_sd)
print(f"95% Prediction interval = [{pi[0]:.2f}, {pi[1]:.2f}]")
# 95% Prediction interval = [30.81, 53.42]

Lines 1–9. We draw 30 points from $\mathcal{N}(42, 7^2)$ . In a real analysis you’d never know the true parameters; this is just so we can sanity-check the intervals against the known mean of 42. The sample mean (42.12) and SD (5.44) are close but not equal to the truth --- a reminder that with $n=30$ , a single sample’s SD can land well off the true population SD by chance.

Lines 11–14 (SE). The standard error of the sample mean is $s/\sqrt{n}$ . With $s \approx 5.44$ and $n=30$ , we get $0.99$ . This is the SD of the estimator (the sample mean), not the SD of the data. That distinction is the entire reason the SE exists as a separate concept from the sample SD.

Lines 16–19 (CI). We use the $t$ -critical value, not $z=1.96$ , because $n=30$ is small enough that the normal approximation overstates our certainty by a hair. The $t$ value for 29 degrees of freedom at the 97.5th percentile is about 2.045. The CI is $\bar x \pm t \cdot \mathrm{SE}$ .

Lines 21–27 (bootstrap CI). We resample 10,000 times with replacement from the data itself (not from a distribution), compute the mean of each resample, and take the 2.5th and 97.5th percentiles of those 10,000 means. No distributional assumption is made --- the empirical distribution of the data is the model. The bootstrap CI is slightly narrower than the parametric CI here by chance; with $B=10{,}000$ the Monte Carlo noise is small enough that this is the real bootstrap answer, not jitter.

Lines 29–32 (credible interval). With a flat (improper) prior and a normal likelihood, the posterior for the mean is $\mathcal{N}(\bar x, \mathrm{SE}^2)$ when the variance is known. Taking the central 95% of that posterior gives an interval numerically very close to the $z$ -based CI. This is a special case: it’s the consequence of the flat prior, not a coincidence. With an informative prior, the credible interval would shrink and shift toward the prior mean.

Lines 34–37 (prediction interval). The extra factor $\sqrt{1 + 1/n}$ is the whole story. The CI for the mean shrinks as $1/\sqrt{n}$ ; the prediction interval for one new point cannot shrink below the data’s SD $s$ , because a new observation carries its own noise. That $\sqrt{1 + 1/n}$ is just $s$ (the noise of the new point) plus a small correction for the uncertainty in $\bar x$ . This is why the prediction interval [30.81, 53.42] is roughly 5.6 times the width of the CI [40.09, 44.15]: the CI is pinning down an average; the PI has to catch a single noisy draw.

Let $\hat\theta$ be an estimator of a parameter $\theta$ , $\mathrm{SE}(\hat\theta)$ its standard error, $n$ the sample size, $s$ the sample SD, and $t_{\alpha/2,\,\nu}$ the upper $\alpha/2$ quantile of a $t$ distribution with $\nu$ degrees of freedom.

Standard error of the mean: $\mathrm{SE}(\bar X) = \frac{s}{\sqrt{n}}$

Confidence interval (t-based, for the mean): $\mathrm{CI}_{1-\alpha} = \left[\bar X - t_{\alpha/2,\,n-1}\,\frac{s}{\sqrt{n}},\ \ \bar X + t_{\alpha/2,\,n-1}\,\frac{s}{\sqrt{n}}\right]$

Credible interval (flat prior, known variance): $\mathrm{CrI}_{1-\alpha} = \left[\bar X - z_{\alpha/2}\,\frac{s}{\sqrt{n}},\ \ \bar X + z_{\alpha/2}\,\frac{s}{\sqrt{n}}\right]$ (The posterior is $\mathcal{N}(\bar X, s^2/n)$ .)

Bootstrap CI (percentile method): $\mathrm{CI}^{\text{boot}}_{1-\alpha} = \left[F^*_{\alpha/2},\ F^*_{1-\alpha/2}\right]$ where $F^*$ is the empirical CDF of the $B$ resampled means.

Prediction interval (for one new observation): $\mathrm{PI}_{1-\alpha} = \left[\bar X - t_{\alpha/2,\,n-1}\,s\sqrt{1+\tfrac{1}{n}},\ \ \bar X + t_{\alpha/2,\,n-1}\,s\sqrt{1+\tfrac{1}{n}}\right]$

Plain English	Statistical symbol	Python equivalent
Sample mean	$\bar X$	`data.mean()`
Sample SD (unbiased)	$s$	`data.std(ddof=1)`
Standard error of the mean	$\mathrm{SE}(\bar X) = s/\sqrt{n}$	`data.std(ddof=1) / np.sqrt(len(data))`
$t$ critical value, 29 df, 95%	$t_{0.025,\,29}$	`stats.t.ppf(0.975, df=29)`
Prediction SD (new point)	$s\sqrt{1 + 1/n}$	`data.std(ddof=1) * np.sqrt(1 + 1/len(data))`

What each band does not answer

This is the confusion the corpus keeps restating, so it’s worth a dedicated section.

Standard error

Does not give you a range for the parameter. It’s a number with units of the parameter. To get a range, you have to multiply by a critical value and add/subtract it from the estimate.

Confidence interval

Does not say “there’s a 95% probability the true mean is in [40.09, 44.15].” Once you’ve computed it, the true mean either is or isn’t in that specific interval; there’s no probability left. What’s 95% is the procedure: if you repeated the experiment infinitely many times and built a CI each time, 95% of those intervals would contain the true mean. This is a subtle but legally different claim. People routinely say the stronger thing anyway; it’s the most common statistical misstatement in science reporting.

Credible interval

Does say “there’s a 95% probability the true mean is in this range” --- but only conditional on the prior. Change the prior, change the interval. This is why Bayesians and frequentists argue: they agree on the numbers more often than they agree on the wording.

Bootstrap CI

Does not eliminate bias. If your estimator is biased (say, it systematically overestimates), the bootstrap will faithfully reproduce that bias in every resample. The bootstrap estimates the variability of your estimator, not its accuracy. It also does not magic away small-sample problems: with $n < 10$ , the empirical distribution is too coarse and the bootstrap CI can be misleadingly narrow or oddly shaped.

Prediction interval

Does not tell you about the population mean. It answers “where does one more draw land?” If you average the next 100 draws, the relevant uncertainty shrinks back toward the CI. Conflating the two is the second-most-common error: people report a CI when they should report a PI, and the CI is far too narrow to cover a new observation.

Edge cases and common mistakes

Interpreting a CI as a probability statement. “There is a 95% probability that the true mean lies in this interval” is a Bayesian statement. A frequentist can only say “95% of CIs constructed this way would contain the true mean.” The distinction matters in writing and in adversarial settings (regulatory, legal).
Using a CI when you need a prediction interval. If you’re forecasting one new patient’s blood pressure, the prediction interval is roughly $\sqrt{1 + 1/n} \approx 1$ times as wide as the data SD, not $\sqrt{n}$ times smaller. A 95% CI of $[40, 46]$ around a mean does not mean 95% of patients fall in $[40, 46]$ . Use the PI.
Bootstrap on tiny samples. With $n = 5$ , the bootstrap resamples from a 5-point empirical distribution. The resampled means can only take a finite, coarse set of values, and the percentile CI is unreliable. Rule of thumb: don’t trust the percentile bootstrap below $n \approx 20$ , and consider the BCa (bias-corrected and accelerated) method or a Studentized bootstrap for skewed statistics.
Too few bootstrap resamples. $B = 100$ gives a noisy estimate of the 2.5th percentile. Use $B \ge 2000$ for a 95% CI, and $B \ge 5000$ if you’re reporting 99% intervals or doing BCa. The bootstrap CI should be stable if you rerun it with a different seed.
Reporting the SE as the uncertainty. “Mean = 42.1 ± 1.0” is a mean $\pm$ SE, which is approximately a 68% CI under normality. Readers may misread ”± 1.0” as a 95% range or as the data SD. Always label which band you’re reporting.
Assuming the credible interval equals the CI. It does only when the prior is flat (or negligibly informative relative to the data). With an informative prior, the credible interval can be narrower and shifted. Report the prior.
Using the normal-approximation CI on heavily skewed data without transformation. If the data are log-normal or heavily right-skewed, the normal-approximation CI for the mean can have poor coverage. Transform (e.g., log), compute the CI on the transformed scale, and back-transform, or use the bootstrap.
Confusing the SE of the mean with the SD of the data. The SD of the data describes spread among observations. The SE describes spread among sample means from repeated experiments. They differ by a factor of $\sqrt{n}$ ; at $n=100$ , the SE is one-tenth the SD. This is the single most common confusion among newcomers.
Power and the CI. A CI that’s wide enough to include both “no effect” and “clinically meaningful effect” is telling you the study was underpowered. Don’t interpret “the CI includes zero” as “no effect”; interpret it as “we don’t know.”
Bootstrap with dependent data. The vanilla bootstrap assumes observations are i.i.d. For time series or clustered data, use a block bootstrap; otherwise the bootstrap will underestimate uncertainty because it breaks the dependence structure.

Reference: The Five Bands of "I'm Not Sure"

Roster

The decision tree: which band for which question

Frequentist vs. Bayesian, restated side-by-side

A worked example: all five bands on one dataset

What each band does not answer

Standard error

Confidence interval

Credible interval

Bootstrap CI

Prediction interval

Edge cases and common mistakes

Cross-references

Further reading

The Bootstrap: How to Measure Uncertainty Without Assuming Normality

Confidence Intervals: What '95% Confident' Actually Means (Without the Math Headaches)

What Does a P-Value Actually Mean? (And Why Practitioners Keep Misusing It)

Reference: Hypothesis Testing

Looking for something else?

Reference: The Five Bands of "I'm Not Sure"

Roster

The decision tree: which band for which question

Frequentist vs. Bayesian, restated side-by-side

A worked example: all five bands on one dataset

What each band does not answer

Standard error

Confidence interval

Credible interval

Bootstrap CI

Prediction interval

Edge cases and common mistakes

Cross-references

Further reading

Related articles

The Bootstrap: How to Measure Uncertainty Without Assuming Normality

Confidence Intervals: What '95% Confident' Actually Means (Without the Math Headaches)

What Does a P-Value Actually Mean? (And Why Practitioners Keep Misusing It)

Reference: Hypothesis Testing

Looking for something else?