Fixed-null calibration

Define what is random before scoring a match

A numerical coincidence acquires a tail probability only after specifying a null distribution and a complete scoring procedure. For a statistic $T$ whose larger values mean a closer match, the null tail is

$$p(t)=\Pr_{H_0}(T\geq t).$$

Under a continuous null distribution, the probability integral transform makes $p(T)$ uniform on $[0,1]$. Discrete tails defined with inclusive ties are conservative: $\Pr(p(T)\leq\alpha)\leq\alpha$.

As a simple conditional example, suppose the offset $R$ from a fixed lattice target is uniform on $[-\Delta/2,\Delta/2]$. A match at least as close as the observed offset has

$$p=\frac{2|R|}{\Delta}.$$

For a multiplicative ladder with ratio $r>1$, a corresponding log-uniform cell has $R=\log(x/x_*)$ and $\Delta=\log r$. This is an exact statement about that chosen cell distribution. A nonuniform ladder, measurement-error distribution, fitted anchor, or search across several targets must be included in its actual null model.

Combining independent continuous tails

Let $U_1,\ldots,U_k$ be independent uniform null probabilities. Then $Y_i=-\log U_i$ obeys $\Pr(Y_i>y)=e^{-y}$ for $y\geq0$. Convolution, or multiplication of the Laplace transforms $\mathbb E[e^{-sY_i}]=(1+s)^{-1}$, gives

$$X=\sum_iY_i\sim\operatorname{Gamma}(k,1),\qquad f_X(x)=\frac{x^{k-1}e^{-x}}{(k-1)!}.$$

For observed $x=-\sum_i\log p_i$, repeated integration by parts yields

$$\boxed{p_{\rm combined}=\Pr(X\geq x) =e^{-x}\sum_{j=0}^{k-1}\frac{x^j}{j!}.}$$

Equivalently $2X$ has a chi-square distribution with $2k$ degrees of freedom. Independence is essential. Several ratios using one measured anchor, or several quantities derived from one fit, generally have a joint null that must be retained.

Probability floors have atoms

Suppose a scoring rule caps the evidential contribution of each match by using $\widehat U_i=\max(U_i,f_i)$ with a fixed $0<f_i<1$. Write $a_i=-\log f_i$. Then

$$\widehat Y_i=-\log\widehat U_i=\min(Y_i,a_i).$$

Its probability measure consists of density $e^{-y}$ on $0\leq y<a_i$ and an atom of mass $f_i=e^{-a_i}$ at $a_i$. Its Laplace transform is

$$\mathbb E[e^{-s\widehat Y_i}] =\frac{1+s e^{-(1+s)a_i}}{1+s},$$

with a removable value at $s=-1$. In particular $\mathbb E[\widehat Y_i]=1-f_i$. The exact distribution of the sum is the convolution of these mixed measures; retaining the atoms matters at an inclusive tail threshold.

For two contributions, let

$$Q_a(t)=\Pr(\min(Y,a)\geq t)= \begin{cases} 1&t\leq0,\\e^{-t}&0<t\leq a,\\0&t>a. \end{cases}$$

Then the exact inclusive tail is

$$\Pr(\widehat Y_1+\widehat Y_2\geq x) =\int_0^{a_1}e^{-y}Q_{a_2}(x-y)\,dy +e^{-a_1}Q_{a_2}(x-a_1).$$

At $x=a_1+a_2$ this equals $f_1f_2$, the mass of the joint atom. For any number of terms, $\widehat X\leq X$ for the same underlying uniform variables, so the ordinary gamma tail evaluated at an observed $\widehat x$ is a conservative upper bound on its capped null tail. A floor chosen after inspecting the result must itself be part of the selection procedure to retain such calibration.

Repeat the whole procedure in simulation

If a numerical analysis searches over depths, index assignments, formulas, anchors, tolerances, or subsets, apply that same search to each simulated null dataset. For example, if the reported statistic is the best score $T=\max_{m\in\mathcal M}T_m$, the null simulation must recompute that maximum. The null model should reproduce shared errors and fitted parameters with the same fitting procedure.

This defines the object being calibrated. A selected formula cannot be treated as independently fixed merely because it can later be written without adjustable parameters. A fresh independent dataset can instead test a formula fixed before those data are examined.

Two finite-simulation quantities

For a fixed observed threshold, let $K$ of $N$ independent null simulations meet or exceed it. Conditional on that threshold, $K\sim\operatorname{Binomial}(N,p)$. If $K=0$, solving $\Pr_p(K=0)=(1-p)^N=\alpha$ gives the one-sided confidence upper limit

$$\boxed{p_{\rm upper}=1-\alpha^{1/N} \simeq\frac{-\log\alpha}{N}.}$$

At 95% confidence the leading numerator is about $2.996$. Zero observed exceedances therefore provide a resolution bound, not a zero tail probability. For numerical stability evaluate the exact expression as $-\operatorname{expm1}(\log\alpha/N)$.

A different question is a valid Monte Carlo hypothesis test. If the observed dataset and $N$ null datasets produce exchangeable statistics, the rank of the observed one is uniform among $N+1$ distinct ranks. Its valid inclusive upper-tail Monte Carlo p-value is

$$p_{\rm MC}=\frac{K+1}{N+1}.$$

Inclusive ties make it conservative. The rank argument and distinction from estimating a fixed tail are discussed by Phipson and Smyth, §§2 and 4. This p-value is not the confidence upper bound above. Exchangeability must be justified; simulating from a fitted model without reproducing the fitting process does not automatically supply it.

Verification and related

spectral_analysis_checks.py checks gamma tails, exact capped convolutions and atoms, finite-rank calibration, and zero-exceedance binomial coverage. These are mathematical calibration examples, with no empirical significance assigned to V2 data.

• Spectral analysis tools — navigation and scope.