A rate gate needs its sample size derived from the rate
The slid-serve test asked for 6 of 10 sliced serves returned. Measured over 200 trials a seed across eight seeds, the receiver's real rate is 0.85..0.92 — so the assertion was sound and the SAMPLE was too small: at p ~= 0.88, 10 trials land at or below 5 with probability ~0.004, one seed in 230, and SEED=80 was it. n is now 40 at the same 60% bar, which puts the threshold five standard deviations below the true rate while still failing instantly if the receiver regresses toward the 0-in-24 the test was born from. It is free: a returned serve breaks the frame loop early, and the suite still runs in ~0.11 s. Green on seeds 1..400. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
bdc3f23f92
commit
131c0fa114
2 files changed
+19
-6
No files matched your search
+12
-5
@@ -431,11 +431,18 @@ Measured, 60 seeds × 150 s of drill each: before, **7 seeds produced overlaps,
|
||||
is still the cap of 7, so the fan is wider, not thinner. Suite green on 20260811 / 1234 / 777 / 99 /
|
||||
424242 / 8, and on 299 of seeds 1..300 — the one straggler is unrelated and still open (see below).
|
||||
|
||||
**OPEN, found by the same fuzz:** `SEED=80` fails *"opponent one returns a slid serve: 5/10"*. That
|
||||
assertion wants 6 of 10 sliced serves returned and the AI's real rate sits right on the threshold, so
|
||||
it is a coin-flip gate of exactly the kind this file warns about. Untouched here because the honest
|
||||
fix is a decision about the opponent, not the test: either widen `n` so the mean settles, or accept
|
||||
5/10 as the real return rate and say so.
|
||||
**The one straggler the same fuzz found, and the arithmetic to copy for any rate gate here.**
|
||||
`SEED=80` failed *"opponent one returns a slid serve: 5/10"*. The receiver is innocent: measured over
|
||||
200 trials a seed across eight seeds, the real return rate is **0.85..0.92**. The gate was simply too
|
||||
small a sample — asking 6 of 10 at p ≈ 0.88 fails with probability ~0.004, one seed in 230, and 300
|
||||
seeds duly produced one. **n is now 40 at the same 60% bar**, which puts the threshold five standard
|
||||
deviations below the true rate while still failing instantly if the receiver regresses toward the
|
||||
0-in-24 the test was born from. It costs nothing — a returned serve breaks the frame loop early, so
|
||||
the whole suite still runs in about a tenth of a second. Green on seeds 1..400.
|
||||
|
||||
The general lesson: **a pass/fail gate on a random rate needs its sample size derived from the rate,
|
||||
not guessed.** If the true rate is p and the bar is b, n has to put b several standard deviations
|
||||
(`sqrt(p(1-p)/n)`) below p, or the gate is a slow coin flip that trains you to re-run the suite.
|
||||
|
||||
## Session log: THE NETPLAY LOBBY — a link to send, a code to type, a name to be (2026-08-15)
|
||||
|
||||
|
||||
Reference in new issue
Block a user