A rate gate needs its sample size derived from the rate
The slid-serve test asked for 6 of 10 sliced serves returned. Measured over 200 trials a seed across eight seeds, the receiver's real rate is 0.85..0.92 — so the assertion was sound and the SAMPLE was too small: at p ~= 0.88, 10 trials land at or below 5 with probability ~0.004, one seed in 230, and SEED=80 was it. n is now 40 at the same 60% bar, which puts the threshold five standard deviations below the true rate while still failing instantly if the receiver regresses toward the 0-in-24 the test was born from. It is free: a returned serve breaks the frame loop early, and the suite still runs in ~0.11 s. Green on seeds 1..400. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
bdc3f23f92
commit
131c0fa114
2 files changed
+19
-6
No files matched your search
+12
-5
@@ -431,11 +431,18 @@ Measured, 60 seeds × 150 s of drill each: before, **7 seeds produced overlaps,
|
||||
is still the cap of 7, so the fan is wider, not thinner. Suite green on 20260811 / 1234 / 777 / 99 /
|
||||
424242 / 8, and on 299 of seeds 1..300 — the one straggler is unrelated and still open (see below).
|
||||
|
||||
**OPEN, found by the same fuzz:** `SEED=80` fails *"opponent one returns a slid serve: 5/10"*. That
|
||||
assertion wants 6 of 10 sliced serves returned and the AI's real rate sits right on the threshold, so
|
||||
it is a coin-flip gate of exactly the kind this file warns about. Untouched here because the honest
|
||||
fix is a decision about the opponent, not the test: either widen `n` so the mean settles, or accept
|
||||
5/10 as the real return rate and say so.
|
||||
**The one straggler the same fuzz found, and the arithmetic to copy for any rate gate here.**
|
||||
`SEED=80` failed *"opponent one returns a slid serve: 5/10"*. The receiver is innocent: measured over
|
||||
200 trials a seed across eight seeds, the real return rate is **0.85..0.92**. The gate was simply too
|
||||
small a sample — asking 6 of 10 at p ≈ 0.88 fails with probability ~0.004, one seed in 230, and 300
|
||||
seeds duly produced one. **n is now 40 at the same 60% bar**, which puts the threshold five standard
|
||||
deviations below the true rate while still failing instantly if the receiver regresses toward the
|
||||
0-in-24 the test was born from. It costs nothing — a returned serve breaks the frame loop early, so
|
||||
the whole suite still runs in about a tenth of a second. Green on seeds 1..400.
|
||||
|
||||
The general lesson: **a pass/fail gate on a random rate needs its sample size derived from the rate,
|
||||
not guessed.** If the true rate is p and the bar is b, n has to put b several standard deviations
|
||||
(`sqrt(p(1-p)/n)`) below p, or the gate is a slow coin flip that trains you to re-run the suite.
|
||||
|
||||
## Session log: THE NETPLAY LOBBY — a link to send, a code to type, a name to be (2026-08-15)
|
||||
|
||||
|
||||
@@ -1108,8 +1108,14 @@ test('a served ball with a slide on it is no longer a free kill', () => {
|
||||
}
|
||||
return false;
|
||||
};
|
||||
// n = 40, NOT 10, and this is the arithmetic to copy for any other rate gate here. The real return
|
||||
// rate is ~0.88 (measured, 200 trials a seed across eight seeds: 0.85..0.92), so a 10-sample gate
|
||||
// asking for 6 fails whenever the tail lands — p ~= 0.004, which is one seed in 230 and duly showed
|
||||
// up as SEED=80. At 40 the same 60% bar sits five standard deviations below the true rate: the gate
|
||||
// still fails instantly if the receiver regresses toward the 0-in-24 this test was born from, and
|
||||
// it costs nothing (a returned serve breaks the frame loop early).
|
||||
let got = 0;
|
||||
const n = 10;
|
||||
const n = 40;
|
||||
for (let i = 0; i < n; i++) if (serve(-0.5)) got++;
|
||||
assert.ok(got >= n * 0.6, `opponent one returns a slid serve: ${got}/${n} (was 0)`);
|
||||
});
|
||||
|
||||
Reference in new issue
Block a user