Edge Score V3b vs same-window PnL: an in-sample diagnostic, not a forward test
Read this first: target leakage, not a forward test
The rolling series below is an in-sample contemporaneous rank correlation (the Edge Score and the realized PnL are computed on the same in-window positions, and the dominant input, conviction = concentration = max|pnl|/total_risk, is PnL-derived). It is NOT a forward test; it overstates predictive skill and should be read as a descriptive diagnostic only.
The honest forward result is the pre-registered per-wallet temporal holdout (AsPredicted #287368): the frozen-coefficient Edge Score held a small positive out-of-sample Spearman of +0.11 (95% CI [0.05, 0.18]) with forward PnL. This did not clear the pre-registered +0.30 threshold, and a refit did not generalize. Forward predictive skill is small and not yet established at the pre-registered bar.
Version scope: everything on this page was computed under the original V3b constants. The live Edge Score constants were refit on 2026-07-13 after a bounded-input correction, so these diagnostics do not describe current scores; the dated method-change note is at /methodology.
Across 26 rolling 30-day windows from October 1, 2025 through April 25, 2026 on the V1-M reference cohort, the V3b composite (frozen coefficients posture +0.7876, conviction +2.7220, discipline -1.1508) produced a mean rolling Spearman of +0.391 and a median of +0.375 against signed-log PnL computed on the same in-window positions. 26 of 26 windows finished above the Brier-only baseline of +0.147. This is an in-sample contemporaneous diagnostic, not a forward test: the score and the realized PnL share the same positions and the dominant input (conviction) is PnL-derived, so it overstates predictive skill. The honest forward result is the pre-registered per-wallet temporal holdout (AsPredicted #287368): out-of-sample Spearman +0.11 (95% CI [0.05, 0.18]), which did not clear the pre-registered +0.30 threshold.
Rolling Spearman (in-sample, same-window PnL)
30-day windows, stepped every 7 days, 10-position minimum per wallet. Score and PnL computed on the same in-window positions (in-sample contemporaneous, not forward). Shaded band is the 95% percentile bootstrap CI (1,000 resamples per window).
How to read this chart
- The teal line is the Spearman correlation between the V3b composite and signed-log PnL within each 30-day window. Each window is scored independently using only the positions that resolved inside it, and the same positions also produce the PnL the score is correlated against. Because the score and the target share positions (and conviction is PnL-derived), the correlation is contemporaneous and in-sample, not forward. Wallets with fewer than 10 resolved positions in a window are excluded for that window.
- The shaded band is the 95% percentile bootstrap CI. A narrow band means the rho estimate for that window is precise; a wide band means the cohort in that window was small or noisy. Both are reported, never just the point estimate, per the credibility-claim audit rules.
- The amber dashed line at +0.514 is the in-sample out-of-fold Spearman published in the V1 paper. Both that benchmark and the rolling series here are in-sample measures, so windows at or above the line are not evidence of forward skill.
- The red dashed line at +0.147 is the Brier-only baseline (calibration alone vs PnL) published in the 10K-wallet study. Windows that finish above this line are adding signal beyond raw calibration.
- The vertical dashed line marks the Control H blinding cutoff at 2025-09-30. The V3b coefficients were fit on a snapshot through April 15-16 2026. The cutoff is retained for context, but it does not make any segment a forward test: every window correlates the score against PnL from the same in-window positions, so the leakage is present on both sides of the cutoff.
Methodology
- Model
- Edge Score V3b (frozen production coefficients).
- Coefficients
- posture 0.7876, conviction 2.7220, discipline -1.1508. Frozen at the V1 paper publication and unchanged.
- Cohort
- V1-M reference cohort, n_unique_positions >= 5 in wallet_analysis_20260425_201800.csv.
- Window
- 30-day rolling, stepped every 7 days, with a minimum of 10 resolved positions per wallet per window.
- Per-window features
- skill_brier is computed window-locally as the wallet’s baseline Brier (under the window’s marginal frequency of wins) minus its observed Brier on the window’s positions. concentration is the largest absolute per-position PnL share over total in-window risk. n_unique_positions is the count of resolved positions inside the window.
- Score
- Edge Score V3b raw composite under the frozen production z-score parameters. Spearman is rank-invariant, so the unbounded raw score is equivalent to the production percentile mapping for correlation purposes.
- Confidence interval
- 95% percentile bootstrap on (Edge Score, signed-log PnL) pairs, 1,000 resamples per window.
- Source data
-
services/api/scripts/output/wallet_all_positions_20260425_201800.csv(V1-M position tape, 542,241 resolved positions) andservices/api/scripts/output/wallet_analysis_20260425_201800.csv(per-wallet features for the cohort filter). - Generator
-
services/api/scripts/forward_validation_rolling_spearman.py. Generated Mon, 27 Apr 2026 23:11:51 GMT. - Underlying JSON
- forward-validation-rolling-spearman.json (one record per window; same schema as the chart data).
Limits of this dashboard
- This is not a forward test. The score and the realized PnL are computed from the same in-window positions, and the dominant input (conviction = concentration = max|pnl|/total_risk) is PnL-derived. The series therefore has target leakage and overstates predictive skill. Read it as a descriptive contemporaneous diagnostic, not as out-of-sample predictive validation.
- Survivor bias still applies. The cohort comes from the leaderboard-ranked snapshot; wallets that blew up and were delisted before the snapshot are absent from every window.
- Window-local skill_brier and the conviction and discipline pillars are computed window-locally, but this does not make the series forward: conviction is derived from in-window PnL, so the score is mechanically correlated with the same-window PnL it is scored against. The z-score reference statistics come from the V1 training cohort and are kept frozen here.
- The min-positions-per-wallet filter excludes thin slices and cuts noise, but it also concentrates the held-out estimate on high-activity wallets. The cohort sizes per window are surfaced in the tooltip and in the underlying JSON.
- A point estimate from a single window is not a methodology verdict, and neither is this whole series: it is in-sample. The real out-of-sample question is answered by the pre-registered per-wallet temporal holdout (AsPredicted #287368), which produced an out-of-sample Spearman of +0.11 (95% CI [0.05, 0.18]) with forward PnL and did not clear the pre-registered +0.30 threshold. See the V1.5 result for the forward finding.
The FDR-screened wallet registry and its forward test
The composite above is one lens. A separate, simpler question is whether any individual wallet’s resolved record beats chance after multiplicity correction. In a retrospective screen of per-wallet resolved Polymarket records (frozen 2026-04-25 position tape), 178 of 3,871 wallets cleared a Benjamini-Hochberg screen at q = 0.10, a cleared proportion of 4.6% (95% Wilson interval [3.98%, 5.30%]). At q = 0.10, the expected number of false discoveries among the 178 cleared wallets is at most 17.8. The screen is in-sample and retrospective: it identifies records that were unlikely under chance, and it does not establish that those wallets keep an edge going forward.
Whether the cleared set keeps its edge is the subject of a strictly prospective test, pre-registered as AsPredicted #294147 before the forward window opened; the public receipt carries the filing timestamp and the frozen content hashes. The candidate set (178 wallets), the control set (3,693 wallets), and the methodology were frozen before any forward evidence accrued. The forward window runs 2026-06-02 through 2026-08-30 and the test matures on time and outcomes alone.
This page reports accrual counts only, never interim pooled effect estimates, while the window is open. The sample floors require at least 40 candidate wallets to clear the 10-position floor and at least 1,000 qualifying candidate forward positions to resolve. The confirmatory verdict is withheld from this page until the pre-registered window closes on 2026-08-30, even once those floors are met, so no mid-window read is surfaced as a result. A null or inconclusive outcome at maturity is a valid pre-registered result and will be reported as such.
Registry dataset
The frozen per-wallet registry behind this screen is being prepared for public release and is pending a data-release review. Nothing is downloadable today. Leave an email and the dataset link is emailed after release.
Amendments to this package, 2026-07-30, dated before the first terminal read
Two defects in our own tooling, found on 2026-07-30 while preparing the third filing in this series, disclosed here in full before window one's terminal read exists. Neither changes any registered set, hash, window, floor, seed, statistic or verdict rule, and nothing below is a reason to delay or qualify the 2026-08-31 readout. The full amendment text below is verbatim from the dated internal record.
AMENDMENT (2026-07-30). A second defect in the extract builder behind our selection stage, disclosed with its direction and its limits. On 2026-07-30, while preparing a third filing in this series, we found a further defect in the extract builder that produced the 2026-04-25 parent extract on which both registered windows (AsPredicted #294147 and #303724) rest. The builder looked up event metadata on the Gamma /events endpoint in batches of 50 slug parameters without sending a page-size parameter. That endpoint's default page size is 20, so a batch of 50 could come back with as few as 20 events answered, and every fill whose event went unanswered was silently dropped from the wallet's record, visible only in a skip counter nothing monitored. The code fix, with a test that fails if the fix is reverted, is commit 30372aca (2026-07-30).
WHERE IT SITS. This is a SELECTION-stage defect, compounding the retention of each wallet's newest 2,000 activity rows disclosed in this package: it degraded the per-wallet records on which the screen computed its in-sample statistics and chose the 178 candidate and 3,693 control wallets. Those sets were frozen on 2026-04-25 and pinned by content hash in both public receipts, before window one opened on 2026-06-02 and before window two's 2026-08-31 open. No set, hash, window, floor, seed, statistic or verdict rule changes with this amendment. The candidate set should be read, as the retention disclosure in this package also states, as the screen's output on the 2026-04-25 extract as it existed.
WHAT THIS CAN AND CANNOT DO TO THE FORWARD TESTS. The forward measurement is arm-uniform: one pipeline scores both arms on the same window, the same frozen measure, the same exclusion rule and the same resolution source, so a selection-stage record defect cannot manufacture a forward difference between the arms. What it can do is degrade the selection itself: incomplete records add noise to the in-sample statistics the screen ranked on, so the frozen candidate set is expected to be a noisier selection than an uncorrupted screen would have produced. The expected direction of that noise on the forward tests is to attenuate the candidate-versus-control contrast toward zero, not to create it. We state the direction and stop there: we do not restate it as a claim that the true effect is larger than whatever the forward tests report. The registered object is the frozen set as it was actually selected, and the forward tests score that object and nothing else.
THE LOSS RATE ON THIS EXTRACTOR IS NOT MEASURED. The 55.90% figure associated with this defect (184,470,230 of 329,998,137 fills discarded) is a census, taken 2026-07-30, of a different extraction run (the 2026-07-28 full-history run), whose code added a run-wide cache shared across wallets that made every miss permanent and global for that run. The 2026-04-25 builder had no such cache and re-queried per wallet, so a slug unanswered for one wallet could still be answered for another. That difference only stops a miss from spreading across wallets; it does not recover the missing wallet's own fills, since each of its batches was answered once, at up to 20 events per batch, on a single pass. And the two runs read different inputs: full histories against the newest 2,000 activity rows per wallet. The 55.90% census therefore neither bounds nor estimates this extractor's loss rate, in either direction, and we will not put a number, or a bound, on it until we have measured one. The measurement is the named follow-up below, not an assumption.
IN-SAMPLE FDR CLAIM, FURTHER WEAKENED. The in-sample claim behind these windows is that 178 of 3,871 discretionary wallets cleared Benjamini-Hochberg at q = 0.10 for positive realized edge over entry prices, expected false discoveries at most about 17.8. That q = 0.10 figure assumes each wallet's record is a faithful sample of its resolved positions. The newest-2,000-rows retention already weakened that assumption; this defect weakens it further. The nominal figure should not be read as carrying its original guarantee for these windows, and the realized false-discovery proportion may exceed it. The forward tests do not inherit this particular weakness, because they score the frozen sets on outcomes that postdate the freeze; it is the in-sample screening claim that is weakened.
FOLLOW-UP M1, NAMED HERE (2026-07-30): measure the 2026-04-25 extractor's enrichment loss rate. Method, in order: (a) if the original build's per-wallet skip counters survive in its logs, the loss rate is the exact ratio of skipped fills to skipped plus retained fills, per wallet and pooled; (b) otherwise, replay: using the corrected full-history extraction over the same frozen wallet universe (a run the third filing already requires), restrict each wallet's stream to fills at or before the 2026-04-25 cutoff, apply the newest-2,000-rows retention to reproduce the original raw input, enrich with the fixed code, and compare the resulting resolved-fill set row for row against the frozen extract (sha256 cf95c5f1b9a957477f6b3c98bdf235744eff8f34230083a4c8bffd4657ef2ec6). Report the pooled loss rate, the per-wallet distribution, and separately the count of frozen-extract rows absent from the replay, which measures API drift since 2026-04-25 and is reported on its own, never folded into the loss rate. Target: report on or before 2026-09-30. This measurement does not gate the window-one terminal read on 2026-08-31; if the terminal readout publishes first, it publishes with the unmeasured status stated above, and this package is amended when the number exists.
AMENDMENT (2026-07-30), SECOND INSTANCE OF THE SAME DEFECT, IN THE MEASUREMENT STAGE, DISCLOSED WITH ITS HANDLING. The missing page-size parameter also existed in a third copy of the slug lookup, inside the production analyzer module that the daily forward-position collector for these windows uses on its primary path. On any single collection pass, fills whose event metadata went unanswered were not scored on that pass, in whichever arm the affected wallet sat, through the same code path. Three properties of the collector bound the consequence, and the second is not even across the arms. The forward archive is append-only with idempotent de-duplication, so a position captured once is never lost. The slug batches are recomposed from run to run, so a position missed on one pass can be captured on a later one; but candidate wallets are collected on every daily pass, while control wallets rotate through a bounded slice and are each visited roughly once every 12 days at the collector's current setting, and a resolved position can only be captured on passes after it resolves. A missed candidate position therefore gets a fresh capture chance every day, a missed control position only when the rotation returns, and a control position that resolves late in a window may see few passes, or none, before a read date. And wallets yielding fewer than 30 resolved positions from the primary path are also read through a second, paginated source. The residual, stated plainly: we have not measured how much in-window evidence remained uncaptured at any given read date, and we have not established the direction, if any, of that residual with respect to outcomes. The code path is shared, so neither arm is exempt, but a shared path does not make the loss equal across arms: a wallet loses nothing on a pass until a lookup batch exceeds the default page size, so per-wallet loss scales with how many distinct events the wallet trades, the design does not match the arms on that, and the visit cadence above gives the two arms unequal recovery chances. Both legs of the verdict rule therefore carry an unmeasured sampling residual whose size and direction we do not claim to know; what bounds it is the recovery mechanics above and the follow-up census below, not an argument.
HANDLING, WITH DATES. The fix to the shared analyzer helper is commit b2146b91, with a test that fails if it is reverted. From the fix date to each window's close, the daily passes re-offer previously unanswered slugs. Because the archive is partitioned by write date, rows first captured after the fix whose resolution predates it are identifiable, and each window's terminal readout will report that count, split by arm, as the measured signature of this defect on the measurement stage (follow-up M2); the split matters because the visit cadence above gives the arms unequal recovery chances, and a pooled count would hide a recovery concentrated in one arm. No read date, window boundary, floor, statistic or verdict rule changes. This is the same class of event as the 2026-07-03 resolution-screen deviation already in this package: an implementation defect corrected toward the filed intent mid-window, documented with dates, with no registered threshold changed. The terminal read uses whatever the archive holds on the read date, exactly as filed.
The CLV-edge forward test: first confirmatory readout (AsPredicted #296461)
A separate, strictly prospective test asked whether ranking wallets by closing-line value predicts their later realized edge. The predictor is each wallet's mean 24-hour CLV over the frozen 2026-04-25 position tape: how far the market price had moved 24 hours after the wallet's fill, relative to its entry price. The forward outcome is each wallet's mean realized edge (won minus entry VWAP probability) over positions resolving on or after 2026-06-13. The test was pre-registered as AsPredicted #296461 on 2026-06-12, before any forward evidence existed. Frozen at filing: the pass bar (Spearman rho at or above +0.10 with the 95% bootstrap CI lower bound above zero), the sample floors (300 wallets and 3,000 pooled forward positions), the per-wallet floors, the exclusion rules, the bootstrap design (wallet-resampled, 10,000 resamples, seeded), and the checkpoint convention. The frozen methodology hash is af24f0826cfc4945d251f68acdb503c373f3c873bdaaaeb1e1a139bad9dfc788; any post-hoc change to a gate, floor, or window flips the hash on the next run.
The frozen floors were first met on 2026-07-03 and the confirmatory read ran on that day's committed artifact. Across 464 wallets clearing both frozen floors, pooling 19,740 forward resolved positions (floors: 300 wallets, 3,000 positions), ranking-period CLV correlated with forward realized edge at Spearman rho +0.159 (95% CI [+0.063, +0.255], wallet-resampled bootstrap, 10,000 resamples). That meets the pre-registered bar. The 464 is the intersection of two counted denominators: 2,572 of 3,509 ranking-side wallets cleared the 30-position and 20-covered ranking floors, and 720 of 1,676 forward-active wallets cleared the 10-position forward floor. Uncovered wallets and excluded positions are counted in the artifact, never dropped silently.
The pre-registered secondary did not go our way
In the same readout, the pre-registered secondary comparison found that CLV did not rank wallets better than plain realized edge. The paired difference of forward Spearmans (CLV rank minus realized-edge rank, the same 464 wallets, the same forward outcome) was -0.092 (95% CI [-0.184, +0.001], wallet-resampled bootstrap, 10,000 resamples). The CI spans zero and the point estimate favors the simpler baseline. Read the two results together: ranking wallets by CLV carried signal into the forward window, and it was not better than simply ranking them by past realized edge.
This is one readout, on one venue, over one in-window period (positions resolving 2026-06-13 through 2026-07-02 in the read artifact). Per the frozen specification, the confirmatory read is the first daily artifact where both floors hold, so this readout is final for this filing; later recomputes are descriptive only, and any further confirmatory claim requires a new pre-registration. A rank correlation of +0.159 (95% CI [+0.063, +0.255], n = 464 wallets) is modest. It is not a forecast, it does not establish that CLV-ranked wallets can be traded profitably, and, given the secondary result above, it does not justify preferring CLV over realized edge as a ranking input. A failed bar would have been reported here the same way.
The verdict artifact, every input archive sha256, the frozen tape sha256, the seed, and the methodology hash are committed at docs/ops/evidence/clv-edge-forward-verdict/latest.json (methodology hash af24f0826cfc4945d251f68acdb503c373f3c873bdaaaeb1e1a139bad9dfc788, master seed 20260613). The forward evidence is the same collector archive the FDR forward test above reads. Exclusions applied to the forward side, all counted in the artifact: 26,617 pre-window, 5,587 micro-market, 1,976 non-objective resolution source, out of 94,125 rows total, leaving 59,945 qualifying positions before the per-wallet floor.
References
- V1 paper: Edge Score Methodology V1 (in-sample OOF Spearman +0.514, Fama-French bootstrap null p < 0.0001).
- V1-M paper: Edge Score V1-M cross-venue extension (defines the V1-M reference cohort used here).
- Brier-only baseline: 10,000-wallet Polymarket study (Spearman +0.147 between Brier and signed-log PnL).