What Convexly's methodology is, and what it isn't

Methodology track

Convexly publishes a full evidence track: methodology committed to the repository before any analysis runs; follow-up tests filed with their pass / fail thresholds before the data is touched; results published whether they pass or fail. The complete filing index, with verdict and audit-chain anchor for each, lives at /research/preregistrations. Failed tests appear in the negative-result registry; everything is auditable client-side at /research/verify.

Edge Score composite

The V3b composite is a frozen-coefficient three-pillar weighted sum. Coefficients (Posture +0.7876, Conviction +2.7220, Discipline -1.1508) were committed to a version-controlled repository before the validation script ran. Validation produced an out-of-fold Spearman rank correlation of +0.514 between Edge Score and signed log realized PnL on the V1 cohort. A Brier-only baseline produces +0.147 on the same cohort.

Each pillar is z-score standardized against the V1 reference cohort. Posture rewards baseline-adjusted Brier (negated), Conviction rewards PnL concentration on the largest event, Discipline penalizes total resolved-position count (fewer larger bets score higher).

Full derivation, OLS fit, OOF folds, and every intermediate statistic is in the V1 paper.

Frozen-coefficient vs ex-ante external filings

V1 is internally ex-ante. The methodology document and pass-fail thresholds were committed to git before the validation script ran. That repository is private, so the timestamped trail a reader can reach is the AsPredicted receipts for the filed tests and the in-browser hash chain, not the commit history.

V1.5 was filed externally, ran on schedule, and is published. The two ex-ante validation experiments that require per-position outcome data (per-wallet temporal holdout E2, per-quarter Information Coefficient stability E7) were filed externally on 2026-04-25 and ran on 2026-04-27. Both primary tests failed their ex-ante pass criteria. The full result is in the V1.5 section below and at the dedicated V1.5 paper.

V2.8.2 was filed externally. Both in-sample and forward-only filings landed before any analysis code ran. Three frozen amendments are committed to the repository: one documenting the closure-caps choice, one substituting the V1-M reference cohort after the original two-hop closure plan exceeded available API capacity, and the audit-trail amendment for the gate script's V1-M cohort-substitution code path (the exact identifier is preserved in the filing index). The primary hypothesis test ran on the V1-M cohort 2026-04-27; result is in the V2.8.2 section below.

Per-filing IDs, receipt status, and full ex-ante text live at /research/preregistrations.

Validation results

• In-sample cross-wallet OOF Spearman: +0.514
• Calibration-only baseline Spearman: +0.147
• Fama-French (2010) bootstrap null at 10,000 permutations: observed Spearman outside every permuted sample, p < 0.0001
• Subgroup stability across six cuts: Spearman range +0.468 to +0.726
• In-sample rolling 30-day rank correlation across 26 monthly windows on the V1-M cohort, 2025-10-01 → 2026-04-25: mean +0.391 (median +0.375). This is an in-sample contemporaneous diagnostic, not a forward test: the score and the realized PnL are computed on the same in-window positions and the dominant input (conviction) is PnL-derived, so it overstates predictive skill. See the in-sample rank-correlation diagnostic for the leakage caveat and the honest forward result.
• V1.5 per-wallet temporal holdout (frozen V1 coefficients, N=805 paired wallets): Spearman = +0.111 (95% CI [+0.046, +0.175]). Ex-ante threshold ρ ≥ +0.30. Failed the ex-ante pass criterion. Positive and significant, but well below the threshold.
• V1.5 partial Spearman of V3b vs PnL controlling for log absolute capital (N=7,805): +0.494 (95% CI [+0.477, +0.512]). Stronger than the marginal +0.322; capital is a suppressor, not a confounder.
• Hill tail index alpha on realized PnL: 1.28 (95% CI 1.20-1.36; variance formally infinite, justifying rank-based rather than parametric inference)
• V1 cohort: 8,656 Polymarket wallets with at least 5 resolved positions (frozen reference)
• V1.5 position tape: 8,778 wallets (V1 cohort plus a small set of wallets that entered the position tape after the V1 freeze; the V1.5 analyses use the larger superset). The full pre-bundle position tape contains 8,997 wallet rows; the additional 219 are filtered out by the V1.5 skill-window inclusion criterion (≥1 resolved position with `last_fill_ts` in the in-sample window). Corrected 2026-08-10: this line previously read 8,998 rows and 220 filtered, which counted the CSV header row as a wallet, and it said all three numbers ship in a public Polymarket data bundle. They do not. The Polymarket bundle that carried them was withdrawn and removed on 2026-08-10; the source pulls are held internally. See Reproducibility below.
• V1-M Manifold cohort: 15,106 users with at least 25 resolved markets in the cross-venue extension

What V1.5 found (the deferred experiments)

V1.5 ran on 2026-04-27 per its ex-ante external filing. Both primary tests failed their pre-committed thresholds. Five exploratory supplementary analyses produced a mixed picture of where V3b holds up.

• E2 per-wallet temporal holdout (frozen V1 coefficients, train 2024-01-01 → 2025-09-30, 14-day embargo, test 2025-10-15 → 2026-04-15, N=805 paired wallets): Spearman = +0.111 [+0.046, +0.175], p ≈ 0.001. Pre-reg pass criterion: ρ ≥ +0.30 AND CI lower > 0. Fail.Refit V3b on the training fold produces ρ = -0.082 with the posture coefficient flipping sign vs the frozen V1 fit.
• E7 per-quarter IC stability (pre-reg quarters Q2 2024 → Q2 2025): median per-quarter Spearman = +0.038. 3 of 5 quarters positive (need ≥5 of 6 per pre-reg). 2025Q1 was strongly negative (ρ = -0.164, p < 0.001, N=1,230). Fail. Per-wallet V3b ranking IS stable across quarters (lag-1 median +0.31, all 8 quarter pairs significant); the stable thing is behavior, not same-quarter alignment with PnL.
• S3 stratified by sample size: V3b predicts strongest at LOW volume and weakens at high volume. Bucket A (5-29 positions, N=3,195): ρ = +0.371. Bucket B (30-100, N=3,089): +0.150. Bucket C (101+, N=1,521): +0.013 (CI crosses zero). Counter to the naive expectation that more data produces a stronger signal. Mechanism: high-volume wallets are more likely market-makers / bots whose PnL is dominated by spread capture rather than directional skill.
• S6 persistent-wallet inversion: 229 wallets active in 4+ quarters show V3b inversely correlated with cumulative signed log PnL, ρ = -0.31 (95% CI [-0.42, -0.20]). Survivorship effect: long-running wallets that stick around include conservative-behavior patterns that protect against ruin without generating large PnL.
• S7 cross-category transfer: V3b refit on one Polymarket category transfers to others with ρ +0.17 to +0.33, comparable to within-category baselines (+0.23, +0.36, +0.27). V3b is more stable across categories than across time.

Honest reframing: V3b's defensible claim moves from “forward predictor of PnL” to cross-sectional ranker of wallet behavior whose temporal alignment with PnL is sample-size and time-window dependent. Full V1.5 paper at /research/edge-score-methodology-v1-5.

What V2.8.2 found (the aggregator stack)

V2.8.2 PRIMARY hypothesis: does W-EXP β=4 + a=2.0 + S-RAW skill-weighted aggregation produce lower mean Brier loss than equal-weighted and market-implied baselines on the held-out window? The in-sample filing and V1-M cohort-substitution amendment are tracked in the public receipt manifest. Ran 2026-04-27.

• PRIMARY result: on 6,256 held-out markets, the skill-weighted aggregator produced a Brier delta of +0.058 (95% CI [+0.054, +0.062]) vs equal-weighted baseline and +0.179 (95% CI [+0.164, +0.193]) vs market-implied baseline. Effect threshold per pre-reg was δ < -0.005. Direction reproduced in all 3 cross-fit folds and all 3 execution tiers. Does not reject H0. The aggregator is significantly worse than both baselines, not equivalent to them.
• 24-aggregator sweep: the V2.8.2 pre-registration listed 24 aggregator specifications (PRIMARY plus 23 others). NONE reject H0_market on the V1-M cohort. Best aggregator was W-MANSKI (Imbens-Manski midpoint), with δ_market = +0.057 [+0.049, +0.066]. W-MANSKI beats the equal-weighted baseline (δ_naive = -0.064, Sharpe +0.336) but does so by shrinking aggregates toward 0.5 (mechanically reducing Brier loss at the cost of being uninformative). PRIMARY reproducible bit-for-bit.
• Pre-data gates G2-G8 with real markets-universe data (re-run 2026-04-27 on 163,710 markets with 99.85% gamma metadata coverage): 5 of 9 gates pass. G1 closure 1.00 (V1-M cohort substitution by construction). G4 reachability 1.00. G5 staleness 0.9951. G8 powered categories satisfied. G2 regime stability 0.407 (vs 0.50 threshold), G3 endDate audit 0.9137 (vs 0.95), G2b per-quarter concentration scoping fixable in V2.9. G6 sigma 0.39 (vs 0.063 ceiling) reflects the PRIMARY aggregator's noise; the negative PRIMARY result remains negative across the noise band (point estimate +0.179, 95% CI [+0.164, +0.193]; the lower bound stays well above the −0.005 effect threshold). G7 N_w power floor 1.98% reflects V1-M cohort thinness; deferred to V2.9 two-hop closure.

Substantive reading: PnL-skill on Polymarket is not a usable forecast-aggregation weight. The Edge Score V1 weighting (BSS_w as the per-wallet skill score) imported into a forecast- aggregation context produces aggregates that are systematically over-extremized vs ground truth, across the entire ex-ante aggregator family. This reproduces and reinforces the V1-M paper's null finding. The full V2.8.2 results note (path D, 2026-04-27) is held in the private Convexly repository and supplied on request rather than reader-inspectable (corrected 2026-08-10, which previously called the repository public); a publishable V2.8.2 paper page is in production.

What the V1 paper does NOT claim

The +0.514 OOF Spearman is moderate, not deterministic. Reference points: IQ tests vs college GPA correlate roughly +0.45, SAT vs first-year GPA roughly +0.5, FICO vs default roughly +0.7-0.8. The Edge Score is a useful rank-ordering signal, not a guarantee of next-period PnL for any individual wallet.

Edge Score is venue-specific by design. The V1-M paper shows that pillar coefficients diverge categorically between Polymarket and Manifold, with the Discipline pillar flipping sign at permutation p = 0.0001. A “Polymarket Edge Score” of 87 is not directly comparable to a “Manifold Edge Score” of 87.

Edge Score does not separate skill from luck on a single wallet's realized PnL history. V1.5 E2 (per-wallet temporal holdout) tested this directly and produced Spearman = +0.111 on N=805 paired wallets, well below the ex-ante +0.30 threshold. Use Edge Score for cross-wallet ranking, not for single-wallet temporal prediction.

Acknowledged limitations

• Survivorship bias. The cohort describes differences among wallets that have ranked on the Polymarket profit leaderboard. Wallets that blew up are not in the cohort. The cohort therefore measures cross-sectional skill differences among survivors, not expected returns for a randomly drawn new trader. Per Peters (2019), ensemble averages on a survivor set do not transfer to the time-average experience of an individual trader. V1.5 adds a supplementary cohort of active-but-unranked wallets to measure the selection shift explicitly.
• V3b vs V1 selection on training cohort. V3b is the shipped composite because its first feature (baseline-adjusted Brier) is structurally harder to game than V1's raw Brier, not because V3b out-beats V1 in a held-out blinded test. The definitive test is a held-out cohort with deliberately injected market-selection gaming, planned as a V1.5 follow-up. The current preference is a principled methodological choice, not a validated empirical one.
• Concentration is size-independent by construction. The Conviction pillar uses share of realized PnL attributable to the wallet's largest single event, not absolute capital. A separate question (does Edge Score correlate with absolute wallet size?) is not formally tested in V1 and remains an open empirical item.
• The rolling rank-correlation diagnostic was initially mislabeled as forward-validation; this is corrected. The mean +0.391 across 26 monthly windows on the V1-M cohort is an in-sample contemporaneous correlation (score and PnL share the same in-window positions; conviction is PnL-derived), not a forward test, and it overstates predictive skill. The honest forward result is the V1.5 per-wallet temporal holdout: out-of-sample Spearman +0.11 (95% CI [0.05, 0.18]), below the +0.30 ex-ante threshold. Diagnostic at /research/forward-validation.
• Half-life of predictive power tested in V1.5 S5. Median refit Spearman across 30/60/90/180-day cutoffs: -0.03, +0.01, +0.06, +0.11 (per-cutoff range [-0.148, +0.149]). The weakly positive trend toward longer horizons does not establish a conventional decay curve; the per-wallet temporal signal is small at every tested horizon and not a basis for forward prediction.
• Cross-category transfer tested in V1.5 S7. V3b refit on one Polymarket category transfers to others with Spearman +0.17 to +0.33, comparable to within-category baselines (+0.23, +0.36, +0.27). V3b is more stable across categories than across time.
• Persistent-wallet inversion is a new V1.5 finding. 229 wallets active in 4+ quarters show V3b inversely correlated with cumulative signed log PnL (ρ = -0.31, 95% CI [-0.42, -0.20]). Survivorship effect: long-running wallets that stick around include conservative-behavior patterns that protect against ruin without generating large PnL.

Anticipated critiques and responses

The V1 paper includes a formal Responses to anticipated critiques section addressing six sophisticated objections that an academic reviewer would raise: bootstrap-under-fat-tails, cross-venue invariance, V3b-vs-V1 circularity, concentration endogeneity, survivorship bias, and pre-registration semantics. Each response is reasoned and cites the relevant literature. Reviewers are welcome to engage with the responses directly.

Reproducibility

Corrected 2026-08-10. This section previously described v1m-data-bundle.tar.gz as 542K position-aggregated trades across 8,998 Polymarket wallets and called it the canonical input for V1, V1-M, and V2.8.2 reproducibility. Both halves of that were wrong. That tarball held the 15,106-user Manifold cohort, as its own README stated, and it was itself withheld on 2026-09-04 pending Manifold's written position on data rights. The separate Polymarket artifact that did carry a 542K position tape was withdrawn and removed on 2026-08-10 and its URL now returns 404; the withdrawal notice, with the reasons and the published sha256, is at README-polymarket-bundle.md.

What the public bundle reproduced, and no longer does: the V1-M Manifold half. That bundle shipped 15,106 aggregated per-user records with hashed identifiers, the refit coefficient tables with bootstrap 95% CIs and permutation null p-values, and a stdlib-only script. Updated 2026-09-04: the bundle and its per-user file were withheld pending Manifold's written position on data rights. The coefficient tables, the window statistics and the script are still published; the per-user records that the script's output was checkable against are not, so the Manifold half is method-disclosed rather than reader-runnable today. The aggregate results in the V1-M paper are unchanged.

What no public artifact reproduces: the 8,656 Polymarket wallets V1 is fit on, and every Polymarket per-wallet result downstream of them. No published bundle has ever contained that cohort. The two source pulls are held internally and the cohort loader in the paper-audit script is version-controlled, so the cohort is reconstructable by us; it is not reader-reproducible today. Publishing it is not a missing step but a decision: the real inputs carry Polymarket display names and raw addresses, so a release would need a rights and identifiability review. The dated correction is at /research/edge-score-methodology-v1.

Methodology code, validation scripts, and frozen amendment commits are version-controlled, and the commit history is what dates a freeze. Corrected 2026-08-10: this line said they are public in the Convexly repository. That repository is private, so the commit history is an internal audit trail rather than something a reader can open. Ask at the methodology contact below for a specific commit or script.

Contact for methodology review

Academic reviewers, journalists with methodology questions, and institutional buyers running due diligence can email research@convexly.app. Methodology critique is welcome; a critique that surfaces an actual error is more valuable to Convexly than another endorsement.