Our LLM Polymarket bot was just reading the market price back to us

We ran an LLM forecaster against Polymarket’s crypto markets this spring. After 319 resolved forecasts, its aggregate Brier score was 0.12, a number a calibration researcher would be pleased with, comfortably better than the 0.25 you’d get flipping coins. By that metric, the system worked.

It also lost to the market in every single bucket we measured. On the forecasts that mattered most - the ones where it disagreed with the market by 15 points or more, the only forecasts you could actually trade - it picked the right side 14.3% of the time. Not “failed to beat the market”: fourteen percent, on calls where a coin flip gets fifty.

A model can be perfectly honest about probabilities and still know nothing the market doesn’t. This post is how we proved that to ourselves, and why the project was dead 21 days after its first session log.

(Research write-up, not financial advice. The system only ever ran in paper mode: no live money was traded on any of this, which, as you’ll see, was the point of the exercise.)

What we built

The forecaster was Kimi K2.5, served over an API at $10/month flat. Cost mattered, because the thesis was cheap continuous forecasting across hundreds of markets. Around it: a market-discovery and orderbook pipeline pulling spread, bid/ask depth, volume, and liquidity from Polymarket’s APIs; a multi-agent orchestrator (researcher → analyst → risk manager → executor → monitor); Postgres for every forecast, resolution, and calibration report; all of it on homelab Linux containers. Live trading was disabled behind a triple safety gate, and every strategy had to survive paper trading before that gate would even be discussed.

One sentence of credibility is all that architecture deserves in this story, because the architecture isn’t what failed.

The red flag: two numbers that couldn’t both be good news

Three weeks in, a routine sanity check produced this table: lifetime performance to that date, bucketed by how much the model disagreed with the market price (“edge”):

Edge bucket	n	Win rate vs market	Brier
< 0.05 (low)	284	25.7%	0.12
0.05-0.15 (med)	21	28.6%	0.10
≥ 0.15 (high)	14	14.3%	0.31

A “win” here means the model was on the correct side of its own disagreement with the market: it said the market was too low and the thing happened, or too high and it didn’t.

Look at the shape of that. Aggregate calibration: excellent. Win rate when it disagrees with the market: worse than random everywhere, and worst where it disagreed most. The question the table forces is: how are both of these true at once?

One hypothesis explains both numbers with nothing left over: the model was reading the market price and emitting noise around it. Mirror the price and you inherit Polymarket’s calibration: prediction markets are well calibrated, so your Brier score looks like skill. But calibration inherited from the market contains zero information the market doesn’t already have, so any genuine disagreement is just your noise term - and noise loses. The low-edge bucket even shows the signature: the model’s average deviation from the market price was 0.011. One percentage point. That’s not a forecast; that’s a tracing.

The investigation

Hypotheses are cheap. We went looking for the mechanism, and it wasn’t hiding.

Exhibit one: the prompt. The evidence string sent to the model is assembled by one function, and its first line is:

def render_db_evidence(market):
    market_price = market.get("outcome_yes_price") or 0.5
    parts = [
        f"Current market price (YES): {market_price:.4f}",  # FIRST LINE
        ...

We fed the model the answer key as the opening line of the exam. By design, originally - the price is legitimate evidence, and we’d reasoned that a good forecaster should treat it as a prior. Hold that thought.

Exhibit two: the model’s own reasoning. We read the stored reasoning from the high-edge forecasts: nineteen unique rows after deduplication. Seventeen of nineteen - 89.5% - explicitly cited the market price as a primary signal. The other two leaned on orderbook or efficient-market logic without naming the price. The number of forecasts reasoning purely from fundamentals: zero out of nineteen.

The stored transcripts are almost painful to read. The money quote, verbatim, from a forecast where the model was asked to independently estimate a probability:

“1. Accept the market price as the best estimate (efficient market hypothesis). 2. Adjust it based on theoretical grounds…”

The model invoked the efficient market hypothesis as its reason for not disagreeing with the market. It’s not wrong, exactly. It’s just a $10/month way to be told the price you already fetched.

Exhibit three: what “disagreement” actually looked like. When the model did break from the price, it didn’t reason its way to a different number. It snapped to one of two degenerate defaults. Seven of the nineteen high-edge forecasts came in at 0.01 - the minimum the output format allowed - and every single one of them resolved YES. Maximally confident, maximally wrong, seven for seven. Another four snapped to 0.5, the “I don’t know” fallback; the market was closer than the model in all four. There is no third mode in this data where the model produces a considered, independent estimate. It either traces the market or it flinches.

The falsification test

At this point the tempting move - the move we almost made - was an ablation study: strip the price from the prompt, rerun, and “see how much the calibrator degrades, then tune with more data.” The problem with ablations as decision tools is that they always produce a number, and the human reading the number always finds a way to keep going.

So instead we pre-registered a kill gate, the day before the experiment ran:

• Strip the market price from the prompt. If the model’s high-edge win rate comes back above 0.55 (n ≥ 15): keep the no-anchor variant and re-enter calibration for 2-4 weeks.
• Below 0.50 (n ≥ 15): kill the phase. No tuning, no “one more run.”
• Sample too small: widen the price band and rerun - but no verdict-flipping after the fact.

The plumbing was more work than the idea: a flag to drop the price line from the evidence string while keeping all other microstructure, and schema migrations so each no-anchor forecast paired against the exact market snapshot its control forecast saw, like-for-like, down to the timestamp. From 57 paired candidate markets, 32 forecasts inserted cleanly; 25 dropped to API timeouts and unparseable responses. (Our internal notes call the decisive run “~13 minutes of wall clock.” Honesty audit: that number was the pre-run estimate, and with a 44% API failure rate the real run was surely longer. Nobody logged the elapsed time. We’re publishing the correction anyway, because rounding your own legend upward is how trust dies.)

The result:

	High-edge win rate	High-edge Brier
Control (price in prompt)	0.357 (n=14)	0.630
Anchor stripped	0.438 (n=16)	0.358

Neither row comes out of the red-flag table above. The control row is the price-in-prompt forecasts this run paired against. Landing on n=14 again is a coincidence of sample size, not the same fourteen forecasts, which is why the control’s win rate and Brier don’t match that table’s high-edge row.

Two things happened at once, and the difference between them is the entire lesson. Stripping the anchor produced a dramatic calibration recovery: Brier collapsing from 0.630 to 0.358 means the model stopped making those insane 0.01-on-a-winner calls. The anchor wasn’t a benign Bayesian prior; it was load-bearing and malignant, actively shoving the model into overconfident wrong answers whenever it deviated.

And the win rate came back at 0.438. Gate says below 0.50 on n ≥ 15: kill. Killed.

Removing the anchor didn’t reveal a suppressed signal underneath. It revealed well-calibrated noise. There was nothing under the anchor at all.

What we took away

1. Calibration is not edge. Brier score measures whether your probabilities are honest. Only win-rate-versus-your-counterparty measures whether they’re worth money. A system can ace the first and be worthless by the second - ours did, and was. Never gate a go/no-go on calibration alone.

2. Pre-register the kill condition. The gate was written down, with thresholds and minimum sample size, before the experiment ran. That one paragraph did what an ablation study would have stretched across months of motivated reasoning. The result never got a chance to negotiate.

3. A kill is a successful experiment. One afternoon of compute settled a question that was about to consume an entire phase of budget. The system worked exactly as designed: the design just included the possibility of this answer.

4. Map the successors before the verdict. The kill note listed four ranked next options - mechanical strategies that don’t need LLM edge, different market universes, different models, or keeping the pipeline as a pure calibration asset - written before the result came in. That’s why the kill was a pivot instead of a stall: the same day the gate fired, the whale-copy-trading paper phase shipped.

5. On LLMs versus efficient markets: crypto prediction markets track the big exchanges too tightly to leave much surface for a language model. If LLM edge exists anywhere on prediction markets, it’s in messier universes - politics, events - where information is textual and slow. We consider that unproven, and we didn’t chase it.

Coda

Full transparency requires reporting what happened to option one, too. In June, the whale-copy successor went through a structured “what is our edge, actually?” review. Nine candidate edge theories were examined. None survived, and that strategy is now recommended for retirement as well. Two falsifications, zero dollars of live money lost, a research pipeline that’s still useful as a calibration instrument - and this post.

This is ArkNode Labs. Experiments here get status chips, and FALSIFIED is a first-class outcome: the epistemics are the product, especially in a niche where every other landing page is a fake win rate. If you want the next experiment’s logs, including the ones that die, the newsletter is below.

---

Paper trading only; no live capital was ever at risk in this system. Nothing here is financial advice - if anything, it’s advice against believing anyone who claims LLM trading edge without showing you a table like the one above.