Calibration
How often it's right
Measured on 10,269 held-out paragraphs from 11 sources, and on 3,000 paragraphs from web pages on sites it never trained on. The numbers include the places it does worst.
Calibration record
Model 2026-09-19.5 · tested 2026-09-19 · 10,269 held-out paragraphs · bars 0.95, machine-ish 0.93
- right when it makes a call
- 96.3%
- of paragraphs get can't tell
- 90%
- of human paragraphs called machine-ish
- 0.3%
| Test | Measured | Required | Result |
|---|---|---|---|
| Calibration error (ECE)0.05 or less | 0.013 | 0.05 or less | pass |
| Top guess right, test setbeat rules alone, 50.8% | 69.1% | beat rules alone, 50.8% | pass |
| Top guess right, held-out attacksbeat rules alone, 5.8% | 55.6% | beat rules alone, 5.8% | pass |
| Separates machine from human (AUROC) | 0.845 | ||
| Machine text, top guess machine-ish | 72.8% | ||
| Human text, top guess machine-ish | 16.6% | ||
| Web pages from sites kept out of training, top guess machine-ish | 20.2% | ||
| Largest probability change from 8-bit weights | 0.010 |
The three figures above are at the shipped bars. The table's top guesses are before them, when the model has to answer every paragraph.
Web check
On pages it never saw
3,000 paragraphs from 2019 web pages, on sites kept out of training. People wrote all of them, before chatbots, so any machine-ish or mixed call here is a mistake.
- called machine-ish or mixed
- 0.7%
- called human-ish
- 2.3%
- can't tell
- 97.1%
Bars
Why it says can't tell so often
The model always has a top guess. It only shows one when the guess clears the bar for that answer, which is 0.93 for machine-ish and 0.95 for the other two. The bars are set so its calls are right 96% of the time on held-out text. Drag either one to see what a lower bar would cost.
- Machine-ish
- 0.93
- Others
- 0.95
- right when it makes a call
- 96%
- of paragraphs get can't tell
- 90%
- of human paragraphs called machine-ish
- 0.3%
Reliability
Its odds mean what they say
Group paragraphs by the probability the model gave an answer, then count how often that answer was true. A perfectly calibrated model lands on the dashed line.
Across is the probability given, up is how often it was true, and bigger dots hold more paragraphs. Human-ish and machine-ish stay within a few points of the line. Mixed strays furthest, by up to 10 points in its upper bins, which hold the fewest paragraphs.
Sources
Where it does well, and where it doesn't
The share of paragraphs whose top guess was right, before the bars. Human text polished by a model is the hardest case. It gets called mixed 25% of the time and machine-ish 33%, which still flags the model's hand, while 42% passes as human. Paraphrasing is the attack that hurts most.
Test sources
English Wikipedia
human734
89%
Reddit posts, 2006 to 2016
human724
88%
Llama 3.1 70B answers (Magpie)
machine386
87%
Yelp reviews
human704
87%
Recent models, plain prompts
machine1,456
76%
RAID: human documents and 11 older models
human and machine3,161
72%
Web pages, 2019 (C4)
human801
70%
ChatGPT replies to real users (WildChat)
machine447
69%
Recent models told to avoid the tells
machine376
67%
Human paragraphs polished by a model
mixed1,298
25%
Human openings finished by a model
mixed182
15%
Held-out attacks
Held-out models told to avoid the tells
machine691
67%
RAID model text, paraphrased
machine1,240
50%
Held-out sites
Web pages, 2019, from sites kept out of training
human3,000
76%
Second opinion
A hosted model did worse, so it's out
Jev is a hosted decision model from typesafe.ai. It got the same numbers the local model computes, never the text, and both made calls on the same 599 held-out paragraphs, 60 of them paraphrased machine text, and on the 2,999 paragraphs of the web check.
| Measure | Local model | Jev |
|---|---|---|
| Says can't tell | 90.0% | 0.5% |
| Right when it makes a call | 96.7% | 67.4% |
| Right when both skip about 30% | 75.5% | 74.9% |
| Calibration error (ECE) | 0.010 | 0.129 |
| Human text called machine-ish | 0.0% | 21.3% |
| Real pages called machine-ish | 0.7% | 21.1% |
| Time to a call | on device | 461 ms median |
Jev almost never says can't tell, and the confidence it states is off from how often it's right by about 12.9 points on average, against 1.0 for the local model. With both skipping about 30% of paragraphs, it was 0.6 points less accurate. The 95% interval runs from 3.4 points worse to 2.3 points better.
On the 2,999 paragraphs of the web check, all written by people, it called 21% machine-ish, against 0.7% for the local model.
So it's out. Neither the extension nor this site uses it. The benchmark stays, so a new version of Jev can be measured the same way.
Data
What it learned from
Labels come from where each paragraph came from, not from annotators. Paragraphs from one source document all land in the same split, so the test never sees a version of something from training.
| Split | human | machine | mixed | Total |
|---|---|---|---|---|
| Training | 21,850 | 21,388 | 7,416 | 50,654 |
| Validation | 2,921 | 3,023 | 1,006 | 6,950 |
| Test | 4,411 | 4,378 | 1,480 | 10,269 |
| Held-out attacks | – | 1,931 | – | 1,931 |
| Held-out sites | 3,000 | – | – | 3,000 |
- Human
- RAID's human documents (abstracts, books, news, poetry), Reddit posts from 2006 to 2016, Yelp reviews and English Wikipedia.
- Machine
- RAID text from 11 older models, ChatGPT replies to real users from WildChat, Llama 3.1 70B answers from Magpie, and paragraphs from nine recent models.
- Mixed
- Human openings finished by a model, and human paragraphs polished by one.
- Held-out attacks
- RAID's paraphrased model text, and text from four models told to avoid the tells. None of it was used in training.