How to build one
- Log every signal with its timestamp, stated confidence and a fixed outcome window, such as 72 hours.
- Group by confidence. Use buckets such as 50 to 60 percent, 60 to 70 percent, and so on.
- Calculate the hit rate in each bucket: correct signals divided by all signals in that bucket.
- Plot stated confidence on the horizontal axis and observed hit rate on the vertical axis. Add the diagonal, where stated and observed match.
An example
| Stated confidence | Signals | Hit rate | Reading |
|---|---|---|---|
| 50 to 60% | 40 | 52% | Roughly honest |
| 60 to 70% | 35 | 63% | Honest |
| 70 to 80% | 22 | 64% | Overconfident |
| 80 to 90% | 9 | 78% | Too few signals to judge |
In the 70 to 80 percent bucket, the model says it is right about three times in four, but it was right about two times in three. That gap is the thing to watch.
How to read the plot
- On the diagonal. The stated confidence can be trusted at that level.
- Below the diagonal. The model is overconfident. Its high-confidence calls win less often than it claims.
- Above the diagonal. The model is underconfident, which is less dangerous but still means the labels are wrong.
- Flat line. Confidence does not predict outcomes at all, so the label is decoration.
Check the sample size
Each point needs a count next to it. A bucket with nine signals can swing by a large amount from one or two results, so treat it as weak evidence. Many small buckets look dramatic and mean little. Combine sparse buckets before drawing conclusions.
Pair it with a score
A reliability diagram shows calibration, not overall skill. Pair it with a score such as the Brier score, which penalises confident wrong calls, and with the full losing record, so the good buckets are not the only ones shown.
Where to see Atradian's numbers
Atradian publishes how accurate its AI has been on the reliability page, and explains how each figure is produced on the methodology page. For the broader checklist, read AI trading signals: how to actually verify they work.