Run 3 · 3 Nov 2026
The State of AI Reliability
142 of 180
tries right against the official source
Six AI apps on free plans, the same ten UK questions, three tries each. 16 were wrong and 22 came back empty.
How each one did
Claude aced run 3.
Where it went wrong
15 of the 16 wrong answers came from two questions.
Run 4 is the first that can call one.
A night shift that crosses the October 2026 clock change and a Highway Code question about footwear
17/36
15 wrong · 4 no answer · 17 right The other eight questions
125/144
1 wrong · 18 no answer · 125 right Who marks it
AI marks it, and a second lab checks.
Claude is one of the six apps, so a second lab re-marks every Claude answer and a sample of the rest; Ben has not checked the marks by hand. Ten questions on one day: a snapshot, not a ranking. How we test
On record
Check it yourself.
The State of AI Reliability (v2 · run 3, captured Tue 3 Nov 2026): DIXON.AI Six consumer AI apps (ChatGPT, Claude, Copilot, Gemini, Grok and Perplexity), each on its free plan, signed in, asked the same 10 fixed UK questions three times in fresh chats: 180 tries. 142 of 180 tries right; 16 wrong; 22 came back with nothing. Of the answers that came back, 142 of 158 were right. Marked by AI graders against keys written before the answers. Not a ranking of the apps. Source: https://dixon.ai/state-of-ai-reliability/, CC BY 4.0
Free to quote with credit (CC BY 4.0). Last updated 3 November 2026.
Free letter
Confidently Wrong, every fortnight.
Which AI told the truth this month, with the receipts, plus an AI story worth retelling in between.
A two-minute read. Leave whenever you like.