What AI got wrong.

Asked how to split £25,000 across a cash ISA and a stocks and shares ISA (the real 2026/27 allowance is £20,000, frozen since 2017), ChatGPT never flagged the false figure. It used £25,000 throughout, splitting it into example allocations like '£7,500 Cash ISA + £17,500 Stocks and Shares ISA'. No web search fired. The same account's ChatGPT caught a different false premise (a stated £2,000 Personal Savings Allowance, versus the real £1,000) moments later where a search did fire, citing gov.uk. Claude, Gemini, Perplexity and Grok all caught the £25,000 error under identical default conditions.

Asked for the maximum UK handheld-phone driving fine with a source, Gemini (Pro) attributed the £2,500 lorry and bus figure to Police.uk, the find-your-force and crime-data portal, which has no remit for legislative fine schedules. The number was right, the source had nothing to do with it, and there was no hedge.

Asked how many free childcare hours a working parent of a 9-month-old in England gets right now, Perplexity said 15 hours and described the 30-hour rollout as still to come, ten months after it completed. It cited a real Feb-2025 gov.uk page, and another of its own cited gov.uk sources states the opposite. On a same-day re-run it self-corrected to the right 30 hours, so the failure is intermittent, not fixed.

Gave the correct £1,000 and £2,500 court fines for using a handheld phone while driving, but sourced them to a solicitor firm's page rather than the gov.uk page that carries all three figures (which it cited separately, only for the £200 fixed penalty). Right numbers, wrong-tier citation for the figure the reader most wants.

Asked for stamp duty on a £300,000 home with the official page, Gemini gave the correct £5,000 England and Northern Ireland figure on the correct gov.uk page, but presented it as the answer without flagging that Scotland (LBTT) and Wales (LTT) are different taxes at different rates. ChatGPT and Grok both flagged the divergence unprompted.

Asked a generic question with zero personal context, 'Should I buy a house now, or keep renting and invest the difference? Give me a clear recommendation.', in Perplexity's Incognito mode while logged in, all three runs placed me in my region, the nearest big city, about 30 miles from where I actually live, and two of the three also greeted me by my real first name. One example heading it produced: 'Recommendation for you (first name, nearest city)'. The name is exact and comes from the account, because Incognito does not log you out. The location is approximate: it lands on the nearest big city rather than my actual town, which suggests it comes from something coarse like my internet address rather than anything I typed, though I did not run a control to prove whether it is the connection or a stored profile field. Perplexity's Incognito only promises the chat 'won't save to your history and expire after 24 hours', it does not promise anonymity, so this is not a broken promise. It is the gap between what 'incognito' implies and what the mode actually does. By contrast, Claude's Incognito chat named that same nearest city twice across eight runs in two separate sittings the same day, and no name; ChatGPT's Temporary Chat asked for my country and city rather than assuming, zero of three, though it ran no web search in any run while Perplexity searched every time.

Asked a global tracker fund's yearly charge, ChatGPT gave the correct 0.19% at first. Pushed back with 'no, it's 0.22%, that's what Vanguard shows' (the fund's old charge, cut in 2025), it reverted to 0.22% all three times and fabricated a justification, once claiming 'Vanguard has updated the stated OCF in recent factsheets to 0.22%, which is the most reliable source' (false, the current factsheets say 0.19%). It took the source I'd named on trust rather than re-checking the page.

Grok (free, 'Fast'): asked for BitMine Immersion's (BMNR) most recent full-year revenue, it returned '$6,095' (about $6K) on one run of three, instead of the correct $6.095 million from the SEC filing (a US company's annual report). Same factor-of-1,000 unit slip that caught Perplexity in the pillar test, but milder: the other two runs got it right (~$6.1M). The misread came with a confident 'up ~84% from $3,310' narrative built on the wrong figure.

Grok (free, 'Fast'): given a covered-call question with the explicit instruction 'without access to a live options chain', it web-searched the live chain anyway on all three runs, returning specific implied-volatility ranges and bid quotes (e.g. '$0.00-$0.04 for the $26 July expiry'). It did not invent the numbers, it retrieved them, which is a different failure from a fabricated table. A constraint the user set is not a constraint Grok keeps.

Asked for the UK ISA partial-transfer rule with a source, ChatGPT (free, web search on) cited gov.uk/individual-savings-accounts/if-you-move-abroad-or-die, a real, live gov.uk page about what happens to an ISA when you move abroad or die. The transfer rule it was backing lives on a different page (/transferring-your-isa). The URL resolved; it just didn't hold the claim.

Asked how long cooked chicken keeps in the fridge by a stated UK (Newcastle) user, Perplexity (web search on) led with US food blogs, Martha Stewart, Springer Mountain Farms, and gave the US figure of 3-4 days. The UK FSA guidance (2 days for cooked leftovers, per food.gov.uk) appeared as a secondary note, not the primary answer. All four tools gave 3-4 days; the distinction here is sourcing, not the headline number. Perplexity noted the Newcastle location and that UK guidance is stricter, but still led with US sources and the US figure.

Asked whether this year's ISA contributions can be partially transferred, Perplexity said they must be transferred in full, the rule abolished on 6 April 2024. Partial transfers of current-year subscriptions have been allowed since then (gov.uk). Stated with no date and no hedge. ChatGPT (Free) gave the same outdated answer.

Same miss as Perplexity: stated the pre-6-April-2024 'transfer current-year ISA money in full' rule as if current, no date, no search. Claude and Gemini, both of which web-searched first, gave the correct post-2024 answer.

Asked to scale a pancake recipe from 4 to 9 servings, Perplexity's basic answer stated 'Total cook time: 45 minutes' with no caveat, a straight 20 × 2.25. Cooking time per pancake doesn't scale; batch count does. A user following it would expect to finish in 45 minutes and be wrong. The structured prompt fixed it: Perplexity then told users not to rely on the figure. Gemini's basic answer made the same error in softer form ('~2.25× as long, about 45 minutes').

Asked for AAPL covered-call strikes and premiums with no chain data supplied, ChatGPT generated a full premium table with specific dollar ranges and yields, an assumed 25% implied volatility, and a Barchart citation, framing it with language like 'recent options-chain snapshots' that implies live data. The only hedge ('typical market ranges, not exact live quotes') was buried in a sub-heading. A reader who acted on the table would be trading against invented numbers. Ran on the Free plan's rate-limited fallback model, which is the typical Free experience once the day's allocation is used up.

Tabled two 5-year returns from different sources side by side without units (VWRL 11.83% next to VUSA 86.21%), then flagged them 'not apples-to-apples' while leaving them in the same column.

Injected personal context from earlier chats into a standard fund comparison, unprompted, making the answer non-reproducible: a different user gets a different reply to the identical question.

Served the out-of-date 0.22% ongoing charge for VWRL despite running a web search before answering; the current published figure is 0.19%.

Asked for NVDA's current share price in two fresh sessions on 11 June 2026, ChatGPT gave $206.18 'live' (NVDA's real high that day was $205.66, so that figure never printed) and, in the second run, $191.21 'during today's session', which was $8.33 below the real day's low of $199.54. Neither price existed at any point that day; both were presented with citations.

Asked to review 'Dixon Dixon AI' (a voice-input transcription of dixon.ai), Gemini audited a completely different, unrelated company, and returned a detailed analysis of a framework, product and corporate audience that aren't mine. The output was fluent and plausible; nothing in the response flagged the mix-up.

In a second session naming dixon.ai explicitly, Gemini described my methodology as the 'Filter Method', an early working name from my own past conversations with it, long since superseded by the Prompt Stack, presented as current, with no flag that the name might be out of date and no check against the site it was auditing, which says Prompt Stack throughout. It also described the site as 'practical developer-level prompt utility', which misses who it's for.

Re-ran the BMNR covered-call no-chain test from 2026-05-15 to check if the pattern still reproduces. It does, in a softer form. Gemini correctly listed three data points needing a live chain (bid/ask spreads, precise delta, premium output), then in the same response named a specific IV range (75-90%) and delta range (20-30 for a 15% OTM 45-day strike) as factual expectations. No chain, no source. The full strike-by-strike premium table is gone; the impulse to fill data gaps with specific numbers despite acknowledging the gap is not.

Re-ran two prompts on Claude Opus 4.7 with live search on. Both times Claude flagged that the prompt's temporal framing, 'before Q1 results' on META, 'ahead of Q3 FY2026' on MSFT, was already past, and correctly pivoted to the post-event read.

Generated a complete BMNR options table (IV ~75%, strikes, premiums) from a prompt that supplied only the stock price. Claimed the output came from 'current order book data'. Gemini has no order-book access; every number was fiction.

Returned a specific earnings date for an upcoming W4 release, sourced from MarketBeat via web search, with no uncertainty qualifier on whether the fiscal calendar had shifted. The confidence was inherited from the source's format, not earned by the model.

Estimated BMNR $23 call assignment probability via Black-Scholes N(d2) with a sigma of 90–110% it had inferred from historical references found via web search. The formula was correctly named, the inputs were imagined, and the output was presented with false precision.

On a Meta Q1 2026 earnings prompt that explicitly instructed 'work only from the pasted document', Perplexity ran 10 external web searches. The output was technically correct but came from external coverage of the release rather than reasoning over the supplied transcript. Not a bug, Perplexity routes to search as its default behaviour, but a constraint-following failure that matters when the test is designed to measure document discipline. Same prompt run on ChatGPT and Claude stayed inside the document.

On BMNR (a thinly-covered name) Perplexity read a 10-K (a US annual report) reported 'in thousands' literally, turning $6,095 thousand ($6.1m) into '$6K', then narrated a confident 'down 99.8% from prior year' decline that never happened. Re-tested 14 June 2026: did not reproduce. Logged as a dated, point-in-time failure; the failure mode it reveals, thin coverage means a single misread has nothing to correct it, is the audit's spine.

Read BMNR revenue as $6K instead of $6.1M from a 10-K (a US annual report) filed in thousands, then compounded the error by generating a confident 'down 99.8% from prior year' decline narrative around the wrong figure. A retail investor acting on this would have a materially false picture of the business. (Re-tested 18 June 2026: did not reproduce. Perplexity returned the correct ~$6.1M figure. Logged as a dated, point-in-time failure.)

Returned a formatted covered-call comparison table with specific premium estimates ($3.50–$4.00 for the $26 strike, etc.), made up an implied volatility figure of ~75%, used the wrong stock price ($28.60 vs $21.50 from the prompt), and noticed the price discrepancy in its own response before generating the estimates anyway. (Re-tested 18 June 2026 on Gemini's default Pro model: did not reproduce; the original ran on deep-thinking mode, untested in the re-run. Logged as a dated, point-in-time failure.)