GLM 5.2 Matches Human Bookkeeper Accuracy on UK VAT Returns - With Some Caveats

Last updated: August 14, 2026

Update (August 14, 2026): This benchmark used GLM 5.2. GLM-5.3 launched today - same base model with scaled-up post-training and selectable reasoning levels, at the same price. The VAT results below have not been rerun on 5.3; treat them as a floor for the model line rather than the current ceiling.

A benchmark published by Toot Books today showed GLM 5.2 preparing quarterly VAT returns for a UK small business with near-human accuracy - and at a fraction of the cost. The Hacker News discussion hit 132 points and 79 comments, with the conversation quickly pivoting from "wow this works" to "but who goes to prison when it doesn't?"

The benchmark is one of the most concrete demonstrations yet of LLMs performing structured financial compliance work. But the details matter more than the headline.

What the benchmark actually tested#

The setup: GLM 5.2 ran on an isolated Google Cloud instance with access to accounting software and a command-line tool. It received bank feeds, receipt PDFs, and two user notes providing context - the same inputs a human bookkeeper would receive.

The task: process 59 transactions and produce a quarterly VAT return.

The numbers:

The model achieved this cost efficiency partly because 93% of prompt tokens hit the provider's cache at reduced rates.

What the model handled well:

• Correct account classification for standard transactions
• Invoice matching to bank entries
• Disambiguating complex scenarios: splits, transfers, duplicate entries

What the model got wrong:

The benchmark documented 20 failures across 18 transactions. The most serious:

1. Misclassified founder capital: A $10,000 founder share capital entry was logged as "Capital Account" instead of "Unpaid Shares" - a distinction with potential legal audit implications
2. VAT category confusion: 14 instances of mixing up zero-rated versus exempt VAT categories
3. Split-transaction VAT errors: 3 cases of incorrect VAT allocation on split entries

The benchmark authors acknowledge a key scope limitation: "The job performed by the humans was broader than what was requested of the model. Humans also had to find the relevant invoices (searching through mailboxes, or requesting them from providers) and reason through circumstances which cannot be inferred from the bank feed and invoices alone."

In other words: the model got the easy version of the task.

What HN is saying#

The Hacker News discussion focused less on whether the tech works and more on what happens when it does not.

The liability question dominated. As one commenter put it: "This is a prime example of a problem space where accuracy matters, but it also matters who ultimately goes to prison. I'm going to go out on a limb and guess it's not the LLM."

The distinction matters. If you hire an accountant and they commit fraud, your liability is limited to some extent - you acted in good faith by engaging a professional. If your LLM decides to commit tax fraud, you are in uncharted legal territory.

The "nearly as accurate as a human" framing drew pushback. One commenter noted: "Humans aren't exactly known for perfect recall" - implying that human bookkeepers make mistakes too, so matching their error rate is not necessarily impressive. Another referenced the classic "60 percent of the time, it works every time" line.

But practitioners were already doing this. Several commenters shared that they are actively using Claude Code, DeepSeek, and other models for bookkeeping in production:

• One commenter uses Claude Code with FreeAgent to match PDFs to invoices and handle VAT
• Another built beansync, a "vibe-coded deepseek bookkeeping" system that parses emails, extracts numbers, and correlates transactions
• A third uses Claude Code with Opus to keep beancount ledgers up to date from Mercury bank feeds

The trust question remained unresolved. "I'd be scared shitless to even try something like this," one commenter wrote, noting that the company behind the benchmark has minimal public presence - "just a company Vineyard Finance LTD that was incorporated last year."

Where this actually matters#

The benchmark makes a compelling case that bookkeeping as pure classification work is largely solved. Take a bank feed, match it to invoices, assign categories, calculate VAT - this is pattern matching with well-defined rules. LLMs are good at this.

But the benchmark also reveals the limits:

Edge cases require domain expertise. The founder capital misclassification would not be caught by someone reviewing outputs casually. You need to know that "Capital Account" and "Unpaid Shares" have different legal meanings.

VAT rules are surprisingly complex. Zero-rated versus exempt is not obvious from the transaction itself - it depends on the nature of the goods or services and the specific regulatory category. The model confused these 14 times out of 59 transactions.

The human loop matters. Every commenter using LLMs for bookkeeping mentioned review steps. The model generates candidates; a human approves. This is not autonomous bookkeeping - it is assisted data entry with smart defaults.

The cost math#

The cost comparison is dramatic on its face: $2.73 versus $1,000-2,800 per quarter. But that comparison elides several factors:

• The human accountant also does invoice retrieval, which the model did not
• The human accountant takes liability for errors
• The human accountant knows when to escalate unusual situations

If you factor in a human review step, the LLM approach still wins on cost - but the margin narrows. You are not eliminating the accountant; you are giving them a first draft that is usually right.

For small businesses with simple books, this might be transformative. For businesses with complex VAT situations, cross-border transactions, or audit risk, the human accountant is not going away.

The bigger picture#

This benchmark is part of a broader pattern: LLMs getting good enough at structured compliance work that the question shifts from "can it do this" to "should it."

The technical capability is clear. GLM 5.2 processed 59 transactions with a 7-pence error on net position. That is better than many humans would do on their first pass.

The harder questions are institutional:

• Who is liable when AI-prepared returns contain errors?
• How do you audit AI-assisted financial records?
• What happens when HMRC (or the IRS) starts using AI to audit everyone?

As one commenter put it: "It's not hard to imagine tax authorities using AI to audit everyone's tax returns every year."

The asymmetry is notable: if the tax authority uses AI to catch errors, and you used AI to make errors, the human in the middle is you.

FAQ#

How accurate was GLM 5.2 at preparing a UK VAT return?#

In the Toot Books benchmark, GLM 5.2 processed 59 transactions in 68 minutes and produced a quarterly VAT return whose net position was off by 7 pence (about 10 cents) against the human accountant's ground truth.

What mistakes did GLM 5.2 make in the bookkeeping benchmark?#

The benchmark documented 20 failures across 18 transactions. The most serious were a $10,000 founder share capital entry logged as "Capital Account" instead of "Unpaid Shares", 14 instances of mixing up zero-rated and exempt VAT categories, and 3 cases of incorrect VAT allocation on split entries.

How much did the AI bookkeeping run cost compared to an accountant?#

The run cost $2.73, partly because 93% of prompt tokens hit the provider's cache, versus typical accounting fees of $1,000-2,800 per quarter. That comparison leaves out invoice retrieval, liability and escalation, which the human accountant also covers, so the margin narrows once you add a human review step.

Can an LLM replace a bookkeeper for VAT returns?#

Not yet. The model got the easier version of the task, edge cases still need domain expertise, and every HN commenter using LLMs for bookkeeping described a review step. The pattern is assisted data entry with smart defaults, where the model drafts and a human approves.

Continue Reading#

• Cheap subagents are better when their work is visible
• The DD Stack Cookbook: Five Recipes That Compose
• Deep Research Agents Need Constraint Ledgers

Sources#

• Toot Books GLM 5.2 VAT Benchmark - full methodology and results
• Hacker News discussion - 79 comments as of this writing
• Digits AI vs Human Bookkeeper Benchmark - referenced in discussion