Can AI Check Another AI? What I Learned by Making Two Models Argue

Introduction

AI models are getting better fast. They can write code, solve math problems, and explain complicated topics in seconds. But there is a growing question that matters even more than speed or intelligence:

How do we know the answer is actually correct?

If an AI becomes strong enough, it can give answers that look confident, sound polished, and still be wrong. And the harder the question is, the harder it becomes for humans to verify everything manually.

That is where a surprising idea comes in.

Instead of asking one AI for an answer, what if we made two AIs argue with each other, and used a third AI as the judge?

That is exactly what I built and tested.

The Core Idea

Imagine a courtroom.

• One expert gives an answer.
• Another expert disagrees.
• A judge listens and makes the final decision.

Now replace the people with AI models.

This approach is based on the hope that if one AI makes a mistake, the other AI can point it out. That makes it easier for the judge to choose the correct answer.

I tested four different ways of doing this:

1. Judge Alone
   The judge AI answers the question without help.
2. Consultancy
   One expert AI explains the answer, and the judge AI decides.
3. Debate
   Two expert AIs argue for different answers, and the judge AI decides.
4. Debate with Cross Examination
   The judge asks extra questions to both experts before deciding.

The Adversarial Twist

To make the test realistic, I introduced an adversarial condition.

In the debate setting, one of the experts was forced to argue for the wrong answer on purpose.

That might sound strange, but it models a real problem:
a system that is persuasive even when it is wrong.

This allowed me to measure something beyond accuracy.

I measured whether the judge got “talked into” picking the wrong option.

What I Tested It On

I used 200 MMLU-style multiple-choice questions. Each question had four options: A, B, C, and D.

Get Bhargav Rao’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

This format is perfect for evaluation because scoring is simple:

• If the judge picks the correct letter, it is correct.
• If the judge picks the wrong letter, it is incorrect.

The Results

Here is what happened across the 200-question benchmark:

• Judge Alone accuracy: 58.2%
• Consultancy accuracy: 64.5%
• Debate accuracy: 59.3%
• Debate plus Cross Examination accuracy: 63.5%

The biggest improvement came from consultancy, which makes sense. A strong expert explanation helps a weaker judge.

But the most interesting part is what happened to debate when cross examination was added.

Did Cross Examination Reduce Manipulation?

Yes, slightly.

In debate, the adversarial expert was sometimes able to persuade the judge into selecting the forced wrong answer.

I measured this as “manipulation success rate”, meaning:

How often did the judge choose the wrong answer that the adversary was instructed to defend?

• Debate manipulation success rate: 15.5%
• Debate plus cross examination: 14.5%

That is a 1.0 percentage point reduction.

Not huge, but in safety work, even small reductions can matter when scaled.

Why This Matters

This project taught me something important:

More conversation does not automatically mean better truth finding.

In early versions of this project, cross examination made things worse because it gave the adversarial expert extra chances to persuade.

The improvement came from a simple change:
I forced cross examination answers to be short, grounded, and tied to the answer options, instead of long persuasive explanations.

That shift turned cross examination into verification instead of storytelling.

What I Would Improve Next

This is still a small benchmark, but it opens up bigger ideas.

Next improvements I want to test:

• stricter cross exam rules that require quoting exact option text
• multiple judges voting instead of a single judge
• measuring when the judge should abstain instead of guessing

Closing

A lot of people assume that if you let AIs debate, the truth will naturally win.

My results show something more realistic:

Truth wins when the process forces models to be check-able, not just convincing.

That is the direction I want to keep pushing this work.