The judgement gap: Are we thinking with AI, or just agreeing with it? 

Something big may have just happened in the behavioural science space. And if you work with AI, or advise organisations that do, it matters.

Most of us are familiar with Kahneman’s dual-process theory. System 1: fast, intuitive, automatic. System 2: slow, deliberate, analytical. It’s one of the most influential frameworks in behavioural science, shaping how we think about decision-making. It’s rarely been seriously challenged.

However, a new paper from Wharton thinks it might need an update.

Meet System 3

In 1980, philosopher John Searle asked us to imagine a person locked in a room, following rules to manipulate symbols in a language they don’t understand, producing perfectly correct outputs with no comprehension of their meaning. His Chinese Room thought experiment was a challenge to the idea that computers could ever truly “think”.

What he didn’t anticipate was that decades later, we’d be outsourcing our own thinking to that room.

Researchers Shaw and Nave argue that in an age where AI is embedded in how we think and decide, there’s a third system at play. System 3: artificial cognition. And unlike Systems 1 and 2, it doesn’t live in your head. It lives in the cloud.

 

However, system 3 doesn’t just assist our thinking; it can quietly replace it too.

They call this cognitive surrender, the tendency to adopt AI outputs without critical evaluation, not because we’ve weighed them up and agreed, but because we’ve stopped thinking altogether. It’s distinct from simply using AI as a tool, it’s the moment the tool becomes the decision-maker.

And they measured it – in three experiments across 1,372 participants, people consulted the AI on more than half of all trials, and among those who did, followed its advice 80% of the time, even when the AI was deliberately programmed to give the wrong answer.

This isn’t just a lab finding

We’re seeing the real-world version of this play out already.

The Microsoft and LinkedIn 2024 Work Trend Index, drawn from 31,000 workers across 31 countries, found that three in four knowledge workers now use AI at work, with email drafting among the most common applications. Tenders, proposals, client communications, and outputs that once required genuine deliberation are increasingly being generated, lightly edited, and sent. Harvard Business Review has since coined a term for the result: “workslop”, AI-generated content that looks polished but lacks the thinking behind it.

The Wharton paper also found that people with higher trust in AI and lower need for critical thinking were most susceptible, which raises an uncomfortable question for organisations rolling out AI tools at scale.

Why this matters for organisations

At Behave, we’ve written about the motivation gap, the proficiency gap, and the ethics gap holding back meaningful AI adoption. Cognitive surrender suggests there’s a fourth gap we haven’t named yet: the judgement gap.

It’s not about whether people use AI. It’s about whether they retain the capacity and the habit of evaluating what it tells them.

The Wharton findings offer a cautious note of optimism here. Cognitive surrender was reduced, though not eliminated, when people were given incentives to get things right and immediate feedback on their answers. In other words, the conditions that activate System 2 thinking can push back against surrender.

For organisations, that translates into some practical questions worth asking:

• • Are our AI tools designed to encourage verification?

• • Do our workflows create moments of deliberate pause, or do they reward speed above all else?

• • Are we building AI literacy that includes knowing when not to defer?

The bottom line

We don’t just use AI. Increasingly, we think with it. The Wharton paper argues this is a structural shift in human cognition, and the early evidence, both in the lab and in the real world, suggests they may be right.

Whether “Tri-System Theory” becomes a lasting framework depends on how well it generalises beyond the lab. But the core finding, that AI can make us more confident and less accurate at the same time, is one that organisations and leaders would be unwise to ignore.

The question isn’t whether your people are using AI. It’s whether they’re still thinking when they do.