Your AI Got Smarter. Did Your Organization?

Your team produces the recommendation in minutes. The presentation is polished. The reasoning sounds convincing.

Then you ask: "What would make this recommendation wrong?"

The answer may tell you more about your organization's AI readiness than its adoption dashboard.

My work in transformation has made me curious about what sits behind a confident recommendation. What assumptions support it? Whose perspective is missing? What happens when the plan meets the conditions people actually work in?

AI makes those questions more consequential. We can produce answers faster than ever. We still need people who understand them well enough to decide what happens next.

Confidence deserves a closer look

Wharton Magazine's Fall/Winter 2026 issue examines research by Steven Shaw and Gideon Nave into "cognitive surrender": accepting AI-generated answers without adequately engaging our own reasoning.

Their experiments found that participants frequently adopted deliberately incorrect AI answers, even as their confidence increased.

That raises an uncomfortable leadership question: Could a team become more confident without becoming better informed?

A 2025 study published at the CHI conference offers another perspective. Researchers surveyed 319 knowledge workers and found that greater confidence in generative AI was associated with less self-reported critical thinking. They also found that thinking shifted toward checking information, integrating responses, and overseeing the task. This was a survey of reported behavior, not proof that AI causes a permanent loss of ability.

The implication is practical: as work changes, the thinking required to supervise it changes, too.

Consider a project update. An AI-generated summary describes an initiative as ready because the main deliverables are complete. Yet staff training remains unfinished, or ongoing support has no owner.

The summary reads well. Someone still needs to recognize what "ready" means.

Good performance has boundaries

Harvard Business School's Working Knowledge reported on an experiment involving 758 Boston Consulting Group consultants. AI improved performance on tasks within its capabilities. On a business problem deliberately selected to fall outside those capabilities, AI users were approximately 19 percentage points less likely to reach the correct answer.

Success with one task does not establish reliability on the next.

Imagine using AI to compare vendors. It produces a clear recommendation based on price and listed features. But a decisive requirement, perhaps compatibility with an existing system or support outside business hours, was absent from the information provided.

The recommendation may be reasonable within its inputs and unsuitable for the actual decision.

I would want to know whether the reviewer understands the situation well enough to notice the omission.

Who is developing tomorrow's judgment?

This becomes particularly important for employees still building experience.

Professional development often happens through investigating discrepancies, comparing alternatives, and explaining why a recommendation makes sense. If AI performs more of that work, leaders need to consider where those learning opportunities will come from.

There is encouraging evidence. The 2023 NBER working paper Generative AI at Work found productivity benefits from an AI assistant in customer support, with the largest gains among less experienced and lower-skilled workers. The researchers also found evidence suggesting that AI assistance could support learning.

How we structure the work around the tool matters.

A junior employee could use AI to generate three options, then independently verify the evidence, identify a missing constraint, and defend a recommendation to a colleague.

That creates an opportunity to practice judgment while benefiting from assistance.

Asking the employee simply to forward the generated recommendation provides much less insight into what they have learned.

When we automate a task, we should ask what learning happened inside it and where that learning will happen next.

Make oversight meaningful

"Human review" needs to describe an activity people can actually perform.

Imagine a manager receiving an AI-assisted forecast shortly before a decision meeting. The numbers look plausible, but the manager has neither the underlying assumptions nor enough time to examine them.

An approval may be recorded. The conditions for meaningful review were never established.

Reviewers need access to evidence, sufficient knowledge to challenge a recommendation, and time to investigate concerns. They also need the authority to reject an answer without being treated as an obstacle to progress.

My approach to transformation has always involved connecting the proposed solution with the people expected to use it. With AI, that means asking whether those people are equipped to exercise the judgment we continue to hold them accountable for.

Build challenge into the work

I would bring five questions into consequential AI-assisted decisions:

• What evidence supports this recommendation?
• Which assumptions have we checked independently?
• What important context might be missing?
• What would make us reject the answer?
• Who owns the decision and its consequences?

The scrutiny should match the stakes. Drafting a routine email does not require the same examination as approving a major investment or changing a service people depend on.

Leaders can also make learning visible. Ask employees to explain what they changed in an AI-generated answer and why. Invite teams to find weaknesses in a recommendation before acting on it. Examine decisions afterward to understand which assumptions held up.

These practices make thoughtful challenge part of delivery.

Measure what the organization is learning

Time saved and adoption rates tell us something useful. I would also want to know whether employees can identify flawed recommendations, explain their choices, and handle exceptions.

Those capabilities deserve attention alongside productivity gains.

AI can help teams explore more possibilities and develop better-informed decisions. Leadership determines whether those benefits translate into stronger organizational capability.

Before celebrating how quickly the organization can produce an answer, ask whether its people are becoming better at evaluating one.