Across teams, organizations, and industries, people are starting with AI instead of the problem they want to solve. As a result, AI outputs from different tools:
- Look and sound very similar and less human
- Lack depth and may sound confident, but fail to stand up to scrutiny
This is a process failure, and it's accelerating.
What is human-in-the-loop AI (HITL AI)?
Human-in-the-loop (HITL) AI is a framework that integrates human oversight directly into the machine learning lifecycle. Rather than relying on fully autonomous systems, HITL uses humans and machines collaboratively to train models, evaluate outputs, and handle complex decision-making. This collaborative relationship empowers people to ensure AI work is accurate and safe.
AI doesn't know it's wrong. And now, neither do you.
AI can create a false sense of confidence in the quality of an output because it is produced quickly and appears polished and authoritative, a phenomenon sometimes called ‘AI-induced confidence bias.’ Combined with pressures to deliver results, it creates a disconnect between how good the output seems and how good it actually is.
But the gap is difficult to identify for both AI and the human in the loop. Since AI is ever-evolving, it can be challenging to find a cascading problem stemming from an early misunderstanding.
If you're familiar with the Dunning-Kruger curve, you'll know the valley of despair, which is known for that humbling dip where you know enough to recognize how much you don't know. It's uncomfortable, but it's where real understanding develops. AI has a habit of launching people straight over it, giving them a false sense of security. My advice is to linger there longer than feels comfortable. The discomfort is what brings awareness and learning that helps mitigate the risks of AI-induced confidence bias.
How does AI-induced confidence bias impact the bottom line?
I've seen overreliance on AI firsthand across customers, partners, and colleagues. I've also experienced it myself. Early in my own AI adoption, I spent a day working with AI on a complex problem and came away feeling like I'd accomplished something significant. The output seemed right. It had structure, confidence, and completeness. Then I stress-tested it and realized I'd produced something with the shape of good work, but not the substance. What was missing was the structured thinking I would have applied if AI hadn't made skipping it feel risk-free.
I later realized that what I had casually attempted in a day was something companies build dedicated practices around. Turning what I produced into something production-grade would require investment, methodology, domain expertise, and hard-won institutional knowledge. That’s the gap AI made invisible: I believed I had solved the problem, but I lacked an understanding of what a real solution required. AI made the output appear more complete than it actually was.
Output is not the same as outcomes
AI has made output effortless, but it hasn't made outcomes more likely—and that distinction is where things break down.
AI-built tools often look impressive but were never pressure-tested by anyone other than their creator. Large, complex problems are handed wholesale to AI without decomposition, defined success criteria, or any of the structured thinking that makes a solution succeed. This content is fluent and well-structured, yet lacks a distinct point of view.
The empty praise these tools and outputs receive only compounds the issue. Curiosity and a willingness to experiment is genuinely valuable. But that value disappears when organizations celebrate the attempt rather than the outcome, regardless of whether it works, scales, or solves a real problem. Effort without outcome is a waste of time and tokens.
These issues trace back to the same root cause: the problem to be solved was never properly defined.
A framework for successful outcomes
Based on my own experience, I developed a principle I apply before every meaningful AI engagement: first, understand how a human would solve this problem manually, step by step. If you can't describe the human process, you don't understand the problem well enough to evaluate what AI produces. You can't spot a wrong answer if you don't know what a right answer looks like. You can't set evaluation criteria if you don't know what “good” looks like at each stage.
This blog post puts that framework into practice. It didn't start with a prompt. It started with an observation I couldn't stop thinking about: a stress-test of ideas and a defined point of view before a single word was drafted.
The migration work I do follows the same structure: inventory the artifacts, define the problem and its constraints, make sense of it manually, and define the target outcome. Then, and only then, is when to bring in AI to accelerate execution.
Different domains. Same discipline.
The dashed line is key. Steps 1–4 are human. AI enters at step 5 to pressure test, accelerate, and generate inside a frame you already own. Not before. And AI is entering and being delegated work, not taking it over without human oversight.
Spec-driven development is one expression of this principle. The specification is evidence that you understood the problem and thought through what solving it actually requires. The same logic applies whether you're modernizing a legacy system or writing a blog post.
AI earns its place as an accelerant inside that frame, not as a replacement for it.
AI makes human judgment more important
There's a narrative in circulation that AI reduces the need for deep expertise, that the tool compensates for gaps in knowledge. The opposite is true, because code quality requires deep expertise and AI lowers the barrier to producing code. This means that the stakes are raised significantly on judgment, because now you're evaluating AI output rather than producing your own. You cannot audit what you cannot understand. If you don't know what a good output looks like, you cannot tell when AI gives you something bad.
For leaders, the question is simple: are your teams thinking first or reaching for AI?
Overreliance on AI is a common pattern. The software development industry has been grappling with one version of this pattern long enough to name it vibe coding. Its purest form looks like writing code in an exploratory manner. You converse with AI using natural language, and AI rapidly generates code in iterations. But you may not understand what the code is doing or whether it solves the right problem. AI produces software that looks functional and gets shipped, but fails in ways its builders can't anticipate.
What do the best AI outcomes have in common?
It comes down to what people bring to the technology.
Consider the Hubble telescope. Getting access to it doesn't make you an astronomer. The instrument is extraordinary, but it amplifies what you bring to it. Bring a deep understanding, a clear plan, and structured execution, and it is transformative. Bring a vague idea and a prompt, and you get something that looks like a discovery but isn't.
This is not new thinking for organizations that have navigated complex legacy modernization. Organizations that succeed don't do so because they have better tools. They succeed because they understand the problem deeply, plan deliberately, and execute with discipline, treating each phase as a gate to pass through rather than a formality to skip. AI is a tool successful organizations use within that approach, not a substitute for it.
AI doesn't change what good problem solving requires. Relying solely on AI makes flawed thinking harder to see—and makes the consequences of skipping the problem-solving discipline more visible, more expensive, and harder to unwind. Organizations that recognize that, and build the process muscle to match the technology, are the ones that will actually close the gap between what AI promises and what it delivers.
