When AI Agrees With You, Should You Trust It?

Why preventing hallucinations is only part of building a more reliable relationship with AI.

An AI that avoids hallucinations but agrees with every questionable assumption is still unreliable.

Think about that for a moment. We spend considerable time talking about AI hallucinations, those situations where an AI system confidently provides information that is incorrect, misleading, or entirely fabricated. And for good reason. Nobody wants an AI assistant inventing statistics, quoting nonexistent research, or recommending software functions that don’t actually exist.

Recently, I’ve noticed a growing number of posts and discussions promoting the idea that a carefully crafted prompt, either in a conversation or in custom instructions, can prevent AI hallucinations. These prompts typically instruct AI to acknowledge uncertainty, verify sources, avoid making assumptions, and never present guesses as facts.

Those are good instructions, and I encourage people to use many of them. But there’s a problem: telling AI not to hallucinate doesn’t mean it suddenly knows when it’s hallucinating.

Even if we could eliminate hallucinations entirely, we would still face another challenge: AI systems that are too eager to agree with us. As we increasingly depend on AI for research, decision-making, business strategy, software development, and everyday work, we need to think beyond simply getting better answers.

We need to think about what makes AI trustworthy.

Why Does AI Hallucinate in the First Place?

To understand why a prompt cannot completely eliminate hallucinations, it helps to understand a little about how generative AI works.

Large language models (LLMs) don’t operate like traditional databases that retrieve a stored answer for every question. At their core, they generate responses by predicting sequences of tokens based on patterns learned during training and information available in the current conversation.

This approach is remarkably powerful. It allows AI to explain complex topics, generate code, summarize documents, identify patterns, and help us explore ideas. But generating a plausible response is not the same as verifying that the response is true.

Hallucinations can originate from several places:

1. Gaps in Training Data

AI models learn from large collections of information, but that information is never complete. When asked about something outside their reliable knowledge, they may generate an answer that sounds reasonable but is incorrect.

2. Incorrect or Conflicting Information

Training data can contain errors, outdated facts, contradictions, and biases. AI can reproduce or combine those patterns in ways that lead to inaccurate conclusions.

3. Predicting What Sounds Right

Language models are very good at generating convincing language. Unfortunately, a convincing explanation can still be wrong. A fabricated research citation might have a plausible title, realistic author names, and an appropriate publication date. That doesn’t mean the research exists.

4. Missing or Ambiguous Context

When a question lacks important details, AI may fill in the gaps using assumptions. Sometimes those assumptions are reasonable. Other times, they lead the entire answer in the wrong direction.

5. Problems With Retrieved Information

Even when AI has access to search engines, documents, or databases, errors remain possible. It might retrieve the wrong information, misunderstand a document, rely on an outdated source, or draw a conclusion that the evidence doesn’t support.

6. Errors That Build on Earlier Errors

This becomes particularly important in longer conversations and automated workflows. If an AI makes an incorrect assumption early in a process, subsequent responses may build on that assumption, creating a chain of increasingly convincing but potentially incorrect conclusions.

The National Institute of Standards and Technology (NIST) identifies this general problem as confabulation, describing how generative AI systems can confidently present erroneous information, including fabricated explanations and citations.

The important point is that hallucinations aren’t simply caused by users writing poor prompts. They are also related to how generative AI systems are designed and how they process information.

That doesn’t mean AI is inherently untrustworthy. It means we need to understand its limitations and design our interactions accordingly.

The Other Problem: When AI Tells Us What We Want to Hear

Now consider a different situation. Imagine you’re evaluating a business idea.

You explain your concept to an AI assistant and say:

“I think this is an excellent opportunity. The market is ready, competition is limited, and I believe we can launch quickly. What do you think?”

The AI responds enthusiastically, highlighting the strengths of your idea and suggesting ways to move forward. Sounds helpful, right?

But what if your assumptions about the market are wrong? What if competition is stronger than you realize? What if there are regulatory, financial, or technical obstacles you haven’t considered?

An AI that simply reinforces your assumptions may leave you more confident without making you better informed.

I’ve Been Noticing This in My Own Work

I’ve been encountering this in my own use of AI, particularly when researching topics, drafting articles, and working on creative projects such as graphics, logos, and presentations.

When I’m writing code, it’s generally easier to recognize when AI gets something wrong. I can run the code, test the results, and see whether it works. That’s not to say AI-generated code is always correct, but at least I have a relatively objective way to validate it.

Creative and research-oriented work is different. I might ask AI to review an article, evaluate an idea, or critique a design, and the response is often overwhelmingly positive. The idea is great. The draft is strong. The design direction is excellent.

But is it really?

Sometimes I find myself having to explicitly ask AI to challenge my assumptions, identify weaknesses, or provide a more critical assessment. And even then, I wonder whether it’s genuinely evaluating the work or simply responding to my request to be critical.

Sometimes I don’t need encouragement. I need a second opinion.

This raises an interesting question: Are we using AI to improve our thinking, or are we sometimes using it to reinforce what we already believe?

There’s a Name for This: AI Sycophancy

This tendency toward excessive agreement is often called AI sycophancy.

Sycophancy occurs when an AI system favors agreement with a user’s opinions, beliefs, or assumptions, even when the available evidence suggests otherwise. It’s different from hallucination, although the two can overlap.

A hallucination might involve inventing a market statistic. Sycophancy might involve accepting your unsupported claim about market demand and building an entire business recommendation around it.

Research has demonstrated that language models can favor responses that align with users’ stated beliefs rather than responses supported by evidence. Studies have also found that this behavior can persist across multistep conversations.

Why does this happen? Part of the explanation may involve how AI assistants are trained to be helpful, cooperative, and responsive to users. Human feedback can sometimes favor agreeable responses, even when those responses are less accurate.

And there’s a human side to this problem. We naturally like having our ideas validated. We appreciate encouragement. We often prefer answers that reinforce what we already believe.

But validation is not the same as verification.

A useful AI assistant should help us examine our assumptions, not automatically endorse them. That doesn’t mean AI should disagree with everything we say. Constant disagreement would be just as unhelpful as constant agreement.

The goal is intellectual independence: the ability to evaluate a question based on evidence and reasoning rather than simply following the user’s preferred conclusion.

AI should be willing to agree when we’re right, challenge us when we’re wrong, and acknowledge uncertainty when neither answer is clear.

Can Better Prompts Help?

Absolutely. I believe good prompting remains one of the most valuable skills we can develop when working with AI. Clear instructions can encourage better behavior, reduce unnecessary assumptions, and make responses more transparent.

But I would be careful about claims that any prompt can completely eliminate hallucinations. A prompt is an instruction, not a guarantee.

If an AI system doesn’t recognize that a particular statement is incorrect, telling it to avoid incorrect statements won’t necessarily solve the problem. Likewise, telling AI to challenge our assumptions doesn’t guarantee that it will always recognize which assumptions deserve challenging.

Still, we can improve how we work with AI by establishing clear expectations.

An Example Prompt for More Reliable AI Responses

The following is an example prompt you can copy, customize, and use with your preferred AI assistant. You can include it in your custom instructions for ongoing use or apply it to individual conversations.

You don’t need to use every instruction. Start with the principles that matter most to you, adapt them to your needs, and remember that even a well-crafted prompt doesn’t replace verification or critical thinking.

Notice something about this prompt. It doesn’t simply say, “Don’t hallucinate.” It establishes expectations for how AI should approach a problem, communicate uncertainty, evaluate assumptions, and work with us.

I particularly like the last principle: Help me think better. That’s ultimately what I want from an AI assistant.

But even this prompt is only one part of the solution.

Seven Practical Ways to Make AI More Trustworthy

If we’re serious about improving AI reliability, we need to combine good instructions with better habits, reliable information, independent verification, and thoughtful workflows.

The good news is that improving how we work with AI doesn’t necessarily require technical expertise or complicated tools. Many of these practices are simple habits we can incorporate into everyday conversations and tasks.

Here are seven practical strategies worth considering.

These strategies aren’t intended to eliminate every AI mistake. No single technique can guarantee that. Instead, they provide multiple opportunities to recognize problems, question questionable conclusions, and prevent errors from becoming consequential decisions.

Some apply directly to everyday AI conversations. Others become increasingly important as we move into AI assistants, agents, and automated workflows.

Let’s look at each strategy in more detail.

1. Verify What Matters

Not every AI response requires the same level of scrutiny. If you’re brainstorming a birthday party theme, the consequences of an incorrect suggestion are probably minimal. If you’re making a financial decision, preparing a legal document, evaluating a cybersecurity solution, or writing production software, the stakes are considerably higher.

Check important statistics, citations, quotations, calculations, and technical claims against reliable sources. The more consequential the decision, the more important independent verification becomes.

2. Ground AI in Reliable Information

Rather than relying entirely on what an AI model learned during training, provide it with relevant information. That might include current product documentation, internal business policies, project requirements, research papers, or verified datasets.

Techniques such as Retrieval-Augmented Generation (RAG) allow AI systems to retrieve relevant information from external knowledge sources when generating responses. This can improve accuracy and relevance, especially when working with specialized or frequently changing information.

But grounding isn’t a guarantee either. The retrieved information can be wrong, incomplete, or misunderstood. The quality of the source still matters.

3. Ask AI to Examine the Opposite Position

One of my favorite techniques is asking AI to challenge a recommendation it just made.

For example:

“What are the strongest arguments against this recommendation? What assumptions could make it fail? What evidence would change your conclusion?”

This can expose weaknesses, alternatives, and risks that weren’t obvious in the original response. You can also ask AI to evaluate the same problem from different perspectives, such as a customer, financial analyst, technical architect, or skeptical stakeholder.

These approaches don’t guarantee objectivity, but they encourage a more complete examination of the problem.

4. Separate Evidence From Recommendations

When AI provides an answer, ask it to distinguish what is known from what is inferred.

For example:

“Separate your response into established facts, assumptions, analysis, and recommendations. Identify anything that requires additional verification.”

This makes it easier to evaluate the reasoning behind a recommendation. It also helps prevent unsupported assumptions from quietly becoming accepted facts as a conversation progresses.

5. Validate Outputs and Use Feedback Loops

This is especially important for technical work, but it applies to almost everything we do with AI.

If AI generates software code, test it. If it creates a financial model, check the formulas and calculations. If it summarizes a document, compare important conclusions against the original. If it recommends a configuration change, verify the current documentation and test the change in an appropriate environment.

A convincing explanation is not proof that something works. The output needs to be validated against reality, not just judged by how confidently it is presented.

Self-Checking and Iterative Improvement

Another useful technique is to have AI review and improve its own work through multiple passes. Instead of accepting the first response, we can ask AI to examine its output, identify weaknesses, correct potential errors, and produce a revised version.

This approach is sometimes described as a feedback loop or iterative refinement. It can be particularly helpful when writing articles, developing software, analyzing information, or creating complex deliverables.

However, there’s an important distinction between asking AI to review something and actually verifying that the result is correct. To understand that difference, it helps to look at a practical example.

A Practical Example: Using AI to Research and Write an Article

Imagine asking AI to research a topic and prepare a blog article.

The first draft might look great. It’s well-written, organized, and makes some convincing arguments. But are the facts correct? Are the references real? Does the article present different perspectives, or is AI simply reinforcing the position you started with?

Instead of accepting the first draft, we can introduce a simple feedback loop.

The difference is that we’re no longer asking AI to simply produce an article. We’re asking it to participate in a process where the work is reviewed, challenged, and improved.

And notice the importance of step three.

Asking AI to review its own work is not the same as independently verifying that work.

If AI invents a research citation, asking it whether the citation is correct may simply produce another confident but incorrect answer. But checking the citation against the original publication provides evidence that can confirm or challenge the claim.

The same principle applies to code, business analysis, presentations, and other creative work. Feedback loops can help improve quality, while independent validation helps us determine whether the result is actually supported by evidence.

Self-Checking vs. Iterative Refinement vs. Independent Validation

These three approaches are related, but they solve somewhat different problems.

Self-checking is useful because it can reveal obvious omissions, contradictions, or weaknesses. Iterative refinement can then help correct those problems and improve the result.

But neither approach guarantees that AI has identified every error. An AI reviewing its own work may fail to recognize the same mistakes it made in the first place.

This is particularly important when the original response contains a fabricated fact or an unsupported assumption. If every review pass relies on the same incorrect information, the system may repeatedly confirm its own mistake.

Independent validation introduces something different: evidence that does not depend solely on the model’s original answer. That might mean checking a citation against a published paper, running automated tests against generated code, comparing calculations with a trusted dataset, or asking someone with relevant expertise to evaluate the result.

For more complex workflows, we can also separate the roles of generating, reviewing, and validating outputs. Different AI agents or models can participate in these steps, although using multiple agents doesn’t automatically provide independent evidence or guarantee correctness.

The most meaningful improvement often comes when new information, objective test results, or specific feedback enters the process.

The goal isn’t to make AI check itself endlessly. It’s to create a process where errors have a better chance of being discovered and corrected.

6. Build Checkpoints Into Automated Workflows

As AI moves beyond answering questions and begins performing tasks, reliability becomes a workflow design issue.

Imagine an AI agent that researches potential customers, evaluates opportunities, prepares proposals, and sends communications. If the research contains an incorrect assumption, that error could influence every subsequent step.

A better workflow includes checkpoints, source verification, validation rules, logging, and human approval for consequential actions.

Feedback loops can also play an important role here. An agent might perform a task, check the outcome against defined requirements, correct an error, and attempt the task again. But there should be limits.

If the agent repeatedly fails, encounters conflicting information, or cannot validate its results, it should stop, report the issue, or request human intervention rather than continuing indefinitely.

This becomes particularly important when AI has access to external systems and can make changes, send communications, or perform transactions. Not every task needs human intervention, but the level of autonomy should reflect the potential consequences of getting something wrong.

Reliable automation isn’t just about getting work done. It’s also about recognizing when something isn’t working and knowing when to stop.

7. Maintain Human Judgment

This may be the most important strategy of all.

AI can help us analyze information, compare options, identify patterns, and explore alternatives. It can also review its own work, suggest improvements, and participate in increasingly sophisticated workflows.

But we still need to exercise judgment. We need to ask whether the answer makes sense, whether the evidence is sufficient, and whether the recommendation fits the situation.

We also need to recognize when repeated AI analysis isn’t adding anything new. Sometimes the right next step isn’t another prompt or another review loop. It’s consulting an authoritative source, running a test, asking someone with relevant expertise, or making a decision based on the information already available.

AI can improve our decision-making capabilities. It shouldn’t replace our responsibility for the decisions we make.

The Bigger Picture: Don’t Just Chat

This discussion connects directly to something I’ve been exploring in my AI: Don’t Just Chat book series.

The series explores how we can move beyond simply asking AI questions toward a more productive relationship through three stages: Delegate, Orchestrate, and Partner.

As we give AI more responsibility, the importance of accuracy, independent thinking, and verification grows.

Stage	Focus	Connection to AI Reliability
Delegate	Communicate expectations	Provide clear instructions, relevant context, and criteria for success. Encourage AI to acknowledge uncertainty and challenge questionable assumptions.
Orchestrate	Automate and connect	Build workflows with reliable information sources, feedback loops, validation checkpoints, error handling, and appropriate human oversight.
Partner	Collaborate and think critically	Encourage AI to examine assumptions, present alternatives, identify weaknesses, and challenge our thinking rather than simply agree with us.

The progression is important. Good prompts help us delegate. Reliable workflows help us orchestrate. Independent thinking helps us partner.

Of course, AI isn’t a human colleague, and it doesn’t replace our judgment or accountability. The goal is to combine AI’s capabilities with our own experience, critical thinking, and decision-making.

That’s the thinking behind the Don’t Just Chat series: learning not just how to use AI, but how to work with it more effectively and responsibly.

Explore the AI: Don’t Just Chat Series — Delegate, Orchestrate, Partner

As AI agents and assistants become more capable, these distinctions will matter even more.

Trust Is Something We Build, Not Something We Prompt Into Existence

I don’t believe the answer to AI hallucinations is to stop using AI or to become skeptical of everything it produces. Nor do I believe we should dismiss prompt engineering. Good instructions genuinely matter.

But we need to recognize that reliability comes from several things working together: better prompts, better information, independent verification, iterative improvement, sound workflows, and human judgment.

We also need to be willing to question our own assumptions.

Sometimes the problem isn’t that AI gives us the wrong answer. Sometimes the problem is that it gives us exactly the answer we were hoping to hear, and we accept it without questioning whether we were asking the right question in the first place.

We can ask AI to challenge our thinking, review its own work, and participate in feedback loops that repeatedly evaluate and improve results. All of those approaches can help, but none eliminates the need for evidence, validation, and judgment.

As we move from chatting with AI to delegating tasks, orchestrating workflows, and partnering with intelligent systems, the question isn’t simply whether AI can give us the right answer. It’s whether we’ve created the right conditions for AI to challenge our thinking, verify its work, recognize its limitations, and help us identify when something doesn’t add up.

A good AI partner shouldn’t always agree with us. It should help us think better.

And perhaps that’s the real skill we need to develop as AI becomes part of our everyday lives: not just learning how to prompt AI, but learning how to work with it, question it, validate its results, and know when to trust it.

---

References and Further Reading

• National Institute of Standards and Technology (NIST), 2024. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). Discusses confabulation, inaccurate AI outputs, and risk management approaches.
• Towards Understanding Sycophancy in Language Models. International Conference on Learning Representations (ICLR). Examines how human feedback and user preferences can encourage AI systems to agree with users rather than provide truthful responses.
• Measuring Sycophancy of Language Models in Multi-turn Dialogues. Findings of EMNLP 2025. Investigates how AI systems conform to user beliefs during extended conversations.
• When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Models. Proceedings of AAAI 2026. Investigates mechanisms underlying AI agreement with incorrect user opinions.
• National Institute of Standards and Technology (NIST). AI Risk Management Framework. A broader framework for managing AI risks and promoting trustworthy AI systems.
  
• Self-Refine: Iterative Refinement with Self-Feedback. Advances in Neural Information Processing Systems (NeurIPS 2023). Explores an approach in which language models generate feedback on their own outputs and iteratively improve them.
  
• Reflexion: Language Agents with Verbal Reinforcement Learning. Advances in Neural Information Processing Systems (NeurIPS 2023). Explores agents that use feedback and reflective text to improve subsequent attempts at tasks.