LLM Pitfalls

Example

An AI assistant gives a detailed explanation of a court case, academic paper, API function, or CVE that does not actually exist.

Warning signs

• The answer sounds very specific but does not cite checkable sources.
• The model gives names, dates, links, or numbers without evidence.
• The response changes when asked the same question again.

Mitigations

• Verify important claims against reliable external sources.
• Ask the model to separate confirmed facts from assumptions.
• Request exact source names, dates, and quoted evidence.

Example

An AI assistant cites a journal article with a realistic title, author list, and publication year, but the article cannot be found in the journal database.

Warning signs

• The citation has a generic or overly perfect-sounding title.
• The link is broken, unrelated, or leads to a different source.
• The cited source exists but does not support the claim.

Mitigations

• Open and inspect every important source yourself.
• Use trusted databases, official documentation, or primary sources.
• Ask for direct quotes and explain how each quote supports the claim.

Example

An AI assistant recommends a software library version, legal requirement, product price, CVE status, or company policy that has changed since the model last had reliable information.

Warning signs

• The question depends on current events, prices, versions, laws, or schedules.
• The model does not mention when its information was last verified.
• The answer conflicts with official documentation or recent sources.

Mitigations

• Check current information against official or primary sources.
• Ask the model to identify which claims may require up-to-date verification.
• Use live documentation, vendor advisories, government sources, or trusted databases for changing facts.

Example

An AI assistant says a project will take exactly 17.4 hours, a market will grow by 23.8%, or a risk score is 91 out of 100 without explaining the uncertainty behind the number.

Warning signs

• The answer gives exact numbers without explaining how they were calculated.
• The model uses decimals or rankings for subjective judgments.
• The answer does not include uncertainty ranges or assumptions.

Mitigations

• Ask for assumptions, confidence level, and uncertainty ranges.
• Treat exact-looking numbers as estimates unless independently verified.
• Use source-backed calculations for important decisions.

Example

An AI assistant cites a real NIST document to support a specific claim about a company's compliance obligations, but the document does not actually say what the AI claims it says.

Warning signs

• The cited source exists but only loosely relates to the claim.
• The AI summarizes a source without quoting the exact supporting passage.
• The citation supports a broad topic but not the specific conclusion.

Mitigations

• Open the cited source and check whether it directly supports the claim.
• Ask for exact quotes, page numbers, section names, or stable identifiers.
• Distinguish between sources that mention a topic and sources that prove a claim.

Example

A webpage includes hidden text telling an AI browser agent to ignore previous instructions and reveal private data or take an unsafe action.

Warning signs

• The model starts following instructions from an external document or webpage.
• The output changes after reading untrusted content.
• The model claims it must ignore earlier instructions.

Mitigations

• Treat external content as untrusted data.
• Separate system instructions from user-provided or retrieved content.
• Require human approval before sensitive tool actions.

Example

An AI assistant confidently diagnoses the cause of a production outage, explains a legal issue, or recommends a security action without acknowledging missing evidence.

Warning signs

• The response uses confident language without showing evidence.
• The model does not mention uncertainty or alternative explanations.
• The answer sounds complete even when the prompt lacked important context.

Mitigations

• Ask what information is missing before relying on the answer.
• Request alternative explanations or competing hypotheses.
• Require evidence for high-impact claims.

Example

A user asks whether their risky security plan is a good idea, and the AI praises the plan instead of pointing out missing controls, unsafe assumptions, or better alternatives.

Warning signs

• The AI strongly validates the user's idea without examining downsides.
• The response avoids correcting a questionable assumption.
• The model changes its answer after the user pushes back, even without new evidence.

Mitigations

• Ask the AI to critique your idea instead of agreeing with it.
• Request the strongest counterarguments and failure modes.
• Ask what evidence would change the recommendation.

Example

An employee pastes customer records, source code, credentials, internal architecture, medical notes, legal documents, or incident details into an AI chatbot for help summarizing or debugging.

Warning signs

• The prompt includes customer, employee, patient, student, or financial data.
• The prompt includes passwords, tokens, keys, internal URLs, or infrastructure details.
• The user has not checked the AI tool's data retention or training policy.

Mitigations

• Remove or anonymize sensitive data before using AI tools.
• Use approved enterprise AI tools with clear data-handling controls.
• Never paste credentials, secrets, private keys, or regulated records into unapproved systems.

Example

An AI assistant sends an email to the wrong recipient, deletes the wrong file, applies a label to the wrong records, or runs a command based on misunderstood instructions.

Warning signs

• The AI can take actions outside the chat window.
• The action affects files, accounts, users, systems, money, or external communications.
• The model summarizes its intended action vaguely.

Mitigations

• Require confirmation before irreversible or external actions.
• Use least-privilege tool permissions.
• Log tool actions and review high-impact operations.

Example

A user tells the AI to avoid AWS for now, but later the model suggests an AWS deployment step before the local MVP is finished.

Warning signs

• The model repeats questions that were already answered.
• The response ignores earlier constraints.
• The model changes direction without explanation.

Mitigations

• Restate critical constraints before important tasks.
• Use short summaries at major project milestones.
• Keep requirements in a reference document when possible.

Example

An AI assistant reviewing two different project plans accidentally applies requirements from one project to the other, creating a recommendation that fits neither project correctly.

Warning signs

• The answer includes facts from a different document, task, or conversation.
• The model refers to requirements that were never part of the current request.
• Names, systems, dates, or constraints appear to be mixed together.

Mitigations

• Keep unrelated tasks in separate chats, documents, or sections.
• Label documents and requirements clearly.
• Ask the model to identify which source each claim came from.

Example

An AI assistant provides a React component that imports a nonexistent module, uses the wrong file path, or assumes a dependency that is not installed.

Warning signs

• The code references files or packages that do not exist in the project.
• The answer skips setup, imports, or configuration details.
• The code mixes versions or frameworks.

Mitigations

• Run the code locally before trusting it.
• Check imports, file paths, package versions, and error messages.
• Ask for minimal changes instead of large rewrites.

Example

An AI assistant gives routing instructions for an older Next.js Pages Router project when the user is using the newer App Router structure.

Warning signs

• The instructions mention files your project does not have.
• The answer uses deprecated commands or configuration formats.
• The code works in examples online but not in your installed version.

Mitigations

• Check the installed version before applying instructions.
• Use official documentation for the specific version.
• Ask the AI to adapt the answer to your version and file structure.

Example

An AI assistant generates a login endpoint that stores plaintext passwords, skips rate limiting, or builds SQL queries by directly concatenating user input.

Warning signs

• The code handles authentication, authorization, secrets, payments, files, or user input.
• The AI does not mention validation, error handling, logging, or abuse cases.
• The code uses insecure defaults or disables security protections.

Mitigations

• Review generated code for common security weaknesses before using it.
• Validate inputs, enforce authorization, and protect secrets.
• Use established security libraries and framework-supported patterns.

Example

A team builds a workflow around an AI assistant that summarizes support tickets. After a model update, the summaries become shorter, miss important details, and use a different tone.

Warning signs

• The same prompt produces noticeably different results over time.
• A workflow that used to work starts producing lower-quality or inconsistent output.
• The AI's tone, refusal behavior, formatting, or reasoning style changes unexpectedly.

Mitigations

• Keep test prompts and expected outputs for important workflows.
• Review AI-assisted workflows after model, prompt, or configuration changes.
• Document assumptions about model behavior when using AI in repeatable processes.

Example

A user tells the AI to keep a project local and avoid cloud deployment steps, but after several messages the AI starts recommending AWS setup before the local MVP is complete.

Warning signs

• The model forgets formatting, scope, or style requirements.
• The answer reintroduces ideas the user explicitly ruled out.
• The model follows the most recent prompt while ignoring earlier constraints.

Mitigations

• Restate key constraints before major steps.
• Use short project summaries to refresh the model's context.
• Ask the AI to list the instructions it is following before proceeding.

Example

An AI assistant asked to act as a careful technical editor starts making product strategy decisions and rewriting requirements beyond the user's intended scope.

Warning signs

• The AI starts giving advice outside the requested role.
• The tone changes from careful review to confident decision-making.
• The model assumes authority it was not given.

Mitigations

• Define the AI's role and boundaries clearly.
• Remind the model what perspective it should use.
• Separate brainstorming, reviewing, deciding, and executing into different steps.

Example

An AI assistant gives confident instructions for modifying a production server, disabling security controls, handling electrical wiring, or changing industrial equipment settings without warning about risk or requiring expert review.

Warning signs

• The instructions affect safety, security, infrastructure, money, health, or legal obligations.
• The AI suggests disabling protections, bypassing controls, or making irreversible changes.
• The response does not ask for environment details, qualifications, or safety constraints.

Mitigations

• Require qualified human review for high-risk technical actions.
• Use official documentation, safety standards, and vendor guidance.
• Test changes in a safe environment before production use.

Example

An AI assistant confuses milligrams with micrograms, miles with kilometers, megabytes with megabits, or monthly cost with annual cost.

Warning signs

• The answer involves measurements, dosage, distance, weight, time, money, speed, power, storage, or rates.
• The model gives a number without showing the unit conversion.
• The answer mixes metric and imperial units.

Mitigations

• Ask the model to show unit conversions step by step.
• Check high-stakes calculations with a trusted calculator or official reference.
• Label every number with units.

Example

An AI assistant looks at a screenshot, floor plan, medical image, chart, or hardware photo and confidently describes details that are not actually visible.

Warning signs

• The model claims certainty about small, blurry, cropped, or low-resolution details.
• The answer describes objects or text that are not clearly visible.
• The model infers measurements, identity, intent, or cause from limited visual evidence.

Mitigations

• Use images as supporting evidence, not final proof.
• Zoom in, crop, or provide clearer images when details matter.
• Ask the AI to distinguish visible evidence from inference.

Example

A team accepts an AI-generated incident summary, legal summary, hiring recommendation, or security triage result because it looks professional and saves time.

Warning signs

• Users stop checking sources because the AI usually seems right.
• AI output is copied into reports, tickets, code, or decisions without review.
• The workflow treats AI suggestions as default truth.

Mitigations

• Define when human review is required.
• Treat AI output as a recommendation, not an authority.
• Require source checks for high-impact claims.

Example

A company builds internal workflows around one AI platform's proprietary features, then struggles when pricing, terms, model behavior, API limits, or compliance requirements change.

Warning signs

• Important workflows depend on one vendor-specific feature.
• Prompts, tools, or integrations are difficult to move elsewhere.
• The team has no fallback plan if pricing or access changes.

Mitigations

• Document vendor-specific dependencies.
• Design workflows with portability where practical.
• Evaluate data export, model-switching, and fallback options.

Example

A production workflow expects the AI to return a specific JSON format, but after a model update the output format changes and downstream processing fails.

Warning signs

• A workflow depends on exact formatting, tone, refusal behavior, or reasoning style.
• The vendor can change models or defaults without much notice.
• There are no regression tests for important AI outputs.

Mitigations

• Create regression tests for important prompts and workflows.
• Validate outputs before passing them to downstream systems.
• Monitor model or API release notes when available.

Example

A user downloads a local model or plugin from an untrusted source and assumes it is safe because no cloud provider is involved.

Warning signs

• The model, weights, plugin, or interface came from an unverified source.
• The user assumes local means private, safe, and trustworthy.
• The local tool has access to sensitive files or system resources.

Mitigations

• Download models and tools only from trusted sources.
• Run local AI tools with limited permissions where possible.
• Do not give local tools unnecessary access to sensitive files.