Day 4 - Limitations and Risks of Generative AI: Hallucinations, Bias, Non-determinism, and Appropriate Use Cases

Generative AI is remarkable, but also risky. It confidently tells lies, incorporates biases from its training data into answers, and gives different answers to the same question each time. Deploying this without understanding in critical areas like healthcare, finance, or law costs both individuals and companies dearly.

The AIF-C01 exam doesn't just ask "generative AI is cool." It constantly asks "where should it be used and where shouldn't it be" and "how do we reduce risks?" Today we'll understand the three major risks of generative AI—hallucination, bias, and non-determinism—and develop the eye to distinguish appropriate use cases from a Responsible AI perspective.

Hallucination: Confidently Making Things Up

Hallucination is the phenomenon where a model generates false content as if it were true, presenting it plausibly. It cites papers that don't exist, makes up fake statistics, or invents legal clauses that don't exist.

Why does this happen? As we saw on Day 1, the essence of an LLM is "predicting plausible next tokens," not "verifying truth." The model produces answers based on "naturalness," not "factuality." So rather than saying it doesn't know something, it tends to fill in plausible content.

Main methods to reduce hallucination:

📚 Case study: In 2023, an American lawyer cited cases generated by ChatGPT in a court filing and was embarrassed when several cited cases didn't exist—they were fakes the model made up. The lawyer faced sanctions. The lesson is clear: generative AI output is a "draft," not "verified fact." In areas where fact-checking is important, human review and source verification are mandatory.

💡 Related theory: Hallucination isn't a bug—it's an essential characteristic of generative models. The model merely samples plausible outputs from a probability distribution and doesn't have built-in capability to verify truth against the outside world. So the realistic goal is not "eliminate hallucinations completely" but "control hallucination's impact." Tying evidence through RAG, having humans review, and assessing risk appropriately for the domain—that's responsible design.

Bias: Learning Society's Prejudice from Data

Bias is when a model produces unfair or discriminatory output regarding gender, race, age, region, etc. Models learn from human-created data like the internet, which contains social prejudices that models absorb directly.

Example: Generating female images for "nurse" and male images for "engineer" more often, or showing negative tone toward specific groups.

Bias is dangerous when used in decisions that affect people's opportunities—like hiring, loan approval, or recommendations. In these areas, biased AI automates and amplifies discrimination.

Ways to address bias:

• Ensure data diversity and balance: Training/evaluation data shouldn't skew toward specific groups.
• Fairness evaluation: Measure performance and outcomes by group to spot gaps.
• Use AWS tools: Amazon SageMaker Clarify measures data and model bias and provides explainability.
• Human oversight and governance: Mandate human review for high-risk decisions.

🔍 Going deeper: AWS provides several capabilities to support responsible AI. Amazon Bedrock Guardrails filters harmful/sensitive content, masks sensitive information (PII), and limits permitted topics. SageMaker Clarify computes bias metrics before and after training plus feature importance. AWS AI Service Cards publicly document an AI service's intended use cases, limitations, and fairness considerations. On the AIF exam, when "measuring bias / blocking harmful content / transparency" appears, connect them to Clarify / Guardrails / AI Service Cards respectively.

Non-determinism: Different Answers Each Time

Non-determinism is the property where the same input can produce different outputs. It's due to the sampling we saw on Day 2 (temperature, Top-P, etc.). Since the model "draws" tokens from a probability distribution each time, the same prompt can select different words.

To reduce non-determinism, lower temperature (close to 0) and narrow Top-P/Top-K. But complete determinism isn't guaranteed, so for areas where 100% reproducibility is essential, it's safer not to rely on generative AI alone.

⚠️ Trap: It's easy to assume "setting temperature to 0 means always the same answer." In practice, even with the same model and settings, infrastructure, version, and internal implementation can cause subtle variations. So for scenarios where "determinism is critical," it's safer not to use generative AI output directly as final decision but include a validation step or rule-based logic.

Other Risks and Considerations

Beyond the three major risks, there are other risks the AIF exam covers.

• Data privacy: When users put sensitive information (personal data, trade secrets) in prompts, there's exposure risk. Filter/mask inputs (e.g., Bedrock Guardrails).
• Copyright/Intellectual property: Generated content might resemble copyrighted works in training data—legal risk.
• Toxic content: Potential to generate violence, hate, or illegal information. Block with guardrails.
• Prompt injection: Malicious input to manipulate the model.
• Environmental cost: Training/inference on large models uses massive electricity.

💡 Related theory: The umbrella covering all these risks is Responsible AI. Core principles usually break down to Fairness, Explainability, Transparency, Robustness, Privacy & Security, and Governance. AWS concretizes these through SageMaker Clarify (fairness, explainability), Bedrock Guardrails (safety), AI Service Cards (transparency), and more. The AIF exam loves testing these principle-and-tool connections.

Criteria for Distinguishing Appropriate Use Cases

Once you know the risks, you see "where it's safe to use." Core decision criteria:

Good use cases: Marketing copy drafts, code autocomplete, customer service drafts, document summaries, internal knowledge search (RAG).

Cases requiring caution: Medical diagnosis without human hands, automatic loan approval/rejection, legal advice without verification, automated news publishing without fact-checking. These MUST have Human-in-the-loop (final human review) and governance.

Wrapping Up

Today's key insight is clear. Generative AI has essential risks: hallucination (plausible falsehood), bias (data prejudice), and non-determinism (different answers each time), plus privacy, copyright, toxicity, and prompt injection. The framework controlling these is Responsible AI, and AWS supports it with SageMaker Clarify (bias/explainability), Bedrock Guardrails (safety), and AI Service Cards (transparency). You must distinguish appropriate use cases by asking "Can humans review if wrong? Is consistency essential? Is it sensitive/regulated?"

In the next session, we'll review Week 3 completely—concepts of generative AI, how LLMs work, prompt engineering, and risks—all at once.

📝 Practice Questions

Click a choice to reveal the answer and explanation.

Question 1

Which most accurately explains "hallucination" in generative AI?

• AA phenomenon where the model's response speed slows
• BA phenomenon where a model generates false content as if it were true, plausibly
• CA phenomenon where the model refuses to respond
• DA model's capability to convert images to text

Question 2

A hiring AI repeatedly produces results disadvantageous to a specific gender. Which combination of the most related concept and the AWS tool for measuring bias is correct?

• AHallucination - Amazon Bedrock
• BBias - Amazon SageMaker Clarify
• CNon-determinism - Amazon S3
• DLatency - AWS Lambda

Question 3

Which is NOT an appropriate method to reduce hallucination?

• AUse RAG, providing trustworthy documents together
• BInstruct in the prompt "Say 'I don't know' if unsure"
• CHave humans finally review important outputs (Human-in-the-loop)
• DIncrease temperature to its maximum to encourage diverse answers

Question 4

Which of the following is the most critical use case (risky) for applying generative AI?

• AWriting draft marketing copy
• BSummarizing documents, then having humans review
• CAutomatically making final loan approval/rejection decisions without human review
• DCode autocomplete suggestions

Question 5

"Non-determinism" means the same prompt can produce different outputs. What is the most appropriate measure to increase consistency?

• ASet temperature low (close to 0)
• BIncrease context window size
• CAdd more examples unconditionally
• DSwitching to a larger model always makes it deterministic