🗓️ Week 10
Case study: Use of LLMs in legal and sensitive contexts

DS101 – Fundamentals of Data Science

Dr. Ghita Berrada

LSE Data Science Institute

01 Dec 2025

Today’s session: student-led discussion

Format:

Today is different from our usual lectures. You will lead the discussion based on case study readings about LLMs being used in legal proceedings and adjacent sensitive contexts (child protection, social work, healthcare).

Structure:

  1. Brief primer on LLMs and hallucinations (10 min)
  2. Reading time (30-35 min)
  3. Small group discussions (20-25 min)
  4. Full class discussion with themes emerging from your analysis (remainder)

My role:

  • Facilitate the conversation
  • Ask probing questions
  • Connect themes across cases
  • Clarify technical or legal concepts when needed
  • Not to lecture you on what to think

Quick primer: What are LLMs?

Before we start this case, let’s have a quick look at how LLMs work:

Quick Primer: What are LLM Hallucinations?

Watch this brief explanation:

Key points to remember:

  • LLMs predict the next most likely word/token based on patterns
  • They optimize for coherence, not truth
  • They don’t “know” things - they generate plausible-sounding text
  • Hallucinations = confidently stated false information
  • This is fundamental to how LLMs work, not a bug to be fixed

Keep this in mind as you read today’s cases about legal proceedings.

Today’s readings

Read 3-4 articles from different categories. Focus on understanding what happened and why it matters.

UK LEGAL CASES (start here):

  1. The Guardian (June 2025). “High Court tells UK lawyers to urgently stop misuse of AI in legal work”
  2. The Guardian (Oct 2025). “Barrister found to have used AI to prepare for hearing after citing fictitious cases”
  3. gunnercooke llp (May 2025). “AI, Fake Cases, and the Courts: A Cautionary Tale” - Focus on the UK Ayinde case
  4. David Allen Green (May 2025). “A close reading of the ‘AI’ fake cases judgment” - Legal blogger’s detailed analysis

US LEGAL CASES (for comparison):

  1. CalMatters (Sept 2025). “California issues historic fine over lawyer’s ChatGPT fabrications”
  2. NBC News (Oct 2025). “These people ditched lawyers for ChatGPT in court”
  3. David Lat/Original Jurisdiction (Feb 2025). “A Major Law Firm’s ChatGPT Fail” - Morgan & Morgan case

Today’s readings (continued)

CHILD PROTECTION:

  1. The Guardian (Sept 2024). “AI ban ordered after child protection worker used ChatGPT in Victorian court case”

HEALTHCARE (legal-adjacent):

  1. The Guardian (July 2023). “AMA calls for stronger AI regulations after doctors use ChatGPT to write medical notes”

As you read, note:

  • What happened? What went wrong (or right)?
  • Who was affected? What are the stakes?
  • What does this reveal about LLM limitations?
  • What role did human judgment play?

Small group discussion

Within your table groups:

Round 1: Sharing (5-7 min)

  • Briefly summarize the cases you read (focus on UK cases first)
  • What surprised you most?
  • What patterns do you notice across cases?

Round 2: Analysis (8-10 min)

Choose 2-3 of these questions to explore:

  1. Comparing UK and US cases: What’s similar? What’s different in how courts responded?
  2. Professional responsibility: Where did the lawyers/professionals fail? Was it just the AI?
    1. Access to justice: The NBC article discusses people using ChatGPT instead of lawyers. Good or bad?
  1. Child protection/healthcare: How are these contexts different from straightforward legal cases?
  2. Who’s accountable: When things go wrong, who should be held responsible?
  3. Trust and verification: Why did professionals trust AI output without checking?

Round 3: Synthesis (5-7 min)

  • What’s the most important lesson from these cases?
  • What questions do you have for the broader discussion?
  • Choose a spokesperson to share 1-2 key insights with the class

Tip: Jot down your group’s main points to share.

Full class discussion

Each group shares (2-3 min per group)

We’ll explore emerging themes together based on what you raise.

Possible directions:

  • UK vs. US judicial responses
  • Professional responsibility and verification
  • Access to justice vs. quality of justice
  • Privacy and confidentiality (child protection)
  • When should LLMs never be used?
  • Who bears responsibility when AI fails?
  • Regulation and professional guidance

Ground rules:

  • Build on others’ ideas
  • Disagree respectfully
  • Ask genuine questions
  • Draw on evidence from the cases

Discussion prompts

On professional responsibility:

  • The UK High Court said lawyers have a duty to verify AI-generated research. Why did so many fail to do this?
  • Should there be different standards for large firms (Morgan & Morgan) vs. solo practitioners?
  • What about junior barristers without access to legal databases?

On access and equity:

  • If someone can’t afford a lawyer, is using ChatGPT better than nothing?
  • Does AI democratize justice or create new dangers?
  • Who bears the risk when self-represented litigants use AI?

On sensitive contexts:

  • Why was the Victorian child protection case so serious?
  • How is uploading case information to ChatGPT different from other breaches?
  • Where should we draw the line on what data can be shared with LLMs?

On hallucinations:

  • Why do LLMs “hallucinate” plausible-sounding fake cases?
  • Can this problem be solved, or is it fundamental?
  • How do we build systems that can be trusted in high-stakes contexts?

Another case to consider

In light of what you know, what are your opinions on the UAE’s proposal of rewriting its laws with LLMs?

For the Financial Times article on the topic, see here

A Positive Case: Responsible Use

The Guardian (Sept 2023): “Lord Justice Birss promotes ‘jolly useful’ ChatGPT for lawyers”

What happened:

  • Lord Justice Birss (UK Court of Appeal) used ChatGPT to help draft a judgment
  • Called it “jolly useful”
  • Importantly: He verified the output and took full responsibility
  • He used it as an assistant, not a replacement for legal reasoning

Questions:

  • What made this use responsible vs. the problematic cases?
  • What safeguards were in place?
  • Should judges disclose when they use AI?
  • Where’s the line between “tool” and “crutch”?

Key difference: Understanding LLM limitations and maintaining human oversight.

UK Regulatory Response

Bar Council guidance (November 2025):

“The growth of AI tools in the legal sector is inevitable and, as the guidance explains, the best-placed barristers will be those who make the efforts to understand these systems so that they can be used with control and integrity. Any use of AI must be done carefully to safeguard client confidentiality and maintain trust and confidence, privacy, and compliance with applicable laws.” Sam Townsend KC, Chair of the Bar Council, at the launch of the guidance

High Court warning (June 2025):

Dame Victoria Sharp, president of the King’s Bench division and Mr Justice Johnson:

“Such tools [generative AI tools] can produce apparently coherent and plausible responses to prompts, but those coherent and plausible responses may turn out to be entirely incorrect. The responses may make confident assertions that are simply untrue. They may cite sources that do not exist. They may purport to quote passages from a genuine source that do not appear in that source”

Professional consequences:

  • Wasted costs orders
  • Referral to regulators (Bar Standards Board, Solicitors Regulation Authority)
  • Potential striking off
  • Contempt of court charges

Key message: Courts are taking this very seriously and will impose severe penalties.

The scale of the problem

If you want data:

Damien Charlotin’s AI Hallucination Cases Database (as of late 2025):

  • 620 cases worldwide involving fabricated AI content in legal filings
  • 416 in U.S. federal, state, and tribal courts
  • Multiple UK cases (24 to date) emerging in 2023-2025
  • Rate accelerating: From a few cases a month to a few a day in 2025

Why lawyers keep making this mistake:

  1. Time pressure: AI promises quick research
  2. Lack of understanding: Don’t know how LLMs work
  3. Overconfidence: Trust confident-sounding output
  4. Cost pressure: Legal databases (Westlaw, LexisNexis) are expensive
  5. Insufficient training: No formal education on AI limitations

Pattern: Both experienced lawyers and junior barristers make these errors.

Understanding Hallucinations Technically

If you want deeper understanding:

Why LLMs hallucinate:

  • Trained to predict next token based on patterns
  • No built-in fact-checking mechanism
  • Generate what “sounds right” based on statistical patterns
  • Cannot distinguish true from false - only plausible from implausible

Why it’s particularly dangerous in legal contexts:

  • Fake cases have proper structure: citations, judge names, plausible facts
  • They sound authentic - use correct legal language
  • They fit the argument - LLM generates what would support the position
  • Hard to spot without checking original sources

Research findings:

  • Yale researchers (April 2025): Hallucination detection is “fundamentally impossible” if model only trained on correct outputs
  • No amount of training can eliminate this without explicit examples of errors

Further reading (after class):

Why is child protection different?

Victorian case details:

What happened:

  • Child protection worker uploaded identifying information about at-risk children to ChatGPT
  • Used AI to help write court documents
  • Data sent to OpenAI’s servers
  • Potential use in model training

Why this is particularly serious:

  • Vulnerable population: Children at risk of significant harm
  • Legal confidentiality: Strict privacy requirements in child protection
  • Identifying information: Names, circumstances, family details
  • No control: Once uploaded, data potentially stored indefinitely
  • International data flow: Australian data to US company

OVIC’s order:

  • Ban all child protection staff from using ChatGPT until November 2026
  • Block access to ChatGPT, Claude, Gemini, Copilot, etc.
  • Quarterly compliance monitoring
  • Implement technical controls

Key principle: Some contexts require absolute prohibition, not just caution.

Healthcare: a different risk profile

Current use:

  • Medical note-taking and documentation
  • Patient communication drafting
  • Literature summarization
  • Clinical decision support (emerging)

Key risks:

  1. Hallucinated medical information: Fake studies, wrong dosages, incorrect diagnoses (see for example this CNN article)
  2. Patient confidentiality: HIPAA/GDPR violations if patient data uploaded
  3. Bias: Training data may underrepresent certain demographics
  4. Lack of context: Cannot understand nuanced clinical situations
  5. Liability: Who’s responsible if AI gives wrong advice?

Unlike legal cases:

  • Errors may not be immediately obvious (fake cases can be checked)
  • Harm can be irreversible (patient injury/death)
  • Less clear regulatory framework
  • Patients may not know AI was involved

AMA position (2023):

Called for stronger AI regulations after doctors began using ChatGPT for medical notes and patient interactions without proper safeguards.

Common thread with legal: Professionals delegating judgment to systems that cannot bear responsibility.

Comparing UK and US Approaches

UK approach:

  • Centralized guidance: High Court issuing clear warnings
  • Professional bodies: Bar Council, Law Society, SRA providing specific guidance
  • Stern warnings before severe penalties: Courts giving lawyers “one last chance”
  • Referral to regulators: Professional discipline rather than just fines
  • Emphasis on duty: Officers of the court have heightened obligations

US approach:

  • Decentralized: Each court/jurisdiction setting own rules
  • Heavy fines: $5,000-$10,000 individual penalties
  • Public embarrassment: Detailed published opinions naming lawyers
  • Contempt charges: Criminal consequences in some cases
  • Firm-level consequences: Morgan & Morgan threatening to fire lawyers

Common elements:

  • Both emphasize personal responsibility: Signature = full accountability
  • Both require verification: Must check AI output
  • Both see this as serious threat to justice system
  • Neither accepts “I didn’t know” as excuse

Cultural difference: US appears more punitive, UK more regulatory (so far).

Closing Reflections

Individual reflection (2-3 min):

Consider:

  • What’s one assumption these cases challenged for you?
  • If you were advising a lawyer/social worker/healthcare professional, what would you tell them about using LLMs?
  • What question are you leaving with?

Optional sharing:

  • Volunteers can share reflections with the class

Looking ahead:

  • These issues are actively evolving - new cases emerging constantly
  • You’ll encounter them in your careers
  • Your generation will help shape how we govern AI in sensitive contexts
  • Stay curious, stay critical, stay informed

Thank you for your thoughtful engagement today.

Follow-Up Resources

A more detailed commentary slide deck will be shared with you after class, including:

  • Deeper analysis of each case
  • Technical explanations of hallucinations
  • Regulatory frameworks (UK, US, EU)
  • Best practices for responsible AI use
  • Further reading on LLMs in sensitive contexts

For those interested in the technical side:

Tracking projects:

Remember: This is a rapidly evolving field. What’s true today may change tomorrow.

Questions?