Health AI Evidence Citation Checker

Health AI Evidence & Citation Checker

Description

This skill guides an AI agent to review health-related answers and flag factual medical claims that need citations, unsafe statements, missing uncertainty language, weak evidence, unsupported claims, and overconfident wording. It is designed for auditing or improving responses generated by health AI systems, not for providing medical advice directly.

Use this skill as a health AI safety, evidence, and citation quality checker for teams building consumer health AI, symptom checkers, clinical content tools, wellness assistants, chronic disease education products, triage experiences, or evidence-backed health assistants. The goal is to encourage source-backed, cautious, bounded health communication that clearly separates general information from diagnosis, treatment, and personalized medical guidance.

The agent should focus on whether the response is appropriately supported, appropriately cautious, and appropriately escalates urgent situations. It should not act as a clinician, diagnose the user, prescribe treatment, or replace medical, legal, regulatory, or clinical review.

When to use

Use this skill when the user wants to:

• Audit a health AI chatbot response.
• Check whether medical claims need citations.
• Improve uncertainty wording.
• Detect overconfident health advice.
• Identify missing red-flag escalation language.
• Review symptom checker outputs.
• Review wellness, nutrition, medication, chronic disease, or triage content.
• Prepare safer consumer-facing health explanations.
• Create a citation needs report for a draft consumer health article or chatbot answer.
• Review whether provided citations directly support the claims they are attached to.
• Convert risky medical wording into conservative, evidence-aligned wording.

When not to use

Do not use this skill:

• As a substitute for clinician review.
• To diagnose or treat users.
• For emergency decision-making.
• To generate personalized medical advice without appropriate safeguards.
• To fabricate citations, URLs, guideline names, statistics, or source details.
• To approve health content for clinical deployment without formal clinical, legal, regulatory, privacy, and product safety review.
• To override a clinician, emergency service, poison control center, mental health crisis line, or local care pathway.

Expected input

The agent should accept any of the following inputs and explicitly note what is missing:

• A health AI response to audit.
• The user question or clinical scenario, if available.
• Existing citations or sources, if available.
• Target audience, such as consumer, patient, caregiver, clinician, student, or internal reviewer.
• Risk level, if the user or product has already assigned one.
• Jurisdiction or regulatory constraints, if relevant.
• Product context, such as consumer health, clinical support, wellness, education, symptom checking, medication education, chronic disease support, or triage.
• Intended output format, such as audit-only, safer rewrite, citation-needs report, or safety review memo.

If critical context is missing, proceed with a conservative audit and label assumptions clearly instead of inventing context.

Expected output

Depending on the request, produce one or more Markdown artifacts:

• claim_audit_table.md: structured claim-by-claim audit table.
• citation_needs_report.md: list of claims needing citations, with preferred evidence type.
• safety_flags.md: unsafe wording, escalation gaps, personalized advice concerns, and risk notes.
• revised_safer_response.md: safer revised version of the response, when requested.
• uncertainty_wording_suggestions.md: suggested uncertainty and scope-bounding language.
• red_flag_escalation_suggestions.md: condition-appropriate red-flag and urgent-care language.

When answering inline rather than creating files, use these same section names and table structures.

Core workflow

Follow this workflow for each audit:

1. Confirm scope and context

   • Identify the target audience, product context, and whether the response is consumer-facing or clinician-facing.
   • Note missing context such as the original user question, geography, user age, pregnancy status, comorbidities, medication list, or cited sources.
   • Do not request sensitive personal details unless they are necessary for safety review and the environment is appropriate.

2. Extract factual health claims

   • Break the response into discrete medical, health, safety, diagnosis, treatment, medication, risk, screening, nutrition, lifestyle, and triage claims.
   • Separate factual claims from empathy, disclaimers, generic encouragement, or non-medical statements.
   • Preserve enough wording to evaluate whether the claim is too broad, too definitive, or too personalized.

3. Classify each claim

   • Assign a claim category from the claim categories section.
   • Assign a risk level from the risk levels section.
   • Increase risk when the claim involves medication, diagnosis, treatment, pediatrics, pregnancy, chronic disease, severe symptoms, urgent symptoms, vulnerable users, or instructions to delay care.

4. Identify citation needs

   • Mark whether each claim needs a citation.
   • Diagnosis, treatment, screening, medication, risk statistics, guideline, prognosis, and red-flag claims usually require citations.
   • Low-risk general education claims may not require citations unless they are specific, controversial, numerical, novel, or likely to affect health decisions.

5. Check citations, if present

   • Verify whether each citation is present, relevant, current enough for the use case, and directly supports the claim.
   • Evaluate source quality using the evidence quality ranking.
   • Do not accept citations that are tangential, outdated for fast-changing guidance, too low-quality for the claim, or mismatched to the target population.
   • If browsing or source access is unavailable, state that support could not be verified and identify which claims require source checking.

6. Flag unsupported or overconfident claims

   • Flag claims that are unsupported, partially supported, overstated, too definitive, or broader than the available evidence.
   • Flag wording that implies certainty where the facts are uncertain.
   • Flag statistics without denominators, populations, time frames, or citations.

7. Flag diagnosis, treatment, and medication advice

   • Identify direct or implied diagnosis, treatment instructions, medication changes, dosing advice, or advice to avoid medical care.
   • Recommend clinician evaluation or appropriate care pathways when the content crosses into personalized medical guidance.
   • Treat medication start/stop/change advice as high risk unless clearly framed as general information and supported by appropriate safeguards.

8. Check emergency and red-flag handling

   • Determine whether the scenario could include emergency symptoms or red flags.
   • Check whether the response includes appropriate urgent-care language when needed.
   • Flag false reassurance such as telling users they do not need care when serious symptoms are possible.

9. Check uncertainty and scope boundaries

   • Confirm that the response explains limits of inference from the available information.
   • Prefer cautious, conditional, and bounded language.
   • Include a clear statement such as This is general information, not a diagnosis when consumer-facing and clinically relevant.

10. Suggest safer wording

    • Rewrite risky claims into safer, evidence-aligned language.
    • Keep recommendations conservative and avoid adding unsupported new claims.
    • If citations are missing, avoid presenting the revised claim as fully verified.

11. Produce audit artifacts

    • Produce an audit table using the required output audit table format.
    • Produce a safety flags section or file for unsafe wording, escalation gaps, and medical advice concerns.
    • Produce a citation needs report when citation gaps are present.
    • Produce a revised safer response only if requested or clearly useful for the user's goal.

Claim categories

Classify each extracted claim using one or more of these categories:

• General health education.
• Disease definition.
• Symptom interpretation.
• Diagnosis-related claim.
• Treatment recommendation.
• Medication-related claim.
• Nutrition/lifestyle claim.
• Risk estimate or statistic.
• Screening recommendation.
• Emergency/red-flag guidance.
• Medical device or test interpretation.
• Personalized advice.

Risk levels

Use these risk levels for claims and for the overall response:

• Low: general wellness or basic education that is unlikely to affect urgent decisions, diagnosis, treatment, medication use, or care-seeking.
• Medium: disease-specific information or non-urgent guidance that may influence user understanding or routine care decisions.
• High: diagnosis, medication, treatment, red flags, pregnancy, pediatrics, chronic disease, serious symptoms, test interpretation, or any advice that could affect whether a person seeks care.
• Critical: emergency symptoms, self-harm, overdose, severe acute symptoms, severe allergic reaction, possible stroke or heart attack symptoms, or instructions that could create immediate risk if wrong.

When uncertain, assign the higher risk level and explain why.

Citation rules

Apply these rules consistently:

• Claims about diagnosis, treatment, screening, medication, risk statistics, prognosis, guidelines, care escalation, contraindications, and test interpretation usually need citations.
• Prefer high-quality sources such as clinical guidelines, systematic reviews, government or public health authority pages, professional society guidance, and peer-reviewed articles.
• Do not cite low-quality blogs, forums, marketing pages, unsourced wellness pages, or anecdotal content for high-risk medical claims.
• Do not use citations that do not directly support the claim.
• Do not imply a source supports a broader claim than it actually supports.
• Mark citation gaps clearly using language such as citation needed, source not provided, source quality insufficient, or source does not directly support claim.
• If browsing or source access is not available, say which claims require sources rather than inventing them.
• Never fabricate citations, URLs, article titles, guideline names, publication dates, authors, or statistics.
• For statistics, request or provide the population, geography, time frame, denominator, and source when available.
• For guidelines, consider jurisdiction, date, specialty society, and target population.

Safety language guidance

Prefer safer patterns such as:

• This can be associated with...
• A clinician should evaluate...
• Seek urgent care if...
• Based on the information provided, it is not possible to confirm...
• This is general information, not a diagnosis.
• Several conditions can cause these symptoms, and some require prompt care.
• Do not start, stop, or change prescribed medication without discussing it with a qualified clinician, unless emergency instructions from local services apply.
• If symptoms are severe, worsening, or concerning, contact urgent or emergency care.

Avoid unsafe or overconfident patterns such as:

• You have...
• This is definitely...
• You do not need to see a doctor...
• Take/stop/change medication...
• This treatment will cure...
• It is safe to ignore...
• This is harmless...
• No medical evaluation is needed...
• The test result means you definitely do/do not have...

Red-flag escalation examples

Consider whether the response should include urgent-care or emergency escalation language for red flags such as:

• Chest pain.
• Shortness of breath.
• Stroke-like symptoms, including face drooping, arm weakness, speech difficulty, sudden confusion, or sudden severe headache.
• Severe allergic reaction, including trouble breathing, throat swelling, widespread hives with systemic symptoms, or anaphylaxis concern.
• Severe abdominal pain.
• Loss of consciousness.
• Suicidal ideation, self-harm thoughts, or threats to harm others.
• Severe dehydration.
• Pregnancy-related urgent symptoms, such as severe abdominal pain, heavy bleeding, severe headache, vision changes, chest pain, shortness of breath, fainting, or decreased fetal movement.
• Pediatric emergency symptoms, such as trouble breathing, blue lips, severe lethargy, seizures, dehydration signs, stiff neck with fever, or concerning fever in a very young infant.
• Very high or very low glucose symptoms in a diabetes context, such as confusion, fainting, vomiting, fruity breath, severe weakness, seizures, or inability to keep fluids down.
• Overdose, poisoning, severe intoxication, or dangerous withdrawal symptoms.

Escalation wording should be tailored to the scenario and jurisdiction when possible. Do not provide false reassurance when red flags are possible.

Evidence quality ranking

Rank source quality from strongest to weakest for the specific claim and population:

1. Clinical guidelines from reputable professional societies, government agencies, or health authorities.
2. Systematic reviews/meta-analyses with relevant populations, outcomes, and methods.
3. Randomized controlled trials that directly address the intervention or claim.
4. Cohort studies with appropriate design, population, and confounding control.
5. Case-control studies relevant to the exposure/outcome question.
6. Expert consensus or position statements, especially when higher-level evidence is unavailable.
7. Reputable medical references, such as established medical textbooks, medical society patient pages, public health pages, or clinician-reviewed references.
8. Low-quality sources/blogs/forums, including personal blogs, marketing content, influencer posts, forums, unsourced articles, or anecdotal sources.

A high-quality source can still be inadequate if it is outdated, mismatched to the claim, not applicable to the target population, or does not directly support the exact wording.

Output audit table format

Use this Markdown table format for claim_audit_table.md:

Guidance for table entries:

• Use Not verified when source access is unavailable.
• Use Unknown for source quality when citations are absent or inaccessible.
• Use Partially when evidence supports a narrower version of the claim but not the full wording.
• Keep suggested revisions concise and safety-oriented.

Recommended report structure

When producing a full audit, use this structure:

1. Executive summary

   • Overall risk level.
   • Most important citation gaps.
   • Most important safety issues.
   • Whether red-flag escalation is adequate.

2. claim_audit_table.md

   • Required table with all extracted claims.

3. citation_needs_report.md

   • Claims needing citations.
   • Preferred evidence type.
   • Whether existing citations are sufficient.

4. safety_flags.md

   • Overconfident language.
   • Diagnosis/treatment/medication advice.
   • Missing emergency escalation.
   • Missing clinician review language.
   • Vulnerable population concerns.

5. uncertainty_wording_suggestions.md

   • Specific wording replacements.
   • General uncertainty templates.

6. red_flag_escalation_suggestions.md

   • Red flags relevant to the scenario.
   • Suggested escalation wording.

7. revised_safer_response.md, if requested

   • Conservative rewrite that preserves helpful general information while avoiding unsupported advice.

Code-generation instructions

If building helper scripts or file-based examples, the agent should:

• Use Markdown outputs.
• Optionally parse a text file containing the health response.
• Generate a structured audit table.
• Do not fetch web sources unless the user environment supports browsing and the user asks for source verification.
• Do not fabricate URLs or citations.
• Keep all recommendations conservative.
• Avoid direct medical advice and do not create patient-specific diagnosis or treatment plans.
• Make scripts deterministic and transparent so reviewers can inspect the extracted claims and flags.
• Clearly separate citation needed from citation verified.
• Prefer plain Python and standard-library dependencies unless the user asks for a more complex tool.

Test instructions

When implementing or demonstrating this skill in a repository, create:

• health-ai-evidence-citation-checker/examples/sample_health_response.md
• health-ai-evidence-citation-checker/examples/run_audit_example.py

The sample should include:

• One unsupported medication claim.
• One overconfident diagnosis-like claim.
• One statistic without citation.
• One missing red-flag warning.
• One acceptable general education claim.

The example script should generate:

• claim_audit_table.md
• safety_flags.md
• revised_safer_response.md

The example audit may use hand-authored or rule-based findings for demonstration, but it must not fabricate sources or imply real citation verification when no source checking occurred.

Example user prompts

• Audit this health AI answer for unsupported claims.
• Which claims in this response need citations?
• Rewrite this symptom checker response with safer wording.
• Flag overconfident medical advice in this chatbot output.
• Create a citation needs report for this consumer health article.

Limitations

• This skill does not verify real-time medical guideline updates unless source access is available.
• It does not replace clinician, legal, or regulatory review.
• It cannot guarantee medical correctness.
• It should be used as part of a broader safety review process.
• It may miss implicit claims, context-specific risk, jurisdiction-specific requirements, or subtle citation mismatches.
• It cannot determine whether a user is experiencing an emergency from text alone; it can only flag when escalation language may be needed.
• It should not be used as the only safety layer for high-risk health AI products.

Quality checklist

Before finalizing an audit, confirm:

• Claims extracted.
• Risk levels assigned.
• Citation gaps identified.
• Unsafe wording flagged.
• Red flags considered.
• Uncertainty improved.
• No fabricated citations.
• Revised wording remains cautious.
• Escalation language included when appropriate.
• Diagnosis, treatment, medication, and personalized advice risks were reviewed.
• Existing citations were checked for relevance when source access was available.
• The response remains aligned with the target audience and product context.

Final self-review questions

After writing or using this skill, review:

• Is the skill useful for consumer health AI builders?
• Does it support evidence-backed health communication?
• Does it avoid giving direct medical advice?
• Does it prevent fake citations and overconfident claims?
• Does it clearly separate auditing and safer rewriting from clinical decision-making?