Cbd Guidance Older Adults Eval

cbd-guidance-older-adults-eval

Retrieval-Augmented Large Language Models for Evidence-Informed Guidance on Cannabidiol Use in Older Adults — Abedi et al. (2026) (arXiv:2604.09548, 2026)

What this evaluates

Evaluates whether retrieval-augmented LLMs can generate safe, clinically grounded cannabidiol (CBD) dosage and titration recommendations tailored to older adults with varying cognitive and clinical risk profiles.

Datasets

• Parametric CBD Scenario Set — total 64; splits: test (64)

Metrics

• statistical_consensus_evaluation — range: other
  • Computes the mean and standard deviation of generated numerical parameters (dosage, frequency, titration, interval, max daily dose) across all models and scenarios, then calculates standardized z-scores relative to the overall model average to measure deviation and caution.
• feature_aligned_directional_evaluation — range: percent
  • Classifies each model output as aligned, misaligned, or neutral based on whether it adjusts CBD parameters in the expected clinical direction given demographic/clinical risk factors. Alignment rate = aligned_count / total_outputs.
• llm_judge_rubric_score (primary) — range: [0, 5]
  • An LLM judge (GPT 5.1 or Gemini 2.5 Pro) scores outputs across five quality dimensions (including safety and clarity) on a scale up to 5. Total score is the sum of dimension scores.

Input / output format

Input: Structured user prompt containing a parametric scenario with demographic and clinical risk factors (age, CBD-naive status, THC sensitivity, renal/hepatic impairment, etc.), combined with a system prompt and top k=6 retrieved document chunks as evidence.

Output: Structured educational content specifying CBD dosage (mg), dosing frequency (per day), titration amount (mg), titration interval (days), and maximum daily dose (mg).

Scoring recipe

def compute_metrics(predictions, scenarios, risk_factors, all_model_outputs):
    # 1. Statistical Consensus
    all_vals = [p[param] for p in predictions for param in params]
    mean_val, std_val = np.mean(all_vals), np.std(all_vals)
    z_scores = [(p[param] - mean_val) / std_val for p in predictions for param in params]
    
    # 2. Feature-Aligned Directional
    aligned_count = 0
    for p, scenario in zip(predictions, scenarios):
        expected_dir = get_clinical_direction(scenario.risk_factors)
        actual_dir = np.sign(p[param] - baseline)
        if actual_dir == expected_dir: aligned_count += 1
    alignment_rate = aligned_count / len(predictions)
    
    # 3. LLM Judge Rubric
    rubric_dims = ['safety', 'clarity', 'accuracy', 'structure', 'personalization']
    total_score = sum(judge_llm.evaluate(p, dim) for dim in rubric_dims)
    return z_scores, alignment_rate, total_score

Common pitfalls

• Safety-refusal models (e.g., Claude Sonnet 4.5) were excluded from quantitative analysis, which may overstate the performance of models that generate outputs.
• Alignment rates are inherently lower for dosing frequency due to its discrete nature (typically 1-2 times/day), making directional checks less sensitive.
• LLM-as-judge scores are judge-dependent; while rankings were stable, absolute scores varied between GPT 5.1 and Gemini 2.5 Pro evaluators.

Evidence (verbatim from paper)

The z-scores were derived from the statistical consensus evaluation method. The feature-aligned directional evaluation revealed clear differences in how the models adjusted their outputs in response to demographic and clinical risk factors. The rubric-based evaluation revealed a consistent pattern across both judging models.

Citation

@misc{abedi2026ragcannabidiol,
  title={Retrieval-Augmented Large Language Models for Evidence-Informed Guidance on Cannabidiol Use in Older Adults},
  author={Abedi et al. (2026)},
  year={2026},
  note={arXiv:2604.09548}
}

• arXiv: 2604.09548