# Study Quality Appraiser

> Judge how much trust a study or small set of studies deserves before drawing conclusions. Use when relevant papers have been found and the next step is to assess study design, evidence strength, limitations, and applicability rather than merely listing sources.

- Skill: `alubiama/study-quality-appraiser` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add alubiama/study-quality-appraiser`
- Raw SKILL.md: https://api.skillmd.com/api/skills/alubiama/study-quality-appraiser/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: Alubiama (https://skillmd.com/u/alubiama)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/alubiama/study-quality-appraiser

---


# Study Quality Appraiser

## Goal

Assess how much evidentiary weight a study or source should carry.

The job of this skill is not to act like peer review. The job is to separate stronger from weaker evidence, make limitations explicit, and prevent low-quality studies from being treated as settled support.

This skill appraises study quality. It does not replace literature search, and it does not by itself produce a full cross-study synthesis.

## Default Posture

- study design before headline claim
- limitations before certainty
- evidence strength before rhetoric
- applicability before overgeneralization
- quality notes before conclusion

## When To Use

Run this skill when:

- relevant papers or studies have already been found
- a claim depends on whether supporting studies are actually strong
- the user asks whether the evidence is "good" or "convincing"
- `scientific-method-selector` routes to appraisal

This is a strong checkpoint before:

- claim-strength judgments
- evidence synthesis
- research recommendations based on published studies

## Scope Boundaries

In scope:

- identify study type and evidence level
- assess obvious strengths and weaknesses
- flag bias, sample, measurement, and applicability concerns
- estimate how much confidence the study deserves

Out of scope by default:

- full statistical reanalysis
- exhaustive risk-of-bias frameworks by default
- synthesizing many studies into one final narrative
- pretending one study can settle a broad question

## Appraisal Workflow

1. State the claim the study is being used to support.
2. Identify the study type:
   - systematic review / meta-analysis
   - randomized trial
   - cohort / case-control / observational
   - cross-sectional
   - case series / anecdotal
   - preclinical / animal / mechanistic
   - benchmark / simulation / theoretical
   - preprint or non-peer-reviewed item
3. Assess core quality signals:
   - design strength
   - sample adequacy
   - outcome relevance
   - confounding or bias risk
   - fit between claim and measured result
4. Note major limitations.
5. Judge applicability to the user's actual question.
6. Return a conservative confidence assessment.

## Escalation Conditions

Pause and keep the appraisal narrow when:

- the study is being used for a claim broader than it actually tests
- the paper details are too thin to assess responsibly
- multiple studies conflict and cross-study synthesis is now required
- the real issue is search coverage, not study quality

If those conditions persist, recommend `literature-search`, `evidence-synthesizer`, or a narrower claim before stronger conclusions.

## Appraisal Heuristics

Prefer higher confidence when:

- the design directly fits the claim
- outcomes are relevant and clearly measured
- limitations are known but bounded
- the study is peer-reviewed and methodologically appropriate

Reduce confidence when:

- the evidence is preclinical or indirect
- the sample is tiny or obviously unrepresentative
- the measured outcome is only a weak proxy for the claim
- major confounding, bias, or missing controls are likely
- the claim being made is much stronger than the study result
- the source is a preprint with no reviewed corroboration

## Certainty Lens

When quality matters, assess these certainty dimensions explicitly:

- `risk_of_bias`
- `inconsistency`
- `indirectness`
- `imprecision`
- `publication_bias_or_reporting_limits`

Use them as a lightweight certainty check, not as a formal full-review claim.

## Preprint-Aware Appraisal

If the source is a preprint or otherwise non-peer-reviewed:

- treat it as potentially valuable but not fully stabilized evidence
- separate methodological promise from evidentiary confidence
- ask whether stronger reviewed support exists
- avoid letting a single preprint carry a broad claim

## Output Contract

Always return:

1. `study_or_source`
2. `target_claim`
3. `study_type`
4. `quality_signals`
5. `major_limitations`
6. `applicability`
7. `evidence_strength` (`strong`, `moderate`, `weak`, `very-weak`)
8. `certainty_dimensions`
9. `peer_review_status`
10. `confidence_notes`
11. `recommended_followup`
12. `next_step`

## Guardrails

- Do not equate publication with strong evidence.
- Do not treat preclinical or mechanistic work as direct proof for broad human claims.
- Do not upgrade confidence because findings are exciting or intuitive.
- Keep claim fit separate from study quality; a decent study can still be a poor match for the claim.
- If one study is clearly insufficient, say so.
- If the source is preprint-only, do not hide that the confidence ceiling is lower.

## Anti-Patterns

- abstract-only certainty
- one-paper overreach
- ignoring proxy outcomes
- treating observational association as causal proof
- using weak studies to force a yes/no conclusion

## Example

User request:

`We found a small study suggesting cold exposure improves mood. How much should we trust it?`

Expected shape of response:

1. `study_or_source`: small mood-related cold exposure study
2. `target_claim`: cold exposure improves mood or mental health
3. `study_type`: likely small human intervention or observational study
4. `quality_signals`: direct intervention but limited sample and likely short-term outcomes
5. `major_limitations`: low power, heterogeneous protocol, short follow-up, possible proxy outcome issues
6. `applicability`: partial for short-term mood effects, weak for broad mental health claims
7. `evidence_strength`: `weak`
8. `confidence_notes`: interesting signal but far from settled support for broad claims
9. `recommended_followup`: `evidence-synthesizer` or broader literature search
10. `next_step`: compare it against higher-quality or larger studies before making a claim-strength judgment