Study Quality Appraiser

Study Quality Appraiser

Goal

Assess how much evidentiary weight a study or source should carry.

The job of this skill is not to act like peer review. The job is to separate stronger from weaker evidence, make limitations explicit, and prevent low-quality studies from being treated as settled support.

This skill appraises study quality. It does not replace literature search, and it does not by itself produce a full cross-study synthesis.

Default Posture

• study design before headline claim
• limitations before certainty
• evidence strength before rhetoric
• applicability before overgeneralization
• quality notes before conclusion

When To Use

Run this skill when:

• relevant papers or studies have already been found
• a claim depends on whether supporting studies are actually strong
• the user asks whether the evidence is "good" or "convincing"
• scientific-method-selector routes to appraisal

This is a strong checkpoint before:

• claim-strength judgments
• evidence synthesis
• research recommendations based on published studies

Scope Boundaries

In scope:

• identify study type and evidence level
• assess obvious strengths and weaknesses
• flag bias, sample, measurement, and applicability concerns
• estimate how much confidence the study deserves

Out of scope by default:

• full statistical reanalysis
• exhaustive risk-of-bias frameworks by default
• synthesizing many studies into one final narrative
• pretending one study can settle a broad question

Appraisal Workflow

1. State the claim the study is being used to support.
2. Identify the study type:
   • systematic review / meta-analysis
   • randomized trial
   • cohort / case-control / observational
   • cross-sectional
   • case series / anecdotal
   • preclinical / animal / mechanistic
   • benchmark / simulation / theoretical
   • preprint or non-peer-reviewed item
3. Assess core quality signals:
   • design strength
   • sample adequacy
   • outcome relevance
   • confounding or bias risk
   • fit between claim and measured result
4. Note major limitations.
5. Judge applicability to the user's actual question.
6. Return a conservative confidence assessment.

Escalation Conditions

Pause and keep the appraisal narrow when:

• the study is being used for a claim broader than it actually tests
• the paper details are too thin to assess responsibly
• multiple studies conflict and cross-study synthesis is now required
• the real issue is search coverage, not study quality

If those conditions persist, recommend literature-search, evidence-synthesizer, or a narrower claim before stronger conclusions.

Appraisal Heuristics

Prefer higher confidence when:

• the design directly fits the claim
• outcomes are relevant and clearly measured
• limitations are known but bounded
• the study is peer-reviewed and methodologically appropriate

Reduce confidence when:

• the evidence is preclinical or indirect
• the sample is tiny or obviously unrepresentative
• the measured outcome is only a weak proxy for the claim
• major confounding, bias, or missing controls are likely
• the claim being made is much stronger than the study result
• the source is a preprint with no reviewed corroboration

Certainty Lens

When quality matters, assess these certainty dimensions explicitly:

• risk_of_bias
• inconsistency
• indirectness
• imprecision
• publication_bias_or_reporting_limits

Use them as a lightweight certainty check, not as a formal full-review claim.

Preprint-Aware Appraisal

If the source is a preprint or otherwise non-peer-reviewed:

• treat it as potentially valuable but not fully stabilized evidence
• separate methodological promise from evidentiary confidence
• ask whether stronger reviewed support exists
• avoid letting a single preprint carry a broad claim

Output Contract

Always return:

1. study_or_source
2. target_claim
3. study_type
4. quality_signals
5. major_limitations
6. applicability
7. evidence_strength (strong, moderate, weak, very-weak)
8. certainty_dimensions
9. peer_review_status
10. confidence_notes
11. recommended_followup
12. next_step

Guardrails

• Do not equate publication with strong evidence.
• Do not treat preclinical or mechanistic work as direct proof for broad human claims.
• Do not upgrade confidence because findings are exciting or intuitive.
• Keep claim fit separate from study quality; a decent study can still be a poor match for the claim.
• If one study is clearly insufficient, say so.
• If the source is preprint-only, do not hide that the confidence ceiling is lower.

Anti-Patterns

• abstract-only certainty
• one-paper overreach
• ignoring proxy outcomes
• treating observational association as causal proof
• using weak studies to force a yes/no conclusion

Example

User request:

We found a small study suggesting cold exposure improves mood. How much should we trust it?

Expected shape of response:

1. study_or_source: small mood-related cold exposure study
2. target_claim: cold exposure improves mood or mental health
3. study_type: likely small human intervention or observational study
4. quality_signals: direct intervention but limited sample and likely short-term outcomes
5. major_limitations: low power, heterogeneous protocol, short follow-up, possible proxy outcome issues
6. applicability: partial for short-term mood effects, weak for broad mental health claims
7. evidence_strength: weak
8. confidence_notes: interesting signal but far from settled support for broad claims
9. recommended_followup: evidence-synthesizer or broader literature search
10. next_step: compare it against higher-quality or larger studies before making a claim-strength judgment