Probast Quality Assessment For Prediction Model Studies

Source: https://github.com/aipoch/medical-research-skills

PROBAST Quality Assessment

This skill evaluates the risk of bias in medical prediction model studies using the PROBAST (Prediction model Risk Of Bias ASsessment Tool) framework. It analyzes the full text of a paper across four domains: Participants, Predictors, Outcome, and Analysis.

When to Use

• Use this skill when you need assess bias in medical prediction model studies using probast tool. use when user wants to evaluate the quality or risk of bias of a medical paper (text or pdf) in a reproducible workflow.
• Use this skill when a data analytics task needs a packaged method instead of ad-hoc freeform output.
• Use this skill when the user expects a concrete deliverable, validation step, or file-based result.
• Use this skill when scripts/extract_pdf.py is the most direct path to complete the request.
• Use this skill when you need the probast-quality-assessment for prediction model studies package behavior rather than a generic answer.

Key Features

• Scope-focused workflow aligned to: Assess bias in medical prediction model studies using PROBAST tool. Use when user wants to evaluate the quality or risk of bias of a medical paper (text or PDF).
• Packaged executable path(s): scripts/extract_pdf.py.
• Reference material available in references/ for task-specific guidance.
• Structured execution path designed to keep outputs consistent and reviewable.

Dependencies

• Python: 3.10+. Repository baseline for current packaged skills.
• Third-party packages: not explicitly version-pinned in this skill package. Add pinned versions if this skill needs stricter environment control.

Example Usage

cd "20260316/scientific-skills/Data Analytics/probast-quality-assessment-for-prediction-model-studies"
python -m py_compile scripts/extract_pdf.py
python scripts/extract_pdf.py --help

Example run plan:

1. Confirm the user input, output path, and any required config values.
2. Edit the in-file CONFIG block or documented parameters if the script uses fixed settings.
3. Run python scripts/extract_pdf.py with the validated inputs.
4. Review the generated output and return the final artifact with any assumptions called out.

Implementation Details

See ## Workflow above for related details.

• Execution model: validate the request, choose the packaged workflow, and produce a bounded deliverable.
• Input controls: confirm the source files, scope limits, output format, and acceptance criteria before running any script.
• Primary implementation surface: scripts/extract_pdf.py.
• Reference guidance: references/ contains supporting rules, prompts, or checklists.
• Parameters to clarify first: input path, output path, scope filters, thresholds, and any domain-specific constraints.
• Output discipline: keep results reproducible, identify assumptions explicitly, and avoid undocumented side effects.

Workflow

To perform the assessment, follow this sequence of operations using the prompts defined in references/probast_prompts.md.

Step 1: Extract Metadata

Extract the first author and year from the paper.

• Prompt: Use "1. Metadata Extraction" from references/probast_prompts.md.

Step 2: Assess Risk Domains (Parallel)

Assess the risk of bias for each of the four domains. For each domain, use the corresponding prompt to generate a risk rating (Low/High/Unclear) and detailed reasoning.

• Domain 1 (Participants): Use "2. Domain 1: Participants" from references/probast_prompts.md.
• Domain 2 (Predictors): Use "3. Domain 2: Predictors" from references/probast_prompts.md.
• Domain 3 (Outcome): Use "4. Domain 3: Outcome" from references/probast_prompts.md.
• Domain 4 (Analysis): Use "5. Domain 4: Analysis" from references/probast_prompts.md.

Step 3: Determine Overall Risk

Combine the risk ratings from the four domains to determine the overall risk of bias.

• Input: The risk ratings (Low/High/Unclear) from Domains 1-4.
• Prompt: Use "6. Overall Risk Assessment" from references/probast_prompts.md.

Step 4: Format Output

Generate a final JSON report containing the risk ratings for all domains and the overall assessment.

• Input: The results from Steps 1-3.
• Prompt: Use "7. JSON Extraction" from references/probast_prompts.md.
• Output Format: A JSON object matching the study_risk_of_bias_schema.

Helper Scripts

PDF Text Extraction

When the user provides a PDF file path, use extract_pdf.py to extract the text content before assessment: