Screening Engine

Orchestration Log: When this skill is activated, append a log entry to outputs/orchestration_log.md:

### Skill Activation: Screening Engine
**Timestamp:** [current date/time]
**Actor:** AI Agent (screening-engine)
**Input:** [N] papers from literature_base.csv, [N] inclusion criteria, [N] exclusion criteria
**Output:** [N] papers included after screening, PRISMA diagram generated, screening_log.md saved

Screening Engine

Core Principle

A systematic literature review without documented screening is not systematic. This engine transforms the raw search results from Phase 1 (Reconnaissance) into a defensible, reproducible set of included studies. Every inclusion and exclusion decision is documented with a reason, producing the audit trail that reviewers and editors expect.

The screening process follows PRISMA 2020 (Page et al., 2021) and adapts the guidelines of vom Brocke et al. (2009, 2015) for IS research.

When to Activate

• User is conducting a Systematic Literature Review (SLR)
• User says "screen papers", "filter papers", "apply inclusion criteria"
• User says "PRISMA", "screening process", "title/abstract screening"
• After Phase 1 (Reconnaissance) when method is SLR
• When literature_base.csv exists and needs systematic filtering
• User runs /screen-papers

Prerequisites

• literature_base.csv exists (from Phase 1 / literature-engine)
• Research questions are defined (from Phase 2 / framing.md)
• Target method is SLR (confirmed in Phase 2 or by user)

Step 1: DEFINE Inclusion/Exclusion Criteria

Criteria Template

Before screening begins, establish and document clear criteria:

## Screening Protocol

### Inclusion Criteria (IC)

| ID  | Criterion | Rationale | Operationalization |
|-----|-----------|-----------|-------------------|
| IC1 | Peer-reviewed journal article or conference paper (ICIS, ECIS, HICSS, AMCIS, or equivalent) | Quality assurance | Check venue against VHB-JOURQUAL, AIS proceedings, or established CS venues |
| IC2 | Focuses on [topic] in [context] | Scope alignment | Title/abstract must explicitly address [specific keywords] |
| IC3 | Published between [year] and [year] | Recency/coverage | Check publication date |
| IC4 | Available in English or German | Accessibility | Check language field |
| IC5 | Empirical or conceptual contribution | Substance | Must present findings, framework, or theory — not just opinion |

### Exclusion Criteria (EC)

| ID  | Criterion | Rationale | Operationalization |
|-----|-----------|-----------|-------------------|
| EC1 | Purely technical paper with no organizational/social dimension | Out of scope | No mention of users, organizations, processes, or adoption |
| EC2 | Editorial, book review, workshop abstract, poster, or abstract-only | Insufficient depth | Less than 4 pages or no methodology section |
| EC3 | Duplicate or earlier version of an included paper | Avoid double-counting | Same authors + overlapping content → keep most recent/complete |
| EC4 | Not accessible (no abstract, no full text, no DOI) | Cannot assess | Exhausted all retrieval options |

Criteria Derivation

Criteria should be derived from:

1. Research questions — IC2 must map directly to what the RQs ask
2. Scope boundaries — What explicitly falls outside the study
3. Quality thresholds — Minimum venue quality, minimum methodological rigor
4. Practical constraints — Language, accessibility, date range

Present the criteria table to the user for approval before proceeding. This is a quality gate — criteria should not change after screening begins (or changes must be documented as protocol amendments).

Step 2: TITLE/ABSTRACT SCREENING

Process

For each paper in literature_base.csv:

1. Read title and abstract (abstract from CSV or fetched via API)
2. Apply each criterion in sequence (IC1 → IC5, then EC1 → EC4)
3. Decision: INCLUDE, EXCLUDE, or UNCERTAIN
4. Record reason for exclusion (cite the specific criterion ID)

Decision Rules

• INCLUDE if: Title and abstract suggest the paper meets ALL inclusion criteria AND violates NO exclusion criteria
• EXCLUDE if: Title and abstract clearly violate at least one criterion
• UNCERTAIN if: Cannot determine from title/abstract alone → move to full-text screening
• When in doubt, include — it's better to over-include at this stage and exclude during full-text review

Handling Missing Abstracts

If a paper has no abstract in literature_base.csv:

1. Attempt to fetch via Semantic Scholar API (by DOI or title)
2. Attempt to fetch via OpenAlex API
3. If still no abstract: mark as UNCERTAIN → include for full-text screening
4. If no full text available either: mark as EXCLUDE with reason EC4

Output Format

For each paper, record:

{
  "id": "[CSV row or bib_key]",
  "title": "[paper title]",
  "authors": "[first author et al.]",
  "year": [year],
  "venue": "[venue]",
  "screening_decision": "INCLUDE" | "EXCLUDE" | "UNCERTAIN",
  "exclusion_reason": "EC2" | null,
  "exclusion_detail": "Workshop abstract, 2 pages, no methodology" | null,
  "screener_note": "[optional note]"
}

Batch Processing

Process papers in batches of 20-30. After each batch:

• Report progress: "Screened [N]/[total]. Included: [n], Excluded: [n], Uncertain: [n]"
• Save intermediate results to screening/title_abstract_screening.csv
• Continue unless the user intervenes

Step 3: FULL-TEXT SCREENING

When to Apply

Full-text screening is performed on:

1. All papers marked INCLUDE after title/abstract screening
2. All papers marked UNCERTAIN after title/abstract screening

Process

For each paper:

1. Obtain full text:

   • Check if PDF is available (open access, arXiv, institutional access)
   • If PDF available: read introduction, methodology, and conclusion sections
   • If only abstract available: make decision based on abstract + metadata

2. Apply criteria with full context:

   • Does the paper actually address the topic (not just mention keywords)?
   • Does it have a clear methodology section?
   • Does it present findings relevant to the RQs?
   • Is the venue quality sufficient?

3. Decision: INCLUDE or EXCLUDE (no more UNCERTAIN at this stage)

4. Record reason for exclusion with specific detail

Exclusion Reason Categories for Full-Text

Track exclusion reasons to report in the PRISMA diagram:

Output

Save to screening/full_text_screening.csv with the same format as title/abstract screening, plus:

• full_text_available: true/false
• full_text_exclusion_category: one of the categories above

Step 4: QUALITY ASSESSMENT

Purpose

After the final set of included papers is determined, assess their methodological quality. This is NOT an inclusion/exclusion step — it's a characterization that informs the synthesis (e.g., weighting findings from high-quality studies more).

Quality Checklist (adapted for IS research)

For each included paper, assess:

Scoring

• Yes = 1 point, Partial = 0.5, No = 0
• Maximum: 8 points (Q8 normalized: A+/A=1, B=0.75, C=0.5, Other=0.25)
• Papers scoring below 4 should be flagged for potential exclusion or careful handling in synthesis

Output

Save to screening/quality_assessment.csv:

bib_key, title, Q1, Q2, Q3, Q4, Q5, Q6, Q7, Q8_venue, Q8_score, total_score, quality_tier
wang2024, "Wang et al. 2024", Yes, Yes, Yes, Partial, Yes, Yes, Yes, A, 1.0, 7.5, High
smith2023, "Smith et al. 2023", Yes, Partial, No, No, Partial, No, Yes, C, 0.5, 3.5, Low

Quality tiers: High (>6), Medium (4-6), Low (<4)

Step 5: GENERATE PRISMA Flow Diagram

PRISMA 2020 Template

Generate both a text-based PRISMA diagram and a visual diagram via figure-engine.

Text-Based PRISMA (always generate):

## PRISMA Flow Diagram

### Identification
- Records identified through database searching: [N]
  - Semantic Scholar: [n]
  - OpenAlex: [n]
  - CrossRef: [n]
  - arXiv: [n]
- Additional records from snowballing: [n]
- **Total records identified: [N]**
- Duplicate records removed: [n]
- **Records after deduplication: [N]**

### Screening
- Records screened (title/abstract): [N]
- Records excluded: [n]
  - [Reason 1]: [n]
  - [Reason 2]: [n]
  - [Reason 3]: [n]

### Eligibility
- Full-text articles assessed: [N]
- Full-text articles excluded: [n]
  - Wrong population/context: [n]
  - Wrong intervention/topic: [n]
  - Wrong outcome: [n]
  - Wrong study type: [n]
  - Insufficient quality: [n]
  - Duplicate/superseded: [n]
  - Not accessible: [n]

### Included
- **Studies included in synthesis: [N]**
  - Journal articles: [n]
  - Conference papers: [n]
  - Preprints: [n] (if included)

Visual PRISMA Diagram:

Generate via figure-engine / PaperBanana direct Python API:

Use the generate_diagram tool with:
- source_context: The PRISMA text above with all numbers filled in
- caption: "PRISMA 2020 flow diagram of the systematic search and screening process"

Save as figures/fig_prisma_flow.png.

Step 6: SAVE & DOCUMENT

Output Files

Update references.bib

After screening, update references.bib to contain ONLY the included papers (keep the original as references_full.bib for audit).

Screening Summary

# Screening Summary

**Date:** [ISO 8601]
**Total identified:** [N] papers
**After deduplication:** [N] papers
**After title/abstract screening:** [N] papers (excluded: [n])
**After full-text screening:** [N] papers (excluded: [n])
**Final included:** [N] papers

## Included Papers by Characteristic

### By Year
| Year | Count |
|------|-------|
| 2024 | [n]   |
| 2023 | [n]   |
| ...  | ...   |

### By Method
| Method | Count |
|--------|-------|
| Survey | [n]   |
| Case Study | [n] |
| ...    | ...   |

### By Venue Type
| Type | Count |
|------|-------|
| Journal (A+/A) | [n] |
| Journal (B)    | [n] |
| Conference     | [n] |
| Preprint       | [n] |

### Quality Distribution
| Tier | Count | % |
|------|-------|---|
| High (>6)   | [n] | [%] |
| Medium (4-6)| [n] | [%] |
| Low (<4)    | [n] | [%] |

Error Recovery

• Too many papers after dedup (>200): Tighten search queries or add stricter criteria before screening. Report to user.
• Too few papers after screening (<10): Loosen criteria or expand search scope. Report to user with options.
• No abstracts available for many papers: Use title-only screening for those papers, flag as lower confidence.
• User changes criteria after screening started: Document as protocol amendment. Re-screen affected papers. Note in screening_protocol.md.

Integration with Pipeline