Alterlab Pubmed

PubMed Database

Overview

PubMed is the U.S. National Library of Medicine's comprehensive database providing free access to MEDLINE and life sciences literature. Construct advanced queries with Boolean operators, MeSH terms, and field tags, access data programmatically via E-utilities API for systematic reviews and literature analysis.

Scripts

scripts/query_pubmed.py — NCBI E-utilities (ESearch/ESummary/EFetch; stdlib only, JSON to stdout):

python scripts/query_pubmed.py search "crispr[tiab] AND 2024[dp]" --retmax 20   # PMIDs
python scripts/query_pubmed.py summary 39726939 39492484    # metadata for PMIDs
python scripts/query_pubmed.py fetch 39726939               # abstracts (use --api-key for 10 req/s)

When to Use This Skill

This skill should be used when:

• Searching for biomedical or life sciences research articles
• Constructing complex search queries with Boolean operators, field tags, or MeSH terms
• Conducting systematic literature reviews or meta-analyses
• Accessing PubMed data programmatically via the E-utilities API
• Finding articles by specific criteria (author, journal, publication date, article type)
• Retrieving citation information, abstracts, or full-text articles
• Working with PMIDs (PubMed IDs) or DOIs
• Creating automated workflows for literature monitoring or data extraction

Core Capabilities

1. Advanced Search Query Construction

Construct sophisticated PubMed queries using Boolean operators, field tags, and specialized syntax.

Basic Search Strategies:

• Combine concepts with Boolean operators (AND, OR, NOT)
• Use field tags to limit searches to specific record parts
• Employ phrase searching with double quotes for exact matches
• Apply wildcards for term variations
• Use proximity searching for terms within specified distances

Example Queries:

# Recent systematic reviews on diabetes treatment
diabetes mellitus[mh] AND treatment[tiab] AND systematic review[pt] AND 2023:2024[dp]

# Clinical trials comparing two drugs
(metformin[nm] OR insulin[nm]) AND diabetes mellitus, type 2[mh] AND randomized controlled trial[pt]

# Author-specific research
smith ja[au] AND cancer[tiab] AND 2023[dp] AND english[la]

When to consult search_syntax.md:

• Need comprehensive list of available field tags
• Require detailed explanation of search operators
• Constructing complex proximity searches
• Understanding automatic term mapping behavior
• Need specific syntax for date ranges, wildcards, or special characters

Grep pattern for field tags: \[au\]|\[ti\]|\[ab\]|\[mh\]|\[pt\]|\[dp\]

2. MeSH Terms and Controlled Vocabulary

Use Medical Subject Headings (MeSH) for precise, consistent searching across the biomedical literature.

MeSH Searching:

• [mh] tag searches MeSH terms with automatic inclusion of narrower terms
• [majr] tag limits to articles where the topic is the main focus
• Combine MeSH terms with subheadings for specificity (e.g., diabetes mellitus/therapy[mh])

Common MeSH Subheadings:

• /diagnosis - Diagnostic methods
• /drug therapy - Pharmaceutical treatment
• /epidemiology - Disease patterns and prevalence
• /etiology - Disease causes
• /prevention & control - Preventive measures
• /therapy - Treatment approaches

Example:

# Diabetes therapy with specific focus
diabetes mellitus, type 2[mh]/drug therapy AND cardiovascular diseases[mh]/prevention & control

3. Article Type and Publication Filtering

Filter results by publication type, date, text availability, and other attributes.

Publication Types (use [pt] field tag):

• Clinical Trial
• Meta-Analysis
• Randomized Controlled Trial
• Review
• Systematic Review
• Case Reports
• Guideline

Date Filtering:

• Single year: 2024[dp]
• Date range: 2020:2024[dp]
• Specific date: 2024/03/15[dp]

Text Availability:

• Free full text: Add AND free full text[sb] to query
• Has abstract: Add AND hasabstract[text] to query

Example:

# Recent free full-text RCTs on hypertension
hypertension[mh] AND randomized controlled trial[pt] AND 2023:2024[dp] AND free full text[sb]

4. Programmatic Access via E-utilities API

Access PubMed data programmatically using the NCBI E-utilities REST API for automation and bulk operations.

Core API Endpoints:

1. ESearch - Search database and retrieve PMIDs
2. EFetch - Download full records in various formats
3. ESummary - Get document summaries
4. EPost - Upload UIDs for batch processing
5. ELink - Find related articles and linked data

Basic Workflow:

import requests

# Step 1: Search for articles
base_url = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/"
search_url = f"{base_url}esearch.fcgi"
params = {
    "db": "pubmed",
    "term": "diabetes[tiab] AND 2024[dp]",
    "retmax": 100,
    "retmode": "json",
    "api_key": "YOUR_API_KEY"  # Optional but recommended
}
response = requests.get(search_url, params=params)
pmids = response.json()["esearchresult"]["idlist"]

# Step 2: Fetch article details
fetch_url = f"{base_url}efetch.fcgi"
params = {
    "db": "pubmed",
    "id": ",".join(pmids),
    "rettype": "abstract",
    "retmode": "text",
    "api_key": "YOUR_API_KEY"
}
response = requests.get(fetch_url, params=params)
abstracts = response.text

Rate Limits:

• Without API key: 3 requests/second
• With API key: 10 requests/second
• Always include User-Agent header

Best Practices:

• Use history server (usehistory=y) for large result sets
• Implement batch operations via EPost for multiple UIDs
• Cache results locally to minimize redundant calls
• Respect rate limits to avoid service disruption

When to consult api_reference.md:

• Need detailed endpoint documentation
• Require parameter specifications for each E-utility
• Constructing batch operations or history server workflows
• Understanding response formats (XML, JSON, text)
• Troubleshooting API errors or rate limit issues

Grep pattern for API endpoints: esearch|efetch|esummary|epost|elink|einfo

5. Citation Matching and Article Retrieval

Find articles using partial citation information or specific identifiers.

By Identifier:

# By PMID
12345678[pmid]

# By DOI
10.1056/NEJMoa123456[doi]

# By PMC ID
PMC123456[pmc]

Citation Matching (via ECitMatch API): Use journal name, year, volume, page, and author to find PMIDs:

Format: journal|year|volume|page|author|key|
Example: Science|2008|320|5880|1185|key1|

By Author and Metadata:

# First author with year and topic
smith ja[1au] AND 2023[dp] AND cancer[tiab]

# Journal, volume, and page
nature[ta] AND 2024[dp] AND 456[vi] AND 123-130[pg]

6. Systematic Literature Reviews

Conduct comprehensive literature searches for systematic reviews and meta-analyses.

PICO Framework (Population, Intervention, Comparison, Outcome): Structure clinical research questions systematically:

# Example: Diabetes treatment effectiveness
# P: diabetes mellitus, type 2[mh]
# I: metformin[nm]
# C: lifestyle modification[tiab]
# O: glycemic control[tiab]

diabetes mellitus, type 2[mh] AND
(metformin[nm] OR lifestyle modification[tiab]) AND
glycemic control[tiab] AND
randomized controlled trial[pt]

Comprehensive Search Strategy:

# Include multiple synonyms and MeSH terms
(disease name[tiab] OR disease name[mh] OR synonym[tiab]) AND
(treatment[tiab] OR therapy[tiab] OR intervention[tiab]) AND
(systematic review[pt] OR meta-analysis[pt] OR randomized controlled trial[pt]) AND
2020:2024[dp] AND
english[la]

Search Refinement:

1. Start broad, review results
2. Add specificity with field tags
3. Apply date and publication type filters
4. Use Advanced Search to view query translation
5. Combine search history for complex queries

When to consult common_queries.md:

• Need example queries for specific disease types or research areas
• Require templates for different study designs
• Looking for population-specific query patterns (pediatric, geriatric, etc.)
• Constructing methodology-specific searches
• Need quality filters or best practice patterns

Grep pattern for query examples: diabetes|cancer|cardiovascular|clinical trial|systematic review

7. Related Articles and Citation Discovery

Find related research programmatically with ELink (pre-calculated neighbors based on title/abstract similarity and MeSH overlap):

elink.fcgi?dbfrom=pubmed&db=pubmed&id=PMID&cmd=neighbor

cmd=prlinks returns publisher full-text URLs; cmd=llinks returns LinkOut URLs.

8. Export for Citation Management

Use EFetch with rettype=medline&retmode=text to export records in MEDLINE/.nbib format for reference managers (Zotero, Mendeley, EndNote):

efetch.fcgi?db=pubmed&id=PMID1,PMID2&rettype=medline&retmode=text

Working with Reference Files

This skill includes three comprehensive reference files in the references/ directory:

references/api_reference.md

Complete E-utilities API documentation including all nine endpoints, parameters, response formats, and best practices. Consult when:

• Implementing programmatic PubMed access
• Constructing API requests
• Understanding rate limits and authentication
• Working with large datasets via history server
• Troubleshooting API errors

references/search_syntax.md

Detailed guide to PubMed search syntax including field tags, Boolean operators, wildcards, and special characters. Consult when:

• Constructing complex search queries
• Understanding automatic term mapping
• Using advanced search features (proximity, wildcards)
• Applying filters and limits
• Troubleshooting unexpected search results

references/common_queries.md

Extensive collection of example queries for various research scenarios, disease types, and methodologies. Consult when:

• Starting a new literature search
• Need templates for specific research areas
• Looking for best practice query patterns
• Conducting systematic reviews
• Searching for specific study designs or populations

Reference Loading Strategy: Load reference files as needed. For basic searches this SKILL.md is usually enough; consult the references for complex query construction (search_syntax.md), API workflows (api_reference.md), or domain templates (common_queries.md).

Tips and Best Practices

• Verify translation: Automatic Term Mapping can silently expand terms; check the query translation (or bypass ATM with field tags / double quotes) when results look off.
• API throughput: keyless is 3 req/s by IP; an API key raises it to 10 req/s. Send a descriptive User-Agent, add exponential backoff on HTTP 429, and cache to avoid redundant calls.
• Large result sets (>~500 records): use the history server (usehistory=y, then page EFetch with retstart/retmax) or EPost, rather than one huge id list (which can hit HTTP 414).
• Evidence quality: prefer systematic review[pt], meta-analysis[pt], and randomized controlled trial[pt]; pair with humans[mh], english[la], and a [dp] range as appropriate.
• Reproducibility: record the exact query string and the search date — PubMed results change as records are added and re-indexed.

Limitations and Considerations

• Scope: peer-reviewed biomedical/life-sciences literature (MEDLINE + PubMed). For preprints use a preprint server; for trial-registry records (NCT IDs, recruitment status) use ClinicalTrials.gov.
• Coverage gaps: pre-1975 articles often lack abstracts; full author names are indexed from 2002 forward.
• Result caps: ESearch returns at most 10,000 UIDs per query (page with retstart, or use the history server).
• Full text: PubMed provides citations/abstracts, not full text. Access depends on publisher, open-access status, or institutional subscription; detailed records require XML parsing.

Support Resources

• PubMed Help: https://pubmed.ncbi.nlm.nih.gov/help/
• E-utilities Documentation: https://www.ncbi.nlm.nih.gov/books/NBK25501/
• NLM Help Desk: 1-888-FIND-NLM (1-888-346-3656)
• Technical Support: eutilities@ncbi.nlm.nih.gov
• Mailing List: utilities-announce@ncbi.nlm.nih.gov