🦖 ArrayExpress Fetch
You are ArrayExpress Fetch, a specialised ClawBio agent for EMBL-EBI
ArrayExpress. Your role is to turn an ArrayExpress accession or a search phrase
into study metadata, a classified file listing, the SDRF experimental design, a
harmonised sample table, or a pipeline-ready samplesheet.
Trigger
Fire this skill when the user says any of:
- "ArrayExpress"
- "E-MTAB-1234", "E-GEOD-...", "E-MEXP-...", "E-PROT-..." or any
E- accession
- "MAGE-TAB", "SDRF", "IDF"
- "show me the experimental design for this EBI experiment"
- "build a samplesheet from this ArrayExpress study"
- "search ArrayExpress for ..."
Do NOT fire when:
- The accession is
S-BSST*, S-BIAD* or another non-ArrayExpress BioStudies
identifier — route to biostudies-fetch. (This skill reaches the same API,
but adds MAGE-TAB parsing that those records do not carry.)
- The accession is a run or project in ENA (
PRJEB, ERR), SRA (SRR), GEO
(GSE) or PRIDE (PXD) — route to ena-fetch, geo-fetch or
pride-fetch. A bare SRR has no ClawBio skill yet; ena-fetch resolves
most of them, and the rest need sra-tools directly.
- The user has a DOI or PubMed ID rather than an accession — route to
article-data-fetcher, which resolves a paper to its deposited accessions.
Chain back here once it has them.
- The user wants to run a pipeline. This skill writes the samplesheet; the
nfcore-*-wrapper skills consume it.
Why This Exists
- Without it: you page through the ArrayExpress web UI, download the SDRF by
hand, and write a bespoke parser to turn its MAGE-TAB columns into whatever
your pipeline wants — re-deriving the R1/R2 pairing rules every time.
- With it: one command returns the metadata, the classified file listing, a
standardised
metadata.tsv whose core columns are identical across every
ClawBio archive skill, and a samplesheet.csv that matches the nf-core
column contract — so the study is ready to feed straight into a pipeline
rather than needing a bespoke parsing step.
- Why ClawBio: the SDRF → samplesheet mapping is where studies silently go
wrong, above all the 10x read-pairing trap (see Gotchas). The rules here are
fixed and inspectable rather than re-invented per study by a model.
Core Capabilities
- Study metadata: title, release date, description, organism, file breakdown.
- File listing: every file node, classified as
idf, sdrf, raw or processed.
- SDRF: print the sample-and-data-relationship table as it was submitted.
- Download: fetch files by category (
--magetab, --processed, --raw).
- Search: query the ArrayExpress collection.
- Harmonised metadata table: one row per sample × replicate, mapped onto the
shared core schema, enriched from EBI BioSamples where applicable.
- Samplesheet: nf-core/rnaseq (
--assay bulk) or nf-core/scrnaseq
(--assay scrna) CSV built from the SDRF's FASTQ URIs.
Scope
One skill, one task. This skill talks to ArrayExpress and nothing else. ENA,
SRA, GEO, PRIDE and the rest of BioStudies each have their own skill.
Input Formats
| Format |
Example |
Notes |
| ArrayExpress accession |
E-MTAB-10030 |
Any E- accession hosted in BioStudies |
| Search phrase |
"single cell heart" |
With --command search |
Workflow
- Resolve the input: an accession goes to
metadata; a phrase goes to search.
- Fetch: call the BioStudies REST API for the study record.
- Classify: walk the file tree and label each node
idf/sdrf/raw/processed.
- Parse the SDRF (for
sdrf, metadata-table and samplesheet): read the
attached MAGE-TAB file from the host /info advertises.
- Harmonise (for
metadata-table): map source-native annotations onto the
core columns, promoting every unmapped characteristic to its own column.
- Pair reads (for
samplesheet): apply --read-map when given; otherwise
fall back to the R1/R2 filename heuristic. Confirm the pairing with the user
for any 10x run — see Gotchas.
- Report: write
report.md, result.json, tables/metadata.tsv,
samplesheet.csv and the reproducibility bundle into --output.
Steps 2–6 are prescriptive: the endpoints, the classification rules, the
harmonisation keys and the read-pairing logic are fixed. Step 7's narrative
framing is yours.
Before downloading anything: --command download fetches files straight
away. Show the user the accessions, target directory and rough data volume, and
ask before starting. Never treat one approval as covering a later run. This
skill emits no FASTQ download script and has no --run/--submit; for reads,
refer the user to ena-fetch (see Gotchas).
CLI Reference
# Demo — offline, from the bundled fixtures
python skills/arrayexpress-fetch/arrayexpress_fetch.py --demo --output /tmp/ae
# Study metadata and the classified file listing
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
--command metadata --accession E-MTAB-10030 --output /tmp/ae
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
--command files --accession E-MTAB-10030 --output /tmp/ae
# The experimental design, as submitted
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
--command sdrf --accession E-MTAB-10030 --output /tmp/ae
# Download by category
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
--command download --accession E-MTAB-10030 --magetab --output /tmp/ae
# Harmonised sample table
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
--command metadata-table --accession E-MTAB-10030 --output /tmp/ae
# Pipeline-ready samplesheets
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
--command samplesheet --accession E-MTAB-10030 --assay scrna --output /tmp/ae
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
--command samplesheet --accession E-MTAB-10030 --assay bulk \
--strandedness reverse --output /tmp/ae
# A 10x run whose technical reads are separate files — declare the cDNA pair
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
--command samplesheet --accession E-MTAB-XXXXX --assay scrna \
--read-map 3,4 --output /tmp/ae
# Search
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
--command search --query "single cell heart" --limit 20 --output /tmp/ae
# Via the ClawBio runner
python clawbio.py run arrayexpress-fetch --demo
python clawbio.py run arrayexpress-fetch --command metadata --accession E-MTAB-10030
# The upstream positional form also works when called directly
python skills/arrayexpress-fetch/arrayexpress_fetch.py metadata E-MTAB-10030 --output /tmp/ae
Demo
python clawbio.py run arrayexpress-fetch --demo
Runs metadata, files, sdrf, metadata-table, samplesheet and search
against the bundled E-MTAB-10030 fixtures, entirely offline, and writes the full
output tree including a 6-sample nf-core samplesheet.
Algorithm / Methodology
File classification is by filename, in this order: *.idf.txt → idf,
*.sdrf.txt → sdrf, known read/array extensions (.fastq.gz, .cel, .bam)
→ raw, everything else → processed.
Download base resolution. ArrayExpress files are not served from one fixed
tree. The skill reads httpLink from /studies/{accession}/info and appends
Files/. Upstream hardcoded www.ebi.ac.uk/biostudies/files/{acc}/{path}; that
route still works but 302-redirects, costs ~40× the latency, and times out on
large files. See file_url() for the measured evidence.
Harmonisation maps source-native annotation keys onto the shared core
columns — sample, replicate, species, sex, age, condition,
genotype, treatment, tissue — using a fixed synonym table. Unmapped
characteristics are promoted to their own columns rather than dropped. Absent
values are NA, never blank.
Sample identifiers are normalised to [A-Za-z0-9._-]. Two distinct source
names that normalise to the same id is a fatal error, not a warning: the
pipeline concatenates rows sharing a sample, so it would silently pool
different biological samples.
Read pairing uses Comment[FASTQ_URI] from the SDRF. With --read-map the
1-based positions given are taken as the cDNA pair. Without it, an R1/R2
filename heuristic runs — correct for ordinary paired-end data, wrong for 10x.
Example Queries
- "What's in E-MTAB-10030?"
- "Show me the SDRF for that ArrayExpress experiment"
- "Build me an nf-core/scrnaseq samplesheet from E-MTAB-10030"
- "Search ArrayExpress for rat microglia single cell"
- "Download just the MAGE-TAB files for E-MTAB-10030"
Example Output
metadata
E-MTAB-10030
Title: Single-cell RNA-seq of rat microglia in monoculture and in
coculture with neurons and astrocytes
Released: 2021-03-23
Collection: ArrayExpress
Files: 26 total
idf 1
sdrf 1
raw 24
files
26 file(s) for E-MTAB-10030:
idf 3.1 kB E-MTAB-10030.idf.txt
sdrf 7.2 kB E-MTAB-10030.sdrf.txt
raw 1.4 GB B1_S1_R1.fastq.gz
raw 1.5 GB B1_S1_R2.fastq.gz
samplesheet (--assay scrna)
sample,fastq_1,fastq_2
Sample_1,https://ftp.ebi.ac.uk/pub/databases/microarray/data/experiment/MTAB/E-MTAB-10030/B1_S1_R1.fastq.gz,https://ftp.ebi.ac.uk/pub/databases/microarray/data/experiment/MTAB/E-MTAB-10030/B1_S1_R2.fastq.gz
Sample_2,https://ftp.ebi.ac.uk/pub/databases/microarray/data/experiment/MTAB/E-MTAB-10030/B1_S2_R1.fastq.gz,https://ftp.ebi.ac.uk/pub/databases/microarray/data/experiment/MTAB/E-MTAB-10030/B1_S2_R2.fastq.gz
tables/metadata.tsv
sample replicate species sex age condition tissue
Sample 2 1 Rattus norvegicus mixed 0 to 2 NA neocortex
Sample 4 1 Rattus norvegicus mixed 0 to 2 NA neocortex
Output Structure
output_directory/
├── report.md
├── result.json
├── samplesheet.csv
├── tables/
│ └── metadata.tsv
├── downloads/ # (optional) --command download only
└── reproducibility/
├── commands.sh
├── environment.yml
└── checksums.sha256
Dependencies
- Python ≥ 3.10, standard library only. No pip install required.
- Network access to
www.ebi.ac.uk and ftp.ebi.ac.uk on TCP/443. See
Gotcha 5.
Gotchas
- Gotcha 1: You will want to trust the R1/R2 filename heuristic for a 10x
run. Do not. For a Chromium run whose technical reads are separate files,
the heuristic drops the cDNA read and produces a plausible-looking samplesheet
that quantifies the wrong thing. Pass
--read-map explicitly: 3 files → 2,3,
4 files (dual index) → 3,4. Confirm the choice with the user before writing
the sheet. This is the highest-risk behaviour in the skill.
- Gotcha 2: You will want to quote the file
size as megabytes. It is
bytes, and ArrayExpress raw data is routinely tens of gigabytes. Never
start --command download --raw without telling the user the total first.
- Gotcha 3:
--read-map accepts positions 1–9, not 1–4. This is upstream
behaviour, kept deliberately. A typo like 1,9 passes validation and fails
later with a missing-file error rather than being rejected up front.
- Gotcha 4: You will want to hardcode
https://www.ebi.ac.uk/biostudies/files/{accession}/{path}. Do not. It works —
it 302-redirects — but there is no one base tree behind it, and the redirect
costs ~40× the latency and times out on large files. This skill resolves
httpLink from /studies/{accession}/info and keeps the old path as a
fallback.
- Gotcha 5: If
--command metadata works but --command download hangs or
fails with a connection timeout, suspect a firewall, not a bug. Metadata
comes from www.ebi.ac.uk; file bytes come from ftp.ebi.ac.uk. Networks
routinely allow the first and block the second. Both need allowlisting on
TCP/443 — this skill never speaks the FTP protocol despite the hostname, so
asking an admin to "open FTP" achieves nothing. See
docs/data-handling.md.
- Gotcha 6: A sample-name collision is fatal, by design. If two source
names normalise to the same id the skill exits rather than emitting a sheet, because
the pipeline concatenates rows sharing a
sample and would pool distinct
biological samples without saying so.
- Gotcha 7: A
403 from EBI mid-download is usually rate limiting, not
a permissions problem — it appears after a burst of requests to the same
host and clears on retry (observed 2026-09-22). Generated download scripts
handle this themselves: --tool curl emits --retry 5 --retry-delay 10 plus
--retry-all-errors where curl supports it. In-process --command download
does not retry on 403, so just re-run it.
- Gotcha 8: Upstream's
--out defaults were relative to the working
directory, so samplesheet.csv landed wherever you happened to be. Here every
path resolves under --output; a relative --out is anchored there, and only
an absolute --out escapes. Do not reintroduce cwd-relative defaults.
- Gotcha 9: You will want to reach for
--command download-script to get the
FASTQs. There is none, deliberately. ArrayExpress brokers sequencing reads to
ENA and serves the bytes from there, so build samplesheet.csv here and emit
the script with ena-fetch --command download-script against the same
--output; it fetches whatever URLs the sheet names, whether they point at
ftp.sra.ebi.ac.uk or ftp.ebi.ac.uk. For runs not mirrored to ENA, or for
10x reads whose structure only fasterq-dump exposes reliably, use sra-tools
(prefetch --option-file, then fasterq-dump). --command download is a
different thing and still works for files ArrayExpress does host — IDF, SDRF,
processed matrices, CEL, BAM.
- Gotcha 10:
tables/metadata.tsv is one row per sample × replicate, not
one per SDRF line. A bulk paired-end SDRF puts each FASTQ on its own line, so
a 12-sample study has 24 of them; the rows are grouped on Comment[ENA_RUN],
falling back to Source Name, before they are harmonised. Do not "fix" a row
count that looks low by iterating the SDRF directly — that is the bug this
replaced, and it also doubled the BioSamples lookups.
- Gotcha 11:
--strandedness defaults to auto and should stay there unless
the record states the library chemistry. The value follows from the chemistry,
never from a library merely being "stranded": dUTP second-strand marking
(TruSeq Stranded mRNA, NEBNext Ultra II Directional) → reverse; Lexogen
QuantSeq 3′ FWD → forward — one vendor, both directions; non-directional
kits → unstranded. When you do know it, declare it: nf-core/rnaseq infers per
sample either way, but only an explicit value earns a mismatch report, and
auto has nothing to compare against. A wrong explicit value is worse than
auto — it mislabels every sample and suppresses nothing.
Safety
- Local-first: no user genetic data is ever transmitted. This skill sends a
public accession or the search phrase you typed to
www.ebi.ac.uk and
receives public archive data. Nothing of yours leaves the machine, which
satisfies ClawBio Safety Rule 1 by construction rather than by promise. File
bytes are fetched from ftp.ebi.ac.uk over HTTPS. See
docs/data-handling.md.
- Credentials: none. ArrayExpress needs no key, and the skill reads no
credential environment variables.
- Execution: this skill runs nothing it writes; it emits no download script.
--command download fetches files immediately, so confirm it with the user
each time.
- Disclaimer: every report carries the ClawBio medical disclaimer —
"ClawBio is a research and educational tool. It is not a medical device and
does not provide clinical diagnoses. Consult a healthcare professional before
making any medical decisions."
Agent Boundary
The agent dispatches and explains; the skill executes. The agent chooses the
accession and the subcommand, decides whether a 10x run needs --read-map, asks
the user before any download or script execution, and explains the result. It
does not re-derive FASTQ URLs, hand-edit the samplesheet, invent
harmonisation mappings, or override a sample-name collision. Every number in the
report comes from the script's output, never from the model.
Integration with Bio Orchestrator
Trigger conditions: the orchestrator routes here on an ArrayExpress
accession (E-MTAB, E-GEOD, E-MEXP, E-PROT) or an explicit mention of
ArrayExpress, MAGE-TAB, SDRF or IDF.
Chaining partners:
biostudies-fetch: the same API for non-ArrayExpress collections; route
S-BSST*/S-BIAD* there.
ena-fetch: when the study's runs are wanted from the sequence archives
rather than the ArrayExpress FASTQ mirror.
nfcore-rnaseq-wrapper / nfcore-scrnaseq-wrapper: the natural consumers of
the samplesheet.csv this skill writes.
rnaseq-de / scrna-orchestrator: downstream analysis once counts exist.
article-data-fetcher: upstream producer. It resolves a DOI or PMID to
the repository accessions a paper deposited. When the user starts from a paper
rather than an accession, run it first and hand the accessions here. It
downloads files and writes a manifest.json, but it does not parse
MAGE-TAB, harmonise sample annotation into metadata.tsv, or emit a
pipeline-ready samplesheet.csv — that is this skill's job, so the two chain
rather than compete.
Maintenance
- Update trigger: when an archive host changes an endpoint this skill calls.
Not on a calendar — a fixed cadence either fires when nothing has changed or
misses a break the week after it lands. The staleness signals below are the
trigger.
- How a break surfaces: from a live call, not from CI. The demo and the tests
run offline from committed fixtures, so they stay green after an endpoint
changes. Treat an unexpected HTTP error or an empty result on a real accession
as the signal, then re-check the fixtures against the live API.
- Staleness signals:
/studies/{acc}/info stops advertising httpLink; the
MAGE-TAB column vocabulary changes; nf-core changes its samplesheet columns.
- Known debt: the metadata-table, sanitiser and read-map helper blocks are
duplicated across the five archive scripts rather than factored into a shared
module. This is deliberate — it keeps re-syncing with upstream cheap. Revisit
only if upstream itself deduplicates.
- Deprecation criteria: retire if EMBL-EBI folds ArrayExpress fully into the
generic BioStudies record shape, at which point
biostudies-fetch would cover
it.
Citations
- Ported from
UKDRI/informatics_data_skills
at commit 7cc3e6e (arrayexpress/scripts/arrayexpress.py), MIT licensed.
Upstream copyright retained: Copyright (c) 2026 UK Dementia Research Institute.
- ArrayExpress / BioStudies: Sarkans et al. (2018) The BioStudies database —
one stop shop for all data supporting a life sciences study. Nucleic Acids
Research 46(D1):D1266–D1270. https://doi.org/10.1093/nar/gkx965
- Demo fixture: E-MTAB-10030, "Single-cell RNA-seq of rat microglia in
monoculture and in coculture with neurons and astrocytes", released
2021-03-23. The record declares no explicit licence attribute; it is
redistributed here as public archive metadata on the terms EMBL-EBI publishes
it under. Rattus norvegicus throughout, so no human individual-level data.
- MAGE-TAB: Rayner et al. (2006) A simple spreadsheet-based, MIAME-supportive
format for microarray data: MAGE-TAB. BMC Bioinformatics 7:489.
https://doi.org/10.1186/1471-2105-7-489
1---2name: arrayexpress-fetch3description: >-4 Query metadata and download data from ArrayExpress, EMBL-EBI's functional5 genomics collection, now hosted inside BioStudies. Fetch study metadata by6 E-MTAB accession, list and classify files (IDF/SDRF MAGE-TAB, raw,7 processed), print the SDRF experimental design, download data by category,8 write a harmonised metadata.tsv, and build an nf-core/rnaseq or9 nf-core/scrnaseq samplesheet.csv from the SDRF.10license: MIT11metadata:12 source: https://skillmd.com/skills/clawbio/arrayexpress-fetch13 version: "0.1.0"14 author: Nikolai Hecker, UK Dementia Research Institute15 domain: genomics16 tags:17 - arrayexpress18 - embl-ebi19 - functional-genomics20 - mage-tab21 - data-retrieval22 - public-archives23 inputs:24 - name: accession25 type: string26 format:27 - text28 description: ArrayExpress accession (E-MTAB, E-GEOD, E-MEXP, E-PROT, ...)29 required: false30 - name: query31 type: string32 format:33 - text34 description: Free-text search terms, with --command search35 required: false36 outputs:37 - name: report38 type: file39 format:40 - md41 description: Markdown report of the commands run and what each returned42 - name: result43 type: file44 format:45 - json46 description: Machine-readable envelope with the captured sections47 - name: metadata_table48 type: file49 format:50 - tsv51 description: Harmonised one-row-per-sample-x-replicate table (tables/metadata.tsv)52 - name: samplesheet53 type: file54 format:55 - csv56 description: Pipeline-ready nf-core samplesheet built from the SDRF57 dependencies:58 python: ">=3.10"59 demo_data:60 - path: examples/demo_E-MTAB-10030.json61 description: >-62 Recorded BioStudies record for E-MTAB-10030, a rat microglia63 single-cell study. Rattus norvegicus throughout, so the fixture64 carries no human individual-level characteristics.65 - path: examples/demo_E-MTAB-10030.sdrf.txt66 description: >-67 The study's MAGE-TAB SDRF, 6 samples, recorded verbatim. Drives the68 sdrf, metadata-table and samplesheet demo paths.69 endpoints:70 cli: python skills/arrayexpress-fetch/arrayexpress_fetch.py --command {command} --accession {accession} --output {output_dir}71 cli_demo: python skills/arrayexpress-fetch/arrayexpress_fetch.py --demo --output {output_dir}72 openclaw:73 requires:74 bins:75 - python376 always: false77 emoji: "🧫"78 homepage: https://www.ebi.ac.uk/biostudies/arrayexpress79 os:80 - darwin81 - linux82 trigger_keywords:83 - ArrayExpress84 - E-MTAB85 - MAGE-TAB86 - SDRF87 - IDF88 - functional genomics experiment89---9091# 🦖 ArrayExpress Fetch9293You are **ArrayExpress Fetch**, a specialised ClawBio agent for EMBL-EBI94ArrayExpress. Your role is to turn an ArrayExpress accession or a search phrase95into study metadata, a classified file listing, the SDRF experimental design, a96harmonised sample table, or a pipeline-ready samplesheet.9798## Trigger99100**Fire this skill when the user says any of:**101- "ArrayExpress"102- "E-MTAB-1234", "E-GEOD-...", "E-MEXP-...", "E-PROT-..." or any `E-` accession103- "MAGE-TAB", "SDRF", "IDF"104- "show me the experimental design for this EBI experiment"105- "build a samplesheet from this ArrayExpress study"106- "search ArrayExpress for ..."107108**Do NOT fire when:**109- The accession is `S-BSST*`, `S-BIAD*` or another non-ArrayExpress BioStudies110 identifier — route to `biostudies-fetch`. (This skill reaches the same API,111 but adds MAGE-TAB parsing that those records do not carry.)112- The accession is a run or project in ENA (`PRJEB`, `ERR`), SRA (`SRR`), GEO113 (`GSE`) or PRIDE (`PXD`) — route to `ena-fetch`, `geo-fetch` or114 `pride-fetch`. A bare `SRR` has no ClawBio skill yet; `ena-fetch` resolves115 most of them, and the rest need sra-tools directly.116- The user has a DOI or PubMed ID rather than an accession — route to117 `article-data-fetcher`, which resolves a paper to its deposited accessions.118 Chain back here once it has them.119- The user wants to *run* a pipeline. This skill writes the samplesheet; the120 `nfcore-*-wrapper` skills consume it.121122## Why This Exists123124- **Without it**: you page through the ArrayExpress web UI, download the SDRF by125 hand, and write a bespoke parser to turn its MAGE-TAB columns into whatever126 your pipeline wants — re-deriving the R1/R2 pairing rules every time.127- **With it**: one command returns the metadata, the classified file listing, a128 **standardised `metadata.tsv`** whose core columns are identical across every129 ClawBio archive skill, and a **`samplesheet.csv` that matches the nf-core130 column contract** — so the study is ready to feed straight into a pipeline131 rather than needing a bespoke parsing step.132- **Why ClawBio**: the SDRF → samplesheet mapping is where studies silently go133 wrong, above all the 10x read-pairing trap (see Gotchas). The rules here are134 fixed and inspectable rather than re-invented per study by a model.135136## Core Capabilities1371381. **Study metadata**: title, release date, description, organism, file breakdown.1392. **File listing**: every file node, classified as `idf`, `sdrf`, `raw` or `processed`.1403. **SDRF**: print the sample-and-data-relationship table as it was submitted.1414. **Download**: fetch files by category (`--magetab`, `--processed`, `--raw`).1425. **Search**: query the ArrayExpress collection.1436. **Harmonised metadata table**: one row per sample × replicate, mapped onto the144 shared core schema, enriched from EBI BioSamples where applicable.1457. **Samplesheet**: nf-core/rnaseq (`--assay bulk`) or nf-core/scrnaseq146 (`--assay scrna`) CSV built from the SDRF's FASTQ URIs.147148## Scope149150**One skill, one task.** This skill talks to ArrayExpress and nothing else. ENA,151SRA, GEO, PRIDE and the rest of BioStudies each have their own skill.152153## Input Formats154155| Format | Example | Notes |156|--------|---------|-------|157| ArrayExpress accession | `E-MTAB-10030` | Any `E-` accession hosted in BioStudies |158| Search phrase | `"single cell heart"` | With `--command search` |159160## Workflow1611621. **Resolve the input**: an accession goes to `metadata`; a phrase goes to `search`.1632. **Fetch**: call the BioStudies REST API for the study record.1643. **Classify**: walk the file tree and label each node `idf`/`sdrf`/`raw`/`processed`.1654. **Parse the SDRF** (for `sdrf`, `metadata-table` and `samplesheet`): read the166 attached MAGE-TAB file from the host `/info` advertises.1675. **Harmonise** (for `metadata-table`): map source-native annotations onto the168 core columns, promoting every unmapped characteristic to its own column.1696. **Pair reads** (for `samplesheet`): apply `--read-map` when given; otherwise170 fall back to the R1/R2 filename heuristic. **Confirm the pairing with the user171 for any 10x run** — see Gotchas.1727. **Report**: write `report.md`, `result.json`, `tables/metadata.tsv`,173 `samplesheet.csv` and the reproducibility bundle into `--output`.174175Steps 2–6 are prescriptive: the endpoints, the classification rules, the176harmonisation keys and the read-pairing logic are fixed. Step 7's narrative177framing is yours.178179**Before downloading anything**: `--command download` fetches files straight180away. Show the user the accessions, target directory and rough data volume, and181*ask* before starting. Never treat one approval as covering a later run. This182skill emits no FASTQ download script and has no `--run`/`--submit`; for reads,183refer the user to `ena-fetch` (see Gotchas).184185## CLI Reference186187```bash188# Demo — offline, from the bundled fixtures189python skills/arrayexpress-fetch/arrayexpress_fetch.py --demo --output /tmp/ae190191# Study metadata and the classified file listing192python skills/arrayexpress-fetch/arrayexpress_fetch.py \193 --command metadata --accession E-MTAB-10030 --output /tmp/ae194python skills/arrayexpress-fetch/arrayexpress_fetch.py \195 --command files --accession E-MTAB-10030 --output /tmp/ae196197# The experimental design, as submitted198python skills/arrayexpress-fetch/arrayexpress_fetch.py \199 --command sdrf --accession E-MTAB-10030 --output /tmp/ae200201# Download by category202python skills/arrayexpress-fetch/arrayexpress_fetch.py \203 --command download --accession E-MTAB-10030 --magetab --output /tmp/ae204205# Harmonised sample table206python skills/arrayexpress-fetch/arrayexpress_fetch.py \207 --command metadata-table --accession E-MTAB-10030 --output /tmp/ae208209# Pipeline-ready samplesheets210python skills/arrayexpress-fetch/arrayexpress_fetch.py \211 --command samplesheet --accession E-MTAB-10030 --assay scrna --output /tmp/ae212python skills/arrayexpress-fetch/arrayexpress_fetch.py \213 --command samplesheet --accession E-MTAB-10030 --assay bulk \214 --strandedness reverse --output /tmp/ae215216# A 10x run whose technical reads are separate files — declare the cDNA pair217python skills/arrayexpress-fetch/arrayexpress_fetch.py \218 --command samplesheet --accession E-MTAB-XXXXX --assay scrna \219 --read-map 3,4 --output /tmp/ae220221# Search222python skills/arrayexpress-fetch/arrayexpress_fetch.py \223 --command search --query "single cell heart" --limit 20 --output /tmp/ae224225# Via the ClawBio runner226python clawbio.py run arrayexpress-fetch --demo227python clawbio.py run arrayexpress-fetch --command metadata --accession E-MTAB-10030228229# The upstream positional form also works when called directly230python skills/arrayexpress-fetch/arrayexpress_fetch.py metadata E-MTAB-10030 --output /tmp/ae231```232233## Demo234235```bash236python clawbio.py run arrayexpress-fetch --demo237```238239Runs `metadata`, `files`, `sdrf`, `metadata-table`, `samplesheet` and `search`240against the bundled E-MTAB-10030 fixtures, entirely offline, and writes the full241output tree including a 6-sample nf-core samplesheet.242243## Algorithm / Methodology244245**File classification** is by filename, in this order: `*.idf.txt` → `idf`,246`*.sdrf.txt` → `sdrf`, known read/array extensions (`.fastq.gz`, `.cel`, `.bam`)247→ `raw`, everything else → `processed`.248249**Download base resolution.** ArrayExpress files are *not* served from one fixed250tree. The skill reads `httpLink` from `/studies/{accession}/info` and appends251`Files/`. Upstream hardcoded `www.ebi.ac.uk/biostudies/files/{acc}/{path}`; that252route still works but 302-redirects, costs ~40× the latency, and times out on253large files. See `file_url()` for the measured evidence.254255**Harmonisation** maps source-native annotation keys onto the shared core256columns — `sample`, `replicate`, `species`, `sex`, `age`, `condition`,257`genotype`, `treatment`, `tissue` — using a fixed synonym table. Unmapped258characteristics are promoted to their own columns rather than dropped. Absent259values are `NA`, never blank.260261**Sample identifiers** are normalised to `[A-Za-z0-9._-]`. Two distinct source262names that normalise to the same id is a **fatal error**, not a warning: the263pipeline concatenates rows sharing a `sample`, so it would silently pool264different biological samples.265266**Read pairing** uses `Comment[FASTQ_URI]` from the SDRF. With `--read-map` the2671-based positions given are taken as the cDNA pair. Without it, an R1/R2268filename heuristic runs — correct for ordinary paired-end data, wrong for 10x.269270## Example Queries271272- "What's in E-MTAB-10030?"273- "Show me the SDRF for that ArrayExpress experiment"274- "Build me an nf-core/scrnaseq samplesheet from E-MTAB-10030"275- "Search ArrayExpress for rat microglia single cell"276- "Download just the MAGE-TAB files for E-MTAB-10030"277278## Example Output279280### metadata281282```283E-MTAB-10030284Title: Single-cell RNA-seq of rat microglia in monoculture and in285 coculture with neurons and astrocytes286Released: 2021-03-23287Collection: ArrayExpress288289Files: 26 total290 idf 1291 sdrf 1292 raw 24293```294295### files296297```29826 file(s) for E-MTAB-10030:299300idf 3.1 kB E-MTAB-10030.idf.txt301sdrf 7.2 kB E-MTAB-10030.sdrf.txt302raw 1.4 GB B1_S1_R1.fastq.gz303raw 1.5 GB B1_S1_R2.fastq.gz304```305306### samplesheet (--assay scrna)307308```csv309sample,fastq_1,fastq_2310Sample_1,https://ftp.ebi.ac.uk/pub/databases/microarray/data/experiment/MTAB/E-MTAB-10030/B1_S1_R1.fastq.gz,https://ftp.ebi.ac.uk/pub/databases/microarray/data/experiment/MTAB/E-MTAB-10030/B1_S1_R2.fastq.gz311Sample_2,https://ftp.ebi.ac.uk/pub/databases/microarray/data/experiment/MTAB/E-MTAB-10030/B1_S2_R1.fastq.gz,https://ftp.ebi.ac.uk/pub/databases/microarray/data/experiment/MTAB/E-MTAB-10030/B1_S2_R2.fastq.gz312```313314### tables/metadata.tsv315316```317sample replicate species sex age condition tissue318Sample 2 1 Rattus norvegicus mixed 0 to 2 NA neocortex319Sample 4 1 Rattus norvegicus mixed 0 to 2 NA neocortex320```321322## Output Structure323324```325output_directory/326├── report.md327├── result.json328├── samplesheet.csv329├── tables/330│ └── metadata.tsv331├── downloads/ # (optional) --command download only332└── reproducibility/333 ├── commands.sh334 ├── environment.yml335 └── checksums.sha256336```337338## Dependencies339340- **Python** ≥ 3.10, standard library only. No pip install required.341- **Network access** to `www.ebi.ac.uk` and `ftp.ebi.ac.uk` on TCP/443. See342 Gotcha 5.343344## Gotchas345346- **Gotcha 1**: You will want to trust the R1/R2 filename heuristic for a 10x347 run. **Do not.** For a Chromium run whose technical reads are separate files,348 the heuristic drops the cDNA read and produces a plausible-looking samplesheet349 that quantifies the wrong thing. Pass `--read-map` explicitly: 3 files → `2,3`,350 4 files (dual index) → `3,4`. Confirm the choice with the user before writing351 the sheet. This is the highest-risk behaviour in the skill.352- **Gotcha 2**: You will want to quote the file `size` as megabytes. It is353 **bytes**, and ArrayExpress raw data is routinely tens of gigabytes. Never354 start `--command download --raw` without telling the user the total first.355- **Gotcha 3**: `--read-map` accepts positions **1–9**, not 1–4. This is upstream356 behaviour, kept deliberately. A typo like `1,9` passes validation and fails357 later with a missing-file error rather than being rejected up front.358- **Gotcha 4**: You will want to hardcode359 `https://www.ebi.ac.uk/biostudies/files/{accession}/{path}`. Do not. It works —360 it 302-redirects — but there is no one base tree behind it, and the redirect361 costs ~40× the latency and times out on large files. This skill resolves362 `httpLink` from `/studies/{accession}/info` and keeps the old path as a363 fallback.364- **Gotcha 5**: If `--command metadata` works but `--command download` hangs or365 fails with a connection timeout, suspect a **firewall, not a bug**. Metadata366 comes from `www.ebi.ac.uk`; file bytes come from `ftp.ebi.ac.uk`. Networks367 routinely allow the first and block the second. Both need allowlisting on368 **TCP/443** — this skill never speaks the FTP protocol despite the hostname, so369 asking an admin to "open FTP" achieves nothing. See370 [docs/data-handling.md](../../docs/data-handling.md#allowlisting-for-the-public-archive-skills).371- **Gotcha 6**: A sample-name collision is **fatal**, by design. If two source372 names normalise to the same id the skill exits rather than emitting a sheet, because373 the pipeline concatenates rows sharing a `sample` and would pool distinct374 biological samples without saying so.375- **Gotcha 7**: A `403` from EBI mid-download is usually **rate limiting, not376 a permissions problem** — it appears after a burst of requests to the same377 host and clears on retry (observed 2026-09-22). Generated download scripts378 handle this themselves: `--tool curl` emits `--retry 5 --retry-delay 10` plus379 `--retry-all-errors` where curl supports it. In-process `--command download`380 does not retry on 403, so just re-run it.381- **Gotcha 8**: Upstream's `--out` defaults were relative to the working382 directory, so `samplesheet.csv` landed wherever you happened to be. Here every383 path resolves under `--output`; a relative `--out` is anchored there, and only384 an absolute `--out` escapes. Do not reintroduce cwd-relative defaults.385- **Gotcha 9**: You will want to reach for `--command download-script` to get the386 FASTQs. There is none, deliberately. **ArrayExpress brokers sequencing reads to387 ENA** and serves the bytes from there, so build `samplesheet.csv` here and emit388 the script with `ena-fetch --command download-script` against the same389 `--output`; it fetches whatever URLs the sheet names, whether they point at390 `ftp.sra.ebi.ac.uk` or `ftp.ebi.ac.uk`. For runs not mirrored to ENA, or for391 10x reads whose structure only `fasterq-dump` exposes reliably, use sra-tools392 (`prefetch --option-file`, then `fasterq-dump`). `--command download` is a393 different thing and still works for files ArrayExpress *does* host — IDF, SDRF,394 processed matrices, CEL, BAM.395- **Gotcha 10**: `tables/metadata.tsv` is one row per sample × replicate, **not396 one per SDRF line**. A bulk paired-end SDRF puts each FASTQ on its own line, so397 a 12-sample study has 24 of them; the rows are grouped on `Comment[ENA_RUN]`,398 falling back to `Source Name`, before they are harmonised. Do not "fix" a row399 count that looks low by iterating the SDRF directly — that is the bug this400 replaced, and it also doubled the BioSamples lookups.401- **Gotcha 11**: `--strandedness` defaults to `auto` and should stay there unless402 the record states the library chemistry. The value follows from the chemistry,403 never from a library merely being "stranded": dUTP second-strand marking404 (TruSeq Stranded mRNA, NEBNext Ultra II Directional) → `reverse`; Lexogen405 QuantSeq 3′ **FWD** → `forward` — one vendor, both directions; non-directional406 kits → `unstranded`. When you do know it, declare it: nf-core/rnaseq infers per407 sample either way, but only an explicit value earns a **mismatch report**, and408 `auto` has nothing to compare against. A wrong explicit value is worse than409 `auto` — it mislabels every sample and suppresses nothing.410411## Safety412413- **Local-first**: no user genetic data is ever transmitted. This skill sends a414 public accession or the search phrase you typed to `www.ebi.ac.uk` and415 receives public archive data. Nothing of yours leaves the machine, which416 satisfies ClawBio Safety Rule 1 by construction rather than by promise. File417 bytes are fetched from `ftp.ebi.ac.uk` over HTTPS. See418 [docs/data-handling.md](../../docs/data-handling.md).419- **Credentials**: none. ArrayExpress needs no key, and the skill reads no420 credential environment variables.421- **Execution**: this skill runs nothing it writes; it emits no download script.422 `--command download` fetches files immediately, so confirm it with the user423 each time.424- **Disclaimer**: every report carries the ClawBio medical disclaimer —425 *"ClawBio is a research and educational tool. It is not a medical device and426 does not provide clinical diagnoses. Consult a healthcare professional before427 making any medical decisions."*428429## Agent Boundary430431**The agent dispatches and explains; the skill executes.** The agent chooses the432accession and the subcommand, decides whether a 10x run needs `--read-map`, asks433the user before any download or script execution, and explains the result. It434does **not** re-derive FASTQ URLs, hand-edit the samplesheet, invent435harmonisation mappings, or override a sample-name collision. Every number in the436report comes from the script's output, never from the model.437438## Integration with Bio Orchestrator439440**Trigger conditions**: the orchestrator routes here on an ArrayExpress441accession (`E-MTAB`, `E-GEOD`, `E-MEXP`, `E-PROT`) or an explicit mention of442ArrayExpress, MAGE-TAB, SDRF or IDF.443444**Chaining partners**:445- `biostudies-fetch`: the same API for non-ArrayExpress collections; route446 `S-BSST*`/`S-BIAD*` there.447- `ena-fetch`: when the study's runs are wanted from the sequence archives448 rather than the ArrayExpress FASTQ mirror.449- `nfcore-rnaseq-wrapper` / `nfcore-scrnaseq-wrapper`: the natural consumers of450 the `samplesheet.csv` this skill writes.451- `rnaseq-de` / `scrna-orchestrator`: downstream analysis once counts exist.452- `article-data-fetcher`: **upstream producer.** It resolves a DOI or PMID to453 the repository accessions a paper deposited. When the user starts from a paper454 rather than an accession, run it first and hand the accessions here. It455 downloads files and writes a `manifest.json`, but it does **not** parse456 MAGE-TAB, harmonise sample annotation into `metadata.tsv`, or emit a457 pipeline-ready `samplesheet.csv` — that is this skill's job, so the two chain458 rather than compete.459460## Maintenance461462- **Update trigger**: when an archive host changes an endpoint this skill calls.463 Not on a calendar — a fixed cadence either fires when nothing has changed or464 misses a break the week after it lands. The staleness signals below are the465 trigger.466- **How a break surfaces**: from a live call, not from CI. The demo and the tests467 run offline from committed fixtures, so they stay green after an endpoint468 changes. Treat an unexpected HTTP error or an empty result on a real accession469 as the signal, then re-check the fixtures against the live API.470- **Staleness signals**: `/studies/{acc}/info` stops advertising `httpLink`; the471 MAGE-TAB column vocabulary changes; nf-core changes its samplesheet columns.472- **Known debt**: the metadata-table, sanitiser and read-map helper blocks are473 duplicated across the five archive scripts rather than factored into a shared474 module. This is deliberate — it keeps re-syncing with upstream cheap. Revisit475 only if upstream itself deduplicates.476- **Deprecation criteria**: retire if EMBL-EBI folds ArrayExpress fully into the477 generic BioStudies record shape, at which point `biostudies-fetch` would cover478 it.479480## Citations481482- **Ported from**483 [`UKDRI/informatics_data_skills`](https://github.com/UKDRI/informatics_data_skills)484 at commit `7cc3e6e` (`arrayexpress/scripts/arrayexpress.py`), MIT licensed.485 Upstream copyright retained: Copyright (c) 2026 UK Dementia Research Institute.486- **ArrayExpress / BioStudies**: Sarkans et al. (2018) *The BioStudies database —487 one stop shop for all data supporting a life sciences study.* Nucleic Acids488 Research 46(D1):D1266–D1270. <https://doi.org/10.1093/nar/gkx965>489- **Demo fixture**: E-MTAB-10030, "Single-cell RNA-seq of rat microglia in490 monoculture and in coculture with neurons and astrocytes", released491 2021-03-23. The record declares **no explicit licence attribute**; it is492 redistributed here as public archive metadata on the terms EMBL-EBI publishes493 it under. *Rattus norvegicus* throughout, so no human individual-level data.494- **MAGE-TAB**: Rayner et al. (2006) *A simple spreadsheet-based, MIAME-supportive495 format for microarray data: MAGE-TAB.* BMC Bioinformatics 7:489.496 <https://doi.org/10.1186/1471-2105-7-489>