# Multi Dataset Unification Pipeline

> When the user requests to find, analyze, and merge multiple datasets from Hugging Face or similar repositories into a unified format. This skill handles the complete pipeline searching for datasets, examining their structure, parsing format specifications, converting diverse dataset formats (ToolACE, Glaive, XLAM, etc.) into a standardized schema, and merging them into a single output file. Key triggers include requests involving 'dataset unification', 'format conversion', 'merge datasets', 'Hugging Face datasets', 'tool-calling datasets', 'JSONL output', or when users provide a format specification document like 'unified_format.md'.

- Skill: `majiayu000/multi-dataset-unification-pipeline` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds add majiayu000/multi-dataset-unification-pipeline`
- Raw SKILL.md: https://api.skillmd.com/api/skills/majiayu000/multi-dataset-unification-pipeline/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: majiayu000 (https://skillmd.com/u/majiayu000)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/majiayu000/multi-dataset-unification-pipeline

---

# Instructions

## 1. Understand the Request
- Identify the target datasets (names, IDs, or search queries).
- Locate and read any provided format specification (e.g., `unified_format.md`).
- Clarify the scope: number of entries per dataset, output file name/location.

## 2. Search and Validate Datasets
- Use `huggingface-dataset_search` to find each dataset.
- Use `huggingface-hub_repo_details` to get metadata and confirm availability.
- Note the dataset size and structure.

## 3. Analyze Dataset Structures
- Load a sample from each dataset using `datasets.load_dataset`.
- Examine the column names and a few sample entries.
- Identify the native format (e.g., ToolACE uses `system`/`conversations`, Glaive uses `conversations`/`tools`, XLAM uses `query`/`answers`/`tools`).

## 4. Convert to Unified Format
- **Always** refer to the provided `unified_format.md` (or equivalent) for the target schema.
- **Key Rules**:
    - `conversation_id`: Format as `{source_short_name}_{index}` (e.g., `toolace_0`).
    - `messages`: List of messages with `role` (`user`/`assistant`/`tool`), `content`, and optionally `tool_calls` or `tool_call_id`.
    - `tool_calls`: For assistant messages, include `id` (format `tool_call_{n}`), `name`, `arguments`.
    - `tools`: List of normalized tool definitions.
    - Remove system messages if they only contain tool instructions.
    - If assistant only made tool calls, set `content` to `null`.
    - Do not add tool return results or assistant replies not in the original data.
- Use the bundled `scripts/convert_and_merge.py` for reliable, deterministic conversion of ToolACE, Glaive, and XLAM formats.
- For new dataset formats, write a custom converter following the patterns in the script.

## 5. Merge and Output
- Limit entries per dataset as requested (default: first 500).
- Merge all converted entries into a single list.
- Write to a JSONL file in the workspace (default: `unified_tool_call.jsonl`).
- Verify the output: count entries, check file size, validate format against the specification.

## 6. Finalize
- Provide a summary: datasets found, entries converted, output location.
- Optionally, show a sample entry from each source for validation.
- Confirm the task is complete.