Data Sufficiency Evaluation And Unanswerability Detection

Skill: data sufficiency evaluation and unanswerability detection

1. Capability Definition & Real Case

• Professional Definition: This capability involves the systematic evaluation of data environments to determine if they contain the necessary variables, cohorts, and granularity to support a specific analytical request. It emphasizes non-verifiability detection, where the agent must critically identify structural data absences and confidently declare the task unanswerable, thereby avoiding hallucinations or proxy-based guessing.
• Dimension Hierarchy: Data Grounding->Data Retrieval & Semantic Grounding->data sufficiency evaluation and unanswerability detection

Real Case

[Case 1]

• Initial Environment: A data sandbox contains clinical archives for cancer patients but lacks mutational burden records or labels mapping the exact locations of distant metastases.
• Real Question: Do patients with brain metastases have a significantly higher mutational burden compared to those with non-brain metastases?
• Real Trajectory: Inspect the clinical tables for columns related to 'brain_metastasis' or 'mutational_burden'; identify that while the patient cohort exists, there is no column defining the location of metastases or the mutational burden metric; instead of attempting a guess, state that the dataset is missing the critical grouping and outcome variables; conclude the request is unanswerable.
• Real Answer: Unanswerable. The dataset lacks mutational burden records and brain metastasis location labels.
• Why this demonstrates the capability: The agent avoids the common error of misidentifying nearest-neighbor columns or hallucinating a result, correctly identifying that the specific comparative evidence required for the analytical request is physically absent from the environment.

[Case 2]

• Initial Environment: An e-commerce dataset containing product names, categories, and review scores, but entirely missing any sales volume, quantity, or revenue metrics.
• Real Question: Which product category generated the highest total revenue in Q3 2023?
• Real Trajectory: Start searching for revenue, sales, or price/quantity columns; read the headers and schemas of all available tables; observe that only review and product metadata information is present; halt the analysis and declare that revenue cannot be calculated without sales volume or unit price data.
• Real Answer: Unanswerable due to missing revenue and sales data.
• Why this demonstrates the capability: The question explicitly asks for a metric that requires data structurally unavailable in the table. The agent must successfully evaluate the schema and recognize the insufficiency before confidently refusing to compute a number, directly testing capability of unanswerability detection.

Pipeline Execution Instructions

To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:

1. Phase 1: Environment Exploration Read the exploration guidelines to discover raw knowledge seeds: references/EXPLORATION.md

2. Phase 2: Trajectory Selection Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: references/SELECTION.md

3. Phase 3: Data Synthesis Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: references/SYNTHESIS.md