Data & Analytics

qhjqhj00 Skill

Evaluates an LLM's ability to generate syntactically correct and semantically accurate SQL queries from natural language questions over large, real-world databases. It probes database schema understanding, value matching, external knowledge incorporation, and query execution efficiency. Use when the user wants to benchmark on BIRD, or asks about evaluating this task. Reports Execution Accuracy (EX).

qhjqhj00 Skill

Evaluates the ability of LLMs to generate correct SQL queries from natural language questions in financial domains. It tests schema linking, cross-database transfer, and output calibration capabilities specific to fund, stock, and macroeconomic data. Use when the user wants to benchmark on BULL, or asks about evaluating this task. Reports execution accuracy (EX).

qhjqhj00 Skill

Evaluates the data loading speed and rendering interactivity of the encube visual analytics framework on a tiled display system under varying data volumes and GPU memory constraints. Use when the user has predictions and gold and needs to compute Load time ($T_{\mathrm{Load}}$).

qhjqhj00 Skill

Evaluates text-to-SQL models on naturally occurring, under-specified user queries from Stack Exchange. Probes the model's ability to handle real-world ambiguity, nested subqueries, parameterized queries, and domain-specific schema knowledge without relying on perfectly specified instructions. Use when the user wants to benchmark on SEDE, or asks about evaluating this task. Reports PCM-F1.

qhjqhj00 Skill

Evaluates conversational text-to-SQL systems on cross-domain database querying. It probes dialogue state tracking via SQL grounding, response generation from query results, and user intent/dialogue act prediction under real-world ambiguity and clarification dynamics. Use when the user wants to benchmark on CoSQL, or asks about evaluating this task. Reports Question Match.

qhjqhj00 Skill

Probes cross-domain semantic parsing in context by requiring models to generate sequential SQL queries across multiple conversational turns. It evaluates the ability to maintain state, handle thematic evolution, and generalize to unseen databases while correctly resolving contextual dependencies. Use when the user wants to benchmark on SParC, or asks about evaluating this task. Reports question match.

qhjqhj00 Skill

Evaluates text-to-SQL models on compositional generalization, out-of-domain robustness, and schema-question alignment across 18 diverse datasets and 12 domains. It probes the model's ability to handle long-form query decomposition and cross-domain SQL pattern diversity. Use when the user wants to benchmark on UNITE, or asks about evaluating this task. Reports accuracy.

qhjqhj00 Skill

Evaluates a model's ability to generate diverse, semantically valid SQL queries when a natural language question is ambiguous. It specifically probes whether the model can cover multiple valid interpretations of the same query within a limited set of top-k outputs. Use when the user wants to benchmark on AmbiQT, SPIDER, Kaggle DBQA, or asks about evaluating this task. Reports BothInTopK.

qhjqhj00 Skill

Evaluates large language models' ability to generate correct Python code for interactive data science notebooks, requiring multi-turn reasoning, grounded understanding of DataFrame schemas, and composition of pandas API calls based on preceding notebook context and natural language intents. Use when the user wants to benchmark on ARCADE, or asks about evaluating this task. Reports pass@k.

qhjqhj00 Skill

Probes LLMs' ability to generate correct SQL queries from natural language questions over complex, enterprise-scale databases. It specifically evaluates handling of high schema complexity, multi-table joins, aggregations, and column-to-table mapping in real-world business contexts. Use when the user wants to benchmark on BEAVER, or asks about evaluating this task. Reports execution accuracy.

qhjqhj00 Skill

This benchmark evaluates an LLM's ability to accurately translate SQL queries across different database systems and dialects. It probes the model's capacity to handle system-specific syntax, semantic equivalence, and edge-case safeguards without relying on superficial string matching. Use when the user wants to benchmark on PARROT, or asks about evaluating this task. Reports Acc_EX.

qhjqhj00 Skill

Evaluates the ability of LLMs and hybrid QA systems to perform tree-structured, multi-hop reasoning over combined text and table data, including complex SQL operations like aggregation, grouping, ordering, and cross-modal retrieval. Use when the user wants to benchmark on SPARTA, or asks about evaluating this task. Reports F1.

qhjqhj00 Skill

Evaluates a model's ability to translate natural language questions into correct SQL queries across diverse database domains. It probes schema linking, lexical matching, and complex query synthesis including joins, aggregations, and subqueries. Use when the user wants to benchmark on Spider, or asks about evaluating this task. Reports exact matching accuracy.

qhjqhj00 Skill

Evaluates a Text-to-SQL system's ability to generate correct SQL from ambiguous natural language queries when integrated with an interactive ambiguity resolution module. It also measures the system's precision, recall, and F1 in detecting and classifying specific types of schema-mapping and reasoning ambiguities. Use when the user wants to benchmark on AmbiSQL Constructed Dataset, or asks about evaluating this task. Reports Exact Match accuracy.

qhjqhj00 Skill

Evaluates clinical text-to-SQL capabilities by requiring models to generate executable BigQuery queries that perform multi-table joins, temporal reasoning, and patient-similarity cohort analysis on electronic health record data. Use when the user wants to benchmark on CLINSQL, or asks about evaluating this task. Reports Execution Score.

qhjqhj00 Skill

Evaluates a model's ability to generate syntactically and semantically correct SQL queries from natural language questions across diverse database schemas. It probes schema linking, handling of complex query structures (joins, nested subqueries, aggregations), and the capacity for iterative self-correction when initial generations fail. Use when the user wants to benchmark on Spider, or asks about evaluating this task. Reports Execution Accuracy (EX).

qhjqhj00 Skill

Evaluates cross-lingual semantic parsing and Text-to-SQL capabilities across Indian languages, probing a model's ability to link natural language queries to complex, high-join-depth relational schemas and generate syntactically and semantically correct SQL. Use when the user wants to benchmark on IndicDB, or asks about evaluating this task. Reports Execution Accuracy (EX).

qhjqhj00 Skill

This evaluation probes a model's ability to formulate correct SQL queries from natural language questions, specifically focusing on capturing structural semantics like GROUP BY, HAVING, ORDER BY, and set operations. It measures how well prompt engineering techniques or fine-tuning improve SQL generation accuracy across different database schemas and question complexities. Use when the user wants to benchmark on StrucQL, Spider, Bird, or asks about evaluating this task. Reports execution accuracy (EX).

qhjqhj00 Skill

Evaluates the privacy-preserving text-to-SQL generation capability of LLMs. It measures execution accuracy against ground-truth SQL while quantifying privacy protection through token abstraction recall and adversarial re-identification resistance. Use when the user wants to benchmark on BIRD, or asks about evaluating this task. Reports Execution Accuracy.

qhjqhj00 Skill

Evaluates LLM-based natural language question answering over heterogeneous data sources by testing the model's ability to generate SQL queries that correctly invoke both database tables and external APIs. It probes complex query planning, API sequencing, and routing across mixed data modalities. Use when the user wants to benchmark on Spider (modified with API-replaced tables), or asks about evaluating this task. Reports execution accuracy.

qhjqhj00 Skill

Evaluates a model's ability to translate natural language questions into executable SQL queries against a given database schema. It probes semantic parsing, table understanding, and the generation of syntactically and semantically correct structured queries. Use when the user wants to benchmark on WikiSQL, or asks about evaluating this task. Reports Acc_ex.

qhjqhj00 Skill

This benchmark probes a model's ability to perform compositional generalization in context-dependent Text-to-SQL. It evaluates whether models can correctly combine previously seen SQL query structures with novel modification patterns (e.g., new WHERE or ORDER BY clauses) in multi-turn dialogues. Use when the user wants to benchmark on CoSQL-CG, or asks about evaluating this task. Reports question match (QM).

qhjqhj00 Skill

Evaluates LLMs' ability to reason about data warehouse graph topologies, specifically focusing on foreign key path enumeration, data lineage impact analysis, and multi-hop graph traversal. It probes whether models can perform structural graph reasoning versus relying on lexical cues, using heterogeneous schema graphs with foreign key and lineage edges. Use when the user wants to benchmark on DW-Bench, or asks about evaluating this task. Reports Micro-EM.

qhjqhj00 Skill

This benchmark evaluates an LLM's ability to perform complex multi-step logical reasoning and chain-of-thought decomposition to generate correct SQL queries from natural language questions. It probes capabilities in mathematical deduction, physical knowledge integration, and cross-domain analytical querying. Use when the user wants to benchmark on LogicCat, or asks about evaluating this task. Reports Execution Accuracy (EX).

qhjqhj00 Skill

Evaluates a model's ability to generate correct SQL queries from natural language clinical questions in the healthcare domain. It specifically probes retrieval-based reasoning, robustness to noisy or ambiguous medical terminology, and generalization under data scarcity conditions. Use when the user wants to benchmark on MIMICSQL, or asks about evaluating this task. Reports Execution Accuracy.

qhjqhj00 Skill

This benchmark probes a model's ability to perform compositional generalization in context-dependent Text-to-SQL. It evaluates whether models can correctly combine previously seen SQL query structures with novel modification patterns (e.g., new WHERE or ORDER BY clauses) in multi-turn dialogues. Use when the user wants to benchmark on SPARC-CG, or asks about evaluating this task. Reports question match (QM).

qhjqhj00 Skill

Evaluates an LLM's ability to generate correct SQL queries from natural language questions over complex, multi-table database schemas. It probes the model's reasoning capabilities and schema generalization by requiring step-by-step rationales and testing on unseen databases. Use when the user wants to benchmark on Spider, or asks about evaluating this task. Reports execution accuracy (EX).

qhjqhj00 Skill

This benchmark evaluates an LLM's ability to generate and execute hybrid relational queries that combine traditional SQL operations with semantic reasoning over textual data. It probes capabilities like semantic joins, information extraction, and multi-hop reasoning by measuring execution accuracy against expert-verified ground truth. Use when the user wants to benchmark on TAG+, or asks about evaluating this task. Reports execution accuracy.

qhjqhj00 Skill

This benchmark evaluates a model's ability to classify the intention of a natural language question relative to a database schema. It probes whether the model can distinguish between answerable queries, questions requiring external knowledge, ambiguous queries, grammatically invalid non-SQL questions, and questions unrelated to the schema. Use when the user wants to benchmark on TRIAGESQL, or asks about evaluating this task. Reports Macro F1.

qhjqhj00 Skill

Evaluates the ability of text-to-SQL models to generate correct SQL or GQL queries for natural language questions across relational and graph databases. It measures execution accuracy by comparing the runtime results of generated queries against reference queries on specific database instances. Use when the user wants to benchmark on Spider, Bird, SQL-Eval, NL2GQL, or asks about evaluating this task. Reports Execution Accuracy (EX).

qhjqhj00 Skill

Evaluates LLM-driven document analysis agents on end-to-end data analytics workflows, including question answering, data visualization, and file generation. It probes multi-step numerical reasoning, cross-data consistency, and long-horizon planning on heterogeneous real-world documents. Use when the user wants to benchmark on AIDABench, or asks about evaluating this task. Reports Pass@3.

qhjqhj00 Skill

Evaluates the quality of English-to-Arabic translations for text-to-SQL tasks and measures the execution accuracy of generated SQL queries on Arabic natural language questions. Use when the user wants to benchmark on AraSpider, or asks about evaluating this task. Reports Execution accuracy.

qhjqhj00 Skill

Evaluates the robustness of text-to-SQL models against semantic-preserving and semantic-changing perturbations across database schemas, natural language questions, and SQL queries. It measures how well models maintain execution accuracy when inputs are altered, revealing architectural vulnerabilities in entity linking, decoder design, and value prediction. Use when the user wants to benchmark on Dr.Spider, or asks about evaluating this task. Reports execution accuracy (EX).

qhjqhj00 Skill

Evaluates an LLM-based agent's ability to automate cohort selection, feature extraction, and clinical code standardization across heterogeneous Electronic Medical Record (EMR) databases. It probes schema-aware SQL reasoning, iterative query planning, and robustness to unseen database structures without manual rule engineering. Use when the user wants to benchmark on MIMIC-III, eICU, SICdb, or asks about evaluating this task. Reports F1.

qhjqhj00 Skill

Evaluates small language models' capability to adapt to and execute diverse tool-use tasks, including REST API calls, complex SQL queries, web navigation, and command-line interactions. It measures both functional correctness and efficiency under few-shot prompting and lightweight adaptation. Use when the user wants to benchmark on Gorilla APIBench (BFCL V4), Spider 2.0 (Enterprise Subset), WebArena, InterCode (Bash & CTF), or asks about evaluating this task. Reports Execution Success Rate (SR).

qhjqhj00 Skill

This benchmark evaluates a model's ability to translate natural language questions into correct SQL queries across diverse, real-world database schemas. It specifically probes the model's capacity to handle complex, multi-operation statements and cross-domain syntax structures that are often underrepresented in traditional benchmarks. Use when the user wants to benchmark on SQL-Synth, or asks about evaluating this task. Reports execution accuracy (EX).