Warbert Web API Recommendation Eval

warbert-web-api-recommendation-eval

WARBERT: A Hierarchical BERT-based Model for Web API Recommendation — Xu et al. (2025) (arXiv:2509.23175, 2025)

What this evaluates

Evaluates a model's ability to recommend relevant Web APIs for a given mashup application based on textual descriptions. It also probes multi-task learning capability through an auxiliary mashup category classification task.

Datasets

• ProgrammableWeb — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Precision@N (primary) — range: [0, 1]
  • Ratio of relevant APIs in the top-N recommended list to N. Formula: |APIs_real ∩ APIs_top-N| / |APIs_top-N|.
• Recall@N — range: [0, 1]
  • Ratio of relevant APIs in the top-N recommended list to the total number of relevant APIs. Formula: |APIs_real ∩ APIs_top-N| / |APIs_real|.
• MAP@N — range: [0, 1]
  • Mean Average Precision across all mashups. AP@N averages Precision@i at each position i where a relevant API is found, then MAP averages AP across all queries.
• NDCG@N — range: [0, 1]
  • Normalized Discounted Cumulative Gain. DCG@N sums relevance (1/0) discounted by log2(i+1) for top-N positions. NDCG normalizes DCG by IDCG (ideal DCG).

Input / output format

Input: Mashup name, description, and category; API name and description. Text sequences are truncated or padded to a maximum length of 256 tokens.

Output: A ranked list of candidate APIs for each mashup, evaluated at fixed top-N cutoffs (N ∈ {1, 5, 10}).

Scoring recipe

def compute_metrics(recommended, relevant, top_n):
    rel = set(relevant)
    top_k = recommended[:top_n]
    hits = len(set(top_k) & rel)
    prec = hits / top_n
    rec = hits / len(rel) if len(rel) > 0 else 0.0
    dcg = sum(1.0 / math.log2(i + 2) for i, api in enumerate(top_k) if api in rel)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(rel), top_n)))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    ap = sum(prec / (i + 1) for i, api in enumerate(top_k) if api in rel) / len(rel) if len(rel) > 0 else 0.0
    return prec, rec, ndcg, ap

Common pitfalls

• Dataset split is performed only on mashups, while all 1647 APIs are retained in the candidate repository for every query.
• Positive pair ratio is extremely low (0.125%), meaning the vast majority of mashup-API pairs are negatives, requiring careful handling of negative sampling or ranking evaluation.
• Metrics are computed at fixed cutoffs (N=1, 5, 10) rather than over the full ranked list, which can mask performance differences at higher ranks.

Evidence (verbatim from paper)

Following MTFM, we use the Mashup and API data crawled in ProgrammableWeb. Since the MTFM model requires data with three attributes: name, description and category. MTFM removes crawled data with missing attribute values. Finally, the dataset contains 8217 mashups and 1647 APIs. To conduct performance evaluation, the dataset is split into the training, validation, and test sets using the ratios of 3:1:1. Note that we only split mashups and all the APIs are used as the API repository. Precision@N refers to the ratio of the number of real hits in the recommended APIs to the number of recommended APIs. Recall@N refers to the ratio of the number of real hits in the recommended APIs to the number of all real APIs. We use N as 1, 5 and 10.

Citation

@misc{xu2025warbert,
  title={WARBERT: A Hierarchical BERT-based Model for Web API Recommendation},
  author={Xu et al. (2025)},
  year={2025},
  note={arXiv:2509.23175}
}

• arXiv: 2509.23175