Iblai Vibe Agent Evals

/iblai-vibe-agent-evals

Add the agent Evals tab -- run your agent against a benchmark (a set of test questions) and score every response, so you can catch weak spots and track improvements as you refine the agent. The tab bundles several surfaces into one component: a benchmark picker, an evaluations table with status badges and a per-row actions menu (View results / New review / Check status / Export CSV / Delete), a New Evaluation dialog, an embedded Manage benchmarks dialog (create benchmarks, add Q&A pairs manually or via CSV upload), a results modal showing every trace with its scores and LLM-as-Judge reviews, and an Evaluate (new review) modal with an LLM provider/model picker. This is one tab in the wider agent-settings family (access, api, datasets, disclaimers, embed, evals, history, llm, memory, prompts, safety, settings, tasks, tools). Each tab is a separate skill. All tabs share the same AgentSettingsProvider wrapper -- set it up once and mount as many tabs as you need.

Common setup (brand, conventions, env files, verification): see docs/skill-setup.md.

Prerequisites

• Auth must be set up first (/iblai-vibe-auth)
• MCP server + skills configured (@iblai/mcp in .mcp.json)
• AgentSettingsProvider must wrap the route (see /iblai-vibe-agent-setting Step 2 if not already set up)
• Ask the user for a real mentorId (agent UUID). Do NOT invent one.

Step 1: Check Environment

Before proceeding, check for an iblai.env in the project root. Look for PLATFORM, DOMAIN, and TOKEN variables. If the file does not exist or is missing these variables, tell the user: "You need an iblai.env with your platform configuration. Download the template and fill in your values: curl -o iblai.env https://raw.githubusercontent.com/iblai/vibe/refs/heads/main/iblai.env"

Step 2: Mount AgentEvaluationTab

AgentEvaluationTab has one required prop: getLLMProviderDetails. This maps a provider name to display info (logo URL, display name) for the LLM picker inside the Evaluate (new review) modal. The host app provides this because logos and display names are host-specific -- same contract as the LLM tab (/iblai-vibe-agent-llm); if that tab is already mounted, reuse the same function.

// app/(app)/agents/[mentorId]/evals/page.tsx
"use client";

import {
  AgentEvaluationTab,
  type LLMProviderDetails,
} from "@iblai/iblai-js/web-containers/next";

function getLLMProviderDetails(
  providerName: string,
  llmName?: string,
): LLMProviderDetails {
  const providers: Record<string, LLMProviderDetails> = {
    openai: { name: "OpenAI", logo: "/logos/openai.svg" },
    anthropic: { name: "Anthropic", logo: "/logos/anthropic.svg" },
    google: { name: "Google", logo: "/logos/google.svg" },
  };
  return (
    providers[providerName] ?? {
      name: providerName,
      logo: "/logos/default.svg",
    }
  );
}

export default function AgentEvalsPage() {
  return (
    <div className="flex h-full flex-col bg-white">
      <AgentEvaluationTab getLLMProviderDetails={getLLMProviderDetails} />
    </div>
  );
}

AgentEvaluationTab reads tenantKey, mentorId, and username from AgentSettingsProvider (via useAgentSettings()) and handles all of its own data fetching and mutations (benchmarks, runs, reviews, scores, export, delete).

With custom pagination

The evaluations table paginates (10 per page). Without a PaginationComponent the built-in web-containers AdvancedPagination is used. Inject your own to match the host app's pagination UI:

<AgentEvaluationTab
  getLLMProviderDetails={getLLMProviderDetails}
  PaginationComponent={({ currentPage, totalPages, onPageChange, disabled }) => (
    <MyPagination
      page={currentPage}
      total={totalPages}
     
      disabled={disabled}
    />
  )}
/>;

Step 3: Customize Labels (Optional)

AgentEvaluationTab renders with the default agent-facing copy (AGENT_EVALUATION_TAB_LABELS). Override any string via the labels prop. Pass a full EvaluationTabLabels bundle (for a full re-skin) or a partial object (for one-off edits).

import {
  AgentEvaluationTab,
  type EvaluationTabLabels,
  AGENT_EVALUATION_TAB_LABELS,
} from "@iblai/iblai-js/web-containers/next";

<AgentEvaluationTab
  getLLMProviderDetails={getLLMProviderDetails}
  labels={{
    header: {
      title: "Benchmarks",
      description: "Score this agent against your test sets.",
    },
    actions: { newExperiment: "Run benchmark" },
  }}
/>;

The label bundle is grouped by surface: header, benchmarkPicker, actions, checkStatusToast, table, startExperimentModal, manageBenchmarksModal, detailModal, evaluateModal, and deleteModal. Override only the keys you need.

Step 4: Use MCP Tools for Customization

get_component_info("AgentEvaluationTab")
get_component_info("AgentSettingsProvider")

Tab surfaces

AgentEvaluationTab is a single mounted component, but it renders several distinct surfaces. The corresponding web-containers source (under web-containers/.../edit-mentor-modal/tabs/evaluation-tab) is listed for reference -- these pieces are internal to AgentEvaluationTab and are not mounted separately.

Evaluations run in the background: the table shows a status badge (COMPLETED green / FAILED red / anything else blue) and Check status refetches a single run and pushes the fresh status into the table. View results, New review, and Export CSV are disabled until the run completes.

<AgentEvaluationTab> Props

Import from @iblai/iblai-js/web-containers/next.

Related Exports

From @iblai/iblai-js/web-containers/next:

• AGENT_EVALUATION_TAB_LABELS -- the default agent-facing label bundle.
• EvaluationTabLabels -- type for the full label bundle.
• AgentEvaluationTabProps -- props type for the tab.
• LLMProviderDetails -- type for the return value of getLLMProviderDetails (shared with /iblai-vibe-agent-llm).

From @iblai/data-layer (the same RTK Query hooks the tab uses, for custom UI): useListEvalDatasetsQuery, useListEvalRunsQuery, useStartEvalRunMutation, useGetEvalRunQuery, useEvaluateEvalRunMutation, useListEvalRunJudgesQuery, useCreateEvalScoreMutation, useDeleteEvalScoreMutation, useExportEvalRunMutation, useDeleteEvalRunMutation, plus the EvalDataset, EvalDatasetRun, EvalDatasetRunDetail, EvalJudgeTask, EvalScore payload types.

Step 5: Verify

Run /iblai-vibe-ops-test before telling the user the work is ready:

1. pnpm build -- must pass with zero errors
2. pnpm test -- vitest must pass
3. Start dev server and touch test:

   pnpm dev &
   npx playwright screenshot http://localhost:3000/agents/<id>/evals /tmp/agent-evals.png
   

Important Notes

• Redux store: Must include mentorReducer and mentorMiddleware -- the evaluations API slice (datasets / runs / judges / scores) ships inside those bundles.
• initializeDataLayer(): 5 args (v1.2+)
• @reduxjs/toolkit: Deduplicated via webpack aliases in next.config.ts
• Peer deps: sonner and @iblai/iblai-web-mentor must be installed (pnpm add sonner @iblai/iblai-web-mentor)
• Shared provider: AgentSettingsProvider must wrap the route at a layout level. See /iblai-vibe-agent-setting Step 2 for the full snippet.
• Benchmarks are organization-wide, runs are agent-scoped: a benchmark can be evaluated against multiple agents. The tab filters runs client-side to the current agent via metadata.mentor_unique_id; pending runs without populated metadata are kept only when the current user initiated them (their freshly-started evaluation).
• Manual scores are 0-1: the Add-a-score form requires a numeric value between 0 and 1. Score writes are ingested asynchronously by the eval store -- the modal patches its cache optimistically and the real record appears on the next refresh.
• CSV format (benchmark Q&A upload): UTF-8, required input column, optional expected_output column, max 10 MB / 10,000 rows; rows with a blank question are skipped.
• First benchmark auto-selected: the combobox picks the first benchmark automatically; New Evaluation stays disabled until a benchmark exists (use Manage benchmarks to create one).
• Brand guidelines: BRAND.md

Evaluations REST API

For custom UI beyond <AgentEvaluationTab>. All endpoints are prefixed with ${dmUrl}/api/ai-mentor/orgs/{org}/users/{user_id}/evaluations where dmUrl is NEXT_PUBLIC_API_BASE_URL.

Benchmarks (datasets)

Evaluation runs

Start-run body (bind the run to the agent being evaluated):

{
  "mentor_unique_id": "agent-uuid",
  "run_name": "qa-eval-v1"
}

run_name is optional -- the backend auto-generates one (e.g. run-091c4937) when omitted.

LLM-as-Judge reviews

Review body:

{
  "criteria": "Evaluate the response on accuracy",
  "score_name": "accuracy",
  "llm_provider": "openai",
  "llm_name": "gpt-5",
  "max_concurrency": 4
}

Scores

Task status polling (used by Check status) goes through /api/ai-mentor/orgs/{org}/users/{user_id}/tasks/{task_id}.