Trust And Methodology Communication

Trust and Methodology Communication

Review Checklist

• Every score shown has a visible signal of its status: provisional scores have an amber badge, final scores have no badge (final is the default/unremarkable state)
• Evidence refs are labeled as "Evidence" (with source) or "Inference" (model reasoning) — these are categorically different and users must be able to distinguish them
• Provisional copy is honest but not alarming — "preliminary" not "unverified"; "may change" not "possibly wrong"
• Methodology disclosure is one click away from every score — a "?" icon or "How is this scored?" link on every lane row
• Methodology explanations are in plain language — no model names, no technical jargon in the user-facing copy
• Dispute acknowledgment shows the specific score/lane the user is questioning — not a generic "contact us" form
• Badge components use aria-label with full text — "Provisional — may change" not just a colored dot
• The methodology drawer closes on Escape and focuses back on the trigger — keyboard accessible
• "How judges score" explainer exists at the bout level, not just per-lane — users need to understand the system before reading lane details
• No score shows "0 confidence" — confidence of null (legacy) renders as N/A; confidence of 0.0 (unlikely but valid) renders as "very low" not "0%"
• Dispute flow captures enough context server-side — submission ID, lane ID, judge ID, user's stated reason
• Final badge is NOT shown on partially-scored results — even if some lanes are complete, partial = provisional

The Trust Gap: Why Black-Box Scoring Fails

Competitors in an AI evaluation platform have one core vulnerability: they invested real effort (and possibly real money) into a submission, and an AI they've never seen gave it a number. Their instinct is suspicion. If you don't proactively explain why and how, you don't get trust — you get disputes, churn, and bad word-of-mouth.

The trust gap has three components:

1. Opacity gap: "I don't know what the AI actually measured"
2. Evidence gap: "I don't know if the score is based on what I actually did"
3. Process gap: "I don't know if the score is final or might change"

Each gap requires a specific design intervention. Closing all three doesn't require a wall of text — it requires the right signal at the right moment.

What builds trust — ordered by impact:

• Specific evidence tied to actual output ("Your agent said: '…' — this reduced the Clarity score")
• Named criteria with plain-language definitions ("Clarity: does the agent communicate its plan?")
• Score lineage ("3 judges scored this; their average was 72.4")
• Honest provisional language ("Preliminary score — may change by up to 5 points")

What destroys trust immediately:

• Generic feedback ("The agent could be more clear")
• Score with no evidence at all
• "Final" label on a score that later changes
• Black-box lane names ("fw_003_composite_eval: 6.2")

Evidence vs Inference Labeling

Evidence and inference are categorically different and must be labeled differently.

// components/trust/EvidenceBadge.tsx
'use client';

import React from 'react';
import { cn } from '@/lib/utils';

export type EvidenceType = 'evidence' | 'inference' | 'metric';

interface EvidenceBadgeProps {
  type: EvidenceType;
  size?: 'sm' | 'xs';
  className?: string;
}

const config: Record<EvidenceType, { label: string; ariaLabel: string; icon: string; styles: string }> = {
  evidence: {
    label: 'Evidence',
    ariaLabel: 'Evidence — directly quoted from agent output',
    icon: '🔎',
    styles: 'bg-blue-50 text-blue-700 border border-blue-200',
  },
  inference: {
    label: 'Inference',
    ariaLabel: 'Inference — judge interpretation, not a direct quote',
    icon: '💭',
    styles: 'bg-purple-50 text-purple-700 border border-purple-200',
  },
  metric: {
    label: 'Metric',
    ariaLabel: 'Measured metric — computed from agent output',
    icon: '📊',
    styles: 'bg-gray-50 text-gray-600 border border-gray-200',
  },
};

export function EvidenceBadge({ type, size = 'sm', className }: EvidenceBadgeProps) {
  const cfg = config[type];
  return (
    <span
      className={cn(
        'inline-flex items-center gap-1 rounded-full font-medium',
        size === 'sm' ? 'px-2 py-0.5 text-xs' : 'px-1.5 py-0.5 text-[10px]',
        cfg.styles,
        className
      )}
      aria-label={cfg.ariaLabel}
      title={cfg.ariaLabel}
    >
      <span aria-hidden="true">{cfg.icon}</span>
      {cfg.label}
    </span>
  );
}

// Usage in evidence ref display
export function EvidenceRefItem({
  type,
  content,
  judgeId,
  confidence,
}: {
  type: EvidenceType;
  content: string;
  judgeId: string;
  confidence: number | null;
}) {
  return (
    <div className="space-y-1.5 py-3 border-b border-gray-100 last:border-0">
      <div className="flex items-center gap-2 flex-wrap">
        <span className="text-xs font-medium text-gray-500 capitalize">{judgeId}</span>
        <EvidenceBadge type={type} size="xs" />
        {confidence !== null && (
          <span className="text-[10px] text-gray-400">
            {(confidence * 100).toFixed(0)}% confidence
          </span>
        )}
      </div>
      {type === 'evidence' ? (
        <blockquote className="text-sm text-gray-700 leading-relaxed pl-3 border-l-2 border-blue-200 italic">
          "{content}"
        </blockquote>
      ) : (
        <p className="text-sm text-gray-700 leading-relaxed pl-3 border-l-2 border-purple-200">
          {content}
        </p>
      )}
    </div>
  );
}

Provisional vs Final Copy Patterns

The scoring lifecycle has three states. Each requires different copy that is honest without being alarming.

// components/trust/ScoreStatusBadge.tsx
'use client';

import React from 'react';
import { cn } from '@/lib/utils';

export type ScoreStatus = 'scoring' | 'provisional' | 'final';

interface ScoreStatusBadgeProps {
  status: ScoreStatus;
  estimatedFinalAt?: Date | null;
}

function formatEta(date: Date): string {
  const diffMs = date.getTime() - Date.now();
  const diffH = Math.round(diffMs / (1000 * 60 * 60));
  if (diffH <= 0) return 'soon';
  if (diffH === 1) return 'in ~1 hour';
  if (diffH < 24) return `in ~${diffH} hours`;
  return `in ~${Math.round(diffH / 24)} days`;
}

export function ScoreStatusBadge({ status, estimatedFinalAt }: ScoreStatusBadgeProps) {
  if (status === 'final') {
    // Final state has no badge — silence signals finality
    return null;
  }

  if (status === 'scoring') {
    return (
      <span
        className="inline-flex items-center gap-1.5 rounded-full bg-amber-50 border border-amber-200 px-2.5 py-1 text-xs font-medium text-amber-700"
        aria-label="Scoring in progress — preliminary results shown"
        role="status"
      >
        <span className="h-1.5 w-1.5 rounded-full bg-amber-400 animate-pulse" aria-hidden="true" />
        Scoring…
      </span>
    );
  }

  // provisional
  const eta = estimatedFinalAt ? formatEta(estimatedFinalAt) : null;
  return (
    <span
      className="inline-flex items-center gap-1.5 rounded-full bg-amber-50 border border-amber-200 px-2.5 py-1 text-xs font-medium text-amber-700"
      aria-label={`Preliminary score${eta ? ` — final expected ${eta}` : ' — may change'}`}
      role="status"
    >
      <span className="h-1.5 w-1.5 rounded-full bg-amber-300" aria-hidden="true" />
      Preliminary{eta ? ` — final ${eta}` : ''}
    </span>
  );
}

// ─── Prose copy patterns for each state ───────────────────────────────────────
// Use these in banners, tooltips, and explanatory text — do not invent new copy.

export const SCORE_STATUS_COPY = {
  scoring: {
    headline: 'Scoring in progress',
    body: 'Your submission is being evaluated. Preliminary scores appear below as judges complete each lane.',
    tooltip: 'These scores are preliminary and may change once all judges complete.',
  },
  provisional: {
    headline: 'Preliminary score',
    body: 'All judges have scored your submission. This score may change slightly during the final aggregation and review period.',
    tooltip: 'Preliminary — final score confirmed after review period closes.',
  },
  final: {
    headline: null,   // no copy needed for final state
    body: null,
    tooltip: 'This score is final.',
  },
} as const;

Methodology Disclosure: Tooltip and Drawer Components

Methodology disclosure must be one click away from every scored element. The pattern: a ? icon triggers a tooltip for brief context and a "Learn more" link opens a drawer with full methodology.

// components/trust/MethodologyTooltip.tsx
'use client';

import React, { useState, useRef, useEffect, useCallback } from 'react';
import { cn } from '@/lib/utils';

export interface LaneMethodology {
  laneId: string;
  laneName: string;
  shortDescription: string;  // 1 sentence, plain language
  whatWeScore: string[];      // 3–5 bullet points, plain language
  whatWeDoNotScore: string[]; // 1–3 things explicitly NOT measured
  maxScore: number;
  weight: number;
}

interface MethodologyTooltipProps {
  methodology: LaneMethodology;
  children: React.ReactNode;
}

export function MethodologyTooltip({ methodology, children }: MethodologyTooltipProps) {
  const [isOpen, setIsOpen] = useState(false);
  const [showDrawer, setShowDrawer] = useState(false);
  const triggerRef = useRef<HTMLButtonElement>(null);

  const close = useCallback(() => {
    setIsOpen(false);
    triggerRef.current?.focus();
  }, []);

  useEffect(() => {
    if (!isOpen) return;
    const handler = (e: KeyboardEvent) => {
      if (e.key === 'Escape') close();
    };
    document.addEventListener('keydown', handler);
    return () => document.removeEventListener('keydown', handler);
  }, [isOpen, close]);

  return (
    <div className="relative inline-flex items-center gap-1.5">
      {children}
      <button
        ref={triggerRef}
        type="button"
        => setIsOpen(!isOpen)}
        className="h-4 w-4 rounded-full bg-gray-100 text-gray-400 hover:bg-gray-200 hover:text-gray-600 transition-colors flex items-center justify-center text-[10px] font-bold"
        aria-label={`How ${methodology.laneName} is scored`}
        aria-expanded={isOpen}
        aria-haspopup="dialog"
      >
        ?
      </button>

      {isOpen && (
        <>
          {/* Backdrop for click-outside close */}
          <div className="fixed inset-0 z-10" aria-hidden="true" />
          {/* Tooltip popover */}
          <div
            role="dialog"
            aria-label={`${methodology.laneName} scoring methodology`}
            className="absolute bottom-full left-0 z-20 mb-2 w-72 rounded-xl bg-white border border-gray-200 shadow-xl p-4 space-y-3"
          >
            <div>
              <h4 className="text-sm font-semibold text-gray-900">{methodology.laneName}</h4>
              <p className="text-xs text-gray-500 mt-0.5">{methodology.shortDescription}</p>
            </div>
            <div>
              <p className="text-xs font-medium text-gray-700 mb-1.5">What we measure:</p>
              <ul className="space-y-1">
                {methodology.whatWeScore.map((item, i) => (
                  <li key={i} className="text-xs text-gray-600 flex items-start gap-1.5">
                    <span className="text-emerald-500 mt-0.5 shrink-0">✓</span>
                    {item}
                  </li>
                ))}
              </ul>
            </div>
            {methodology.whatWeDoNotScore.length > 0 && (
              <div>
                <p className="text-xs font-medium text-gray-700 mb-1.5">Not scored:</p>
                <ul className="space-y-1">
                  {methodology.whatWeDoNotScore.map((item, i) => (
                    <li key={i} className="text-xs text-gray-400 flex items-start gap-1.5">
                      <span className="shrink-0 mt-0.5">—</span>
                      {item}
                    </li>
                  ))}
                </ul>
              </div>
            )}
            <div className="flex items-center justify-between pt-1 border-t border-gray-100">
              <span className="text-xs text-gray-400">
                Max: {methodology.maxScore} · Weight: ×{methodology.weight}
              </span>
              <button
                type="button"
                => { setShowDrawer(true); close(); }}
                className="text-xs text-indigo-600 hover:text-indigo-800 font-medium"
              >
                Full methodology →
              </button>
            </div>
          </div>
        </>
      )}

      {showDrawer && (
        <MethodologyDrawer
          methodology={methodology}
          => {
            setShowDrawer(false);
            triggerRef.current?.focus();
          }}
        />
      )}
    </div>
  );
}

// Full methodology drawer
function MethodologyDrawer({
  methodology,
  onClose,
}: {
  methodology: LaneMethodology;
  onClose: () => void;
}) {
  useEffect(() => {
    const handler = (e: KeyboardEvent) => {
      if (e.key === 'Escape') onClose();
    };
    document.addEventListener('keydown', handler);
    return () => document.removeEventListener('keydown', handler);
  }, [onClose]);

  return (
    <>
      <div className="fixed inset-0 z-30 bg-black/30" aria-hidden="true" />
      <div
        role="dialog"
        aria-label={`Full scoring methodology for ${methodology.laneName}`}
        aria-modal="true"
        className="fixed right-0 top-0 z-40 h-full w-full max-w-md bg-white shadow-2xl overflow-y-auto"
      >
        <div className="p-6 space-y-6">
          <div className="flex items-start justify-between">
            <div>
              <h2 className="text-lg font-bold text-gray-900">{methodology.laneName}</h2>
              <p className="text-sm text-gray-500 mt-1">{methodology.shortDescription}</p>
            </div>
            <button
              type="button"
             
              className="text-gray-400 hover:text-gray-600 text-xl leading-none"
              aria-label="Close methodology"
            >
              ×
            </button>
          </div>

          <section>
            <h3 className="text-sm font-semibold text-gray-900 mb-3">What we measure</h3>
            <ul className="space-y-2">
              {methodology.whatWeScore.map((item, i) => (
                <li key={i} className="flex items-start gap-2 text-sm text-gray-700">
                  <span className="text-emerald-500 mt-0.5 shrink-0">✓</span>
                  {item}
                </li>
              ))}
            </ul>
          </section>

          {methodology.whatWeDoNotScore.length > 0 && (
            <section>
              <h3 className="text-sm font-semibold text-gray-900 mb-3">Not scored in this lane</h3>
              <ul className="space-y-2">
                {methodology.whatWeDoNotScore.map((item, i) => (
                  <li key={i} className="flex items-start gap-2 text-sm text-gray-500">
                    <span className="shrink-0 mt-0.5">—</span>
                    {item}
                  </li>
                ))}
              </ul>
            </section>
          )}

          <section className="rounded-lg bg-gray-50 p-4">
            <h3 className="text-sm font-semibold text-gray-900 mb-2">Scoring weight</h3>
            <p className="text-sm text-gray-600">
              This lane contributes{' '}
              <strong>{((methodology.weight / 1) * 100).toFixed(0)}%</strong>{' '}
              to your overall score, with a maximum of{' '}
              <strong>{methodology.maxScore} points</strong>.
            </p>
          </section>
        </div>
      </div>
    </>
  );
}

Scoring Dispute Acknowledgment

When a user thinks a score is wrong, what they need is:

1. To feel heard (acknowledge their question is legitimate)
2. To understand what the platform can and can't do
3. A specific, low-friction path to flag it

// components/trust/DisputeButton.tsx
'use client';

import React, { useState } from 'react';
import { z } from 'zod';

const DisputeFormSchema = z.object({
  submissionId: z.string().uuid(),
  laneId: z.string().min(1),
  judgeId: z.string().optional(),
  reason: z.string().min(20, 'Please describe what you believe is incorrect (20+ characters)').max(500),
  userContact: z.string().email().optional(),
});

type DisputeFormData = z.infer<typeof DisputeFormSchema>;

interface DisputeButtonProps {
  submissionId: string;
  laneId: string;
  laneName: string;
  score: number;
  maxScore: number;
}

export function DisputeButton({
  submissionId,
  laneId,
  laneName,
  score,
  maxScore,
}: DisputeButtonProps) {
  const [open, setOpen] = useState(false);
  const [reason, setReason] = useState('');
  const [submitted, setSubmitted] = useState(false);
  const [submitting, setSubmitting] = useState(false);
  const [error, setError] = useState<string | null>(null);

  async function handleSubmit(e: React.FormEvent) {
    e.preventDefault();
    setError(null);

    const parsed = DisputeFormSchema.safeParse({
      submissionId,
      laneId,
      reason,
    });

    if (!parsed.success) {
      setError(parsed.error.errors[0]?.message ?? 'Invalid input');
      return;
    }

    setSubmitting(true);
    try {
      const res = await fetch('/api/disputes', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify(parsed.data),
      });
      if (!res.ok) throw new Error('Failed to submit');
      setSubmitted(true);
    } catch {
      setError('Failed to submit. Please try again.');
    } finally {
      setSubmitting(false);
    }
  }

  if (!open) {
    return (
      <button
        type="button"
        => setOpen(true)}
        className="text-xs text-gray-400 hover:text-gray-600 underline transition-colors"
      >
        Question this score
      </button>
    );
  }

  return (
    <div className="mt-3 rounded-lg bg-gray-50 border border-gray-200 p-4 space-y-3">
      {submitted ? (
        <div className="space-y-1">
          <p className="text-sm font-medium text-gray-900">Your question has been logged.</p>
          <p className="text-xs text-gray-500">
            We review disputes weekly. You'll receive an update via email if we identify a scoring issue.
            Scores are rarely changed, but every question is reviewed.
          </p>
          <button
            type="button"
            => { setOpen(false); setSubmitted(false); setReason(''); }}
            className="text-xs text-indigo-600 hover:text-indigo-800 font-medium mt-1"
          >
            Close
          </button>
        </div>
      ) : (
        <>
          <div>
            <p className="text-sm font-semibold text-gray-900">Question this score</p>
            <p className="text-xs text-gray-500 mt-0.5">
              {laneName}: <span className="font-medium text-gray-700">{score}/{maxScore}</span>
            </p>
          </div>
          <p className="text-xs text-gray-500">
            Describe what you believe is incorrect or missing from this evaluation.
            Be specific — vague questions can't be reviewed effectively.
          </p>
          <form className="space-y-3">
            <textarea
              value={reason}
              => setReason(e.target.value)}
              placeholder="e.g., 'The judge cited my agent as missing a verification step, but in line 3 of my output, the agent explicitly calls verify_result() before responding.'"
              className="w-full text-xs rounded-lg border border-gray-200 bg-white px-3 py-2 text-gray-700 placeholder-gray-300 focus:border-indigo-400 focus:outline-none resize-none"
              rows={4}
              maxLength={500}
              aria-label="Describe the scoring issue"
            />
            {error && <p className="text-xs text-red-500">{error}</p>}
            <div className="flex items-center gap-2 justify-end">
              <button
                type="button"
                => setOpen(false)}
                className="text-xs text-gray-400 hover:text-gray-600"
              >
                Cancel
              </button>
              <button
                type="submit"
                disabled={submitting || reason.length < 20}
                className="rounded-lg bg-indigo-600 px-3 py-1.5 text-xs font-medium text-white hover:bg-indigo-700 disabled:opacity-50 disabled:cursor-not-allowed"
              >
                {submitting ? 'Submitting…' : 'Submit question'}
              </button>
            </div>
          </form>
        </>
      )}
    </div>
  );
}

Anti-Patterns

❌ Generic "AI may make mistakes" disclaimer instead of specific methodology

// BAD — builds zero trust, just creates anxiety
<p className="text-xs text-gray-400">
  Scores are generated by AI and may not be perfectly accurate.
</p>

// GOOD — specific, names what is measured
<MethodologyTooltip methodology={laneMethodology}>
  <span className="text-sm font-medium text-gray-700">{lane.laneName}</span>
</MethodologyTooltip>

❌ Showing "Final" on partially-scored results

// BAD — if any judge is pending, the score is NOT final
const isFinal = result.overallScore !== null;  // WRONG
{isFinal && <span>Final score</span>}

// GOOD
const isFinal = result.isFullyScored && !result.inReviewPeriod;
{isFinal ? null : <ScoreStatusBadge status={result.scoreStatus} />}
// Final state has NO badge — silence signals finality

❌ Hiding dispute option because you're afraid of disputes

// BAD — no dispute path = users feel powerless = distrust compounds
// (just don't render DisputeButton at all — common omission)

// GOOD — dispute option is visible, even if rarely used
<DisputeButton submissionId={id} laneId={lane.laneId} ... />
// The existence of a dispute path signals confidence in the scoring

Common Failures to Catch in Review

Changelog

• 2026-03-31: Created for Bouts premium feedback system build