← all publishers

ByteStack-Labs

@bytestack-labs source repo

Report this profile Claim this profile

• 4 skills
• 0 followers
• 7 hours ago last updated

1. ▌
   Tool Eval · bytestack-labs

   Verifies the tools an agent or orchestrator depends on, by re-deriving a tool evaluation's real accuracy instead of trusting a single pass/fail score. Separates a formatting miss (a correct value scored wrong) from a real failure (a wrong value scored right), recomputes the expected answer from the raw inputs rather than a stored ground truth, and surfaces the silent-wrong cases a naive exact-match scorer rubber-stamps. Produces a reproducible tool-reliability receipt and keeps any model's qualitative feedback on tool naming, docs, or errors as a clearly labeled judge layer, never mixed into the number. Use this whenever someone runs a tool or function-calling eval and the score looks fine, when an agent's tool results are "mostly right," when a calculator, extractor, API, or MCP tool eval reports a pass rate you are about to trust, or when you suspect a scorer is grading format instead of value. Trigger on mentions of tool eval, function-calling accuracy, exact-match scoring, ground-truth mismatch, format fa
   0
   installs
2. ▌
   Trajectory Eval · bytestack-labs

   Evaluates whether an agent or a multi-step or multi-agent system actually completes the task, not just whether each step or the final answer looks right. Captures full trajectories (every step, tool call, and intermediate state), measures per-step success against end-to-end success, exposes the compounding-error curve where high per-step reliability still collapses over a long trajectory, attributes failures to the step that introduces them, and tests whether composing or chaining agents helps or hurts. Use this whenever an agent "works in the demo but fails with real users," passes its individual steps but fails the task overall, when multi-agent "synergy" is assumed rather than measured, for long-horizon or tool-using agents, or to benchmark agents head to head on real tasks. Trigger on mentions of agent reliability, tool calls, multi-step, multi-agent, orchestration, compounding errors, trajectory, long-horizon tasks, or agent evaluation. This is the agent depth that production-autopsy hands off to. Every
   0
   installs
3. ▌
   Calibration Guard · bytestack-labs

   Detects and quantifies confidently-wrong behavior: where a model, classifier, agent, or LLM judge is highly confident and incorrect, and where the coupling between confidence and correctness breaks down under distribution shift. Measures calibration on the in-distribution set and again on the production distribution, builds reliability diagrams, reports expected calibration error and the high-confidence error rate, and produces a selective-prediction or escalation rule you can wire into monitoring. Use this whenever someone says a system is "confident but wrong," "sure of itself and failing," or "silently wrong," when accuracy looks fine but trust is eroding, when an LLM judge's scores stop tracking human labels, or when production errors arrive at high confidence with nothing flagging them. Trigger on mentions of calibration, reliability diagram, ECE, overconfidence, false reassurance, abstention, selective prediction, or confidence thresholds. This is the deep confidence workup that production-autopsy hands
   0
   installs
4. ▌
   Production Autopsy · bytestack-labs bundle

   Start here. Audits a deployed ML or LLM or agent system that scores well on evaluation but fails, regresses, or behaves unexpectedly in production. Runs a reproducible root-cause "autopsy": frames the eval-to-deployment gap, reproduces the production failure, quantifies it by slice, tests confidence calibration under distribution shift, isolates root cause by ablation, and produces a prioritized diagnostic report with next steps. It also triages across reliability areas and hands off to the deeper skills in this plugin (calibration-guard and trajectory-eval today, with coverage-audit, load-readiness, and drift-watch planned) where a finding needs more depth. Use this whenever someone says a model "passes every benchmark but fails in prod," "the eval numbers don't hold up," "it's confidently wrong," an agent "works in the demo and breaks with real users," or asks to investigate silent failures, distribution shift, calibration problems, a benchmark-vs-reality discrepancy, or why a fine-tuned model or agent degr
   0
   installs