Hiring Scorecard Builder

Hiring Scorecard Builder

Unstructured interviews have poor predictive validity and amplify the biases of whoever talks most. A scorecard forces interviewers to decide what matters before meeting the candidate and to gather evidence, not impressions.

Step 1 - Define the Role's Top 4-6 Signals

Start from the first 90-day outcomes, not a list of desired traits. Ask: what does success look like concretely at 30, 60, and 90 days? Then work backward to the signals a candidate would have to demonstrate to make those outcomes likely. Good signals are observable in a conversation. Bad signals are adjectives ('smart', 'passionate', 'culture fit').

Step 2 - Map Signals to Interview Slots

Each interviewer owns 1-2 signals. Never have two interviewers independently assess the same signal with no calibration plan - duplication wastes time and creates conflicting data. Assign technical depth signals to engineers, cross-functional signals to PMs or peers, and leadership/judgment signals to the hiring manager.

Step 3 - Write the Scoring Rubric

For each signal, write what a 1 (does not meet bar), 3 (meets bar), and 5 (exceeds bar) looks like in observable terms. Example for 'navigates ambiguity': a 1 asks for requirements to be fully specified before acting; a 3 defines scope, makes assumptions explicit, and validates them; a 5 proactively creates structure where none exists and coaches others through it. The rubric must be written before interviews start.

Step 4 - Require Evidence, Not Ratings

The scorecard submission form has two fields per signal: a numeric score AND a required evidence quote from the candidate. No evidence, no score. This prevents post-hoc rationalization and makes debrief discussions concrete.

Step 5 - Run a Calibrated Debrief

Collect scores independently before the debrief meeting. Start by reading scores aloud before anyone speaks. Then discuss only the signals where scores diverge by 2 or more points - those are the interesting ones. The hiring manager does not share their view first; they synthesize last.

Escape Hatches

For a high-volume pipeline, reduce to 3 signals and a 3-point scale. For a senior or staff role, add a written exercise component and score the artifact, not the person's delivery style.

Deliverable

Produce a complete scorecard package: the 4-6 named signals each traced to a 30/60/90-day outcome, the signal-to-interviewer slot map, a written 1/3/5 rubric per signal in observable terms, the two-field submission form (score plus required evidence quote), and the debrief protocol (independent submission, scores read aloud first, hiring manager synthesizes last).

Do NOT

• Do not write the rubric after interviews begin - a bar defined mid-loop bends toward the candidates already seen, which is exactly the post-hoc rationalization scorecards exist to prevent.
• Do not use adjective signals ('strong communicator', 'culture fit') - anything not observable in a conversation becomes a proxy for likability and similarity bias.
• Do not let interviewers share impressions before scores are submitted; early opinions anchor the group and erase the independent-signal value of a panel.
• Do not average divergent scores into a middle number - a 1 and a 5 on the same signal means the interviewers saw different things or hold different bars, and that disagreement is the data.
• Do not assign more than 2 signals to one interviewer; coverage gets shallow and the interview becomes a checklist sprint instead of an evidence-gathering conversation.
• Do not carry a scorecard between materially different roles; signals derived from one role's 90-day outcomes do not transfer to another's.

Quality bar

• Every signal traces to a concrete 30/60/90-day outcome and is observable in an interview.
• The 1/3/5 rubric exists in writing before the first interview is scheduled.
• Every submitted score has a verbatim evidence quote attached; scores without evidence are returned, not accepted.
• The debrief discusses only divergences of 2+ points, and the hiring manager speaks last.