Use this article to add a Scorecard to an Eval, create a new Scorecard, add criteria, and set each criterion's weighting and Critical setting.
How to add a Scorecard to an Eval
A Scorecard adds quality scoring to an Eval, such as Clarification or Efficiency, on top of its pass/fail checks. Until an Eval has a Scorecard, it shows only Simulation results and no Quality evaluation. A Scorecard applies to every Simulation in the Eval, with the same criteria for each.
Go to Test > Evals and open the Eval.
Click Add scorecard at the top right of the Eval, next to New simulation.
After you add a Scorecard, a Quality evaluation section appears at the top of the Eval, next to Simulation results. The criteria shown come from the Scorecard you added, so the section lists whichever criteria that Scorecard contains. It shows the overall Review score and the average for each of those criteria. The list of simulated conversations also gets a Review score column and a column for each criterion.
You can also add a Scorecard from Eval settings. To open Eval settings on an existing Eval, click the options menu (...) next to New simulation, then click Edit eval. In Eval settings, open the Quality evaluation dropdown, then choose an existing Scorecard or click Create new scorecard. The default is No scorecard.
Note: Scorecards are shared with Monitors, so changes to a Scorecard apply everywhere it's used. To write criteria that score reliably, see the How to write effective Monitor & Scorecard criteria article.
How to create a Scorecard for an Eval
To create a Scorecard for an Eval, name it and add the criteria Fin is scored on. Eval Scorecards always evaluate Fin, so the Who's evaluated setting is fixed to Fin AI Agent, and Human teammates assigned isn't available.
In Eval settings, open the Quality evaluation dropdown and click Create new scorecard. The dropdown then shows New scorecard, and the New scorecard form appears below it.
Under Scorecard name, enter a name.
Under Evaluation criteria, click + Criteria. The Edit scorecard > New criteria panel opens. To fill it in, see "How to add criteria to a Scorecard" below.
After you add scored criteria, the Review score (%) section shows a weighted average of them. It's calculated by multiplying each criterion's score (%) by its weight, adding the results, then dividing by the total weight. Until then, it shows Not scored.
The Auto-review scorecard toggle is available only when every criterion uses AI evaluation.
How to add criteria to a Scorecard
In the Edit scorecard > New criteria panel, under Criteria name, enter a clear name for reference, for example Sentiment.
Under Description, describe what should be evaluated, in up to 255 characters. The AI reviewer uses this description to decide which rating applies.
Under Rating options, click Add option until the criterion has at least 2 rating options. The Not applicable option is always there and doesn't count.
Optionally, turn on Rating reasons.
Click Save to save the criterion, then click Save changes in Eval settings to attach the Scorecard to the Eval.
Evals score every criterion automatically with AI, using the criterion description and the Simulation transcript (all messages, excluding events). Score manually isn't available, because Simulations have no human reviewer. Fin content, Procedures, and guidance aren't available to the AI reviewer as context yet.
How to set score weighting and Critical for a criterion
In the Evaluation criteria table, each criterion has a Score weighting and a Critical setting:
Score weighting: how much the criterion counts toward the Review score (%). Criteria that show Not scored have no weighting.
Critical: if a Critical criterion fails, the whole scorecard fails, regardless of the review score. The Critical checkbox is unavailable for criteria that show Not scored.
How to read Scorecard results for a Simulation
To see how Fin scored on a single conversation, open a Simulation from an Eval that has a Scorecard. Alongside its Handoff check and Simulation checks, the Simulation shows its Review score and, for each criterion, the rating the LLM (large language model) judge gave. If Rating reasons is turned on for a criterion, you also see the judge's reason, so you can see why a response scored the way it did and turn that into a fix.
Need more help? Get support from our Community Forum
Find answers and get help from Intercom Support and Community Experts