Use this article to ask Operator to create, run, and analyze Evals, review the changes it proposes, and schedule recurring runs. Operator can also recommend improvements to your Evals and Simulations or to Fin's configuration. For the basics, including what Evals and Simulations are, see the Evals explained article.
Note: To use Evals with Operator, your workspace needs the Pro add-on and you need access to Operator.
How to create an Eval with Operator
To create an Eval with Operator, open Operator in the left-hand navigation and ask it to create an Eval for your use case. For example, you could ask it to test:
The Release you're working on: the changes to Fin's configuration in your current Release, before they go live.
Specific conversations: share one or more conversation IDs, and Operator builds Simulations that recreate them.
Edge cases you haven't covered yet: ask Operator to look for gaps in your existing Simulations and suggest new ones.
A new or updated Procedure: check that Fin follows each step, including the points where it should hand off.
A topic: name a topic, such as refunds, and Operator creates Simulations covering the range of scenarios customers are likely to contact you about.
More variations of an existing Eval: ask Operator to add Simulations with different customers, tones, or languages.
You can tell Operator what the Simulations, Simulation checks, and Handoff checks should be, or let it work them out. If anything is unclear, such as whether to add Simulations to an existing Eval or which team a handoff should go to, Operator asks you first. Choose an option, type your own answer under Something else, or click Skip to let Operator decide.
Tip: The more specific your request, the fewer questions Operator needs to ask.
Example prompt: Create a new Eval called "Refund requests" with 5 Simulations: a straightforward refund request, a customer asking about a refund outside the refund window, a frustrated customer who's been charged twice, a customer asking in Spanish, and one where Fin should hand off to the Billing team. To adapt it, replace the Eval name and the scenarios with your own topic and cases.
How to review and save what Operator creates
Operator doesn't save anything until you review it. New Evals and Simulations appear in the "changes pending" panel above the message box, each marked "New". Click any item to review it before saving.
When you're ready, choose where to save the changes:
Publish to Fin Main: saves all pending changes to your live Evals straight away.
Save to release draft: saves the changes to an existing release draft, so you can test them alongside other changes before they go live. Select a draft from the list, or click New release draft to create one.
To discard the changes instead, click the delete icon.
How to run an Eval with Operator
To run an Eval with Operator, ask it to run the Eval, for example: Run the Refund requests Eval. Operator shows you the results as each Simulation runs.
You don't need Operator to run an Eval. Go to Test > Evals and run it yourself. Runs started from Operator appear there too, so you can review every result in one place.
How to read the results Operator gives you
Once every Simulation has run, Operator lays out the results and explains them:
A results table: each Simulation's overall result (Passed, Failed, or Error) and the individual checks behind it, so you can see why it landed that way. For example, a Simulation called "Bulk workspace export" can fail because the Handoff check failed (Fin escalated when it shouldn't have), even though the Fin reply check passed.
A summary: the headline counts (3 passed, 1 failed, 2 errored), followed by a plain-language breakdown of what went wrong and why. For example, Operator can trace a failed export back to a guidance rule that hands off to a teammate whenever the word "failure" appears.
How to fix issues with Operator
After analyzing the results, Operator can recommend improvements, either to your Evals and Simulations or to Fin's configuration, and make those changes for you. For example, when a Simulation fails because of a guidance rule, Operator traces the failure back to that rule so you can see what to fix. Operator doesn't apply fixes on its own. Each change it proposes appears in the "changes pending" panel for you to review before it's saved, as described in "How to review and save what Operator creates" above.
How to schedule Evals with Operator and Routines
To run Evals on a regular schedule, such as every Monday morning, set up a Routine in Operator. You can't schedule Evals from the Evals page.
To schedule an Eval:
Open Operator in the left-hand navigation and click Routines.
Under Set up a routine, describe what you want to run and when. For example: "Run the Refund requests Eval every Monday at 9am and summarize the results."
Operator drafts the Routine and shows it in a panel on the right, with its trigger, schedule, and instructions. If anything is missing, such as the time or time zone, Operator suggests a default and asks you to confirm or change it.
Check the Routine's details, then click Accept to turn it on, or Discard to cancel it.
Each time the Routine runs, it reports how many Simulations passed and failed, which ones failed and why, and any recommended fixes to Fin's content, guidance, or Procedures.
Need more help? Get support from our Community Forum
Find answers and get help from Intercom Support and Community Experts