The Agency, Control, Reliability (ACR) Tradeoff for Agents

The ability to perform high-agency tasks is important, but it is just as important to ensure that agents can execute tasks competently, reliably, and consistently, when deploying them in high value use cases.

Fig.1 : Balancing the AI agent’s agency and control with reliability for complex tasks is hard. Customers typically want high reliability, even if it comes at a cost of some agency. “Give Fin a Task” is a variant of AI agent that fits this requirement, and with its composable nature, customers can progressively solve more complex tasks.

Assessing the competence of customer support agents is challenging because performance depends on both task complexity and the level of agency an agent has (Fig. 1).

A practical way to measure reliability is to evaluate how consistently an agent completes a given task when repeated multiple times. This is particularly relevant for customer support, where agents frequently handle repetitive requests, such as processing refunds or cancellations.

Simulated Task Testing

To measure agent performance, we simulate a “Task” where the agent interacts with a simulated “end user” over multiple turns. The simulated user, powered by another LLM, follows a predefined script to mimic real customer interactions while attempting to resolve a specific support issue. Each set of such interactions forms a “test,” which can be repeated as needed. A test consists of:

1. Expected outcomes – Clearly defined success criteria for the test, such as an expected order status, a specific value like a refund amount, or required API calls (e.g., getShopifyOrders) to retrieve additional information.
2. Simulated user prompt – Defines how the user interacts with the agent, including their persona and communication style.
3. Stopping condition – Reached when the agent fails to meet the task requirements or exceeds the maximum number of interaction turns.

The test concludes when either the expected outcomes are met or the stopping condition is reached. For example, in an order shipment inquiry, the agent must extract the correct user ID, call the appropriate function to retrieve the order status, interpret the response, and clearly communicate it to the user. Only when all these expectations are met, the test would be considered a pass.

Realistic User Simulation

To reflect real-world interactions with Fin, we model customer behaviors such as impatience, brevity, and incomplete information. These factors, combined with task complexity and imperfect API specifications, create a realistic testing environment.

By simulating diverse agentic tasks, we gain insights into how different LLM agent architectures perform under real-world conditions.

The Metric

Most benchmarks use pass@k, which measures the percentage of tasks successfully completed at least once in k repetitions. However, this is not a useful measure of reliability for customer support agents.

Instead, we need the total percentage of tasks successfully completed every time when repeated k times. This metric is widely known as pass^k. This is a much stricter metric, as it requires consistent success rather than a single correct attempt. The choice of k is indicative rather than absolute—it helps identify trends but does not perfectly reflect real-world expectations. In practice, no two customer interactions are identical, making absolute replication unlikely.

Fig 2. Task pass rates across all tests for high agency function calling agents, with different values of K for repetitions. The dotted trend line indicates that the reliability of these agents falls rapidly.

To get a deeper insight into the agent’s behaviour, we classified the tasks in our benchmark into three categories,

• Simple Tasks: Involve independent context gathering or straightforward action calls. This might include a one-off status check for an order.
• Moderate Tasks: Require conditional branching or chaining, such as using one action’s output as input for another. This might involve use cases like validation of some information from the user, but first authenticating them, checking their accounts, and then gathering context for validation.
• Complex Tasks: Involve multiple pathways to achieve a goal, imperfect information (e.g., a user forgetting credentials), and extensive context gathering and chaining. These are much more complicated tasks where we are dealing with a user who has incomplete information, or a user that wants to do several changes to their state, or simply a difficult user.

When we run the experiments just on simple and complex tasks as classified by the above rubric, the metric of pass^k looks very different. While agents perform in line with the overall performance as seen in Fig 2 for simple tasks, their performance drops by a huge margin on complex tasks. Function calling agents using older models like Claude sonnet, perform slightly better on simple tasks compared to reasoning models like O3. Reasoning models gain some advantage on complex tasks, but not enough to be optimistic.

Fig 3: Difference in the agent’s pass^k performance on simple vs complex tasks

The above results show that in the context of customer support, the LLMs and their agentic capabilities are not yet at a level of reliability to solve the full range of task complexities that our customers might experience. Low reliability might directly impact CSAT and escalations, while making agent interactions more frustrating for our end users.

Fig 4: Performance on tasks strictly structured as steps with two variants of agents: an agent designed to strictly follow a steps based execution for a steps based task structure (GFAT agent), and a conventional function calling agent but one that still uses a steps based task structure. We have used Claude 3.5 Sonnet V2 to generate the above results

This agentic product would strongly lean on the following key concepts :

• Control: Since the agent leverages the pre-existing, and familiar workflow-like environment, a teammate would find it easy to build complex workflows with agentic blocks that do narrower well defined tasks, while achieving a bigger and more complex end goal via composition.
• Agency: Although we will trade off agency for control, we would build these agents to be highly expressive via steps based instructions for the agent. The Give Fin a Task agents would be configured using steps based instructions to fulfill a task, where each step is an executable instruction. The agent is designed to follow these steps reliably, allowing high levels of reliability for narrower and moderately complex tasks, while still maintaining all the conversational benefits of an agent powered by a LLM.
• Composability: : The product would allow teammates to break down highly complicated tasks into narrower and more reliable tasks, thereby pushing the overall reliability of the individual task to a much higher level, when compared to an unconstrained agentic approach. This would also allow teammates to compose much more complex workflows, with lots of such individual “Give Fin a Task” blocks.
• Feedback loops: Based on the research done in the team, we will build tooling that
  • Helps the teammate to write task prompts that maximise reliability, without meaningfully compromising the agency.
  • Provides insights into the reliability of a configured task prompt to the teammate, and provides suggestions for improvements
  • Provide meaningful metrics around the agentic blocks in the workflow, once they are in production.

This bet has shown a lot of promise in our early benchmarking tests, as seen in Fig 4. The Give Fin a Task agent shows significantly higher levels of Pass^k performance (k = 3,4,5), on tasks configured using the steps based structure and executed by the steps executor agent (GFAT), which has a tempered level of agency. The steps executor agent is restricted in what it can do, by the steps based structure of its instructions, therefore reducing its agency when compared against pure function calling agents.

The performance on simple tasks show a very high degree of reliability, and we see a considerable gain in performance for moderate tasks for the steps executor. The previous, almost exponential, trend in the drop in the performance with the increase in repetition almost vanishes, indicating a much more stable and reliable agent performance. The benefit of the steps based task structure is also extended to pure function calling agents, where their reliability also receives a considerable boost across the three task categories, and especially for simple tasks.