When choosing and evaluating an AI Agent, there are a number of factors you need to consider. You need to understand if it can integrate with your business, handle complexity, operate at scale, perform well, and deliver value.

Entry Criteria

1. Capabilities

Does the AI Agent support the use cases you care about most? Look for:

• Ability to answer complex, multi-touch queries, not just FAQ-style questions.

• Personalization using customer data.

• Ability to guide and control the behavior of the AI Agent.

• Multilingual, multi-channel support.

• AI insights for real-time customer experience analysis.

• Seamless handovers to human agents.

• Transparency and control over the end-to-end experience.

Why it matters: Capabilities only create value if they lead to real outcomes. The most important measure of any AI Agent is its ability to fully resolve conversations – consistently, accurately, and at scale. That’s the foundation for ROI, customer trust, and long-term success. Just as critical is transparency and control: teams need to understand how the AI Agent makes decisions, guide its behavior, and ensure it’s operating in line with brand, policy, and quality standards. Without this, even advanced features can introduce risk and undermine trust.

2. Platform fit

Determine if the AI Agent can integrate with your existing systems, meet your compliance standards, and scale with your team.

Integrations and customization

Can it connect with your existing systems? How easily can it be customized to fit your workflows?

Check that:

• It connects with your helpdesk, knowledge base, CRM, and analytics tools.

• It’s extensible via APIs, webhooks, or SDKs.

• It can reflect your business logic, data sources, and routing rules.

Security and compliance

Does the vendor meet your data privacy obligations? Confirm that:

• The vendor meets privacy and data protection obligations (e.g., GDPR, CCPA, HIPAA.).

• Certifications like SOC 2, ISO 27001, or ISO/IEC 42001 are in place.

• It can isolate or redact PII.

• It can support SSO, RBAC, and audit logs.

Why it matters: The AI Agent you choose needs to meet your company’s standards. Platform fit ensures a safe, scalable deployment.

Evaluation criteria

1. Business performance

This is where operational value becomes real. Beyond conversation quality, you need to measure the actual results the AI Agent is producing for your business and customers.

Some metrics to measure:

• Resolution rate Conversations fully handled by the AI Agent with no human involvement.

• Deflection or containment rate Conversations handled without reaching your team.

• Time saved Hours of manual work offloaded from your support team.

• CSAT If you’re testing in a live environment, track how comparable customer satisfaction with AI is to human-handled interactions.

When it comes to measuring AI's impact, we recommend:

1. Focusing less on deflection, and more on resolution

While helpful as an early signal, deflection is a limited, and potentially misleading, measure of success. What matters is whether the issue was resolved. With AI Agents like Fin now capable of resolving the majority of queries, we need to shift our focus from deflection to outcomes.

2. Measuring customer satisfaction across all conversations with AI

Across all Intercom customers who use CSAT, just 8% of conversations get scored, whereas AI can analyze 100% of customer conversations (involving both AI Agents and humans) in real time. If you're evaluating Fin, it assesses three critical dimensions of customer conversations:

• Resolution status Was the issue actually resolved? And if there were multiple issues, were each of them resolved?

• Customer sentiment How did the customer feel throughout the interaction?

• Service quality Was the response clear, helpful, and efficient?

Fin uses these inputs to generate a “CX rating” from 1–5 for each conversation. CX Score Reasons provide clear explanations of what drove each rating, whether it was product feedback, sentiment, or answer quality. These individual ratings contribute to a broader “Customer Experience Score,” based on real-time insights from every support conversation.

Why it matters: Business value comes from resolving the right volume, reliably, securely, and at scale.

2. Conversation quality

Quality isn’t just about how good the AI Agent sounds. It’s about how clearly and accurately it helps customers. You need to look at:

• Accuracy Does it understand intent and deliver the right answer, every time?

• Behavior Does it know when to clarify or hand over to a human agent? Can it represent your brand’s tone of voice and adhere to your company’s policies?

• Experience Does it create a smooth, fast, and satisfying experience for both customers and support teams?

Why it matters: Great AI Agents build trust by communicating clearly, responding appropriately and accurately, and staying on brand. Incorrect or vague answers, poor tone, or robotic behavior can undermine confidence.