BlogBuilding Fin
When should skills become Agents?
We turned a Claude skill into an AI SRE Agent that now completes 41% of our weekly dead-letter queue fixes. Here's what we learned.
In April 2026, we paused normal delivery for a day to explore how AI could change engineering at Fin. Over the day, engineering teams from across the company converged on the same problem spaces, resulting in us building Claude skills for triage, on-call monitoring, and issue fixing.
All great ideas solving real problems. But it also raised a question: when should skills become Agents?
Initially, we didn’t have a clear mental model of what problems would be best tackled by an Agent, but we wanted to find out. At Fin, one of the core principles we apply to building products is “think big, start small.” We didn’t want to boil the ocean, so we decided to pick one problem and build an Agent for it.
We chose dead-letter queue (DLQ) alarms, which are notifications in our system that background jobs have repeatedly failed. We already had a Claude skill that could investigate these alarms and suggest a fix, but engineers still had to trigger it, check its work, and take action. The next step was to see if it should become an Agent that could own more of the workflow, take action, and be tested and improved as a system.
Our hypothesis was that an Agent was only justified if it could safely move from an alarm to a measurable outcome and materially reduce the work left for an engineer. Since rolling it out in August, it now completes 41% of our DLQ fixes.
Here’s how we built it and what we learned.
Picking the right problem
At Fin, we have always been a product-led company. We fight hard to ensure shipping the product is not seen as an end. When it came to building an Agent, we quickly identified that the same principles applied.
You need to think of your Agent like your product:
- What problem has the characteristics that an Agent solution should solve for, like a clear trigger, repeatable steps, measurable outcomes and actions that can be taken safely?
- Could you deliver 80% of the impact by running the skill manually?
- When the model changes (which happens frequently), does the value of autonomous action justify the additional evaluations, monitoring, and safeguards required to maintain quality?
There has been an abundance of excitement around the potential of Agents in the AI SRE (site reliability engineering) space, with many of the top vendors releasing their own general Agents (e.g. AWS Devops Agent and incident.io AI SRE).
Having experimented internally with some, we found there were challenges a generic Agent could struggle with:
- Integration between internal systems like Datadog or Honeycomb, whether via APIs or MCPs, is relatively immature. The ability for your Agent to self-improve and test consistently is a prerequisite, and to do so, it needs access to company context.
- Every company (and often team) has a different approach to managing alarms. These internal processes are difficult to map directly to an out-of-the-box solution. You can read more about our processes in this piece about building a healthy incident culture from Mark Gorman, a technical program manager here at Fin.
So we decided to build our own.
Why we started with DLQ alarms
DLQ alarms take up a huge amount of our Infra Platform Team’s time, because our Customer Agent Fin runs on queues.
A queue is a list of work to be done later. One part of the system puts a message in it – send this email, reindex this conversation, regenerate this report – and a separate fleet of workers picks messages off and does the work. The point is decoupling. The thing asking for the work doesn't wait around for it, and when the workers fall behind, the queue absorbs the backlog instead of the request failing.
Most companies use a handful of queues like this for the obviously slow things. We went a good deal further than that. In Fin’s monolith, we have roughly 1,400 distinct queues in our queue config, served by around 600 worker classes. That entire topology then exists three times over, once in each of our US, EU, and Australia regions, in separate AWS accounts.
Fin leans hard on this infrastructure – about a hundred of these queues are for Fin and other AI work. We did this intentionally as LLM calls are slow, content ingestion is even slower, and a customer conversation can't wait on either. Nearly everything Fin does that isn't the direct reply to a person happens in a queue.
When work in a queue has failed enough times that the system gives up trying (by default, it makes three attempts), it gets sent to a DLQ. DLQs are essential architecture for maintaining consistency while scaling.
When a DLQ starts to fill up, an alarm fires, and someone has to answer three questions:
- What’s causing this message to fail processing?
- What needs to change for it to be processed?
- Does this message still need to be processed?
Because of Fin’s reliance on queues we alarm our DLQs aggressively, and we have to be good at responding when those alarms fire. But there are a lot of them – we receive 50 new alarms a week, rising to 325 if you include re-trigger notifications.
What made DLQs a good fit for an Agent
Agents are only useful when their scope is carefully defined, and DLQs were well suited to an Agent solution because they’re frequent, bounded, measurable, and actionable.
Frequent
Every queue created through our shared Terraform module gets a dead-letter queue automatically, with alarms configured based on how important the queue is. On our most critical tier, a single dead-lettered message raises a warning, while ten sends a page to an engineer. With roughly 1,400 queues, that all adds up.
DLQ alarms accounted for around 80% of the pages reaching the Infra Platform team's secondary on-call. With the team clearing these manually, new alarms often arrived faster than they could be handled.
Bounded
What lands in the Infra Platform team’s queue tends to be one of three things:
- Something was briefly unavailable. For example, a dependency, timeout, or database replica running behind.
- The worker has a genuine bug, or hit data it can't handle.
- The job died before it could report why. For example, it was killed for memory, or stopped by a platform limit.
Measurable
Every DLQ alarm looks slightly different, but resolving one usually follows the same path. Because it’s repeatable, we could test each decision against previous alarms, measure whether it was correct, and gradually allow the Agent to move from explaining the problem to safely resolving it.
Actionable
Most resolutions to DLQ alarms are concrete actions. For example:
- If the failure was temporary, put the messages back on the queue to try again.
- If the job simply needs more attempts or more time, adjust how retries work.
- If there is a bug, find the responsible code and prepare a fix for its owning team.
- If the evidence is unclear, hand the case to an engineer with a concise summary of what has already been established.
All of these factors are what made DLQ alarms such a good candidate for an Agent. The work still required judgment, but the questions, evidence, and possible outcomes were consistent.
Building the DLQ Agent
Version 1
Our first attempt at a DLQ Agent was not good.
Our intent was to reduce the cognitive load for engineers responding to alarms at the first point of contact via an expert Claude skill. Instead, the Agent wrote a long report for every alarm that we couldn’t effectively verify for accuracy.
When it did correctly identify a root cause, it frequently missed critical details or presented plausible yet inaccurate analysis. Combined with verbose responses, engineers found it tedious to parse useful insights from noise. Consequently, the outputs lost credibility and were largely ignored in day-to-day operations.
It also became clear that for the Agent to be a real step change in tackling on-call toil, it needed to take actions. At this point, it only suggested fixes rather than making them.
Version 2
The first version of the Agent failed because we asked one prompt to do five jobs.
A single skill was expected to gather evidence, decide which evidence mattered, infer a cause, recommend a fix, and explain everything to an engineer. When its answer was wrong, we could not tell which part had failed. When we changed the prompt, we could not measure whether the system had improved. A system you cannot measure is a system you cannot improve.
None of this was new to us. It is the same lesson we learned building Fin.
Probabilistic systems need a tighter development loop than traditional software. You need real examples, known outcomes, repeatable evaluations, and a way to notice when improving one behaviour has inadvertently made another one worse.
We grew Fin's resolution rate one measured improvement at a time, and distilled this process into Evals and Releases, so customers can use it when shipping changes in their own Fin setup.
We needed to apply the same process to our DLQ Agent.
Going back to the drawing board
We started over by consulting the people doing the work: the Infra Platform team. Here’s the process we followed.
Start with the outcome
Speaking with the team gave us a much sharper definition of a useful outcome, which informed what the Agent should drive towards. Every alarm should end in one of three outcomes:
- The queue is safely redriven.
- A reviewable fix is open against the owning code.
- A human gets a short escalation, grounded in evidence, that says what we know and what we do not.
Backtest against past scenarios
For the DLQ Agent, we built backtests from real alarm investigations. The initial correlation corpus began with 63 alarm fires, reduced to 33 distinct scenario shapes so that repeated versions of the same incident did not distort the results.
For code fixes, we could go further: replay the investigation against the repository as it existed immediately before the human fix was shipped. The defect was present, but the answer was not available to the Agent. That let us test whether it could genuinely find and fix the problem rather than discover the result after the fact.
Give the model less to do
The architecture followed the same principle – make each component responsible for one narrow decision.
The model is good at one thing here that code is not: reading an exception and working out what it means. Everything around that judgment is deterministic, testable code. Fin is built in the same way.
The model should never be asked to guess a fact it could have been handed.
What that architecture looks like inside our DLQ Agent
EMS, our event management system, acts as the deterministic control plane. It receives the Datadog alarm and resolves facts the model should never have to guess: the queue and region, retry configuration, backlog depth and trend, worker fleet, recent production changes, and any known redrive history.
Buzz then runs the investigation in an isolated container. It collects an error breakdown from Honeycomb and Datadog before invoking two bounded judgment steps. The first asks whether the observed errors actually explain the alarm. The second classifies the relevant failure as transient, non-transient, or infrastructure-related.
Those results are persisted as typed artifacts and returned to EMS. A Ruby gate validates the handoff, suppresses duplicate work, and chooses exactly one action lane:
- A transient failure goes to the retry lane, which decides whether to redrive now, wait, change retry configuration, or leave the queue alone.
- A non-transient failure goes to the code-fix lane, which can prepare a pull request against the owning code.
- An infrastructure failure goes to the infrastructure lane, which can propose a bounded configuration or capacity change.
- Insufficient or contradictory evidence results in no automatic action. The uncertainty is reported explicitly to a human.
And every outcome is posted to Slack on the original alarm:
Every failure becomes a test
Just as importantly, each part could now be improved independently. If correlation was wrong, we could add the case to the correlation suite. If classification was wrong, we could test that decision without rerunning the entire workflow. If a proposed fix was rejected, that outcome could become a new evaluation case.
The goal was to build a system that could take actions as reliably as an engineer.
What has been the impact
We launched the DLQ Agent in August 2026 with the north star of eliminating secondary on-call for the Infra Team. Currently, 41% of DLQ fixes each week are completed by the Agent when measured by merged PRs. Five weeks ago, it stood at 4%.
We will continue to iterate, but early signs suggest we have a path to achieving our north star.
We’ve also had the unexpected highlight of product teams noticing the Agent “doing the work” right under their proverbial noses.
Not all skills should be Agents
These days, it’s tempting to treat "build an Agent" as the goal. But Agents are only useful when their scope is carefully defined and their architecture is built for them to succeed.
DLQs worked because the problem was frequent, bounded, measurable and actionable. We could supply the facts deterministically, give the model narrow judgment calls, gate its actions, and turn every failure into another test.
We said from the beginning that an Agent was only justified if it could safely move from an alarm to a measurable outcome and materially reduce the work left for an engineer. The DLQ Agent is now completing 41% of weekly DLQ fixes, so it has cleared that bar.
That's the same bar we’ll use when deciding whether to build the next Agent.