Interview Questions on the Security Alert Triage Case

The triage agent now gathers evidence, returns a structured verdict, and routes every risky action through code. Analysts approve isolation, high-severity alerts never close on the model’s word, and log text stays in its place as evidence. Once a candidate presents that design, the interviewer starts testing it.

The questions come in three kinds. The first asks for proof that the agent works. The second probes cost and failure, then changes a requirement to see how the design bends. The last tests values, with a request that sits on the edge of what Anthropic allows. Each section below takes one question and gives an answer a candidate can say out loud.

Proving the agent works

The first question is usually the most direct one. “How do you know this works?” A strong answer names the metrics that matter and then measures them. Two metrics fit a triage agent.

Recall is the share of real attacks that the agent sends to a person by escalating them or requesting isolation. Precision is the share of escalated alerts that turn out to be real attacks. For Harborline, a missed attack costs far more than an extra escalation, so recall matters most. Precision shows how much analyst time the escalations cost.

Suppose the interviewer asks us to measure both on a small labeled set. We already built the triage agent for this case in triage_agent.py, with its evidence tools, its structured verdict, and the route function that decides what happens next. Now we test that agent without changing it. For this, we create a new file, eval_triage.py, which adds four alerts from past nights to the agent’s data, runs all eight alerts through triage and route, and compares each ...