Guardrails for Production AI Agents: A Design Guide

Start from reversibility, not from risk

The usual framing for agent safety is risk: which actions are dangerous. That framing produces long debates and inconsistent answers, because risk is contextual and everyone weighs it differently. A more tractable question is reversibility. Can this action be undone, and if so, how quickly and by whom?

Reversibility sorts actions cleanly. Reading data is fully reversible in the sense that it changes nothing. Drafting a message for review is reversible. Sending that message is not. Issuing a refund is technically reversible and practically embarrassing. Deleting a record may be irreversible depending on your backups. Once you have this list, the guardrail design mostly follows from it.

The practical rule that emerges is that irreversible actions require either a human in the loop or a constraint tight enough that the action cannot be wrong. Everything else can be allowed with monitoring and correction. This is a far easier conversation to have with a product team than an abstract discussion about model reliability.

The three layers of constraint

The first and strongest layer is tool availability. An agent cannot take an action for which no tool exists. This sounds trivial and is routinely underused: teams give an agent broad database access because it is convenient, then attempt to constrain behaviour through prompting. Removing the capability is far more reliable than instructing against using it.

The second layer is argument validation. Where a tool must exist, constrain what can be passed to it. Bounded amounts, allow-lists of recipients, required references to prior state. This is ordinary input validation, applied at the boundary between the model and your systems, and it catches the large majority of plausible-but-wrong tool calls.

The third layer is post-condition checking. After the tool runs, verify that the result is consistent with what the agent intended. This catches the cases where the arguments were individually valid but collectively wrong, and it is the layer most implementations omit entirely because it requires knowing what the intent was.

Designing escalation that actually works

Most escalation paths are built as a failure exit: the agent cannot proceed, so it returns an error and a human starts over. This is better than nothing and it wastes almost everything the agent had accumulated. A good escalation hands over the full context — what was attempted, what was learned, what remains uncertain — so the human resumes rather than restarts.

The second design question is who receives the escalation and how quickly. An escalation path that routes to a queue nobody watches is functionally identical to a dead end, and the agent's caution becomes indistinguishable from failure. Deciding the recipient and the expected response time is part of designing the guardrail, not an operational detail to sort out later.

The third is what happens to the customer during the handover. An agent that goes silent while awaiting human review is a worse experience than one that never engaged. The handover needs a user-facing story, and writing that story often reveals that the escalation threshold was set in the wrong place.

Guardrail checklist for an agent taking real actions

• List every action by reversibility before designing anything else
• Remove tools for irreversible actions unless a human approves each one
• Validate every argument at the boundary, with bounds and allow-lists rather than instructions
• Add post-condition checks that verify the result matched the intent
• Define who receives escalations and how fast they are expected to respond
• Preserve full context on handover so a human resumes rather than restarts
• Give the user a visible story during handover, not silence
• Log every constrained attempt, because blocked actions are your best signal about drift

Where guardrail design usually goes wrong

The most common failure is treating guardrails as a launch checklist rather than as an evolving part of the system. A team designs constraints against the failures they can imagine, ships, and then never revisits them as real usage reveals failures nobody anticipated. Guardrails that are not reviewed become a snapshot of what the team feared six months ago.

The second failure is setting the escalation threshold by intuition and never measuring it. Set it too tight and the agent escalates constantly, humans stop reading the escalations carefully, and the guardrail becomes ceremonial. Set it too loose and the cases that most needed review are exactly the ones that pass through. The only way to calibrate is to sample escalated and non-escalated cases and check whether the split matches what a reviewer would have chosen.

The third is applying uniform constraints across a system where the stakes vary enormously. An agent that both answers questions and issues refunds does not need one guardrail policy; it needs a permissive one for the reading path and a strict one for the writing path. Uniform policies are always wrong somewhere, usually in the direction of blocking useful work while still permitting the expensive mistake.

Part of the AI Engineering Practice cluster · Read the pillar page