AI Agent Guardrails, Evaluation, and Production Readiness

Step 1: Apply least privilege

Give the agent only the tools and access the task needs, with read-only access wherever possible and narrowly scoped credentials. An agent that cannot delete data cannot delete it by mistake or under attack.

Step 2: Require approval for consequential actions

Pause for a human before irreversible or sensitive steps such as refunds, deletions, payments, or outbound messages. Make the approval request show exactly what will happen.

NEEDS_APPROVAL = {"issue_refund"}

def run_tool(name: str, args: dict) -> tuple[str, bool]:
    if name in NEEDS_APPROVAL:
        answer = input("Allow " + name + " with " + str(args) + "? [y/N] ")
        if answer.strip().lower() != "y":
            return "The user declined this action.", False
    function = FUNCTIONS.get(name)
    if function is None:
        return "Unknown tool: " + name, True
    try:
        return function(**args), False
    except Exception as error:
        return "Error: " + str(error), True

Step 3: Limit turns and spend

Cap the number of steps and total tokens per task, and stop with a clear message when a limit is hit. Log each turn so you can see what the agent did and what it cost.

import logging

MAX_TURNS = 10
MAX_TOTAL_TOKENS = 200_000

def run_agent_guarded(user_input: str) -> str:
    messages = [{"role": "user", "content": user_input}]
    used = 0

    for turn in range(MAX_TURNS):
        response = client.messages.create(
            model=MODEL, max_tokens=16000, system=SYSTEM,
            tools=TOOLS, messages=messages,
        )
        used += response.usage.input_tokens + response.usage.output_tokens
        logging.info("turn=%s stop=%s tokens=%s", turn, response.stop_reason, used)

        if used > MAX_TOTAL_TOKENS:
            return "Stopped: token budget exceeded."
        if response.stop_reason != "tool_use":
            return next((b.text for b in response.content if b.type == "text"), "")

        # ... run tools and append results as in the loop lesson ...

    return "Stopped: too many steps."

Step 4: Defend against injection and bad arguments

Treat tool results and fetched content as untrusted, since they can contain instructions aimed at the model. Validate tool arguments in code, and never let the model's output control security decisions, such as which files it may touch, without checks.

Step 5: Evaluate agents on whole tasks

Build a set of realistic tasks with clear success criteria, run the agent on each, and measure task success rate, correct tool use, number of steps, and cost per task. Review failed runs step by step, add them as regression tests, and re-run the set whenever you change prompts, tools, or models. Roll out gradually with monitoring and an easy way to turn the agent off.