AI agents move the conversation from generating content to taking action. They can plan across multiple steps, call tools, update systems and adapt to intermediate results. That creates real operational potential—and a larger blast radius when identity, authority, evaluation and ownership are unclear.
Start with the work, not autonomy
The first question is not where the organisation can deploy an agent. It is which outcome requires a sequence of information gathering, judgement and action that current workflows handle poorly. Good candidates have a clear objective, observable completion, accessible tools and boundaries that can be expressed. Work that is ambiguous, politically sensitive or impossible to verify is a weak starting point.
Use the least autonomy needed. A workflow with deterministic steps may be more reliable and easier to govern than a free-running agent. An assistant may recommend actions while a person executes them; a constrained agent may act only within an approved system and limit; a more autonomous agent may plan across tools. These are operating choices, not maturity levels that every use case should climb.
Separate the model from the system
An agent is not just a language model. It is a system of instructions, context, memory, tools, identity, permissions, orchestration, evaluation and monitoring. Reliability depends on the complete design. Treating model capability as system capability leads teams to overlook what happens when information is stale, a tool returns an error, a task is interrupted or an instruction conflicts with policy.
Architecture should make each boundary visible. Which data can enter context? What persists between sessions? Which tools can be called? What inputs can those tools accept? Where are outputs validated? How is state recovered? A simpler, composable design makes behaviour easier to test and allows higher-risk actions to receive stronger controls.
Give every agent an identity
Shared service accounts destroy accountability. An enterprise agent should have a distinguishable identity, an owner and permissions limited to its purpose. NIST's work on software-agent identity emphasises identification, authorisation, auditing and non-repudiation. Those familiar security concepts become more important when software can choose and sequence actions dynamically.
Apply least privilege to data, tools and transactions. Use short-lived credentials where possible. Separate read from write access. Limit amounts, destinations and rates. Require additional confirmation for consequential or irreversible actions. Permissions should be reviewed as the workflow changes; an agent that gains a new tool is materially different even if its model and user interface remain the same.
Design meaningful human control
Human-in-the-loop is not a sufficient control if the reviewer lacks time, context or authority. Decide which moments require human judgement and provide the evidence needed to exercise it. A confirmation screen that simply repeats the agent's recommendation can produce automation bias. A useful review shows source information, assumptions, uncertainty, proposed actions and likely consequences.
Control can also be structural: budgets, transaction limits, approved destinations, sandboxed tools and reversible actions. High-frequency low-impact work may rely more on monitoring and sampling; rare high-impact decisions may require explicit approval. The design should state when the system stops, escalates and hands work back to a person.
Defend against hostile context
Agents consume emails, documents, websites and system outputs that may contain malicious instructions. NIST describes agent hijacking as indirect prompt injection that causes an agent to take unintended harmful actions. The risk grows when an agent can both read untrusted content and use powerful tools.
Do not rely on a system prompt as the security boundary. Separate trusted instructions from untrusted data; sanitise and label content; restrict tools; validate parameters; isolate execution; and require confirmation for sensitive actions. Test with adversarial inputs that resemble the organisation's real environment. Logging should preserve enough detail to reconstruct what the agent saw, decided and attempted without exposing unnecessary sensitive data.
Evaluate trajectories, not just answers
A final answer can be correct even when an agent took an unsafe route, and an unsuccessful outcome can reveal a sound recovery attempt. Evaluation therefore needs to examine the trajectory: tool selection, sequence, intermediate state, policy compliance, efficiency, recovery and final result. Anthropic's guidance on agent evaluations similarly stresses that multi-turn, state-changing behaviour demands more than conventional one-shot tests.
Build representative tasks with verifiable outcomes. Include routine cases, edge cases, tool failures, ambiguous instructions, conflicting data and attempted hijacking. Track task success, unsafe actions, unnecessary actions, human interventions, latency and cost. Evaluations should run before release and after changes to models, prompts, tools, permissions or workflows.
Operate agents as production services
Agents need service ownership, monitoring, incident response, change control, support and retirement plans. Observability should show task status, tool calls, failures, policy events, resource use and human overrides. Teams need a kill switch and a tested way to suspend one action, one agent or the wider service without losing necessary records.
Model and tool dependencies will change. Version instructions and evaluations, canary significant updates and define rollback. Review whether stored memory remains accurate and appropriate. Monitor for drift in task mix as users discover new behaviours; a system may gradually be used for decisions outside its original approval even when the code has not changed.
Operational cost also needs active management. Agent loops can generate variable model use, repeated tool calls and long-running tasks. Set budgets and timeouts, detect loops, cap retries and expose cost at the task level. Capacity planning should include downstream systems because an agent can create bursts of activity that a human-paced process never produced. Service-level objectives should distinguish model availability from successful task completion: an agent can return a response while failing to achieve the operational outcome.
Prepare people and accountability
Domain expertise remains central. Recent Anthropic research on agentic coding found that people made most planning decisions and that greater domain expertise was associated with more successful work. The implication reaches beyond software: agents can absorb execution, but people still need to define the objective, judge trade-offs and recognise when the result does not fit reality.
Roles will change. Process owners become designers of goals and boundaries; operational teams supervise exceptions and improve the system; risk and security teams need evidence from agent trajectories; leaders need to decide where autonomy is appropriate. Training should use real tasks and failure scenarios, not only prompt techniques.
Accountability must remain clear when several agents or suppliers collaborate. Name the owner of the end-to-end outcome and record how responsibility passes between components. Users should have a straightforward route to question, correct or escalate an action. Teams need exercises that simulate bad data, unavailable tools, compromised content and unexpected behaviour, just as they rehearse other operational incidents. The objective is not to make people monitor every step, but to ensure they can recognise material failure and recover control quickly.
Scale authority gradually
Begin in a contained environment with read-only access and observable tasks. Move to recommendations, then reversible actions, then bounded transactions as evidence grows. Expand users, tools, data and autonomy separately so the effect of each change can be understood. A capability that is reliable in one domain should not inherit trust automatically in another.
The goal is not maximum autonomy. It is dependable delegation: the organisation knows what the agent can do, why it is allowed to do it, how success is measured and how control is recovered. Enterprises that establish those foundations can move faster because each new agent does not require identity, assurance and operations to be invented again.