INSIGHTS / AI

Designing agentic AI systems that stay within bounds.

Agents that call tools can take real actions. This article sets out the engineering controls — permissions, confirmation, evaluation and observability — that make them safe enough for enterprise use.

From answering to acting

A retrieval-augmented assistant reads and summarises. An agent goes further: it plans, calls tools and changes state in other systems. That shift moves the main risk from a wrong answer to a wrong action, and the design has to change accordingly.

Treat every tool as an API with a security boundary

The model should never hold more authority than the user it acts for. Each tool needs a narrow contract, server-side validation and authorisation evaluated against the end user's identity — not against a powerful shared service account.

Expose small, purpose-specific tools rather than generic 'run query' or 'call any API' tools
Validate tool arguments server-side; never trust the model to format them safely
Pass the user's identity (for example with the on-behalf-of flow) so existing permissions apply
Make write operations idempotent so a retried plan cannot duplicate an action

Put a person in the loop where it matters

Classify actions by consequence. Reading and drafting can be automatic; sending, paying, deleting or changing access should require explicit confirmation with a clear summary of what will happen. The confirmation step belongs in the application, not in the prompt, so the model cannot talk its way past it.

Assume prompt injection

Any content the agent reads — emails, documents, web pages, tool results — can contain instructions. Defences are layered: keep untrusted content separate from instructions, restrict which tools are available after untrusted content has been read, and rely on authorisation rather than on the model's judgement to prevent harmful actions.

Measure quality continuously

Agents are non-deterministic, so quality must be measured statistically. Build an evaluation set of realistic tasks with expected outcomes, run it on every prompt, model or tool change, and track task success, tool-call accuracy, cost and latency over time.

Start with 50–100 representative tasks drawn from real usage
Score tool selection and arguments, not just the final answer
Gate releases on regression thresholds, as you would with tests

Make behaviour observable

Trace each run end to end — prompts, retrieved context, tool calls, results and decisions — with sensitive data redacted. Without traces, diagnosing a bad outcome becomes guesswork, and governance teams have nothing to review.

Summary

Bounded agents are an engineering discipline rather than a prompt. Narrow tools, user-scoped authorisation, confirmation for consequential actions, injection-aware design, continuous evaluation and tracing together make agentic AI something an enterprise can operate responsibly.

Contact: contact@nexaarislabs.com