AI agents with human approval: how to keep a person in control

Give an AI agent access to your CRM, your inbox or your accounting system, and sooner or later it will propose something you did not expect: a discount outside policy, a reply drafted to the wrong customer, a duplicate payment. Deloitte's 2026 survey of more than 3,200 IT and business leaders found that only 21 percent of organizations have a mature governance model in place for agentic AI, which means most agents are already running ahead of the controls meant to catch that moment (Deloitte). Human approval is how you close that gap without turning a capable agent back into a suggestion nobody acts on.

What does "human approval" mean for an AI agent?

Human approval means deciding, in advance, which of an agent's actions a person must see before or after they happen, and how. An AI agent is software that interprets a request and chooses its own steps and tools to complete it, rather than following a single path fixed in advance. Most working programs use one of four patterns, not a single on or off switch:

• Approve every action. A person reviews and approves each proposed action before it runs. Slowest, and right for anything consequential or hard to reverse.
• Approve above a threshold. Routine items under a set limit, such as a dollar amount or a known counterparty, run on their own; anything above the limit stops for approval.
• Sample and audit. The agent acts on its own, every action is logged, and a named person reviews a regular sample plus anything flagged as uncertain.
• Exception review. The agent handles the routine case itself and only stops for a person when it hits a case it was not built to judge.

The NIST AI Risk Management Framework treats this as a governance requirement rather than an implementation detail. Under GOVERN 3.2, "policies and procedures are in place to define and differentiate roles and responsibilities for human-AI configurations and oversight of AI systems." In practice that means writing down, before the agent goes live, who holds each of the four patterns above for each action it can take. This is what a human in the loop AI workflow actually is: not a vague promise that "a person is involved," but a named person holding a specific pattern for a specific action.

Which actions need a person, and which do not?

Default to a person for anything that touches money, legal terms, a customer facing message, an irreversible change, or a decision about a person's job, credit or safety. Default to automation for internal, reversible, rule based work at real volume, where an occasional mistake is cheap to catch and fix.

The table below maps common action types to the same four control levels used in Kastling's human approval planner, so you can place a specific task on it directly.

Illustrative example: Consider a regional distributor whose AI agent reviews inbound supplier invoices and prepares them in the accounting system. Invoices under a set amount from an approved vendor post automatically, logged for a weekly sample review by the accounts payable lead. Anything above that amount, from a new vendor, or with a detail that does not match the purchase order is prepared and held in a queue for the same lead to approve, edit or reject, with a four hour escalation to a named deputy when the lead is unavailable. Six months in, the sampled tier still catches the occasional coding error, so the threshold stays where it started rather than climbing further.

Two categories deserve a second look even when they seem routine. Anything customer facing carries reputational risk that a private, internal mistake does not, so a first draft usually needs approval even if the underlying task is simple. Anything that changes who has access to what, such as granting a permission or resetting a credential, is both consequential and hard to reverse cleanly, which puts it above the line by default. For the wider design of a workflow the agent sits inside, see the practical guide to AI workflow automation.

How do you design the approval step so a reviewer can actually judge it?

Building an AI approval workflow that a reviewer can actually use starts with what they see on screen. At minimum, that means the proposed action itself, the source it came from, a confidence score or a stated reason, and a way to approve, edit or reject it in one action rather than a support ticket in another system. Amazon Bedrock's user confirmation feature works this way: the developer "receives information about the function (or API) and parameter values that an agent wants to use," and Bedrock notes that requesting confirmation "may safeguard your application from taking actions due to malicious prompt injections" as well as ordinary mistakes (AWS). AWS's own walkthrough of the pattern shows the reviewer a plain description of the action and its parameters, with a simple choice to proceed or not (AWS).

The queue around that decision matters as much as the decision itself:

• A named owner and a backup. If the only person who can approve is unavailable, either the queue stalls or someone approves items they should not. Microsoft Copilot Studio's approval stages let you require "First to respond" or "Everyone must approve" from named stakeholders, which is one way to build in a backup rather than a single point of failure (Microsoft Learn).
• A time limit and an escalation path. Decide up front how long an item can wait before it goes to someone else, rather than discovering the answer when a customer complains.
• A real pause, not a notification. OpenAI's Agents documentation describes the same mechanic: a tool call that needs review "pauses" the run, and "your application approves or rejects the pending items" before execution resumes from the same state (OpenAI). The action does not happen quietly in the background while the notification sits unread.

Anthropic's own guidance on agent design makes the same point about checkpoints in less formal terms: well built agents "can pause for human feedback at checkpoints or when encountering blockers," rather than running to completion regardless of what they meet along the way (Anthropic). The mechanism matters less than the discipline of building a real stop into the process, wherever the action would otherwise happen.

How do permissions and least privilege keep an agent inside its limits?

An approval step protects you when the agent proposes the wrong thing. Least privilege protects you when it does not even get the chance: the agent should hold only the access its defined tasks need, nothing wider, so a mistake or a manipulated input cannot reach systems the task was never meant to touch. Amazon Bedrock's action groups are one way to draw that line structurally. Each group exposes a specific, named set of functions or API calls the agent may invoke, and the return of control option lets the application, not the model, decide whether and how a given action actually executes (AWS). That separation between what the agent may propose and what the surrounding system is willing to execute is the same idea as least privilege applied to people: scope access to the task, not to the person or system doing it.

In practice this means naming, for each action group or tool the agent can call, which systems it reaches, what it can read versus write, and what it is never allowed to do regardless of what it is asked. Review that list on the same schedule you review the approval levels, because access that made sense for a pilot rarely still makes sense once a task has scaled to real volume.

What should the log and audit trail capture?

Enough that someone who was not in the room can reconstruct what happened and why. That means the proposed action, the source data behind it, any confidence or reasoning the system produced, and who approved, edited or rejected it, and when. NIST's framework treats this as an ongoing operating requirement, not a one time setup task. MANAGE 4.1 calls for "post-deployment AI system monitoring plans," including "mechanisms for capturing and evaluating input from users and other relevant AI actors, appeal and override," while MANAGE 2.4 requires that "mechanisms are in place and applied, and responsibilities are assigned and understood, to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use" (NIST). In plain terms: know how to turn a specific behavior off without turning off everything else, and make sure that decision does not depend on one person's memory of what happened.

When do you move from full review toward sampling, and when do you move back?

Start every new task at full review, whatever level the process will eventually run at. Widen it only as evidence builds: a defined number of reviewed items, or a defined period, during which the approval rate stays high and the corrections reviewers make stop teaching you anything new. How to test AI before it touches your operations covers the evaluation work that produces that evidence in more detail, and the same frozen test set you build for launch is what you rerun before widening a review level, not just before the first release.

Moving back is the less comfortable half of this, and the one teams skip. Pull a task back to full review the moment the error rate rises, the task starts handling a new kind of input, the underlying model or prompt changes, or a reviewer starts disagreeing with a pattern they used to agree with. None of these are failures of the program. They are the program working as designed, because the alternative is finding out about the regression from a customer instead of a reviewer.

What are the common ways human approval breaks down?

Four failure modes account for most of the trouble, and none of them are really about the model.

1. Rubber stamping. A reviewer clicks approve without actually checking, usually because the interface gives them nothing to check against. If the agent cannot show its confidence or its source, a reviewer has no way to spend their attention where it is needed, which is exactly why that gap should push a task down toward fully manual rather than being waved through.
2. A queue nobody owns. Approval requests pile up because no single person is named as accountable, and everyone assumes someone else is handling it. This is the practical failure of skipping the human-AI configuration roles that NIST's GOVERN function asks organizations to define up front.
3. Approval fatigue. Too many low value items reach a person, so they stop reading closely, and the one item that matters gets the same distracted click as the ninety nine that did not. This is usually a sign the threshold or sampling rate was set too conservatively for how the task actually performs.
4. An agent with broad credentials. The agent can reach far more of your systems than any of its actual tasks require, so a single bad decision, or a manipulated input, has a blast radius nobody scoped for. This is the least privilege failure from earlier in this guide, and it is the one most often discovered during an incident rather than a review.

How do you test an approval workflow before it goes live?

Test the queue, not just the agent. Run a batch of real cases, including the awkward ones, through the full path: proposal, review interface, approval or rejection, escalation on a missed deadline, and the log entry that results. A short checklist before go live:

1. A named owner is assigned, with a backup who can act when they are away.
2. The reviewer sees the proposed action, its source and a confidence or reason, not a raw log line.
3. Approving, editing or rejecting takes one action, not a ticket in a different system.
4. An escalation path and a time limit exist for anything left unreviewed.
5. Every proposal, decision and approver is logged and can be pulled up later.
6. The agent's credentials reach only the systems and actions this task needs, nothing wider.
7. A test set of real cases, including the ones that used to cause arguments, has run through the approval step end to end.

How Kastling approaches human approval for AI agents

We treat human approval as part of the design of the agent, not an afterthought bolted on before launch. During discovery and, where the work calls for it, a separately scoped paid audit, we map which of an agent's actions touch money, legal terms, customer communication or an irreversible change, and agree with the business owner who holds the approval for each one. Development is tested against real cases from the client's own history, with the people who will review the output involved before go live.

We scope permissions to the task rather than granting broad access by default, and we agree logging and escalation as part of the build, not as a later addition. A named person stays in control of consequential decisions, and how that control loosens over time is something we agree with the business owner rather than something the system decides for itself. Read more about the AI Integration & Automation service.