How to Build an AI Agent Loop: Triggers, Retries, Checkpoints, and Human Handoffs
Learn how to design a production-grade ai agent loop architecture with proper triggers, state management, retry logic, checkpoints, and human handoff patterns that don't break silently under real workloads.
AI agentsAgent architectureAgentic workflowAutonomous agentsLLM engineering
Agent design patterns
Claude Code
Production AI
Last month a team shipped an agent that scraped competitor pricing, formatted a report, and emailed it to the sales team every morning. It worked perfectly in testing. In production, it ran for three weeks before anyone noticed it had been sending the same report repeatedly because the pricing page had changed its HTML structure. The agent never errored. It just quietly looped and delivered stale data.
That is the failure mode that kills production agents. Not dramatic crashes — silent drift. The loop kept running, the termination condition was never triggered, and there was nothing in the output that looked wrong at a glance.
Building an agent loop that works in a demo is easy. Building one that holds up in production requires thinking through four primitives, three checkpoint rules, a retry strategy, and a clear policy on when to stop and ask a human. This guide covers all of it.
What an agent loop actually is in production
A demo agent loop is usually: call LLM, get a response, maybe call a tool, repeat. That works for a five-minute walkthrough.
The trigger starts the loop — a schedule, a webhook, a user action, or a prior agent completing its own loop. The task queue holds work units if multiple items need processing. Inside the loop body, the agent reads its current state, runs the executor (the LLM call plus any tool calls), writes updated state, and evaluates whether to continue or stop. Checkpoints and human gates live inside the loop body, not outside it.
The 4 primitives every agent loop needs
1. Trigger
The trigger answers: what starts this loop, and what data does it carry?
A trigger can be scheduled (cron), event-driven (webhook, message queue), or chained (prior agent output). The critical design question is what the trigger payload contains and whether the loop can validate it before starting execution.
python
@dataclassclassTriggerPayload:
task_id: str# stable, used as idempotency key root
task_type: str# determines which executor branch runs
input_data: dict# the actual work input
triggered_by: str# "schedule" | "webhook" | "agent:research-runner"
triggered_at:
max_iterations: =
Validate the payload immediately. If task_id is missing, the idempotency system breaks downstream and you'll get duplicate side effects on retries.
2. State
State is what carries information across iterations. It is not the LLM's context window — it is an external store that persists even if the loop crashes.
python
@dataclassclassAgentState:
task_id: str
iteration: int
status: str# "running" | "waiting_for_human" | "done" | "failed"
working_memory: dict# structured data the agent has collected
action_log: list# every action taken, for idempotency checks
last_checkpoint: str# ISO timestamp of last successful checkpoint
error_count: int
last_error: str | None
State should be serializable to JSON and written to a durable store (database, Redis with persistence, object storage) after every iteration. If the agent process crashes, the loop can be resumed by reading state back in.
3. Executor
The executor is the LLM call plus tool execution. It takes the current state and the task context as input, runs a step, and returns a result plus any state mutations.
The executor should not make decisions about whether to continue. That belongs to the terminator.
4. Terminator
The terminator evaluates whether the loop should continue, stop successfully, or stop with an error. Keep termination logic explicit and separate from the executor.
The is_stuck check deserves special attention. Maintain a rolling window of the last N tool calls and their arguments. If the same call appears twice with identical arguments, the agent is not making progress. Stop it.
Where to put checkpoints
A checkpoint is a persisted snapshot of agent state at a known-good moment. Checkpoints make loops restartable. Without them, a crash at step 18 of a 20-step research task means starting from zero.
Three rules for checkpoint placement:
Before irreversible actions. If the next step will send an email, write to a database, call a payment API, or post to an external system — checkpoint first. If the action fails mid-way, you can retry from the pre-action snapshot with the same state.
After expensive operations. If a step burned 10,000 tokens on a complex analysis or made several API calls to aggregate data, checkpoint immediately after. Losing that work to a downstream failure is expensive.
At confidence thresholds. If your executor returns a confidence score or uncertainty signal, checkpoint when the agent transitions from high-confidence to uncertain territory. That boundary is where you most often need to resume with human input.
Most agent failures are transient: rate limits, brief network issues, a tool API returning a 503. A well-designed retry strategy handles these without human involvement.
Exponential backoff with jitter. Start with a short base delay and double it on each retry, adding a random jitter to prevent thundering herd if multiple agents fail simultaneously.
python
import random
import time
defretry_with_backoff(fn, max_retries=3, base_delay=1.0):
for attempt inrange(max_retries):
try:
return fn()
except RetryableError as e:
if attempt == max_retries - 1:
raise
delay = base_delay * (2 ** attempt) + random.uniform(0, 1)
time.sleep(delay)
Idempotency keys. Every write operation should carry an idempotency key before it's retried. Derive the key from a stable hash of the task ID and step index — not from a timestamp, which changes on retry.
What NOT to retry. Do not retry: LLM calls that returned a valid response (even an unexpected one), actions that succeeded but returned an error in the response body, or any operation you can't verify was idempotent. The safest rule is to only retry operations that raised a clear infrastructure exception (timeout, rate limit, network error) and that you can prove are idempotent.
Human handoffs: the decision framework
The hardest design question in any agent loop is where to put human gates. Gate too aggressively and you've built an expensive email drafting assistant. Gate too loosely and you've given an agent unsupervised authority over consequential actions.
The decision axis is: reversibility times blast radius.
Action
Reversible?
Blast radius
Gate it?
Read a file
Yes
None
No
Search the web
Yes
None
No
Write a draft
Yes
Low
No
Send an email
No
Medium
Yes
Post to social media
No
High
Yes
Delete database records
No
High
Yes
Approve a financial transaction
No
Very high
Yes
Update a config in production
No
Very high
Yes
When you do gate, the gate must be non-blocking for the agent. The loop pauses, writes a human-approval task to a queue, and the agent state is persisted in "waiting_for_human" status. When the human acts, the queue triggers the loop to resume from that state.
The approval URL pattern — where a human can review context and click approve or reject — is far better than email-based approval. Email gets buried. A dedicated approval UI surfaces the full agent context.
Three loop patterns with code
Pattern 1: Simple task-runner loop
For single-task, single-pass work with clear completion criteria.
python
defrun_task_loop(trigger: TriggerPayload, db, llm_client) -> AgentState:
state = load_or_init_state(trigger, db)
whileTrue:
# Execute one step
step_result = execute_step(state, trigger.input_data, llm_client)
# Update state
state.iteration += 1
state.working_memory.update(step_result.state_mutations)
state.action_log.append(step_result.summary())
# Checkpoint if neededif should_checkpoint(state, step_result):
save_checkpoint(state, db)
# Evaluate termination
decision = evaluate_termination(state, step_result)
if decision.should_stop:
state.status = decision.status
save_checkpoint(state, db)
return state
return state
Pattern 2: Multi-step research loop with state persistence
For tasks that span multiple research passes, where each pass informs the next.
python
defrun_research_loop(trigger: TriggerPayload, db, llm_client) -> AgentState:
state = load_or_init_state(trigger, db)
# Research loop: gather, synthesize, verify
phases = ["gather", "synthesize", "verify"]
for phase in phases:
state.working_memory["current_phase"] = phase
phase_iterations = 0while phase_iterations < 5:
step_result = execute_step(state, {**trigger.input_data, "phase": phase}, llm_client)
state.iteration += 1
phase_iterations += 1
state.working_memory.update(step_result.state_mutations)
state.action_log.append(step_result.summary())
# Checkpoint after every gather step (expensive)if phase == "gather":
save_checkpoint(state, db)
if step_result.phase_complete():
break
decision = evaluate_termination(state, step_result)
if decision.should_stop:
state.status = decision.status
save_checkpoint(state, db)
return state
state.status = "done"
save_checkpoint(state, db)
return state
The key difference: state carries the current phase, so if the process crashes mid-gather, it resumes in the gather phase rather than starting from scratch.
Pattern 3: Human-gated approval loop
For loops where certain actions require explicit human sign-off before proceeding.
python
defrun_approval_loop(trigger: TriggerPayload, db, llm_client, approval_queue) -> AgentState:
state = load_or_init_state(trigger, db)
whileTrue:
step_result = execute_step(state, trigger.input_data, llm_client)
state.iteration += 1
state.working_memory.update(step_result.state_mutations)
state.action_log.append(step_result.summary())
# Check if this step wants to take an irreversible actionif step_result.proposed_action and is_irreversible(step_result.proposed_action):
save_checkpoint(state, db) # checkpoint before gate
request_human_approval(state, step_result.proposed_action, approval_queue)
return state # loop suspends hereif should_checkpoint(state, step_result):
save_checkpoint(state, db)
decision = evaluate_termination(state, step_result)
if decision.should_stop:
state.status = decision.status
save_checkpoint(state, db)
return state
return state
defresume_after_approval(task_id: str, approved: bool, db, llm_client, approval_queue) -> AgentState:
state = load_state(task_id, db)
ifnot approved:
state.status = "failed"
state.working_memory["rejection_reason"] = "human_rejected_action"
save_checkpoint(state, db)
return state
# Inject approval into state so the agent knows it was cleared
state.working_memory["last_approved_action"] = state.action_log[-1]
state.status = "running"# Resume the loopreturn run_approval_loop(
TriggerPayload(task_id=task_id, **state.working_memory["original_trigger"]),
db, llm_client, approval_queue
)
This pattern is central to building agents that can handle consequential work without requiring a human to babysit every step. The agent runs autonomously until it hits a gate, suspends cleanly, and picks up exactly where it left off after approval.
Silent failures. The agent returns a result that looks valid but is wrong. Fix: add output validation after every executor call. Define a schema for what a valid step result looks like and reject results that don't match before updating state.
Infinite loops. The terminator never fires because the success signal is ambiguous. Fix: always set max_iterations at the trigger level, enforce it in the terminator as a hard cap, and log a warning (not a silent exit) when the cap is hit.
State corruption on concurrent runs. Two instances of the same loop run simultaneously and write conflicting state. Fix: use optimistic locking on the state store — include a version field in the state and reject writes where the version doesn't match the current database version.
Retry storms. A transient failure causes the loop to retry aggressively, which overloads the downstream service, which causes more failures. Fix: cap total retries at the task level (not per-step), and implement circuit breakers on high-failure-rate operations.
Context window drift. In long loops, the message history passed to the executor grows until it degrades generation quality or hits the token limit. Fix: summarize old turns rather than passing the raw history — keep the last N turns verbatim and compress earlier turns into a structured summary injected at the top of the message history.
Building agent loops as a career skill
Knowing how to design a loop that doesn't break silently in production is not a trivial skill. It requires understanding LLM behavior across multi-turn execution, distributed systems patterns (idempotency, circuit breakers, state machines), and human-computer interaction design for approval workflows.
This is also increasingly a skill that forward-deployed engineers are expected to bring. If you're interviewing for FDE or AI engineering roles, expect questions about agent loop design and failure handling — the ability to reason through what happens when an agent hits an unexpected state is now a standard interview topic.
The patterns in this guide are practical starting points. What takes more time is implementing them against real infrastructure, debugging the edge cases that only appear under production load, and learning to tune termination conditions for your specific task type.
Loop Engineering workshop: July 20, 2026
If you want to build all three loop patterns from this guide as working systems — not pseudocode, but actual running loops with state persistence, checkpoint saves, and a human approval UI — the Loop Engineering workshop at explainx.ai covers exactly that.
What the session covers:
Wiring up triggers: scheduled, webhook, and chained agent triggers
State design and persistence with a real database backend
Retry logic with idempotency keys that actually work
Checkpoint placement and resume-from-checkpoint logic
Human-gated approval UI with a real queue-and-resume pattern
Three working loop templates you leave with: multi-step research runner, human-gated content pipeline, failure-resilient task runner
Details: July 20, 2026 — one four-hour live session.
The core insight from every production agent loop that has failed: the problem is almost never the LLM. It's the infrastructure around the LLM — the missing checkpoint that forces a full restart, the retry logic that doesn't account for idempotency, the terminator that relies on the model to decide it's done rather than enforcing a hard cap. Get the loop architecture right and the LLM part becomes much easier to reason about.
