DigitalOcean Signals and Simulations: Catching AI Agents That Fail Silently

DigitalOcean has launched a set of tools aimed at one of the hardest problems in production agents: failures that do not look like failures. Signals, Simulations and Insights are in public preview, and Agent Data Services is in private preview. According to DigitalOcean's announcement, Signals watches live traffic for frustrated users and failing tools, and Simulations lets you replay scenarios in a sandbox to test a fix before it ships.

If you run agents for customers, the pitch is familiar and the details matter: how do you turn a bad conversation into a test, and who approves the fix? Here is what DigitalOcean says, what it leaves out, and how it compares to other agent observability approaches.

TL;DR: the questions people will ask

The problem: agents that look healthy while failing

Classic monitoring catches crashes and latency. Agents fail differently: the server returns 200, the model produces fluent text, and the user is still wrong, stuck or annoyed. Teams find out from support tickets, or never. DigitalOcean calls the output of this class of problem a silent failure, and its new services are built around turning those into a test set.

This follows the company's September launch of Managed Agents with pay-per-use CPU and 16,000 tools: once agents are running on the platform, the next product is telling you how they behave.

Observability trace beads under a magnifying lens showing DigitalOcean Signals reviewing agent sessions for silent failures

Signals: finding the sessions that matter

Signals monitors production traffic and groups conversation turns into sessions. It flags three kinds of trouble: Interaction (frustrated or confused users), Execution (failing tools) and Environmental (infrastructure constraints). Trends appear on a Signals view in Harness Runtime and Serverless Inference, and you can drill down into the sessions behind them.

Some details from the announcement:

• Cheap first pass. Signals runs a low-cost triage to select informative sessions, so you are not paying a large model to read every transcript. In a controlled study on τ-bench, DigitalOcean reports an 82% informativeness rate and a 1.52x efficiency gain over random sampling. That is the vendor's own number on one benchmark.
• Opt-in and private. Signals is designed to redact PII before processing, keeps data within your DigitalOcean Team boundary, and the post says the data is never used for cross-customer model training.
• Setup. You enable it per agent, and data processing can take up to an hour.

DigitalOcean's post does not say exactly how the first-pass triage works beyond calling it inexpensive and pointing to its paper, "Signals: Trajectory Sampling and Triage for Agentic Interactions". Flagged sessions can be turned into evaluation scenarios in one click.

Simulations: test the fix before anyone else hits the bug

Simulations builds scenarios from your agent's configuration, weighted by Signals data when available, or from a plain-language description. A user simulator then plays each persona against your candidate agent in an isolated sandbox. Mock tools can stand in for systems that write data or cost money, so a test run does not email real customers or charge a card. An LLM judge scores each run on user-goal completion, knowledge retention and tool-use accuracy, plus any custom metrics you define.

Isolated sandbox dome with a gripper at work, illustrating DigitalOcean Simulations testing agent fixes safely

In setup you choose the candidate agent, the user simulator model, metrics, episodes per scenario and a turn limit. Simulations work with any harness Harness Runtime supports, including Claude Code, Codex and OpenCode. Several widely used frameworks are not supported in this preview.

Each scenario plays several times, and because every trajectory records each tool call, DigitalOcean says Simulations can tell a reasoning failure from a tool-use failure. Running a baseline against a candidate shows what improved and what regressed. The post does not describe a human-approval step; keeping one is our recommendation, not a product feature.

Insights and Agent Data Services

Insights is the third public-preview piece: built-in observability across agents, applications and infrastructure, according to DigitalOcean's separate launch post. Agent Data Services, in private preview, is the data layer: managed data flows that process, enrich and vector-index incoming data; a catalog where structured records and vectors share governed lineage; SQL and API querying; and sources such as Spaces and third-party cloud storage. Example uses include capturing agent execution traces and session memory, making video libraries searchable, and unifying images, audio, call transcripts and product documentation in one endpoint. Access is governed by per-workspace scoping, team roles, API tokens and audit logging.

Private preview terms are blunt: services are provided as-is, with no SLA or support commitment, and they are not covered by DigitalOcean's standard compliance certifications. Pricing is still in development, per the Agent Data Services post.

Agent approval handoff with a gripper offering a bead to a human hand, representing human review of proposed agent fixes

A worked example of the loop

Imagine a support agent that answers refund questions. A customer asks whether a digital purchase can be refunded after seven days, the agent replies confidently with the wrong window, and the customer rephrases the question three times before giving up. Nothing errors. In a Signals-style workflow, that session is flagged as an Interaction problem because of the repeated rephrasing, and it lands in a list of informative sessions rather than in a haystack of thousands.

You then write a scenario from that session: a persona who asks about refunds on digital goods after a week. Simulations runs that persona against your current agent and against a candidate with a corrected policy document or a changed prompt, using mock tools for the order system so nothing real is touched. The judge scores both runs for goal completion and tool-use accuracy. If the candidate wins across several episodes, you ship the change after a human signs off. The same scenario then stays in your test set, so a later prompt edit cannot quietly reintroduce the bug.

That last step is the real value: a library of regression tests that grows from actual incidents instead of from guesses made at design time. It is the same discipline software teams use when they add a test for every production bug, applied to conversations.

What to measure before trusting it

Because both the triage and the judging involve models, validate them before you rely on them. Take fifty sessions, label them yourself as good or bad, and see how many Signals flags that you would also have flagged, and how many it misses. Then take ten scenarios where you know the right answer and check whether the judge scores agree with you. If agreement is low, tighten the metrics or write custom ones. Track the cost of simulations as well: each episode is a conversation between two models, so large scenario sets can add up even when individual runs are cheap.

How it compares and when it makes sense

The idea of converting production failures into evals is not unique. Other observability vendors have been shipping issue-detection features, as we covered in LangSmith Engine's issue detection. What is distinctive here is the integration with the hosting platform: if your agents already run on DigitalOcean's Managed Agents, Signals and Simulations need no extra instrumentation, and the data stays in your account boundary.

It makes sense if you already host on DigitalOcean and have enough traffic that manual transcript review no longer scales. It makes less sense if your agents run elsewhere, use an unsupported framework, or you need an open-source, self-hosted pipeline.

Limits and open questions

• Preview status. Public preview for three products, private preview for the data service, with no stated pricing or SLAs.
• LLM judges. Simulation scores depend on a judge model, which can be wrong or inconsistent; validate it on cases you grade by hand.
• Simulated users are not real users. A persona played by a model finds some bugs and misses others.
• Framework support. LangChain, Hermes and CrewAI are not supported in this preview.
• Security. Agents that act on real systems need controls beyond observability; failure detection finds problems after the fact.

What to do this week

1. Pick one agent with real traffic and enable Signals on it.
2. Review the flagged sessions manually for a few days to see whether the categories match your pain.
3. Turn three real failures into Simulations scenarios, apply a candidate fix, and compare judge scores before and after.
4. Keep a human approval step for any change that ships.

Related reading

• DigitalOcean Managed Agents launch
• LangSmith Engine issue detection
• OpenCode 2.0: Code Mode and background agents
• What is an agent harness?
• Fork-and-Flush for long-running agents
• Agent teams vs a single agent: cost and quality
• Agent Learning Services announcement

Features and availability are accurate as of October 12, 2026 and may change during preview.