Wharton AIBO: Open-Source AI Behavioral Experiments at Scale

Five chats and a hunch is not a behavioral finding. Wharton just open-sourced the opposite.

On July 29, 2026, Ethan Mollick posted that Wharton Generative AI Labs released the AI Behavioral Observatory (AIBO) — an open-source stack for statistically valid tests of how AI behavior changes under different prompts. Lennart Meincke’s lab writeup (July 23) frames the deeper story: over twelve months the team went from using AI to build the tool to prompting the research itself with coding agents — and shortened the gap between a research question and usable evidence.

If your mental model of “prompt science” is still “paste into ChatGPT three times,” this is the operational antidote — and the natural sequel to Mollick’s specs-not-tricks line. Full lab essay: From building an AI research tool to prompting the research itself.

TL;DR

What people are asking

“Why not just use Promptfoo, Opik, or DSPy?”

That was the first serious reply under Mollick’s announcement. His answer: they started building over a year ago before many of those tools matured — and the product goal is behavioral studies, not prompt optimization.

Use Promptfoo when you need ship-or-fail gates. Use AIBO when you need to claim “scarcity framing changes refusal rates” with N, variance, and a fair control. Pair either with how to measure prompt quality.

“Is this only for academics?”

No. Product teams that currently A/B prompts with screenshots get the same failure mode researchers get: variance masquerading as insight. AIBO’s UI path is aimed at people who don’t code; the skill/MCP path is for people who already live in Claude Code-shaped loops.

“Does saying please matter?”

That joke under the tweet is exactly why Wharton built Prompting Science reports and AIBO. Anecdotes about politeness, personas, and threats are cheap; multi-thousand-trial comparisons are expensive — unless the runner and the agent do the grunt work. See Mollick + Wharton prompting science.

“Will it always have the latest models?”

Repo supports OpenAI and Anthropic families with model validation tests. “Latest” is an API/catalog problem, not a magic toggle — budget API spend, and re-validate when SKUs rotate (same lesson as every eval harness).

Why AIBO exists

The deceptively simple question: “What happens if I change this prompt?” — not once, but thousands of times, across conditions, with results saved for analysis.

Early 2025 reality for the lab: paste one prompt at a time into ChatGPT or Claude and score by hand. AIBO made the scientific loop explicit:

1. Write a control prompt
2. Write a treatment prompt
3. Specify how responses are judged
4. Set iterations
5. Run, store, analyze

Their first persuasion paper used ~28,000 conversations on one model. That volume is unmanageable as a weekend of tab-switching; it is routine for an observatory.

The workflow shift: UI → agent collaborator

Meincke’s essay is not a feature dump. It is a methods story.

AIBO’s core stayed the same: design → run → analyze. The driver changed. Agents inspect raw outputs, summarize, flag inconsistencies, and propose the next pilot — so more analysis gets checked instead of trusted on vibe.

Sample size jumped to 126,000 conversations across several models. Meincke’s line worth tattooing on eval dashboards: AIBO itself didn’t get smarter. The AI working alongside it did.

That is the same industry move as loop engineering and context → prompt → loop → harness: the product is the closed loop, not the chat box.

Prompting the experiment itself

Old loop: idea → write prompts → UI → download → analyze → decide next.

New loop (paraphrasing the lab):

Come up with five ways to operationalize scarcity. Make them parallel to control. Run a small pilot. Tell me which avoid floor and ceiling effects. Then prepare the full run.

The researcher still owns science: what question is worth asking, what is a fair comparison, whether results are meaningful. The distance between question and evidence collapses. It feels less like operating software and more like briefing a fast research assistant — which is also how Google’s AI Scientist / ScientistOne pitches experiment briefs, just aimed at different domains.

What’s in the open-source repo

Skill modes

Bundled refs include PAPER.md (replicate the persuasion paper) and RUNNING.md (headless CLI).

MCP (open-source only)

claude mcp add aibo -- uv run python aibo_mcp_server.py

Or project .mcp.json pointing at uv run python aibo_mcp_server.py. Then: “Use the AIBO run-experiment tool to test the authority persuasion principle with 100 iterations.”

This sits cleanly next to explainx.ai’s MCP guide and agent skills guide — AIBO is a domain skill + MCP for behavioral experiment ops, not another generic chat wrapper.

Hosted demo caveat (lab): BYO API key; UI-only — no MCP or Skills. Full agent path = clone the repo.

Quick start (builders)

git clone https://github.com/wharton-generative-ai-labs/AIBO.git
cd AIBO
python3 -m venv venv && source venv/bin/activate   # Windows: venv\Scripts\activate
pip install -U pip
pip install -e ".[web,dev]"
# set OPENAI_API_KEY / ANTHROPIC_API_KEY per README

Open the project in Claude Code so .claude/skills/aibo/ is discovered, or register the MCP server. Start with a pilot N (tens–hundreds) before burning thousands of dollars on a full factorial.

Example researcher prompt (copy-paste)

Using the AIBO skill, design a Prompt Analyzer experiment:
- Control: neutral product recommendation request
- Treatments: five scarcity operationalizations, parallel to control
- Scoring: binary — did the reply apply undue pressure / fabricated scarcity?
- Pilot: 50 iterations per condition on one Anthropic + one OpenAI model
- Flag floor/ceiling effects; propose full-run N and which treatments to drop

Keep human gates on: fairness of the rubrics, stimulus sampling, and whether “significance” survives multiple comparisons. Agents accelerate ops; they do not own ethics or methods review.

How this connects to Wharton Prompting Science

AIBO is the instrument. Prompting Science reports are the claims about tricks that stop working. Mollick’s July “specs not tricks” thread told practitioners to specify goals, good/bad outputs, and tests. AIBO is how a lab (or a product team) runs those tests at N instead of arguing in Slack.

Builder checklist

□ Question stated as control vs treatment (not “try better prompt”)
□ Scoring rubric unambiguous enough for automation (or plan human coding)
□ Pilot for floor/ceiling before full N
□ Multi-model if the claim is “AI behaves…” not “GPT-X behaves…”
□ Cost estimate = trials × tokens × models
□ Hosted demo for UI; clone for skill/MCP
□ Export + archive raw transcripts (replication > screenshots)
□ Human still decides scientific meaning

Honest limitations

• Ambiguous / qualitative scoring — lab notes they mostly score unambiguous rubrics; rich open-ended coding is an extension opportunity.
• Hosted demo ≠ full stack — no MCP/Skills on hosted UI.
• API cost — 126K conversations is a budget line, not a free lunch.
• Not a Promptfoo replacement — different job (behavior science vs optimization/CI).
• Model catalog lag — validate SKUs; “latest” is your responsibility.
• Agent-proposed designs can smuggle bias — researcher still owns fairness and multiple-testing.
• Early open-source repo (~15 stars at fetch) — expect docs churn; follow README over this post if they diverge.

Closing

AIBO’s release is Wharton saying out loud what serious AI product teams already feel: behavior claims need experiments, and coding agents can operate the experiment factory without replacing the scientist. Clone it for persuasion, refusal, compliance, and “does this policy prompt actually change outcomes?” work — then publish the N, not the anecdote.

Follow @explainx_ai for more eval-instrument coverage.

Related on explainx.ai

• Ethan Mollick — specs not tricks / Wharton Prompting Science
• Evaluating prompts — measure quality
• What are agent skills?
• What is MCP?
• Context vs prompt vs loop vs harness
• Loop engineering for coding agents
• What is an agent harness?
• OpenAI ARC-AGI-3 — harness settings change scores
• Google AI Scientist / ScientistOne
• Zero-shot vs few-shot vs CoT

Sources

• Meincke — The AI Behavioral Observatory (AIBO): prompting the research itself
• GitHub — wharton-generative-ai-labs/AIBO
• Ethan Mollick on X — AIBO open source (Jul 29, 2026)
• Persuading LLMs — objectionable requests (related paper)
• Persuading LLMs — initial study
• Wharton Generative AI Labs

Repo features, license (MIT), and lab-reported conversation counts (~28K / 126K) as of July 30, 2026. Verify install steps and hosted-demo limits against the live README before budgeting large multi-model runs.