Expert guidance for designing statistically valid A/B tests and experiments.
Works with
Provides a structured hypothesis framework, sample size calculations, and metrics selection (primary, secondary, guardrail) to ensure rigorous test design
Covers test types (A/B, A/B/n, MVT, split URL), traffic allocation strategies, and implementation approaches (client-side vs. server-side)
Includes pre-launch checklists, guidance on avoiding common pitfalls like early peeking, and frameworks for analyzing
AI-first code editor with Composer
Before installing skills in Cursor, ensure your development environment meets these requirements:
node --versionab-test-setupExecute the skills CLI command in your project's root directory to begin installation:
Fetches ab-test-setup from coreyhaines31/marketingskills and configures it for Cursor.
The CLI shows a list of agents. Use arrow keys and space to select Cursor:
Confirm successful installation by checking the skill directory location:
Restart Cursor to activate ab-test-setup. Access via /ab-test-setup in your agent's command palette.
We perform automated surface-level scans (Gen AI Scanner, Socket, Snyk) during installation. These checks detect common vulnerabilities but do not guarantee complete security. Always review skill source code and verify the publisher's reputation before production use.
Skills execute code in your environment. Always review source, verify the publisher, and test in isolation before production.
Submit your Claude Code skill and start earning
Automate repetitive workflows and reduce manual effort
Example
Generate reports, summarize documents, draft communications
Save 3-5 hours per week on routine tasks
Learn new skills, understand complex topics, get expert guidance
Example
Explain concepts, provide examples, suggest learning resources
Accelerate learning and skill development by 2x
Enhance output quality through reviews, suggestions, and refinements
Example
Review drafts, suggest improvements, catch errors
Improve work quality by 30-40% with less effort
0
total installs
0
this week
19.2K
GitHub stars
0
upvotes
Run in your terminal
0
installs
0
this week
19.2K
stars
You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.
Check for product marketing context first:
If .agents/product-marketing-context.md exists (or .claude/product-marketing-context.md in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task.
Before designing a test, understand:
Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].
Weak: "Changing the button color might increase clicks."
Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."
| Type | Description | Traffic Needed |
|---|---|---|
| A/B | Two versions, single change | Moderate |
| A/B/n | Multiple variants | Higher |
| MVT | Multiple changes in combinations | Very high |
| Split URL | Different URLs for variants | Moderate |
| Baseline | 10% Lift | 20% Lift | 50% Lift |
|---|---|---|---|
| 1% | 150k/variant | 39k/variant | 6k/variant |
| 3% | 47k/variant | 12k/variant | 2k/variant |
| 5% | 27k/variant | 7k/variant | 1.2k/variant |
| 10% | 12k/variant | 3k/variant | 550/variant |
Calculators:
For detailed sample size tables and duration calculations: See references/sample-size-guide.md
| Category | Examples |
|---|---|
| Headlines/Copy | Message angle, value prop, specificity, tone |
| Visual Design | Layout, color, images, hierarchy |
| CTA | Button copy, size, placement, number |
| Content | Information included, order, amount, social proof |
| Approach | Split | When to Use |
|---|---|---|
| Standard | 50/50 | Default for A/B |
| Conservative | 90/10, 80/20 | Limit risk of bad variant |
| Ramping | Start small, increase | Technical risk mitigation |
Considerations:
DO:
Avoid:
Looking at results before reaching sample size and stopping early leads to false positives and wrong decisions. Pre-commit to sample size and trust the process.
| Result | Conclusion |
|---|---|
| Significant winner | Implement variant |
| Significant loser | Keep control, learn why |
| No significant difference | Need more traffic or bolder test |
| Mixed signals | Dig deeper, maybe segment |
Document every test with:
For templates: See references/test-templates.md
Prerequisites
Time Estimate
15-45 minutes depending on use case complexity
Steps
Common Pitfalls
✓ Do
✗ Don't
💡 Pro Tips
✓ Use when
Use when skill capabilities match your task, clear ROI on time saved, and you can validate outputs. Best for repetitive tasks, learning, and quality improvement.
✗ Avoid when
Avoid when task requires deep expertise you can't validate, involves sensitive decisions, or when learning process is more valuable than speed of completion.
coreyhaines31/marketingskills
coreyhaines31/marketingskills
cexll/myclaude
github/awesome-copilot
github/awesome-copilot
sickn33/antigravity-awesome-skills
Solid pick for teams standardizing on skills: ab-test-setup is focused, and the summary matches what you get after install.
ab-test-setup fits our agent workflows well — practical, well scoped, and easy to wire into existing repos.
ab-test-setup is among the better-maintained entries we tried; worth keeping pinned for repeat workflows.
Keeps context tight: ab-test-setup is the kind of skill you can hand to a new teammate without a long onboarding doc.
Solid pick for teams standardizing on skills: ab-test-setup is focused, and the summary matches what you get after install.
Useful defaults in ab-test-setup — fewer surprises than typical one-off scripts, and it plays nicely with `npx skills` flows.
Registry listing for ab-test-setup matched our evaluation — installs cleanly and behaves as described in the markdown.
Keeps context tight: ab-test-setup is the kind of skill you can hand to a new teammate without a long onboarding doc.
I recommend ab-test-setup for anyone iterating fast on agent tooling; clear intent and a small, reviewable surface area.
We added ab-test-setup from the explainx registry; install was straightforward and the SKILL.md answered most questions upfront.
showing 1-10 of 73