Back to Skills

Ab Test Analysis

Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results, checking if a test reached significance, interpreting split test data, or deciding whether to ship …

By phuryn
25k2.6kUpdated 1 month agoMIT

Skill Content

## A/B Test Analysis

Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.

### Context

You are analyzing A/B test results for **$ARGUMENTS**.

If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.

### Instructions

1. **Understand the experiment**:
   - What was the hypothesis?
   - What was changed (the variant)?
   - What is the primary metric? Any guardrail metrics?
   - How long did the test run?
   - What is the traffic split?

2. **Validate the test setup**:
   - **Sample size**: Is the sample large enough for the expected effect size?
     - Use the formula: n = (Z²α/2 × 2 × p × (1-p)) / MDE²
     - Flag if the test is underpowered (<80% power)
   - **Duration**: Did the test run for at least 1-2 full business cycles?
   - **Randomization**: Any evidence of sample ratio mismatch (SRM)?
   - **Novelty/primacy effects**: Was there enough time to wash out initial behavior changes?

3. **Calculate statistical significance**:
   - **Conversion rate** for control and variant
   - **Relative lift**: (variant - control) / control × 100
   - **p-value**: Using a two-tailed z-test or chi-squared test
   - **Confidence interval**: 95% CI for the difference
   - **Statistical significance**: Is p < 0.05?
   - **Practical significance**: Is the lift meaningful for the business?

   If the user provides raw data, generate and run a Python script to calculate these.

4. **Check guardrail metrics**:
   - Did any guardrail metrics (revenue, engagement, page load time) degrade?
   - A winning primary metric with degraded guardrails may not be a true win

5. **Interpret results**:

   | Outcome | Recommendation |
   |---|---|
   | Significant positive lift, no guardrail issues | **Ship it** — roll out to 100% |
   | Significant positive lift, guardrail concerns | **Investigate** — understand trade-offs before shipping |
   | Not significant, positive trend | **Extend the test** — need more data or larger effect |
   | Not significant, flat | **Stop the test** — no meaningful difference detected |
   | Significant negative lift | **Don't ship** — revert to control, analyze why |

6. **Provide the analysis summary**:
   ```
   ## A/B Test Results: [Test Name]

   **Hypothesis**: [What we expected]
   **Duration**: [X days] | **Sample**: [N control / M variant]

   | Metric | Control | Variant | Lift | p-value | Significant? |
   |---|---|---|---|---|---|
   | [Primary] | X% | Y% | +Z% | 0.0X | Yes/No |
   | [Guardrail] | ... | ... | ... | ... | ... |

   **Recommendation**: [Ship / Extend / Stop / Investigate]
   **Reasoning**: [Why]
   **Next steps**: [What to do]
   ```

Think step by step. Save as markdown. Generate Python scripts for calculations if raw data is provided.

---

### Further Reading

- [A/B Testing 101 + Examples](https://www.productcompass.pm/p/ab-testing-101-for-pms)
- [Testing Product Ideas: The Ultimate Validation Experiments Library](https://www.productcompass.pm/p/the-ultimate-experiments-library)
- [Are You Tracking the Right Metrics?](https://www.productcompass.pm/p/are-you-tracking-the-right-metrics)

How to use

  1. Copy the skill content above
  2. Create a .claude/skills directory in your project
  3. Save as .claude/skills/pm-skills-ab-test-analysis.md
  4. Use /pm-skills-ab-test-analysis in Claude Code to invoke this skill

GitHub stars License: MIT PRs Welcome Tests Companion: pm-skills Companion: burnstop Companion: claude-usage

PM Skills Marketplace: The AI Operating System for Better Product Decisions

68 PM skills and 42 chained workflows across 9 plugins. Claude Code, Cowork, and more. From discovery to strategy, execution, launch, growth, and shipping AI-built code.

PM Skills marketplace: skills, commands, and all 9 plugins at a glance

Designed for Claude Code and Cowork. Skills compatible with other AI assistants.

Start Here

New idea? → /discover
Need strategic clarity? → /strategy
Writing a PRD? → /write-prd
Planning a launch? → /plan-launch
Defining metrics? → /north-star

If this project helps you, ⭐ the repo.

Why PM Skills Marketplace?

Generic AI gives you text. PM Skills Marketplace gives you structure.

Each skill encodes a proven PM framework — discovery, assumption mapping, prioritization, strategy — and walks you through it step by step. You get the rigor of Teresa Torres, Marty Cagan, and Alberto Savoia built into your daily workflow, not sitting on a bookshelf.

The result: better product decisions, not just faster documents.

How It Works (Skills, Commands, Plugins)

Example prompts: a skill and two commands (/write-prd, /ship-check) in action

Skills are the building blocks of the marketplace. Each skill gives Claude domain knowledge, analytical frameworks, or a guided workflow for a specific PM task. Some skills also work as reusable foundations that multiple commands share.

Skills are loaded automatically when relevant to the conversation — no explicit invocation needed. If needed (e.g., prioritizing skills over general knowledge), you can force loading skills with /plugin-name:skill-name or /skill-name (Claude will add the prefix).

Commands are user-triggered workflows invoked with /command-name. They chain one or more skills into an end-to-end process. For example, /discover chains four skills together: brainstorm-ideas → identify-assumptions → prioritize-assumptions → brainstorm-experiments.

Plugins group related skills and commands into installable packages. Each plugin covers a PM domain — discovery, strategy, execution, and so on. Installing the marketplace gives you all 9 plugins at once.

Commands use skills. Some skills serve multiple commands. Some skills (like prioritization-frameworks or opportunity-solution-tree) are standalone references that Claude draws on whenever relevant — no command needed.

Commands are designed to flow into each other, matching the PM workflow. After any command completes, it suggests relevant next commands — just follow the prompts.

Installation

Claude Cowork (recommended for non-developers)

  1. Open Customize (bottom-left)
  2. Go to Browse pluginsPersonal+
  3. Select Add marketplace from GitHub
  4. Enter: phuryn/pm-skills

All 9 plugins install automatically. You get both commands (/discover, /strategy, etc.) and skills.

Installing PM Skills in Claude Cowork

Claude Code (CLI)

# Step 1: Add the marketplace
claude plugin marketplace add phuryn/pm-skills

# Step 2: Install individual plugins
claude plugin install pm-toolkit@pm-skills
claude plugin install pm-product-strategy@pm-skills
claude plugin install pm-product-discovery@pm-skills 
claude plugin install pm-market-research@pm-skills 
claude plugin install pm-data-analytics@pm-skills
claude plugin install pm-marketing-growth@pm-skills
claude plugin install pm-go-to-market@pm-skills
claude plugin install pm-execution@pm-skills
claude plugin install pm-ai-shipping@pm-skills

Codex CLI (OpenAI)

Codex reads the same plugin marketplace file as Claude Code, so you can install PM Skills natively — no conversion or file-copying needed:

# Step 1: Add the marketplace
codex plugin marketplace add phuryn/pm-skills

# Step 2: Install the plugins you want
codex plugin add pm-toolkit@pm-skills
codex plugin add pm-product-strategy@pm-skills
codex plugin add pm-product-discovery@pm-skills
codex plugin add pm-market-research@pm-skills
codex plugin add pm-data-analytics@pm-skills
codex plugin add pm-marketing-growth@pm-skills
codex plugin add pm-go-to-market@pm-skills
codex plugin add pm-execution@pm-skills
codex plugin add pm-ai-shipping@pm-skills

What you get: every skill (the PM frameworks), available to Codex and invocable by name. Install whole plugins rather than cherry-picking individual skills — a workflow usually relies on several skills that ship together.

What's different from Claude Code: the /slash commands (/discover, /write-prd, …) install but don't run as Codex slash commands — Codex plugins don't expose commands. To run a workflow, just describe the steps in plain language, for example:

Run product discovery on [your idea]: brainstorm options, map assumptions, prioritize the risky ones, then design experiments — pause between each step.

Optional — let Codex turn the workflows into skills. Because the command files ship inside each installed plugin, you can ask Codex to convert the ones you use most:

Read the command files in the pm-execution plugin and create equivalent Codex skills for the workflows I use most often.

This is a best-effort, model-driven conversion (some Claude-specific command syntax won't translate), but it's a quick way to get the guided workflows on Codex without leaving the CLI.

Other AI assistants (skills only)

The skills/*/SKILL.md files follow the universal skill format and work with any tool that reads it. Commands (/slash-commands) are Claude-specific.

ToolHow to useWhat works
Gemini CLICopy skill folders to .gemini/skills/Skills only
OpenCodeCopy skill folders to .opencode/skills/Skills only
CursorCopy skill folders to .cursor/skills/Skills only
KiroCopy skill folders to .kiro/skills/Skills only
# Example: copy all skills for OpenCode (project-level)
for plugin in pm-*/; do
  mkdir -p .opencode/skills/
  cp -r "$plugin/skills/"* .opencode/skills/ 2>/dev/null
done

# Example: copy all skills for Gemini CLI (global)
for plugin in pm-*/; do
  cp -r "$plugin/skills/"* ~/.gemini/skills/ 2>/dev/null
done

Available Plugins

<details> <summary><strong>1. pm-product-discovery</strong> — Ideation, experiments, assumption testing, OSTs, interviews (13 skills, 5 commands)</summary>

Skills (13):

  • brainstorm-ideas-existing — Multi-perspective ideation for existing products (PM, Designer, Engineer)
  • brainstorm-ideas-new — Ideation for new products in initial discovery
  • brainstorm-experiments-existing — Design experiments to test assumptions for existing products
  • brainstorm-experiments-new — Design lean startup pretotypes for new products (Alberto Savoia)
  • identify-assumptions-existing — Identify risky assumptions across Value, Usability, Viability, and Feasibility
  • identify-assumptions-new — Identify risky assumptions across 8 risk categories including Go-to-Market, Strategy, and Team
  • prioritize-assumptions — Prioritize assumptions using an Impact × Risk matrix with experiment suggestions
  • prioritize-features — Prioritize a feature backlog based on impact, effort, risk, and strategic alignment
  • analyze-feature-requests — Analyze and categorize customer feature requests by theme and strategic fit
  • opportunity-solution-tree — Build an Opportunity Solution Tree (Teresa Torres) — outcome → opportunities → solutions → experiments
  • interview-script — Create a structured customer interview script with JTBD probing questions
  • summarize-interview — Summarize an interview transcript into JTBD, satisfaction signals, and action items
  • metrics-dashboard — Design a product metrics dashboard with North Star, input metrics, and alert thresholds

Commands (5):

  • /discover — Full discovery cycle: ideation → assumption mapping → prioritization → experiment design
  • /brainstorm — Multi-perspective ideation (ideas|experiments × existing|new)
  • /triage-requests — Analyze and prioritize a batch of feature requests
  • /interview — Prepare an interview script or summarize a transcript (prep|summarize)
  • /setup-metrics — Design a product metrics dashboard

Examples:

Skills:

  • What are the riskiest assumptions for our AI writing assistant idea?
  • Help me build an Opportunity Solution Tree for improving user activation
  • Prioritize these 12 feature requests from our enterprise customers [attach CSV]

Commands:

  • /discover AI-powered meeting summarizer for remote teams
  • /brainstorm experiments existing — We need to reduce churn in our onboarding flow
  • /interview prep — We're interviewing enterprise buyers about their procurement workflow
</details> <details> <summary><strong>2. pm-product-strategy</strong> — Vision, business models, pricing, competitive landscape (12 skills, 5 commands)</summary>

Product strategy, vision, business models, pricing, and macro environment analysis. Covers the full strategic toolkit from vision crafting through competitive landscape scanning.

Skills (12):

  • product-strategy — Comprehensive 9-section Product Strategy Canvas (vision → defensibility)
  • startup-canvas — Startup Canvas combining Product Strategy (9 sections) + Business Model — an alternative to BMC and Lean Canvas for new products
  • product-vision — Craft an inspiring, achievable, and emotional product vision
  • value-proposition — 6-part JTBD value proposition (Who, Why, What before, How, What after, Alternatives)
  • lean-canvas — Lean Canvas business model for startups and new products
  • business-model — Business Model Canvas with all 9 building blocks
  • monetization-strategy — Brainstorm 3–5 monetization strategies with validation experiments
  • pricing-strategy — Pricing models, competitive analysis, willingness-to-pay, and price elasticity
  • swot-analysis — SWOT analysis with actionable recommendations
  • pestle-analysis — Macro environment: Political, Economic, Social, Technological, Legal, Environmental
  • porters-five-forces — Competitive forces analysis (rivalry, suppliers, buyers, substitutes, new entrants)
  • ansoff-matrix — Growth strategy mapping across markets and products

Commands (5):

  • /strategy — Create a complete 9-section Product Strategy Canvas
  • /business-model — Explore business models (lean|full|startup|value-prop|all)
  • /value-proposition — Design a value proposition using the 6-part JTBD template
  • /market-scan — Macro environment analysis combining SWOT + PESTLE + Porter's + Ansoff
  • /pricing — Design a pricing strategy with competitive analysis and experiments

Examples:

Skills:

  • Compare Lean Canvas vs Business Model Canvas vs Startup Canvas for my marketplace startup
  • Design a value proposition for our AI writing assistant targeting non-native English speakers
  • Run a Porter's Five Forces analysis for the project management SaaS market

Commands:

  • `/strategy B2B project manage

View source on GitHub