PostHog Handbook Library / Engineering

792 words. Estimated reading time: 4 min.

Running experiments on PostHog.com

These are the defaults for any A/B test on the website. Most of an experiment's validity lives in the PR, not in the PostHog UI, so get the code right first.

If you use an agent, point it at the website-experiments skill in the PostHog skills store. It has the same rules, a code template, and the exact metric definitions.

Planning

Settle these before any code ships.

  1. Hypothesis in the description: the change, why it should work, the primary metric, and the PR link.
  2. Audience: PostHog.com shares project 2 and the person record with the app, so exclude existing app users in code.
  3. Exposure point: if the change is below the fold, expose on scroll, not page load.

Implementing the flag

  1. Ship dark, with control as the default. Deploy the flag at 0% initially.
  2. Don't read the flag for people who are out of scope. If we need to filter out existing users, check the ph_current_instance cookie first (getPostHogInstanceCookie in src/hooks/usePostHogInstance.ts). If it is set, render control and never call getFeatureFlag. No flag call means no exposure. Filters on $is_identified can't do this job: existing users on a logged-out browser look anonymous when they are exposed.
  3. Fire a custom exposure event where the change becomes visible (if below the fold). Send it once per page view when the top of the experiment slot scrolls into view, with $feature/<flag-key> on the event. Then set it as the experiment's exposure event.
  4. Make the exposure condition identical in both arms. Wrap control and test in the same slot component with the same marker. If only the test arm tracks visibility, the arms are no longer comparable.
  5. Count assignment, not successful rendering. Keep a test visitor in test even if the variant fails to load, and put an error boundary around a lazy-loaded variant.
  6. Remove preview scaffolding before merge. Layout switchers and debug toggles used to compare designs on the deploy preview don't belong in production.

PR #21732 is a good reference implementation of rules 2 to 5.

Metrics and analysis

  1. Primary metric. user signed up with is_organization_first_user = true counts new orgs, not invited teammates.
  2. Add the four signup-quality metrics as secondaries. They show whether extra signups are worth having. All are funnels that start at a new-org signup:
  1. Use Bayesian stats with a 90% credible interval by default. Use 95% for high-impact pages, such as the pricing page or anything above the fold on the homepage.

Running and cleaning up

  1. Check the first day's exposures, persons, and recordings. Problems like existing users leaking into the test show up within a day.
  2. Fix exposure in code, then reset. Editing exposure criteria on a running experiment can't remove people already enrolled, so a reset is cheaper than a contaminated result.
  3. Clean up within a sprint. Once concluded, remove the losing branch and the flag check, delete the exposure event code, and archive the flag. A flag left at 100% with dead code behind it is flag debt.

Pre-launch checklist

Copy this into the PR description of every website experiment.

Canonical URL: https://posthog.com/handbook/engineering/posthog-com/running-experiments

GitHub source: contents/handbook/engineering/posthog-com/running-experiments.md

Content hash: dc353e8c0364be90

Static reader notes
  • MDX_COMPONENT_STATIC_ADAPTER: Adapted interactive MDX components for static reading: PrivateLink.