PostHog Handbook Library / Engineering

2,096 words. Estimated reading time: 10 min.

# On-call and escalation

Auto TL;DR

## At a Glance

This long page covers these main areas. The list is generated from the article headings, so it updates with every handbook rebuild.

1. Escalation schedules
2. Team schedules
3. Manual escalation schedules
4. When to use manual escalation
5. How to trigger manual escalation
6. Global on call schedule
7. Why is the on call rotation spread across all engineers?
8. Before going on call

At PostHog, every engineer is responsible for maintaining their team's services and systems. That includes:

- Tracking and visualizing key performance metrics
- Configuring automated alerting
- Documenting runbooks for on-call resolution
- First-responder to team-owned services during working hours

In addition, every engineer regardless is part of the global follow-the-sun on-call rotation.

### Escalation schedules

#### Team schedules

Every team has 2 schedules in incident.io

- `On call: {team}`
- This is the working-hours rotation. Each engineer should have their working hours in place here Mon-Fri with a sensible working day
- For example 8:00-17:00 for EU based engineers is likely preferable as there will be US engineers who can take 17:00 onwards
- Each member is responsible for ensuring this is up-to-date with PTO. You can create an override for your schedule simply assigned to "No one".
- `Support: {team}`
- This is a weekly or bi-weekly rotation (teams can decide) that covers both who is assigned to the support hero rotation as well as the out of-hours-escalation for the extreme case

#### Manual escalation schedules

In addition to the regular on-call schedules, teams that own production-critical services have a manual escalation schedule in incident.io:

- `Escalation: {team}`
- This is the "everything is broken" escalation path for critical out-of-hours emergencies
- Only teams that own production-critical services are required to have this schedule - other teams don't need one
- Everyone relevant on the team should be on this schedule - it's not a rotation like regular on-call
- There should be no gaps in coverage since this is the last resort when normal escalation paths fail
- This schedule is triggered manually by whoever is handling an incident when they need additional help

##### When to use manual escalation

Manual escalation should be used when:

- The primary on-call person is unresponsive or unavailable
- The situation is critical and requires immediate additional expertise
- You're the on-call responder and need help from a specific team outside normal working hours

##### How to trigger manual escalation

1. From within an incident in incident.io, use the escalation options to page the relevant `Escalation: {team}` schedule
2. This will notify all members on that escalation schedule simultaneously
3. Any available team member can then respond and assist with the incident

> 💡 Manual escalation is a safety net, not a shortcut. Always try the normal escalation paths first before manually escalating to an entire team.

#### Global on-call schedule

Schedule in incident.io

> 💡 You can use @on-call-global in Slack to reach out to whoever is on call! This syncs automatically with the incident.io schedule. This group is also automatically added to all incidents.

PostHog Cloud doesn't shut down at night (_whose_ night anyway?) nor on Sunday. As a 24/7 service, our goal is to be 100% operational 100% of the time. The global on-call is the last line of defense and is escalated to:

- if nobody at the `On call: {team}` level is available
- if the alert is critical but has no team assignment (for whatever reason)

This schedule has 3 week day layers:

- Europe (06:00 to 14:00 UTC) - (8 hours)
- Americas East (14:00 to 22:00 UTC) - (8 hours)
- Americas West (22:00 to 06:00 UTC) - (8 hours)

And 2 weekend layers:

- Europe Weekend (06:00 to 18:00 UTC) - (12 hours)
- Americas Weekend (18:00 to 06:00 UTC) - (12 hours)

#### Why is the on-call rotation spread across all engineers?

If you're in a product team, it's tempting to think that service alerts don't apply to you, or that when you're on call you can just hand everything off to the infrastructure team. That's not the case, because it's important that every engineer has a basic understanding of how our software is deployed, where the weak points in our systems are, and what the failure modes look like. This understanding should be all that's needed to follow the runbooks, and if you follow the causes of alerts, ultimately you'll be less likely to ship code that takes PostHog down.

Besides knowledge, being on call requires availability – including weekends. If teams had their own separate rotations, there would be more people on call in total, and each would have to stand by 24/7 as our teams aren't big enough to follow the sun. This would be more stressful because of availability constraints, while being less productive because of the rare alerts being spread across multiple people.

### Before going on call

#### Mindset

Read: Jos Visser: Ten things not to worry about regarding oncall (Worth the read, even if you're an on-call veteran.)

#### Be prepared

Because the stability of production systems is critical, on-call involves weekends too (unlike Support Hero). More likely than not, nothing will happen over the weekend – but you never know, so the important thing is to keep your laptop at hand.

Before going on call, make sure you have the Incident.io mobile app Android / iOS installed and configured. This way it'll be harder to miss an alert.

> TRICKY: We use Slack auth for incident.io and Slack really doesn't like you using the mobile web version. Make sure to choose `Sign in with Slack` and then use your email to login to Slack, not google auth as that seems to cause redirect issues for some people.

> Still having redirect issues signing up with Slack? Create a Slack password instead of using Google SSO, then log in with that password.

To get a calendar with all your on-call shifts from incident.io go to the schedules section, select `Sync calendar` at the top right and copy the link for the webcal feed. In google calendar, add a new calendar from URL and paste the link in there.

#### Make sure alerts can break through Do Not Disturb

The incident.io app does not configure these settings for you on install. By default your phone's Do Not Disturb, Sleep, or Focus mode will silence pages. Configure both the incident.io app and phone contact so alerts always come through.

##### iOS

1. Save the incident.io On-call contact to your phone. In the incident.io mobile app, go to `Settings` → `Contacts` and enable the toggle to add the contact to your address book.
2. Enable Emergency Bypass for that contact. Open the `Contacts` app → `Lists` → `All Contacts`, find `incident.io On-call`, tap `Edit` → `Ringtone` → toggle on `Emergency Bypass`. Repeat for `Text Tone` so SMS pages also break through. On iOS 18+, edit from `All Contacts` rather than the auto-grouped "incident.io" section to avoid a known Apple bug.
3. Allow critical notifications. `Settings` → `incident.io` → `Notifications` → enable `Critical Alerts`. These bypass silent and Focus mode for push notifications.
4. Allowlist incident.io in every Focus mode. `Settings` → `Focus` → for each mode (Sleep, Do Not Disturb, Work, etc.) → `Apps` → add `incident.io` to the allowed list.
5. Optional: Watch for sound redirects. Apple Watch pairing, AirPods Announce Notifications, and Screen Time can route incident.io alerts away from your phone speaker – disable the incident.io app in the Watch app's notification list and add incident.io to Screen Time's "Always Allowed" apps.

##### Android

Exact menu paths vary by manufacturer (Pixel, Samsung, OnePlus, etc.), but the same setup applies:

1. Grant Do Not Disturb access to the app. Accept the prompt during incident.io onboarding, or set it manually via `Settings` → `Apps` → `incident.io` → `Special app access` → `Do Not Disturb access`.
2. Disable battery and sleep restrictions for the app. `Settings` → `Apps` → `incident.io` → disable `Pause app activity if unused`, and set `App battery usage` to `Unrestricted`. Otherwise Android may kill the app in the background and you won't get paged.
3. Allowlist incident.io in each Do Not Disturb / Focus mode. On Pixel/stock Android: `Settings` → `Sound & vibration` → `Do Not Disturb` → `Apps` → add `incident.io`. On Samsung: `Settings` → `Modes and routines` → each mode → `Allowed apps` → add `incident.io`.
4. Star the incident.io On-call contact and allow starred contacts to bypass DND. Save and star/favorite the contact, then under `Do Not Disturb` → `People` (or `Exceptions`), allow `Calls` and `Messages` from `Starred contacts`. This lets the phone calls and SMS pages ring through.
5. Consider enabling Alarm-style notifications. In the incident.io app's advanced settings, toggle `Alarm style notifications` and make sure alarms are allowed during Do Not Disturb – this routes pages through the alarm channel, which most phones treat as un-silenceable.

> 💡 Test it! Once configured, put your phone into Do Not Disturb / Sleep mode and ask a teammate to send you a test page (or use the `Send test notification` button in the incident.io app). If you don't hear it, something still isn't right.

### Make sure your availability is up-to-date

If you are unavailable for any of your schedules you need to act!

1. For your `On call: {team}` schedule simply click on your name in your layer, click `create an override` and then remove yourself from the list so it shows `No one`
2. For your `Support: {team}` schedule or `On call: {global}` schedules click `Request cover` at the top right. This will notify selected team members automatically to find someone to cover you (you should probably do a shout out in #ask-posthog-anything as well). You can trade whole weeks, but also just specific days. Remember not to alter the rotation's core order, as that's an easy way to accidentally shift the schedule for everyone.

### Make sure you have all the access you might need

To be ready, make sure you have access to:

- PostHog Cloud admin interfaces (🇺🇸 US / 🇪🇺 EU) - post in #ask-posthog-anything to be added
- Grafana (🇺🇸 US / 🇪🇺 EU)
- ArgoCD - this is where 99% of cluster operations take place such as restarting pods, scaling things up and down etc.
- Metabase (🇺🇸 US / 🇪🇺 EU) - post in #ask-posthog-anything to be invited

#### More advanced access

If you are part of a team that looks after more critical infrastructure such as infra, ingestion, workflows, error-tracking etc. then you are expected to dive deeper than the usual on-call engineer.

As well as the above access you should ensure you have access and feel comfortable working with:

- EKS over `kubectl` / `k9s`, in case you need to run Kubernetes cluster operations (such as restarting a pod) – follow this guide to get access
- Our tailnet, which gates our internal services (such as Grafana, Metabase, or runbooks) – follow this guide to join

### Responding to alerts when on-call

Image: alert-example

Critical alerts will trigger per-team escalation policies which go like this:

1. If available, a member of the team associated with the alert is paged first
2. If nobody is available or nobody responds within the configured time then the `On call: global` schedule is paged

> If at any point you get paged - always respond! Even if you are unavailable you should respond as such (either via the app or the personal Slack notification). That way the escalation can continue to the next available person.

By default if you are being paged, especially as the global on-call, the alert is considered critical, meaning it almost definitely requires attention.

Every alert should have associated Grafana and Runbook links allowing you to quickly get more visual details of what is going on and how to respond.

When an alert fires, find if there's a runbook for it. A runbook tells you what to look at and what fixes exist. In any case, your first priority will be to understand what's going on, and the right starting point will almost always be Grafana.

Sometimes alerts are purposefully overly-sensitive and might already be fixing themselves by the time you see them. Use your best judgement here. If the linked graph has a spike that is clearly coming down, watch it closely and give it time for the alert to auto-resolve.

If the alert is starting to have any noticeable impact on users or you are not sure whether to raise an incident - go raise an incident. It's that simple.

If you're stumped and no resource is of help, get someone from the relevant team to shadow you while you sort the problem out. The idea is that they can help you understand the issue and where to find how to debug it. The idea is _not_ for them to take over at this point, as otherwise you won't be able to learn from this incident.

Canonical URL: https://posthog.com/handbook/engineering/operations/on-call-rotation

GitHub source: contents/handbook/engineering/operations/on-call-rotation.md

Content hash: `7aaa84aeb7d1e015`