Runbook template: turn an incident doc into a workflow

This page has a runbook template to copy, then the same runbook written as a ShiftLabs workflow. The read-only steps run on their own. The step that changes production waits for a person.

What is a runbook?

A runbook is the written procedure for one operational task, such as restarting a service or rotating a key. It lists when to use it, the checks to run, the change to make, how to confirm it worked and how to undo it. A good runbook is specific enough that someone new can follow it at 3 a.m.

Runbook vs playbook

A runbook covers one task, step by step. A playbook covers a whole situation, such as a major outage, and says who does what and which runbooks to use. A playbook often points to several runbooks.

The template

Copy this into your docs and fill in one task.

Text

Runbook: <task name>
Owner: <team or person>
Last tested: <date>

When to use
- <alert name or symptom>

Before you start
- Access needed: <cluster, role, tool>
- Who approves the change: <role or names>

Check (read only)
1. <command or dashboard to confirm the problem>
2. <command to capture current state>

Change (needs approval)
3. <the one command that changes production>

Confirm
4. <command or metric that shows it worked>

Roll back
5. <command to undo step 3>

Notes
- <links to dashboards, past incidents, related runbooks>

The same runbook as a workflow

Here the template is a restart. Steps 1 and 2 run without asking. Step 3 waits for an admin. Every command runs on a runner inside your network.

TypeScript

// workflows/restart-api.ts
import { workflow } from "shiftlabs";

export default workflow<{ namespace: string; deployment: string }>(async ({ payload, step }) => {
  const { namespace, deployment } = payload;

  // Check (read only)
  const pods = await step.exec("list pods", { command: ["kubectl", "get", "pods", "-n", namespace] });
  await step.exec("rollout history", {
    command: ["kubectl", "rollout", "history", `deployment/${deployment}`, "-n", namespace],
  });

  // Change (needs approval)
  const decision = await step.waitForApproval("restart", {
    message: `Restart ${deployment} in ${namespace}?\n\n${pods.stdout.slice(0, 1500)}`,
    approvers: ["role:admin"],
  });
  if (!decision.approved) return { restarted: false, note: decision.note };
  await step.exec("restart", {
    command: ["kubectl", "rollout", "restart", `deployment/${deployment}`, "-n", namespace],
  });

  // Confirm
  await step.exec("wait for rollout", {
    command: ["kubectl", "rollout", "status", `deployment/${deployment}`, "-n", namespace, "--timeout=5m"],
  });
  return { restarted: true, by: decision.by };
});

Approval messages hold up to 2,000 characters, so the example trims the pod list.

Line by line

Check.
step.exec runs a command on your runner and returns its exit code and output. A non-zero exit code throws ExecError, so the run stops before the change.
Change.
step.waitForApproval puts the request in the Approvals inbox and in Slack if it's connected. Only the approvers you name can answer. No answer in 72 hours by default counts as denied.
Confirm.
If the rollout doesn't finish, the command fails and the run shows where.
Roll back.
Add a second workflow for the undo, or run it by hand. Keep it behind an approval too.

Each step is saved as it finishes, and the run keeps the command output, who approved and the note they wrote.

What to automate first

Pick a runbook your team runs by hand at least weekly.
Write only the read-only checks first. Run them for a week.
Add the change behind an approval.
Give the runner only the rights this runbook needs, and limit its token to this workflow. See What a runner runs.

Limits that apply

On Free, a command can run up to 30 minutes, with 2 runner tokens. On Pro, 12 hours and 20 tokens. Commands run on your machines, so they use no CPU time of your plan. See Runner limits.

Start free