Skip to content

Operating

Runbooks

When something is wrong, start here.

Agent not checking in

  1. 01Confirm the service is running: agent status, or Get-Service AgentAgent.
  2. 02Check egress to ingest.getkoop.ai:443. A TLS-inspecting proxy is the single most common cause — the agent pins our intermediate and refuses an intercepted connection by design.
  3. 03Check the certificate: agent cert --show. An expired device certificate needs re-enrollment.
  4. 04Read agent.log around the last successful check-in. The failure reason is logged in full.
  5. 05If the device was de-enrolled, its certificate is revoked and it must re-enrol with a fresh Enrollment token.

An action fails repeatedly

  1. 01Open the incident in Ledger and read the execution stage. The failing step, its exit code and its stderr are recorded.
  2. 02Confirm the device satisfies the action’s preconditions — they are listed on the Playbook page for that action.
  3. 03Check the action’s fleet-wide success rate. If it dropped recently, it is us and not you; the Playbook channel can ship a fix without an agent update.
  4. 04Deny with a structured reason. It feeds the action-quality metrics and is the fastest way to get it fixed.
  5. 05If it is reverting on many devices at once, narrow the Guardrail rule for that action while it is investigated.

Ticket sync drift

  1. 01Check the connector sync health panel: inbound rate, webhook versus polling, error rate, last error.
  2. 02If webhooks are failing, Koop falls back to polling automatically. Resolutions appear in your system up to 60s later, but nothing is lost.
  3. 03Use Replay failed writes for writebacks rejected by a business rule. Idempotency keys make replay safe.
  4. 04If your correlation rate has dropped, a field mapping has probably changed on your side.

Rolling back an agent version

# pin a device group to a known-good version
# Fleet → select group → Pin agent version → 1.8.1

# or per device, locally
agent pin --version 1.8.1

Pinning stops Updater from updating that group. Updater also rolls back automatically on a failed post-update health check, so in most cases you will find this has already happened.

A completely broken agent

If the agent will not start, will not stop, has been partially removed, or is crash-looping faster than Updater can intervene, the recovery path is the standalone repair installer. It is a normal MSI, PKG or shell installer, published alongside every release and signed with the same release keys. Deploy it with whatever you already use to deploy software.

  • It needs no running agent — that is the whole point of it.
  • It needs no working updater, and does not talk to one.
  • It needs no network. Everything it installs is in the package.
  • It needs no enrollment token. The device certificate on disk is reused if it is still valid.
  • It needs no console access, so it works when you have lost visibility of the machine entirely.
text
# Windows — Intune, SCCM, or any MDM that runs a command
msiexec /i KoopRepair.msi /qn

# macOS — Jamf, Kandji, or any MDM that runs a package
installer -pkg KoopRepair.pkg -target /

# Linux — Ansible, Salt, or a shell
sh koop-repair.sh --quiet
  1. 01Push the repair installer to the affected devices through your MDM. There is no Koop-specific tooling to install first.
  2. 02It stops whatever is running, forcibly if the service is refusing to stop, and replaces the agent and the updater together.
  3. 03It preserves the device certificate, the device identifier and the local state, so the machine reappears as itself rather than as a new enrollment.
  4. 04The device checks in within a minute. If it does not, the agent log written by the installer says why.
WHY THIS EXISTS
Updater can replace a wedged agent on its own, and usually does. The repair installer is the floor below that: the case where Updater itself is gone. Without it, the recovery for a fully broken machine is a desk-side visit, which on a fleet of any size is not a recovery plan.

Emergency Halt

Halt is in the top bar of every console screen. Scope it globally, per group or per device. Every agent in scope stops acting within five seconds; telemetry continues, so you keep visibility.

It travels on a channel independent of orchestration and works when Act is down. Agents that are offline receive the stand-down as their first message on reconnect, before any work is dispatched.

AFTER A HALT
Nothing resumes on its own. You re-enable deliberately, per group, and the whole sequence — halt, scope, who, when, and resume — is in Ledger.