Monitoring & alerting: a practical framework

Core philosophy

Monitoring is a core part of running infrastructure, but monitoring is not the same thing as paging. We monitor broadly — anything worth managing is worth watching — but we're deliberate about what actually wakes someone up. Responsibility for a system's monitoring sits with its owner, but an alert firing is never "not my problem" for the rest of the team; everyone shares operational responsibility.

1. What gets monitored, and who decides

Every system the team manages is assumed to be important enough to monitor at least the basics: for a server, that's CPU, disk, memory, and network; for a network device, that's availability and throughput. The system owner is responsible for making sure monitoring exists and is fit for purpose — but "the basics" is a floor, not a ceiling.

2. The toolchain, and what each tool is actually for

No single tool covers everything well, so the team leans on a small set of tools chosen for what they're each strong at, rather than one platform stretched to do everything:

Darktrace — coverage in detail

Darktrace is the one tool in the stack that spans network, endpoint, and SaaS in a single pane, so it's worth breaking out what it actually covers rather than treating it as a single line item:

3. Notifying vs. paging

Both are used only when genuinely required, and the difference is treated as a hard line rather than a suggestion:

• Notification — non-intrusive channel (email, team chat), fine to arrive during office hours only, for things worth knowing but not worth interrupting anyone for.
• Paging — intrusive channel (SMS), reserved for genuine emergency response.

Both carry a real failure mode if overused: alerts that fire too often or too casually get treated as "crying wolf" and eventually ignored — which defeats the entire point of having them. Paging is watched even more carefully, since constant intrusive alerts reliably produce alert fatigue.

Alert lifecycle: from raised to closed

Beyond the notify-vs-page decision, every alert that does get raised follows the same escalation lifecycle, so nothing sits open and unowned indefinitely:

Production component raises an alert
              │
              ▼
   Alert Manager promotes it to a managed alert
              │
              ▼
   Managed alert — status: open, escalation 0
              │
              ▼
   Notification sent (Manager + Alert Operations, email)
              │
      ┌───────┴────────┐
      ▼                ▼
User takes action   User takes no action
      │             (timeout elapses)
      ▼                ▼
Problem resolved   Alert Monitor escalates
      │                │
      ▼                ▼
Managed alert       Managed alert — status: open,
closed, esc. 1      escalation 1
                        │
                        ▼
              Reminder notification sent
                        │
                        ▼
              User solves problem, closes alert
                        │
                        ▼
              Managed alert closed, escalation 1

The escalation step matters as much as the initial page: an alert that's acknowledged but never actually resolved shouldn't quietly age out — escalation 1 exists specifically to catch that case and put it back in front of someone.

Deciding if something deserves an alert at all

Component raises a signal
        │
        ▼
Does it require an action from a human?
        │
   ┌────┴────┐
  No          Yes
   │           │
   ▼           ▼
Don't alert   Is the action urgent / time-sensitive?
              │
         ┌────┴────┐
        No          Yes
         │           │
         ▼           ▼
     Notification    Page
   (email, office     (SMS, any time,
      hours ok)        emergency response)

An alert should never exist just because it's technically possible to create one. Every alert requires a defined action, owned by the system owner — and wherever that action can be automated, it should be, freeing a human for the things that genuinely need judgment. Automating the response doesn't mean ignoring the root cause, though: treating the cause is always better than repeatedly managing the symptom.

4. Action & alert review

An action can be as simple as clearing disk space or as involved as running a remediation script — but there always has to be one, or the alert has no reason to exist. Periodically, every alerting system gets reviewed: is it still in production, are we still watching the right signals, is the threshold still sane. This is what keeps the signal-to-noise ratio from decaying over time as systems get decommissioned or repurposed.

5. Daily backup verification (Veeam)

Backup status is checked by a person every morning, not left purely to automated alerting — the backup dashboard is reviewed for any errors in the last 24 hours, and any failed job gets drilled into.

Worked example: a real resolved incident

A daily job run showed 2 errored jobs out of 16. Drilling into one showed a specific server failing to back up, with the underlying error pointing at a VSS (Volume Shadow Copy Service) snapshot failure:

Failed to create snapshot (Microsoft CSV Shadow Copy Provider)
(mode: Hyper-V child partition snapshot)
Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at
'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.

The fix was to restart the affected VSS-related services on the host — a good illustration of the general rule for this kind of error: the exact error string is usually specific enough to search directly and find a known resolution, rather than needing to reason about VSS internals from scratch. Anything the team can't resolve internally gets raised with the backup vendor's support as a ticket.

Operating checklist

• Every managed system has at least baseline monitoring in place, owned explicitly by someone
• No alert exists without a defined action behind it
• Notifications and pages use genuinely separate channels — never blur the two
• Automatable responses are automated; root causes are still tracked down regardless
• Alerting systems are reviewed periodically for continued relevance, not left to run forever untouched
• Backup dashboard reviewed by a person every morning, not just trusted to alert on failure