Zero-touch configuration backup for a multi-vendor network

Architecture overview

A dedicated Linux host runs Oxidized as a systemd background service. It fetches configuration from every device in scope over SSH/REST, and — instead of leaving backups sitting on a local disk — pushes them straight into a Git repository over SSH, authenticated with a repo-scoped deploy key rather than a personal access token or a shared password.

[ Network devices ]  ◄── SSH / REST API ──  [ Oxidized service ]  ── SSH / deploy key ──►  [ Git repository ]
 Palo Alto · Juniper         (systemd,                                                         (protected branch,
 Aruba · Cisco IOS-XE       user: oxidized)                                                     force-push allowed
                                                                                                  for this key only)

1. Scope of monitored infrastructure

Devices are mapped in Oxidized's inventory (router.db) using their OS model, management IP, and environment metadata. The full active backup scope for this deployment — hostnames and IPs below are illustrative, not the real addressing:

Twelve devices across five OS types, spanning core/distribution/edge switching, dual-node virtual chassis routing, an HA firewall pair backed up independently on each node, and carrier-facing edge routers reached over a public management path where no private one exists. New device types are added to router.db as a one-line mapping — the pipeline itself doesn't change.

2. Oxidized server configuration

Oxidized runs under a dedicated, non-personal system account (oxidized). The systemd unit keeps it running across reboots and restarts it automatically if the process dies.

[Unit]
Description=Oxidized - Network Configuration Backup
After=network-online.target

[Service]
User=oxidized
ExecStart=/usr/local/bin/oxidized
Restart=on-failure
RestartSec=10s

[Install]
WantedBy=multi-user.target

The application config sets a 24-hour fetch interval and defines a post-cycle hook that pushes the freshly-fetched configs to Git over SSH — deliberately over SSH rather than HTTPS, to sidestep MTU/TLS interception issues seen on some managed network paths.

interval: 86400
log: /home/oxidized/.config/oxidized/logs

hooks:
  push_to_git:
    type: exec
    events:
      - nodes_done
    cmd: "git --git-dir=/opt/oxidized/.config/oxidized/network-backups.git --work-tree=/opt/oxidized/.config/oxidized/network-backups.git push git@git.internal.example.com:it-ops/network.git main --force"
    async: true

3. Authentication & security (Git hosting)

No personal access tokens are used anywhere in this pipeline — it's entirely independent of any individual's account, which matters both for auditability and for continuity when people change roles.

1. Generate a dedicated ed25519 SSH key locally on the server, under the oxidized account:
   bash

   sudo su - oxidized
   ssh-keygen -t ed25519 -C "oxidized@backup-host01"

2. Add the public key (~/.ssh/id_ed25519.pub) to the Git project as a deploy key under Settings → Repository → Deploy Keys.
3. Grant that deploy key write permission — and name it clearly (e.g. "Oxidized Agent — backup-host01") so a future admin knows exactly what it's for and where to revoke it.

Protected branch configuration

To let Oxidized overwrite history when needed (--force, since a config backup repo doesn't need merge history), the target branch is configured to explicitly allow force pushes:

• Project → Repository → Protected branches
• Setting: "Allowed to force push" enabled for the main branch

4. Vendor-specific gotcha: Juniper redaction

By default, Junos hides sensitive configuration (passwords, keys) behind /* ACCESS-DENIED */ placeholders unless the account pulling the config has sufficient view rights — which silently produces backups that look complete but aren't. Fixed with a dedicated, minimal privilege class:

# 1. Create a dedicated privilege class for backups
set system login class oxidized-class permissions [ view view-configuration ]

# 2. Create the local user and link it to the class
set system login user oxidized-agent class oxidized-class

# 3. Set authentication (password or the SSH public key from the Oxidized server)
set system login user oxidized-agent authentication plain-text-password

commit and-quit

This grants exactly enough visibility to pull a complete config — nothing more — which is the point of a dedicated backup account over reusing an existing admin login.

5. Administration & troubleshooting

Managing the service

# Restart the service after making configuration changes
sudo systemctl restart oxidized

# Verify that the service is running (Active: active (running))
sudo systemctl status oxidized

# View live logs from Oxidized
sudo journalctl -u oxidized -f

Manual test run (debug mode)

To force an immediate backup cycle outside the schedule and see verbose output directly in the terminal:

# Stop the background daemon first
sudo systemctl stop oxidized

# Run a single one-shot backup with debugging enabled
OXIDIZED_FLAGS="one_shot" OXIDIZED_REST="false" oxidized --debug

Git directory permissions

If manual runs hit Permission denied or unable to access .gitattributes, it's almost always ownership drift from running a manual test as a different user:

sudo chown -R oxidized:oxidized /opt/oxidized/.config/oxidized/network-backups.git

Rollout checklist

• Dedicated oxidized service account created, no shared/personal credentials
• systemd unit installed with restart-on-failure
• Repo-scoped SSH deploy key generated and added with write access only
• Protected branch configured to allow force-push from the automation key only
• Vendor-specific least-privilege accounts created where redaction is a risk (e.g. Junos)
• Post-cycle Git push hook tested end to end with a manual one-shot run
• Backup interval and retention confirmed against change-management requirements