NETMapper operations guide

Upgrades, backups, restore, rollback, diagnostics, and where your data lives.

View raw Markdown

Everything after the first successful login. All commands assume Docker Compose unless a bare-metal variant is spelled out; run them in the directory that holds docker-compose.yml and .env.

Upgrade

An upgrade is three lines in .env plus new containers. Migrations run themselves.

Before you start:

  1. Take a backup. Mandatory before every upgrade, see the next section.
  2. Read the CHANGELOG entries between your version and the target. Look for breaking changes, security entries that call for credential rotation, and migration notes.
  3. Check disk space. A rule of thumb is 2 GB free on the Docker host, or 500 MB on bare metal.
  4. Note the running version: curl -fsS http://localhost/health.

Then replace the three release lines in .env with the ones from the new release notes:

NETMAPPER_VERSION=<new version>
NETMAPPER_BACKEND_DIGEST=sha256:...
NETMAPPER_FRONTEND_DIGEST=sha256:...

With scripts/upgrade.sh

scripts/upgrade.sh from the repository does the whole sequence. Run it from the directory holding docker-compose.yml and .env, after editing the three lines above:

sh scripts/upgrade.sh

It takes a pre-upgrade backup with scripts/backup.sh and prints its path, pulls the images named in .env, runs alembic upgrade heads in a one-off backend container, recreates the backend and frontend containers, and then polls GET /ready for up to 60 seconds. If a step fails it prints the step number and the backup path, and tells you to run scripts/rollback.sh <backup>.

Running the HTTPS overlay? Point the script at both files first:

export COMPOSE_FILE=docker-compose.yml:docker-compose.https.yml

By hand

docker compose pull
docker compose up -d
docker compose logs -f backend --tail=50

The backend entrypoint runs the migrations and the container does not become Ready until they finish. You should see alembic upgrade heads report the new head revision, then “Application startup complete”. Verify with curl -fsS http://localhost/ready.

On bare metal the shape is the same: back up, git pull, stop the service, reinstall the backend dependencies, rebuild the frontend, run alembic upgrade heads, start the service.

Downtime

NETMapper does not support zero-downtime rolling upgrades. The migration step needs exclusive DDL access and the backend has to be stopped while it runs. A typical upgrade takes two to five minutes, so schedule it in a quiet window.

Backup

The database is the only durable state, and pg_dump is the whole backup.

docker compose exec db pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB" \
  > "backup-$(date +%F).sql"

scripts/backup.sh does the same thing on a schedule. It reads DATABASE_URL, writes into NETMAPPER_BACKUP_DIR (default /var/lib/netmapper/backups) and keeps the most recent NETMAPPER_BACKUP_RETENTION files (default 30). A daily cron entry:

0 2 * * * /opt/netmapper/scripts/backup.sh >> /var/log/netmapper/backup.log 2>&1

The dump does not contain NETMAPPER_ENCRYPTION_KEY. Store the key separately from the dump, and store both somewhere you can actually reach during an incident. A dump restored without its key gives you an inventory whose switch passwords and notification secrets cannot be decrypted.

Restore

Before you start: have the backup file, have the encryption key that was active when it was taken, confirm the file is not truncated, and make sure there is roughly twice the backup size free in temp space.

scripts/restore.sh wraps the procedure:

sh scripts/restore.sh /path/to/backup-2026-05-28.sql

It checks the file, checks that NETMAPPER_ENCRYPTION_KEY is set (a warning, not a block), stops the backend, drops and recreates the database, pipes the dump in, runs alembic upgrade heads, restarts the backend and polls GET /ready. If a step fails the backend stays stopped until you fix the cause and re-run.

By hand, with Docker Compose:

docker compose stop backend
grep NETMAPPER_ENCRYPTION_KEY .env

docker compose exec db psql -U postgres \
  -c "DROP DATABASE IF EXISTS ${POSTGRES_DB};" \
  -c "CREATE DATABASE ${POSTGRES_DB} OWNER ${POSTGRES_USER};"

cat backup-2026-05-28.sql | \
  docker compose exec -T db psql -U "${POSTGRES_USER}" "${POSTGRES_DB}"

docker compose start backend      # migrations run on start
curl -fsS http://localhost/ready

After every restore, check that /ready reports the database as ok, that the switch count is what you expect, that a manual scan on one switch succeeds (an SSH authentication failure here means the encryption key does not match the dump), that the audit log still holds its history, and that the Alembic head matches the running code:

docker compose exec backend alembic -c alembic.ini current

Restoring onto a different host is the same procedure, with the key copied over and set in .env before the backend starts. A dump from an older version is fine, because the migrations run forward. A dump from a newer version than the running code is not: upgrade the code first.

Rollback

If an upgrade goes wrong, roll back to the pre-upgrade backup:

sh scripts/rollback.sh /path/to/pre-upgrade-backup

The script stops the stack, restores the database from that backup, brings the stack back up against the current compose file, and health-checks /ready with a 60 second retry window. Put the previous release’s three lines back into .env before you bring the stack up, so the images match the schema you just restored.

The Alembic downgrade path exists but is only needed when the migration that ran cannot be replayed forward on the restored schema. Read the migration’s downgrade() function before using it: not every migration implements a safe downgrade.

Diagnostics

When you file a bug, attach a diagnostics bundle:

sh scripts/collect-diagnostics.sh

It writes ./diagnostics-<timestamp>.tar.gz holding the last 24 hours of container logs, the container environment with secrets redacted, a schema-only database dump, the current Alembic revision, the last 200 audit log rows, the NETMapper version, OS and Docker versions, and disk and memory figures.

Redaction covers NETMAPPER_ENCRYPTION_KEY, NETMAPPER_JWT_SECRET, POSTGRES_PASSWORD, SMTP_PASSWORD, and any variable whose name contains KEY, SECRET, PASSWORD, TOKEN or API_KEY. Review the bundle before you send it anyway: audit rows carry usernames and resource ids, and application logs can carry hostnames and IP addresses from your network.

Not enough detail in the logs? Set LOG_LEVEL=DEBUG in .env, restart the backend, reproduce, collect again, then set it back to INFO.

GitHub accepts attachments up to 25 MB on an issue. If the bundle holds something you cannot post publicly, ask through the contact form and a private transfer will be arranged.

Where the data lives

What Where
PostgreSQL data The named Docker volume pg_data in the Compose project. This is the only durable state.
Configuration and secrets .env next to docker-compose.yml, or /etc/netmapper/.env on bare metal. Back it up: it holds the encryption key.
Application logs The backend’s stdout, captured by the Docker log driver, or the systemd journal on bare metal.
Frontend assets Built into the image, no persistent volume.
TLS certificates The certbot_conf volume when the HTTPS overlay is in use.

Audit log retention

Every privileged and mutating action writes a persistent AuditLog row, and admins can read the trail at GET /admin/audit-log or in the audit log UI.

Retention defaults to 90 days. The knob is one environment variable:

NETMAPPER_AUDIT_RETENTION_DAYS=365

A prune job runs daily at 03:30 local time and deletes rows older than the window. Admins can also prune on demand from the audit log UI. If you need the trail kept longer than you want to keep it in the database, export it before the prune window closes: CSV export ships in Core, and JSON Lines plus RFC 5424 syslog export are Pro features.