# NETMapper operations guide (netmapper.nx1xlab.dev/docs/operations) Everything after the first successful login. All commands assume Docker Compose unless a bare-metal variant is spelled out; run them in the directory that holds `docker-compose.yml` and `.env`. ## Upgrade An upgrade is three lines in `.env` plus new containers. Migrations run themselves. Before you start: 1. **Take a backup.** Mandatory before every upgrade, see the next section. 2. **Read the CHANGELOG** entries between your version and the target. Look for breaking changes, security entries that call for credential rotation, and migration notes. 3. **Check disk space.** A rule of thumb is 2 GB free on the Docker host, or 500 MB on bare metal. 4. **Note the running version:** `curl -fsS http://localhost/health`. Then replace the three release lines in `.env` with the ones from the new release notes: ``` NETMAPPER_VERSION= NETMAPPER_BACKEND_DIGEST=sha256:... NETMAPPER_FRONTEND_DIGEST=sha256:... ``` ### With scripts/upgrade.sh `scripts/upgrade.sh` from the repository does the whole sequence. Run it from the directory holding `docker-compose.yml` and `.env`, after editing the three lines above: ```bash sh scripts/upgrade.sh ``` It takes a pre-upgrade backup with `scripts/backup.sh` and prints its path, pulls the images named in `.env`, runs `alembic upgrade heads` in a one-off backend container, recreates the backend and frontend containers, and then polls `GET /ready` for up to 60 seconds. If a step fails it prints the step number and the backup path, and tells you to run `scripts/rollback.sh `. Running the HTTPS overlay? Point the script at both files first: ```bash export COMPOSE_FILE=docker-compose.yml:docker-compose.https.yml ``` ### By hand ```bash docker compose pull docker compose up -d docker compose logs -f backend --tail=50 ``` The backend entrypoint runs the migrations and the container does not become Ready until they finish. You should see `alembic upgrade heads` report the new head revision, then "Application startup complete". Verify with `curl -fsS http://localhost/ready`. On bare metal the shape is the same: back up, `git pull`, stop the service, reinstall the backend dependencies, rebuild the frontend, run `alembic upgrade heads`, start the service. ### Downtime NETMapper does not support zero-downtime rolling upgrades. The migration step needs exclusive DDL access and the backend has to be stopped while it runs. A typical upgrade takes two to five minutes, so schedule it in a quiet window. ## Backup The database is the only durable state, and `pg_dump` is the whole backup. ```bash docker compose exec db pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB" \ > "backup-$(date +%F).sql" ``` `scripts/backup.sh` does the same thing on a schedule. It reads `DATABASE_URL`, writes into `NETMAPPER_BACKUP_DIR` (default `/var/lib/netmapper/backups`) and keeps the most recent `NETMAPPER_BACKUP_RETENTION` files (default 30). A daily cron entry: ``` 0 2 * * * /opt/netmapper/scripts/backup.sh >> /var/log/netmapper/backup.log 2>&1 ``` **The dump does not contain `NETMAPPER_ENCRYPTION_KEY`.** Store the key separately from the dump, and store both somewhere you can actually reach during an incident. A dump restored without its key gives you an inventory whose switch passwords and notification secrets cannot be decrypted. ## Restore Before you start: have the backup file, have the encryption key that was active when it was taken, confirm the file is not truncated, and make sure there is roughly twice the backup size free in temp space. `scripts/restore.sh` wraps the procedure: ```bash sh scripts/restore.sh /path/to/backup-2026-05-28.sql ``` It checks the file, checks that `NETMAPPER_ENCRYPTION_KEY` is set (a warning, not a block), stops the backend, drops and recreates the database, pipes the dump in, runs `alembic upgrade heads`, restarts the backend and polls `GET /ready`. If a step fails the backend stays stopped until you fix the cause and re-run. By hand, with Docker Compose: ```bash docker compose stop backend grep NETMAPPER_ENCRYPTION_KEY .env docker compose exec db psql -U postgres \ -c "DROP DATABASE IF EXISTS ${POSTGRES_DB};" \ -c "CREATE DATABASE ${POSTGRES_DB} OWNER ${POSTGRES_USER};" cat backup-2026-05-28.sql | \ docker compose exec -T db psql -U "${POSTGRES_USER}" "${POSTGRES_DB}" docker compose start backend # migrations run on start curl -fsS http://localhost/ready ``` After every restore, check that `/ready` reports the database as ok, that the switch count is what you expect, that a manual scan on one switch succeeds (an SSH authentication failure here means the encryption key does not match the dump), that the audit log still holds its history, and that the Alembic head matches the running code: ```bash docker compose exec backend alembic -c alembic.ini current ``` Restoring onto a different host is the same procedure, with the key copied over and set in `.env` before the backend starts. A dump from an older version is fine, because the migrations run forward. A dump from a newer version than the running code is not: upgrade the code first. ## Rollback If an upgrade goes wrong, roll back to the pre-upgrade backup: ```bash sh scripts/rollback.sh /path/to/pre-upgrade-backup ``` The script stops the stack, restores the database from that backup, brings the stack back up against the current compose file, and health-checks `/ready` with a 60 second retry window. Put the previous release's three lines back into `.env` before you bring the stack up, so the images match the schema you just restored. The Alembic `downgrade` path exists but is only needed when the migration that ran cannot be replayed forward on the restored schema. Read the migration's `downgrade()` function before using it: not every migration implements a safe downgrade. ## Diagnostics When you file a bug, attach a diagnostics bundle: ```bash sh scripts/collect-diagnostics.sh ``` It writes `./diagnostics-.tar.gz` holding the last 24 hours of container logs, the container environment with secrets redacted, a schema-only database dump, the current Alembic revision, the last 200 audit log rows, the NETMapper version, OS and Docker versions, and disk and memory figures. Redaction covers `NETMAPPER_ENCRYPTION_KEY`, `NETMAPPER_JWT_SECRET`, `POSTGRES_PASSWORD`, `SMTP_PASSWORD`, and any variable whose name contains `KEY`, `SECRET`, `PASSWORD`, `TOKEN` or `API_KEY`. Review the bundle before you send it anyway: audit rows carry usernames and resource ids, and application logs can carry hostnames and IP addresses from your network. Not enough detail in the logs? Set `LOG_LEVEL=DEBUG` in `.env`, restart the backend, reproduce, collect again, then set it back to `INFO`. GitHub accepts attachments up to 25 MB on an issue. If the bundle holds something you cannot post publicly, ask through the [contact form](https://nx1xlab.dev/contact) and a private transfer will be arranged. ## Where the data lives | What | Where | |---|---| | PostgreSQL data | The named Docker volume `pg_data` in the Compose project. This is the only durable state. | | Configuration and secrets | `.env` next to `docker-compose.yml`, or `/etc/netmapper/.env` on bare metal. Back it up: it holds the encryption key. | | Application logs | The backend's stdout, captured by the Docker log driver, or the systemd journal on bare metal. | | Frontend assets | Built into the image, no persistent volume. | | TLS certificates | The `certbot_conf` volume when the HTTPS overlay is in use. | ## Audit log retention Every privileged and mutating action writes a persistent `AuditLog` row, and admins can read the trail at `GET /admin/audit-log` or in the audit log UI. Retention defaults to 90 days. The knob is one environment variable: ``` NETMAPPER_AUDIT_RETENTION_DAYS=365 ``` A prune job runs daily at 03:30 local time and deletes rows older than the window. Admins can also prune on demand from the audit log UI. If you need the trail kept longer than you want to keep it in the database, export it before the prune window closes: CSV export ships in Core, and JSON Lines plus RFC 5424 syslog export are Pro features.