Backup UISP Application Backup Files with Rsync

UISP runs inside of a docker container. To copy out the backup files we need to use the “docker cp” command.

sudo docker cp unms:/home/app/unms/data/unms-backups ./uisp-backups

This will copy the backups into ./uisp-backups directory.

On an Ubuntu system, docker needs sudo permissions. If you copy the backups with the above command, the backup files will be assigned to the root user and you will not be able to use your normal user to manipulate the files.

You can either add your current user to the Docker group, or change the files owner

sudo chown username:username -R ./uisp-backups/

We can now copy all the automatic backups with rsync

sudo rsync -a ./uisp-backups -e "ssh -p 22" backupuser@backuphost:/backups

You can also automate this with Cron by doing something like

1 1 * * 1 docker cp unms:/home/app/unms/data/unms-backups ~/uisp-backups && rsync -a ~/uisp-backups -e "ssh -p 22" backupuser@backuphost:/backups

Every Monday at 1:01AM, copy the current UISP automatic backups, then use rsync to copy them to a remote server.

This expects that the current user has permissions to call Docker without sudo.

Give Ubuntu User Access to Run Docker?

By default on Debian based systems, Docker needs the sudo command to run. We can add a normal user to the Docker group so we don’t have to.

sudo usermod -aG docker username

Change out the username to your Ubuntu username.

The -a option means append the group to the username. It does not remove the user from current groups.
the -G option means add the specified group.

Setting up RRDReST on CentOS 8 or AlmaLinux 9

There are some differences on setting up RRDReST on CentOS 8, Almalinux 9 vs CentOS 7

If you are setting this up to use with LibreNMS and Grafana, check out the rest of the this article. https://www.incredigeek.com/home/setting-up-grafana-on-librenms/

Installing RRDReST

All the docker commands have been swapped out for podman.

  1. Install Docker
  2. Create a compose file
  3. Run compose file to create container

Install docker

Podman is default on CentOS 8 and later and is, for the most part, a drop in replacement for Docker.

sudo yum install -y podman podman-compose
sudo systemctl enable podman

Create a Podman / Docker network to use. We’ll use this to assign a static IP address to the container. We’ll call the network rrdnet, and we’ll use the 10.89.2.0/24 range.

sudo podman network create --subnet=10.89.2.0/24 rrdnet

Create podman-compose file

Create a docker compose file

vi podman-compose.yml

Add the following

version: "3.5"
services:
  rrdrest:
    image: michaelwadman/rrdrest:latest
    container_name: rrdrest
    restart: always
    volumes:
      - "/opt/librenms/rrd:/opt/librenms/rrd:Z"
    environment:
      - TZ=America/Denver
    networks:
      rrdnet:
        ipv4_address: 10.89.2.2
        ipam:
          driver: default
          config:
            - subnet: 10.89.2.0/24
networks:
  rrdnet:
    external: true

Change the TZ to your time zone. If you have issues with the graphs, most likely something is off with the time zone between this container and Grafana/LibreNMS server

Note that the :Z is needed for SELinux to allow RRDReST to access the sub folders. AKA. the rrd files.

The container should have a 10.89.2.2 IP address. You can take all the networking sections out, and the container will receive DHCP. The problem is that the IP can change, breaking our graphs in Grafana.

Run RRDReST Container

Save the file. Then start and setup the container with

sudo podman-compose up -d

You will need your docker container IP address to setup the connection in Grafana. If you used the above docker-compose config, then it should be 10.89.2.2.

sudo docker exec -it rrdrest ip addr | grep eth0

Configure RRDRest to start on system boot with systemd

The “restart: always” option does not appear to work on systems with podman. We can create a systemd service instead.

Use the following command to automatically create a systemd file.

sudo podman generate systemd rrdrest

Copy the contents to a new file in /etc/systemd/system/

/etc/systemd/system/rrdrest.service

If you end up deleting the rrdrest container, you’ll need to update the systemd file again. You may need also need to run “systemctl daemon-reload”

Enable the new service with

systemctl enable rrdrest

Congratulations. RRDReST is now setup and running.

You can verify it’s running by checking with Podman / Docker.

sudo podman ps

You can also ping it

ping 10.89.2.2

Restart UniFi services on UDM

Mark manages the Ubiquiti UniFi applications at Incredigeek Inc. and is unable to access the UniFi controller. It starts loading and then stops. The URL bar shows that it is trying to load a null network site.

Thankfully the WiFi is still working, Mark thinks to himself, but how am I supposed to manage the network? I am able to access the UniFi Core application, so maybe I can login using a secure shell and check on the application.

ssh root@192.168.1.1

Once logged in, and after using the google, he finds that unifi-os restart will restart the UniFi applications. But I just need to restart the Network application. Running “unifi-os” –help reveals the following options.

# unifi-os --help
Usage: /usr/sbin/unifi-os [stop start restart shell 'update url']

Oh shell!

unifi-os shell

Alternatively, we know that on the UDM’s the UniFi Applications are run inside a Docker container. We could run “docker ps” to show the containers and then “docker exec -it unifi-os bash”

Now we can restart just the UniFi Network application.

systemctl restart unifi

It can take a little bit to restart.

Setting up Grafana on LibreNMS

Thanks to the guys who put together the information at the following links.

https://wadman.co.nz/2021/01/02/Viewing-LibreNMS-data-in-Grafana/
https://www.reddit.com/r/LibreNMS/comments/ojc8cc/how_to_almost_natively_integrate_librenms_and/

I ran into some issues trying to get this to work. So here are some of my notes. I already had a LibreNMS installation set up.

  1. Install RRDReST
    1. Install Docker
    2. Configure Docker compose file
  2. Configure LibreNMS API User and Key
  3. Set up and Configure Grafana
    1. Install Grafana
    2. Install JSON Data Source Plugin
    3. Add LibreNMS API Data Source
    4. Add RRDReST Data Source
    5. Import Dashboard into Grafana
  4. View graphs

Installing RRDReST

NOTE FOR CENTOS 8, ALMALINUX 8 and 9
The steps for installing RRDReST are slightly different. Check out the following post.
https://www.incredigeek.com/home/setting-up-rrdrest-on-centos-8-or-almalinux-9/

I had issues installing RRDReST. I am guessing it had to do with it accessing files. I was able to install it in a docker container.

  1. Install Docker
  2. Create a compose file
  3. Install container

Install docker

sudo yum install -y docker docker-compose
sudo systemctl enable docker

Create docker compose file with the following options

vi docker-compose.yml

Change the TZ to your time zone. If you have issues with the graphs, most likely something is off with the time zone between this container and Grafana/LibreNMS server

version: "3.5"

services:

  rrdrest:
    image: michaelwadman/rrdrest:latest
    container_name: rrdrest
    restart: always
    volumes:
      - "/opt/librenms:/opt/librenms"
    environment:
      - TZ=America/Denver

Save the file and start and setup the container with

sudo docker-compose up -d

You will need your docker container IP address to setup the connection in Grafana

sudo docker exec -it rrdrest ip addr | grep eth0

Congratulations. You should now have a RRDReST docker container that will auto start on system boot and has the correct time zone.

Configure LibreNMS API User and Key

  1. Create a Grafana user in LibreNMS. (Settings Gear -> Manage Users -> Add Users)
    You could technically skip this step and use an existing user.
  2. Create API token for the newly created user (Setting Gear -> API -> API Settings)
Create a Grafana user in LibreNMS
Setup API Key for Grafana in LibreNMS

Set up and Configure Grafana

Basic steps are as follows

  1. Install Grafana
  2. Install JSON Data Source Plugin
  3. Configure Data Sources
    1. LibreNMS API
    2. RRDRest API
  4. Import Dashboard into Grafana

Install Grafana

There is not anything special with installing Grafana on the same server as LibreNMS. You can follow the official guide to install it

https://grafana.com/docs/grafana/latest/installation/

After Grafana is installed, install the JSON API data source. You can do this using the grafana-cli

grafana-cli plugins install marcusolsson-json-datasource

A note on SSL/TLS certificates. If you have an SSL certificate for LibreNMS, you can use it for grafana. If you run into issues, try copying the cert (fullchain.pem, privkey.pem) to /etc/grafana/

You’ll most likely need to change owner

sudo chown root:grafana /etc/grafana/*.pem

And maybe the file permissions.

sudo chmod 640 /etc/grafana/*.pem

Install JSON Data Source Plugin

This is fairly straight forward.

grafana-cli plugins install marcusolsson-json-datasource

Add LibreNMS API Data Source

In Grafana, go to Configuration -> Data Sources -> Add data source

  • Set Name for Data Source
  • URL should be https://your_librenms_url/api/v0
  • Add Custom HTTP Header
    • Header field should be “X-Auth-Token”
    • Value field should contain the API token we created in LibreNMS
  • Save and Test
    If you receive any errors, refer to the Troubleshooting part at the end.
Adding LibreNMS API Data Source in Grafana

Add RRDReST Data Source

In Grafana, go to Configuration -> Data Sources -> Add data source

  • Set Name for Data Source
  • URL needs to be your docker container IP address (Steps above)
  • Save and Test (Should return “Unprocessable Entity”)
Adding RRDReST API Data Source in Grafana

Import Dashboard into Grafana

Now we need a dashboard to present our data.

  • Go to Create -> Import
  • Upload JSON file (Download from here or PasteBin )
  • Under RRDReST API , select our RRDReST Data Source
  • Under LibreNMS API , select our LibreNMS Data Source
  • Click Import

You should now be able to view your dashboard and use the drop down menus to select devices

Grafana viewing bandwidth on device being monitored by LibreNMS

Troubleshooting

There were a couple of issues I ran into while trying to get everything working together.

RRDReST shows 404 Not Found

Issue: When trying to run RRDReST with uvicorn, I was never able to access the rrd files, even the test rrd files that are included when installing RRDReST. I am guessing it is either a permisions issue, or something unable to access the files.
Work around: Install RRDReST via Docker container.

Error Running uvicorn RRDReST

Error Adding LibreNMS API

Issue: Get a “JSON API: Bad Request” when trying to set up the LibreNMS API Data Source in Grafana.

Work around: Install a valid SSL Certificate and set up a DNS record so you can access LibreNMS at librenms.yourdomain.com.

More info: I would assume that “Skip TLS Verify” would work with or without a valid certificate, but it would not work for me. There are potentially some other options with modifying how Nginx or Apache is set up that would get this working. If you setup Grafana to use a SSL certificate, you may need to copy the certificate files (fullchain.pem, privkey.pem) to /etc/grafana/ and run “chown root:grafana *.pem” to let grafana have access to the files.

Grafana LibreNMS API JSON API: Bad Request

Backup Matrix Synapse PostgreSQL Database

This is part of a series of posts on backing up and restoring a backup for Matrix Synapse server. Synapse was installed using the matrix-docker-ansible deployment which while a little complicated can greatly ease management later on down the road. All the main components are in docker containers so we need to use docker to access.

https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/maintenance-postgres.md#backing-up-postgresql

As the root user run

docker exec --env-file=/matrix/postgres/env-postgres-psql matrix-postgres pg_dumpall -h matrix-postgres | gzip -c > /matrix/postgres.sql.gz

This will dump the Postgres database in /matrix/postgres.sql.gz
We can use this later to restore to a new server or keep as a backup.

Install UniFi 5.6.42 in Docker

Pull UniFi 5.6.42 docker image

sudo docker pull jacobalberty/unifi:5.6.42

Install image

sudo docker run -d --init --restart=unless-stopped --name=unifi-controller --net=host --volume=/docker/unifi:/var/lib/unifi -p 8080:8080/tcp -p 8081:8081/tcp -p 8443:8443/tcp -p 8843:8843/tcp -p 8880:8880/tcp -p 8883:8883/tcp -p 3478:3478/udp jacobalberty/unifi:5.6.42

Access and setup the unifi controller from a browser.

https://ip-address:8443

Auto renew SSL Cert with UniFi running in Docker

Setting up the SSL cert for UniFi service when running in docker is fairly easy to do. All you have to do is modify the UniFi SSL renew script to use the UniFi Docker directory and change the start and stop service to start and stop the Docker container. The script below should be ready to go.

Download, chmod +x it, and run, drop it in cron to auto renew.

In the below script, change (unifiDir=”/docker/unifi”) to your UniFi directory.

Note: this triggers calling the teams.sh script that will send an update to Microsoft Teams to let you know that the certs should be renewed. Check here for more info.

#!/usr/bin/env bash
# Added support to do UniFi and UniFi controllers at the same time using the same cert.
# Original script from https://git.sosdg.org/brielle/lets-encrypt-scripts/raw/branch/master/gen-unifi-cert.sh
# More info here https://www.reddit.com/r/Ubiquiti/comments/43v23u/using_letsencrypt_with_the_unifi_controller/ 
# And here https://www.reddit.com/r/Ubiquiti/comments/43v23u/using_letsencrypt_with_the_unifi_controller/
# Modified script from here: https://github.com/FarsetLabs/letsencrypt-helper-scripts/blob/master/letsencrypt-unifi.sh
# Modified by: Brielle Bruns <bruns@2mbit.com>
# Download URL: https://source.sosdg.org/brielle/lets-encrypt-scripts
# Version: 1.7
# Last Changed: 04/10/2020
# 04/10/2020: Changed directories and commands to work with a UniFi Docker install
# 02/02/2016: Fixed some errors with key export/import, removed lame docker requirements
# 02/27/2016: More verbose progress report
# 03/08/2016: Add renew option, reformat code, command line options
# 03/24/2016: More sanity checking, embedding cert
# 10/23/2017: Apparently don't need the ace.jar parts, so disable them
# 02/04/2018: LE disabled tls-sni-01, so switch to just tls-sni, as certbot 0.22 and later automatically fall back to http/80 for auth
# 05/29/2018: Integrate patch from Donald Webster <fryfrog[at]gmail.com> to cleanup and improve tests
# 09/26/2018: Change from TLS to HTTP authenticator

# Location of LetsEncrypt binary we use.  Leave unset if you want to let it find automatically
# LEBINARY="/usr/src/letsencrypt/certbot-auto"

# Change to your UniFi Docker directory
unifiDir="/docker/unifi"

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

function usage() {
  echo "Usage: $0 -d <domain> [-e <email>] [-r] [-i]"
  echo "  -d <domain>: The domain name to use."
  echo "  -e <email>: Email address to use for certificate."
  echo "  -r: Renew domain."
  echo "  -i: Insert only, use to force insertion of certificate."
}

while getopts "hird:e:" opt; do
  case $opt in
    i) onlyinsert="yes";;
    r) renew="yes";;
    d) domains+=("$OPTARG");;
    e) email="$OPTARG";;
    h) usage
       exit;;
  esac
done

DEFAULTLEBINARY="/usr/bin/certbot /usr/bin/letsencrypt /usr/sbin/certbot
  /usr/sbin/letsencrypt /usr/local/bin/certbot /usr/local/sbin/certbot
  /usr/local/bin/letsencrypt /usr/local/sbin/letsencrypt
  /usr/src/letsencrypt/certbot-auto /usr/src/letsencrypt/letsencrypt-auto
  /usr/src/certbot/certbot-auto /usr/src/certbot/letsencrypt-auto
  /usr/src/certbot-master/certbot-auto /usr/src/certbot-master/letsencrypt-auto"

if [[ ! -v LEBINARY ]]; then
  for i in ${DEFAULTLEBINARY}; do
    if [[ -x ${i} ]]; then
      LEBINARY=${i}
      echo "Found LetsEncrypt/Certbot binary at ${LEBINARY}"
      break
    fi
  done
fi

# Command line options depending on New or Renew.
NEWCERT="--renew-by-default certonly"
RENEWCERT="-n renew"

# Check for required binaries
if [[ ! -x ${LEBINARY} ]]; then
  echo "Error: LetsEncrypt binary not found in ${LEBINARY} !"
  echo "You'll need to do one of the following:"
  echo "1) Change LEBINARY variable in this script"
  echo "2) Install LE manually or via your package manager and do #1"
  echo "3) Use the included get-letsencrypt.sh script to install it"
  exit 1
fi

if [[ ! -x $( which keytool ) ]]; then
  echo "Error: Java keytool binary not found."
  exit 1
fi

if [[ ! -x $( which openssl ) ]]; then
  echo "Error: OpenSSL binary not found."
  exit 1
fi

if [[ ! -z ${email} ]]; then
  email="--email ${email}"
else
  email=""
fi

shift $((OPTIND -1))
for val in "${domains[@]}"; do
        DOMAINS="${DOMAINS} -d ${val} "
done

MAINDOMAIN=${domains[0]}

if [[ -z ${MAINDOMAIN} ]]; then
  echo "Error: At least one -d argument is required"
  usage
  exit 1
fi

if [[ ${renew} == "yes" ]]; then
  LEOPTIONS="${RENEWCERT}"
else
  LEOPTIONS="${email} ${DOMAINS} ${NEWCERT}"
fi

if [[ ${onlyinsert} != "yes" ]]; then
  echo "Firing up standalone authenticator on TCP port 80 and requesting cert..."
  ${LEBINARY} --server https://acme-v01.api.letsencrypt.org/directory \
              --agree-tos --standalone --preferred-challenges http ${LEOPTIONS}
fi

if [[ ${onlyinsert} != "yes" ]] && md5sum -c "/etc/letsencrypt/live/${MAINDOMAIN}/cert.pem.md5" &>/dev/null; then
  echo "Cert has not changed, not updating controller."
  exit 0
else
  echo "Cert has changed or -i option was used, updating controller..."
  TEMPFILE=$(mktemp)
  CATEMPFILE=$(mktemp)

  # Identrust cross-signed CA cert needed by the java keystore for import.
  # Can get original here: https://www.identrust.com/certificates/trustid/root-download-x3.html
  cat > "${CATEMPFILE}" <<'_EOF'
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----
_EOF

  md5sum "/etc/letsencrypt/live/${MAINDOMAIN}/cert.pem" > "/etc/letsencrypt/live/${MAINDOMAIN}/cert.pem.md5"
  echo "Using openssl to prepare certificate..."
  cat "/etc/letsencrypt/live/${MAINDOMAIN}/chain.pem" >> "${CATEMPFILE}"
  openssl pkcs12 -export  -passout pass:aircontrolenterprise \
          -in "/etc/letsencrypt/live/${MAINDOMAIN}/cert.pem" \
          -inkey "/etc/letsencrypt/live/${MAINDOMAIN}/privkey.pem" \
          -out "${TEMPFILE}" -name unifi \
          -CAfile "${CATEMPFILE}" -caname root

  docker container stop ${dockerContainerId}
  sleep 10
  dockerContainerId=$(sudo docker container list | grep unifi-controller | awk '{print $1}')
  echo "Removing existing certificate from Unifi protected keystore..."
  keytool -delete -alias unifi -keystore ${unifiDir}/keystore -deststorepass aircontrolenterprise

  echo "Inserting certificate into Unifi keystore..."
  keytool -trustcacerts -importkeystore \
          -deststorepass aircontrolenterprise \
          -destkeypass aircontrolenterprise \
          -destkeystore ${unifiDir}/keystore \
          -srckeystore "${TEMPFILE}" -srcstoretype PKCS12 \
          -srcstorepass aircontrolenterprise \
          -alias unifi

  sleep 2
  echo "Starting Unifi controllers..."
  docker container start ${dockerContainerId}
  ./teams.sh -b "$(hostname) - UniFi service is restarting, ssl cert should be renewed."

  echo "Done!"
fi

Basic Docker commands

In the following commands, 367c7a1465ec = Docker container ID

Start/stop Docker service

systemctl start docker
systemctl stop docker

Automatically start docker on system boot

systemctl enable docker

List docker containers

sudo docker container list
sudo docker container list
367c7a1465ec        jacobalberty/unifi:latest   "/usr/loca/bin/dockā€¦" 15 minutes ago      Up 14 minutes (healthy) unifi-controller

The bold part is your Docker container ID

List docker images on system

sudo docker images
sudo docker images
jacobalberty/unifi latest baebbe301633 9 days ago 711MB

Stop container

sudo docker stop 367c7a1465ec

Start container

sudo docker stop 367c7a1465ec

Remove/Delete a Docker Image

Need to stop the container first.

sudo docker rmi 367c7a1465ec

Get a Shell on a Docker Container

We can connect to a Docker container with the following, replace DockerContainerName with the Docker container name.

docker exec -it DockerContainerName sh