How to Fix UISP “Application is loading. 1 min remaining…”

After a recent update, UISP may stop working.

If we run sudo ~unms/app/unms-cli status, we receive the following error:

WARN[0000] /home/unms/app/docker-compose.yml: the attribute version is obsolete, it will be ignored, please remove it to avoid potential confusion
Error response from daemon: Container 174b07e5e39d311a326c69497f1b2e1ae0eedcb067c9dada6e546ab556aad371 is restarting, wait until the container is running

It appears that there is a bug upgrading to 2.4.155.
https://community.ui.com/questions/UISP-v2-4-143-and-v2-4-155-UISP-Fails-to-start-Geomagnetism-date-comparison-BUG/0cb32012-9afb-41d4-9f37-6a9ba94c4c36

The Fix

To fix the issue, manually update from the command line again.

sudo ~unms/app/unms-cli update

Collecting Logs

There are a couple ways to check logs. We can view docker logs for specific containers using

docker logs unms

We can collect all of the logs with

sudo tar -cvjSf /tmp/uisp-logs.tar.bz2 /home/unms/data/logs

If you receive an error about bzip2 not being found, either install it sudo apt install bzip2 or change tar to tar -cvz

This will put all the logs in /tmp. You can download them with scp, sftp, winscp or something.

https://help.ui.com/hc/en-us/articles/115015690107-UISP-How-to-Find-Logs-Report-Bugs

LibreNMS Error “.sock: rrd_fetch_r failed:”

The following error showed up after adding a new device to LibreNMS.

.sock:rrd_fetch_r failed: 

It was not displaying any graph data, but the device was up and connected.

Looks like the error is SELinux related. You can fix the error by resetting the security context with the following command.

sudo restorecon -RFv /opt/librenms

If that does not work, try running all the following

Running the following commands will fix the issue most of the time:

sudo chown -R librenms:librenms '/opt/librenms'

sudo setfacl -d -m g::rwx /opt/librenms/bootstrap/cache /opt/librenms/storage /opt/librenms/logs /opt/librenms/rrd

sudo chmod -R ug=rwX /opt/librenms/bootstrap/cache /opt/librenms/storage /opt/librenms/logs /opt/librenms/rrd

sudo semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/bootstrap/cache(/.*)?'

sudo semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/storage(/.*)?'

sudo semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/logs(/.*)?'

sudo semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/rrd(/.*)?'

sudo restorecon -RFv /opt/librenms

LibreNMS Devices Disappeared from Web Interface

What do you do when all your devices disappear from the web interface?

Everything still appears to be working. Alerts work.

Running a MySQL command to check if the devices are in the database returns all the devices

mysql -u librenms -p librenms -e 'use librenms ; select hostname,sysName,status from devices where status=1'

If we try going to /addhost we are greeted with an Error You have insuffecient permissions to view this page.

Running ./validate.php returns everything good

Potentially it could be an issue with SElinux or with Apache/NGinx

Running

audit2why < /var/log/audit/audit.log

Doesn’t return anything

No errors pop up in the Logs

Could be something happened with the LibreNMS user.

Test a different LibreNMS user and all the devices show up.

We’ve now isolated the issue to being something with out user.

Server Logs not Showing up in LibreNMS

The problem: Linux servers have been configured to send their local syslogs to LibreNMS, but are not showing up under the LibreNMS -> DEVICE -> Logs-> Syslog

After a bit of troubleshooting, found that the issue is the hostname being sent with the logs is different than what LibreNMS has for the device. It appears that some Linux distributions will or can use an abbreviated system hostname. There is a section in the LibreNMS docs about this

https://docs.librenms.org/Extensions/Syslog/#matching-syslogs-to-hosts-with-different-names

We can either do what the docs say, or we can set the host name in the rsyslog.conf file on each of the servers.

Log into the server and open up

sudo vi /etc/rsyslog.conf

At the very top, add the following line to set the hostname

$LocalHostName host.server_name_fqdn.com

Save the file and restart rsyslog

sudo systemctl restart rsyslog

Refresh the page to verify the logs are showing up in LibreNMS.

If you are still having issues, you may want to check the following

  1. SELinux on LibreNMS SELinux Audit Commands and Links, Setup LibreNMS as Syslog Server
  2. Firewall on LibreNMS FrDual Zones in Firewalld (Public/Private or External/Internal), Install LibreNMS on CentOS
  3. Read the documentation entirely through

Setup LibreNMS as Syslog Server

Using the LibreNMS documentation for setting up syslog-ng so LibreNMS can ingest logs from Cisco, Mikrotik, Ubiquiti etc. equipment.

https://docs.librenms.org/Extensions/Syslog/

Enable Syslog in LibreNMS settings

First thing we need to do is enable syslog for LibreNMS. Edit the /opt/librenms/config.php and add or enable

$config['enable_syslog'] = 1;

Install and Configure syslog-ng

Install syslog-ng with dnf or yum.

sudo dnf install -y syslog-ng

Create a config file for LibreNMS

vi /etc/syslog-ng/conf.d/librenms.conf

Put the following in the config file

source s_net {
        tcp(port(514) flags(syslog-protocol));
        udp(port(514) flags(syslog-protocol));
};

destination d_librenms {
        program("/opt/librenms/syslog.php" template ("$HOST||$FACILITY||$PRIORITY||$LEVEL||$TAG||$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC||$MSG||$PROGRAM\n") template-escape(yes));
};

log {
        source(s_net);
        source(s_sys);
        destination(d_librenms);
};

Restart and enable syslog-ng

sudo systemctl restart syslog-ng
sudo systemctl enable syslog-ng

SELinux

If we are running SELinux, we’ll need to make and apply a module to let the logs show up in the web interface.

vi librenms-rsyslog.te

Put the following in the file

module mycustom-librenms-rsyslog 1.0;

require {
        type syslogd_t;
        type httpd_sys_rw_content_t;
        type ping_exec_t;
        class process execmem;
        class dir { getattr search write };
        class file { append getattr execute open read };
}

#============= syslogd_t ==============
allow syslogd_t httpd_sys_rw_content_t:dir { getattr search write };
allow syslogd_t httpd_sys_rw_content_t:file { open read append getattr };
allow syslogd_t self:process execmem;
allow syslogd_t ping_exec_t:file execute;

Now run the following commands to make and apply our SELinux module.

checkmodule -M -m -o librenms-rsyslog.mod librenms-rsyslog.te
semodule_package -o librenms-rsyslog.pp -m librenms-rsyslog.mod
sudo semodule -i librenms-rsyslog.pp

Setting up HTTPS SSL/TLS Certificate for Grafana

Prerequisites

  1. Grafana Installed (Install guide)
  2. SSL/TLS Certificate

In this example, the server is already using Let’s Encrypt to create the certificate for a LibreNMS server. So all we are doing is copying the certificate to a Grafana directory, putting the correct permissions on it, and updating the Grafana config file to use the certificate.

Steps

  1. Copy Certificate to Grafana Directory
  2. Configure Grafana Config File
  3. Automate Certificate Copy to Grafana Directory

Copy Certificate files

In the following commands, change librenms.incredigeek.com to the directory that Let’s Encrypt is using for your fully qualified domain name (FQDN). Usually it is just your FQDN, but could also have -0001 or something appended to the end.

cp -f /etc/letsencrypt/live/librenms.incredigeek.com/privkey.pem 
/etc/grafana/ 
cp -f /etc/letsencrypt/live/librenms.incredigeek.com/fullchain.pem /etc/grafana/ 
chown root:grafana /etc/grafana/*.pem
chmod 640 /etc/grafana/*.pem Enable grafana on system bootup

In the above, we are copying the privkey.pem and fullchain.pem to /etc/grafana. We are then setting the correct owner/permissions on the files so that the Grafana service can read the certificate.

Configure Grafana Config File

This is super easy. Open up the Grafana config file in /etc/grafana.ini

vi /etc/grafana.ini

Find the following variables and configure them like so

protocol = https
cert_file = /etc/grafana/fullchain.pem
cert_key = /etc/grafana/privkey.pem

Restart Grafana

systemctl restart grafana-server.service

You should now have a working SSL certificate for the site.

Automate Certificate Copy

Let’s Encrypt certificates need to be updated frequently. This means that we should automate the above steps to avoid any down time. After all, a monitoring tool with down time defeats the purpose of monitoring.

We’ll need to create a root crontab

sudo crontab -e

Add the following changing out the FQDN to your FQDN.

0 0 1 * * cp -f /etc/letsencrypt/live/librenms.incredigeek.com/privkey.pem /etc/grafana/ && cp -f /etc/letsencrypt/live/librenms.incredigeek.com/fullchain.pem /etc/grafana/ && chown root:grafana /etc/grafana/*.pem && chmod 640 /etc/grafana/*.pem 

This is set to run once a month. Change if desired. Also change out librenms.incredigeek.com with your FQDN.

Note about domain name and IP addresses. Let’s Encrypt will not create a certificate for an IP address. You should be using a domain name instead (i.e. networkmonitoring.yourdomain.com) If the certificate is installed, and you access it via the IP address, you will receive a HTTPS error in your browser.

LibreNMS Error – Renaming of 192.168.1.20 failed. Does your web server have permission to modify the rrd files?

Renaming of 192.168.1.20 failed . Does your web server have permission to modify the rrd files?

First thing to check is verify that the IP address is not already being monitored.

If you are getting the above error while trying to rename a device in LibreNMS, you may need to rerun some of the SELinux commands from the installation.

SSH into the server and run

restorecon -RFvv /opt/librenms

Now try renaming the device. Note that you can’t rename the device if the name/ip to a name that is being used by a different device.

If you continue to have issues, check the permissions from the installation guide (Official guide here)

You can also check for SELinux errors with

audit2why < /var/log/audit/audit.log

More SELinux info here

Setting up RRDReST on CentOS 8 or AlmaLinux 9

There are some differences on setting up RRDReST on CentOS 8, Almalinux 9 vs CentOS 7

If you are setting this up to use with LibreNMS and Grafana, check out the rest of the this article. https://www.incredigeek.com/home/setting-up-grafana-on-librenms/

Installing RRDReST

All the docker commands have been swapped out for podman.

  1. Install Docker
  2. Create a compose file
  3. Run compose file to create container

Install docker

Podman is default on CentOS 8 and later and is, for the most part, a drop in replacement for Docker.

sudo yum install -y podman podman-compose
sudo systemctl enable podman

Create a Podman / Docker network to use. We’ll use this to assign a static IP address to the container. We’ll call the network rrdnet, and we’ll use the 10.89.2.0/24 range.

sudo podman network create --subnet=10.89.2.0/24 rrdnet

Create podman-compose file

Create a docker compose file

vi podman-compose.yml

Add the following

version: "3.5"
services:
  rrdrest:
    image: michaelwadman/rrdrest:latest
    container_name: rrdrest
    restart: always
    volumes:
      - "/opt/librenms/rrd:/opt/librenms/rrd:Z"
    environment:
      - TZ=America/Denver
    networks:
      rrdnet:
        ipv4_address: 10.89.2.2
        ipam:
          driver: default
          config:
            - subnet: 10.89.2.0/24
networks:
  rrdnet:
    external: true

Change the TZ to your time zone. If you have issues with the graphs, most likely something is off with the time zone between this container and Grafana/LibreNMS server

Note that the :Z is needed for SELinux to allow RRDReST to access the sub folders. AKA. the rrd files.

The container should have a 10.89.2.2 IP address. You can take all the networking sections out, and the container will receive DHCP. The problem is that the IP can change, breaking our graphs in Grafana.

Run RRDReST Container

Save the file. Then start and setup the container with

sudo podman-compose up -d

You will need your docker container IP address to setup the connection in Grafana. If you used the above docker-compose config, then it should be 10.89.2.2.

sudo docker exec -it rrdrest ip addr | grep eth0

Configure RRDRest to start on system boot with systemd

The “restart: always” option does not appear to work on systems with podman. We can create a systemd service instead.

Use the following command to automatically create a systemd file.

sudo podman generate systemd rrdrest

Copy the contents to a new file in /etc/systemd/system/

/etc/systemd/system/rrdrest.service

If you end up deleting the rrdrest container, you’ll need to update the systemd file again. You may need also need to run “systemctl daemon-reload”

Enable the new service with

systemctl enable rrdrest

Congratulations. RRDReST is now setup and running.

You can verify it’s running by checking with Podman / Docker.

sudo podman ps

You can also ping it

ping 10.89.2.2

Add Custom OID to device in LibreNMS

In the post, we’ll be adding a custom OID for a Ubiquiti Solar Charge Controller.

  • Find the device OID
  • Add the Custom OID in LibreNMS
  • Check the graph

Find the custom OID

Check out the following post if you are trying to add a Ubiquiti Solar Charge controller graph to LibreNMS. Otherwise you may need to do some googling around looking for the OID.

Add Custom OID in LibreNMS

Go to your device -> Settings(Little Gear) -> Custom OID -> +Add New OID

Adding Custom OID in LibreNMS for Ubiquiti UI Charge Controller

Couple notes about the information.
– There needs to be a 0 after the end of the OID.
– Data Type needs to be Gauge, Not Counter. A gauge can go up and down. A counter just counts up.

Hit “Test OID” and you should get a little notification saying it got a value for the OID. In this case 25572.

Test Custom OID in LibreNMS for Ubiquiti UI Charge Controller

Now Hit “Save OID”

Check the Graph

Now LibreNMS should start graphing our Custom OID. You may need to give it a minute to start showing data.

The graphs show up under Graphs -> Custom OID

Where are Custom OID Graphs?
LibreNMS Custom OID Graph