LibreNMS Error “.sock: rrd_fetch_r failed:”

The following error showed up after adding a new device to LibreNMS.

.sock:rrd_fetch_r failed: 

It was not displaying any graph data, but the device was up and connected.

Looks like the error is SELinux related. You can fix the error by resetting the security context with the following command.

sudo restorecon -RFv /opt/librenms

If that does not work, try running all the following

Running the following commands will fix the issue most of the time:

sudo chown -R librenms:librenms '/opt/librenms'

sudo setfacl -d -m g::rwx /opt/librenms/bootstrap/cache /opt/librenms/storage /opt/librenms/logs /opt/librenms/rrd

sudo chmod -R ug=rwX /opt/librenms/bootstrap/cache /opt/librenms/storage /opt/librenms/logs /opt/librenms/rrd

sudo semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/bootstrap/cache(/.*)?'

sudo semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/storage(/.*)?'

sudo semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/logs(/.*)?'

sudo semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/rrd(/.*)?'

sudo restorecon -RFv /opt/librenms

Setup LibreNMS as Syslog Server

Using the LibreNMS documentation for setting up syslog-ng so LibreNMS can ingest logs from Cisco, Mikrotik, Ubiquiti etc. equipment.

https://docs.librenms.org/Extensions/Syslog/

Enable Syslog in LibreNMS settings

First thing we need to do is enable syslog for LibreNMS. Edit the /opt/librenms/config.php and add or enable

$config['enable_syslog'] = 1;

Install and Configure syslog-ng

Install syslog-ng with dnf or yum.

sudo dnf install -y syslog-ng

Create a config file for LibreNMS

vi /etc/syslog-ng/conf.d/librenms.conf

Put the following in the config file

source s_net {
        tcp(port(514) flags(syslog-protocol));
        udp(port(514) flags(syslog-protocol));
};

destination d_librenms {
        program("/opt/librenms/syslog.php" template ("$HOST||$FACILITY||$PRIORITY||$LEVEL||$TAG||$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC||$MSG||$PROGRAM\n") template-escape(yes));
};

log {
        source(s_net);
        source(s_sys);
        destination(d_librenms);
};

Restart and enable syslog-ng

sudo systemctl restart syslog-ng
sudo systemctl enable syslog-ng

SELinux

If we are running SELinux, we’ll need to make and apply a module to let the logs show up in the web interface.

vi librenms-rsyslog.te

Put the following in the file

module mycustom-librenms-rsyslog 1.0;

require {
        type syslogd_t;
        type httpd_sys_rw_content_t;
        type ping_exec_t;
        class process execmem;
        class dir { getattr search write };
        class file { append getattr execute open read };
}

#============= syslogd_t ==============
allow syslogd_t httpd_sys_rw_content_t:dir { getattr search write };
allow syslogd_t httpd_sys_rw_content_t:file { open read append getattr };
allow syslogd_t self:process execmem;
allow syslogd_t ping_exec_t:file execute;

Now run the following commands to make and apply our SELinux module.

checkmodule -M -m -o librenms-rsyslog.mod librenms-rsyslog.te
semodule_package -o librenms-rsyslog.pp -m librenms-rsyslog.mod
sudo semodule -i librenms-rsyslog.pp

LibreNMS – Could not ping 192.168.1.20 (192.168.1.20)

LibreNMS uses fping to check if devices are up or not. So if something is broken with fping, say a SELinux permission, you can receive the “Could not ping” error, while trying to add a new device.

LibreNMS unable to ping device

First we need to verify that fping is working. SSH into the LibreNMS server and try pinging an address.

fping 192.168.1.20

There was an issue with fping working if ipv6 was disabled. If fping is not working at all, check out this thread.

If you get an alive or unreachable message, then we know fping is working and can move on to the next stage of troubleshooting.

If you are using SELinux, then there is a good chance the problems has to do with that. You can try rerunning all the SELinux commands from the install guide. Note that it has a specific portion for fping.

https://docs.librenms.org/Installation/Install-LibreNMS/#selinux

If it is still not working, we can take a look at the issue with the audit2why command and feed in the audit log.

audit2why < /var/log/audit/audit.log

Here is some example output.

[root@librenms ~]#
type=AVC msg=audit(1676192040.183:404404): avc:  denied  { bind } for  pid=128555 comm="fping" lport=1 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=rawip_socket permissive=0

        Was caused by:
                Missing type enforcement (TE) allow rule.

[root@librenms ~]#

Another, perhaps more effective way to check the log is to follow it using the “tail -f” command.

tail -f /var/log/audit/audit.log | grep denied

And then in the web browser, try adding a new device. If SELinux is blocking it, it should throw a denied entry.

Example output

type=AVC msg=audit(1676192040.183:404404): avc:  denied  { bind } for  pid=128555 comm="fping" lport=1 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=rawip_socket permissive=0

Now we have verified that the issue is SELinux permissions related. We can create a module to allow it.

audit2allow -a -M fping_http < /var/log/audit/audit.log

And apply the module with

semodule -i fping_http.pp

You may need to do this a couple times. Check the audit log again to see if anything new shows up. Notice the slight difference in this error compared to the above error.

# tail -f /var/log/audit/audit.log | grep denied
type=AVC msg=audit(1676192613.121:404409): avc: denied { node_bind } for pid=153257 comm="fping" scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=rawip_socket permissive=0

We’ll create a new module for this and apply it

audit2allow -a -M node_http < /var/log/audit/audit.log
semodule -i node_http.pp

Not sure that is the best way to fix the problem. But it appears that SELinux is keeping Apache “httpd” from running fping which is why we need to create and load the modules.

LibreNMS Error – Renaming of 192.168.1.20 failed. Does your web server have permission to modify the rrd files?

Renaming of 192.168.1.20 failed . Does your web server have permission to modify the rrd files?

First thing to check is verify that the IP address is not already being monitored.

If you are getting the above error while trying to rename a device in LibreNMS, you may need to rerun some of the SELinux commands from the installation.

SSH into the server and run

restorecon -RFvv /opt/librenms

Now try renaming the device. Note that you can’t rename the device if the name/ip to a name that is being used by a different device.

If you continue to have issues, check the permissions from the installation guide (Official guide here)

You can also check for SELinux errors with

audit2why < /var/log/audit/audit.log

More SELinux info here

SELinux Audit Commands and Links

You can install audit2why by installing the policycoreutils package

sudo dnf install policycoreutils-python-utils

Show what and why something is failing

audit2why < /var/log/audit/audit.log

Search with ausearch

ausearch -m avc --start recent

Create and apply a module to fix the failure

This creates two files, a .pp and .te. The .pp is the compiled version of the .te

audit2allow -M mymodule < /var/log/audit/audit.log
semodule -i mymodule.pp

Note that “mymodule.pp” will replace any previous “mymodule.pp”. If your needing to create multiple modules/allow multiple exceptions, you can change the name of each module.

You can also add the rules together then manually compile it. Refer to the first link for more details.

Links with more info

https://danwalsh.livejournal.com/24750.html

http://selinuxgame.org/tutorials/ausearch/index.html

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security-enhanced_linux/sect-security-enhanced_linux-fixing_problems-allowing_access_audit2allow

semanage Allow and Delete ports in CentOS

The commands are for CentOS, but should work on Fedora and RedHat.

If semanage is not installed refer to here.

You would typically use this along with the systems firewall to allow a port through.  Guide for firewalld and iptables.  If you change it in the firewall and fail to add/edit it in semanage you can potentially get weird behavior like sshd not wanting to start after changing the port.

Add port

semanage port -a -t ssh_port_t -p tcp 2222

The above command allows the sshd service to start, using port 2222.

List allowed ports

semanage port -l

You can use grep to filter the results

Example:

[admin@localhost ~]# semanage port -l | grep ssh
ssh_port_t tcp 2222, 22
[admin@localhost ~]#

Delete port

semanage port -d -p tcp 2222

Other examples

Allow SNMP

semanage port -a -t snmp_port_t -p udp 161

 

LibreNMS not able to ping device and can’t change device name

If your having issues in the LibreNMS web interface changing a devices IP address or you can’t add a new one because it says it can’t ping the device, it is probably because SELinux is causing issues.

As far as changing a devices ip address, it looks like SELinux is not allowing Apache write access to the /opt/librenms/rrd directory

Temporary fix is to run

setenforce 0

You can permenatly disable SElinux by opening up “/etc/selinux/config” and changing “SELINUX=enforcing” to “SELINUX=disabled” and rebooting.

 

Renaming of x.x.x.x failed . Does your web server have permission to modify the rrd files?

It looks like there may be a bug where you’ll try to change the IP of a device and get the above error.  It looks like the issue has something to do with the ip address.  If your trying to change the IP on a router that has multiple IP addresses, try a different address in LibreNMS.

 

How To Enable/Disable SELinux

Disable SELinux

You can manually edit the SELinux config file in /etc/selinux/config and change the variable SELINUX=enforcing to disabled

vi /etc/selinux/config
...
SELINUX=disabled
...

or you can use this little command

sed -i.bak -e 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config

Enable SELinux

You can enable SELinux manually the same as above but set SELINUX=disabled to SELINUX=enforcing

vi /etc/selinux/config
...
SELINUX=enforcing
...

or

sed -i.bak -e 's/^SELINUX=.*/SELINUX=enforcing/g' /etc/selinux/config