Tag Archives: linux

Server Logs not Showing up in LibreNMS

Posted on by
Reply

The problem: Linux servers have been configured to send their local syslogs to LibreNMS, but are not showing up under the LibreNMS -> DEVICE -> Logs-> Syslog

After a bit of troubleshooting, found that the issue is the hostname being sent with the logs is different than what LibreNMS has for the device. It appears that some Linux distributions will or can use an abbreviated system hostname. There is a section in the LibreNMS docs about this

https://docs.librenms.org/Extensions/Syslog/#matching-syslogs-to-hosts-with-different-names

We can either do what the docs say, or we can set the host name in the rsyslog.conf file on each of the servers.

Log into the server and open up

sudo vi /etc/rsyslog.conf

At the very top, add the following line to set the hostname

$LocalHostName host.server_name_fqdn.com

Save the file and restart rsyslog

sudo systemctl restart rsyslog

Refresh the page to verify the logs are showing up in LibreNMS.

If you are still having issues, you may want to check the following

  1. SELinux on LibreNMS SELinux Audit Commands and Links, Setup LibreNMS as Syslog Server
  2. Firewall on LibreNMS FrDual Zones in Firewalld (Public/Private or External/Internal), Install LibreNMS on CentOS
  3. Read the documentation entirely through

Setup LibreNMS as Syslog Server

Posted on by
Reply

Using the LibreNMS documentation for setting up syslog-ng so LibreNMS can ingest logs from Cisco, Mikrotik, Ubiquiti etc. equipment.

https://docs.librenms.org/Extensions/Syslog/

Enable Syslog in LibreNMS settings

First thing we need to do is enable syslog for LibreNMS. Edit the /opt/librenms/config.php and add or enable

$config['enable_syslog'] = 1;

Install and Configure syslog-ng

Install syslog-ng with dnf or yum.

sudo dnf install -y syslog-ng

Create a config file for LibreNMS

vi /etc/syslog-ng/conf.d/librenms.conf

Put the following in the config file

source s_net {
        tcp(port(514) flags(syslog-protocol));
        udp(port(514) flags(syslog-protocol));
};

destination d_librenms {
        program("/opt/librenms/syslog.php" template ("$HOST||$FACILITY||$PRIORITY||$LEVEL||$TAG||$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC||$MSG||$PROGRAM\n") template-escape(yes));
};

log {
        source(s_net);
        source(s_sys);
        destination(d_librenms);
};

Restart and enable syslog-ng

sudo systemctl restart syslog-ng
sudo systemctl enable syslog-ng

SELinux

If we are running SELinux, we’ll need to make and apply a module to let the logs show up in the web interface.

vi librenms-rsyslog.te

Put the following in the file

module mycustom-librenms-rsyslog 1.0;

require {
        type syslogd_t;
        type httpd_sys_rw_content_t;
        type ping_exec_t;
        class process execmem;
        class dir { getattr search write };
        class file { append getattr execute open read };
}

#============= syslogd_t ==============
allow syslogd_t httpd_sys_rw_content_t:dir { getattr search write };
allow syslogd_t httpd_sys_rw_content_t:file { open read append getattr };
allow syslogd_t self:process execmem;
allow syslogd_t ping_exec_t:file execute;

Now run the following commands to make and apply our SELinux module.

checkmodule -M -m -o librenms-rsyslog.mod librenms-rsyslog.te
semodule_package -o librenms-rsyslog.pp -m librenms-rsyslog.mod
sudo semodule -i librenms-rsyslog.pp

Mongo “illegal hardware instruction mongo” on Linux

Posted on by
Reply

While trying to install and run mongo on Kali Linux, I encountered the following error.

zsh: illegal hardware instruction mongo

Using a Bash shell it returns the following instead.

Illegal instruction

It appears that the issue is from running mongo in a virtual machine.

https://www.mongodb.com/community/forums/t/mongodb-community-5-0-12-illegal-instruction-core-dumped-ubuntu-20-04-5-lts/204332

https://askubuntu.com/questions/699077/how-to-enable-avx2-extensions-on-a-ubuntu-guest-in-virtualbox-5

Resolution? Run on bare metal or find a way to enable avx2 in the virtual machine.

Running Node App as systemd Service

Posted on by
Reply

In this post we will be using systemd to run a node application. This is helpful as it will automatically start the app when the server starts so we don’t have to manually. These steps can easily be modified to run a bash script, or any other application.

  • Create systemd file
  • Customize systemd file
  • Enable systemd file

We’ll be creating a service for the Simple Whisper Web Interface as an example. Chang things as needed.

Create systemd file

This is super simple. We create a .service file in /lib/systemd/system. When we enable the service, it will create a symlink to this file.

sudo vim /lib/systemd/system/whisperweb.service

Customize systemd file

Change the settings as appropriate. It would be a good idea to run any service as a limited user that only has the rights needed to get the job done. Do note that you will need to have any prerequisites installed and available for that user to use. I.e. libraries installed with npm etc. 

[Unit]
Description=Simple Whisper Web Interface Service File
After=network.target

[Service]
Type=simple
User=whisperuser
ExecStart=/usr/bin/node mainssl.js
WorkingDirectory=/home/whisperuser/
Restart=on-failure

[Install]
WantedBy=multi-user.target

Enable systemd file

Enabling the service will create a symlink that will then run this service file on system boot.

sudo systemctl enable whisperweb.service

And now we can start the service.

sudo systemctl start whisperweb.service

We can verify that the service is running by running

sudo systemctl status whisperweb.service

The following article has some great explanations on what different options in the unit file mean and do.

https://nodesource.com/blog/running-your-node-js-app-with-systemd-part-1/

Creating a Simple systemd Service to Launch Shell Script on System Boot

Posted on by
Reply

We will setup a simple systemd service to automatically run a bash script on system boot.

Create systemd file

Create the service file with

vi /etc/systemd/system/multi-user.target.wants/bashscript.service

Now fill out the file. Change the Description and ExecStart. After= means only start this unit after the other units have finished. This is needed if we need to make a network connection. If our script runs before the network is up, the connection will fail.

[Unit]
Description=systemd Unit File to Launch Bash Script on System Boot
After=network.target
After=syslog.target

[Install]
WantedBy=multi-user.target

[Service]
ExecStart=/home/user/script.sh

Change the ExecStart to your bash script and save the file

Enable systemd file

Now that the file is created, we need to enable the service so it starts on system boot 

systemctl enable bashscript.service

You should get the following output.

Created symlink /etc/systemd/system/multi-user.target.wants/bash.service → /etc/systemd/system/bash.service.

Now to test, reboot your system, or start the service with

systemctl start bashscript.service

How to Bypass NVIDIA NVENC Limits on RTX Cards on Linux

Posted on by
Reply

It appears that NVIDIA has limited the number of NVEncoding streams on consumer GPUs. Guess it is so people have to buy the more expensive professional cards.

Fortunately, the limit is only applied to the driver, and there is a patch available that let’s us bypass the limiter.

https://github.com/keylase/nvidia-patch

Install Patch

This assumes you already have the driver installed. If you do not, or run into issues with the commands below, refer to the above link.

Download the tool

https://github.com/keylase/nvidia-patch/archive/refs/heads/master.zip

wget https://github.com/keylase/nvidia-patch/archive/refs/heads/master.zip

Unzip the file

unzip nvidia-patch-master.zip

Run the patch script

cd nvidia-patch-master
sudo bash ./patch.sh

And we are finished!

Further reading

NVIDIA has a matrix of which cards support how many streams etc.

https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new

And while we are on the topic of artificial limits, check out the vGPU license bypass

https://github.com/KrutavShah/vGPU_LicenseBypass

Exim Troubleshooting Email Logs

Posted on by
Reply

The following is a great article explaining the main Exim log.

https://forums.cpanel.net/resources/reading-and-understanding-the-exim-main_log.383/

The following are some helpful tips from the post.

Search log by email address

You can search for specific addresses with the exigrep. Replace email@address with the email address of interest.

exigrep email@address /var/log/exim_mainlog

Message Direction

Looking at entries in the main log, some of the messages will have an indicator from the following table that tell us the status of the message and/or where it came from or went.

<=Indicates the arrival of a message to Exim for handling
=>Shows a normal message delivery
->Additional address for the same delivery, i.e. an Email forwarder.
>>cutthrough is a router precondition
This option requests delivery be attempted while the item is being received. It is usable in the RCPT ACL and valid only for single-recipient mails forwarded from one SMTP connection to another. If a recipient-verify callout connection is requested in the same ACL it is held open and used for the data, otherwise one is made after the ACL completes.
*>delivery suppressed by -N
**delivery failed; address bounced
==delivery deferred; temporary problem
<>For “<>” from the exim manual; Additionally, you will often find A bounce message is shown with the sender address “<>”, and if it is locally generated, this is followed by an item of the form
R=<message id>

Some other posts that may be helpful while troubleshooting mail deliveries.

View messages by ID

Bulk Delete Messages in Queue

Unable to launch Flatpaks on Fedora using Hardened Kernel

Posted on by
Reply

If you have installed the hardened Linux Kernel on Fedora, you may have encountered the following error when trying to launch Flatpak applications.

bwrap: No permissions to creating new namespace, likely because the kernel does not allow non-privileged user namespaces. On e.g. debian this can be enabled with 'sysctl kernel.unprivileged_userns_clone=1'.
error: Failed to sync with dbus proxy

https://security.stackexchange.com/questions/209529/what-does-enabling-kernel-unprivileged-userns-clone-do

https://github.com/containers/bubblewrap/issues/324

The issue looks to arise from the fact that the hardened Linux Kernel disables unprivileged name space and Fedora does not have setuid on by default on the bubblewrap executable.

Enabling setuid on bubblewrap

You can set the setuid permission on the bubblewrap executable with

sudo chmod u+s /usr/bin/bwrap

Allow Unprivileged Name Space (Alternative work around)

You could also allow unprivileged name space by running 

sysctl kernel.unprivileged_userns_clone=1

Note that setting the setuid seems the safer/recommended option.

It looks like using the setuid binary for bubblewrap would be better to use then enabling unprivileged user space.

https://madaidans-insecurities.github.io/guides/linux-hardening.html#sysctl-kernel

Remove setuid on bubblewrap

If you would like to remove the setuid permission for any reason, you can with the following command.

sudo chmod u-s /usr/bin/bwrap

Dual Zones in Firewalld (Public/Private or External/Internal)

Posted on by
Reply

In Firewalld we can use multiple zones for different types of traffic. For instance, we can setup an “internal” zone with our local IP addresses that are trusted, and then setup the public facing interface to the “drop” or “block” zone to block everything not from our internal network.

  1. Setup trusted IP addresses in the “internal” zone
  2. Configure services/ports that should be allowed on our “internal” zone
  3. Set “drop” zone as the default for all other traffic
  4. Reload firewall

1. Setup trusted IP addresses in “internal” zone

Add all of our trusted IP addresses to the internal zone. The following example adds all of the private IP addresses “RFC 1918” to the internal zone. Change as needed.

firewall-cmd --zone=internal --add-source=192.168.0.0/16 --add-source=172.16.0.0/12 --add-source=10.0.0.0/8 --permanent

2. Configure services/ports that should be allowed on our “internal” zone

Next we need to specify which services or ports should be accessible in our trusted zone.

Here is an example to allow https, ssh, and cockpit services

firewall-cmd --zone=internal --add-service=https --add-service=ssh --add-service=cockpit --permanent

Here is an example to allow port 8080 tcp

firewall-cmd --zone=internal --add-port=8080/tcp --permanent

3. Set “drop” zone as the default for all other traffic

The final configuration piece we need to do is set the default zone. Anything not specified in other zones will get processed by the default zone.

firewall-cmd --set-default-zone=drop

The drop zone drops everything.

4. Reload firewall

Reload the firewall with 

firewall-cmd --reload


Verifying changes

Let’s verify the changes with the firewall-cmd –get-active-zones command

# firewall-cmd --get-active-zones
drop
  interfaces: en0
internal
  sources: 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8

You can also use

firewall-cmd --list-all-zones

to list all the zones. Active zones show (active) next to them.

You can verify that your changes worked by doing an internal and external nmap scan.

If you have issues with services still being accessible from the outside, try disabling Network Manager for that specific interface

You can edit the ifcfg-eth0 file and add

NM_CONTROLLED=no