Using Auditd to monitor changes to Linux

Install and enable auditd with

sudo dnf install auditd
sudo systemctl enable auditd
sudo systemctl start auditd

Add a file or directory to monitor with

auditctl -w /etc/passwd -k password

-w is watch path
-k is a filter key we can use later to search through logs

Now we can search with ausearch

ausearch -k password

Using Preconfigured Rules

There are already some preconfigured rules in /usr/share/audit/sample-rules/

We can copy those to /etc/auditd/rules.d/ and use them.

cd /usr/share/audit/sample-rules/
cp 10-base-config.rules 30-stig.rules 31-privileged.rules 99-finalize.rules /etc/audit/rules.d/
augenrules --load

Note on the 31-privileged.rules file. You’ll need to run the commands in the file which will create a new file. Then we can copy that to “/etc/auditd/rules.d/”

find /bin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' > priv.rules
#find /sbin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#find /usr/bin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#find /usr/sbin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#filecap /bin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /sbin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /usr/bin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /usr/sbin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules

And Copy priv.rules to /etc/audit/rules.d/31-privileged.rules. Overwrite the file there if needed.

cp ./priv.rules /etc/audit/rules.d/31-privileged.rules

Load the rules.

augenrules --load

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/auditing-the-system_security-hardening

Exim Troubleshooting Email Logs

The following is a great article explaining the main Exim log.

https://forums.cpanel.net/resources/reading-and-understanding-the-exim-main_log.383/

The following are some helpful tips from the post.

Search log by email address

You can search for specific addresses with the exigrep. Replace email@address with the email address of interest.

exigrep email@address /var/log/exim_mainlog

Message Direction

Looking at entries in the main log, some of the messages will have an indicator from the following table that tell us the status of the message and/or where it came from or went.

<=Indicates the arrival of a message to Exim for handling
=>Shows a normal message delivery
->Additional address for the same delivery, i.e. an Email forwarder.
>>cutthrough is a router precondition
This option requests delivery be attempted while the item is being received. It is usable in the RCPT ACL and valid only for single-recipient mails forwarded from one SMTP connection to another. If a recipient-verify callout connection is requested in the same ACL it is held open and used for the data, otherwise one is made after the ACL completes.
*>delivery suppressed by -N
**delivery failed; address bounced
==delivery deferred; temporary problem
<>For “<>” from the exim manual; Additionally, you will often find A bounce message is shown with the sender address “<>”, and if it is locally generated, this is followed by an item of the form
R=<message id>

Some other posts that may be helpful while troubleshooting mail deliveries.

View messages by ID

Bulk Delete Messages in Queue

LibreNMS – Could not ping 192.168.1.20 (192.168.1.20)

LibreNMS uses fping to check if devices are up or not. So if something is broken with fping, say a SELinux permission, you can receive the “Could not ping” error, while trying to add a new device.

LibreNMS unable to ping device

First we need to verify that fping is working. SSH into the LibreNMS server and try pinging an address.

fping 192.168.1.20

There was an issue with fping working if ipv6 was disabled. If fping is not working at all, check out this thread.

If you get an alive or unreachable message, then we know fping is working and can move on to the next stage of troubleshooting.

If you are using SELinux, then there is a good chance the problems has to do with that. You can try rerunning all the SELinux commands from the install guide. Note that it has a specific portion for fping.

https://docs.librenms.org/Installation/Install-LibreNMS/#selinux

If it is still not working, we can take a look at the issue with the audit2why command and feed in the audit log.

audit2why < /var/log/audit/audit.log

Here is some example output.

[root@librenms ~]#
type=AVC msg=audit(1676192040.183:404404): avc:  denied  { bind } for  pid=128555 comm="fping" lport=1 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=rawip_socket permissive=0

        Was caused by:
                Missing type enforcement (TE) allow rule.

[root@librenms ~]#

Another, perhaps more effective way to check the log is to follow it using the “tail -f” command.

tail -f /var/log/audit/audit.log | grep denied

And then in the web browser, try adding a new device. If SELinux is blocking it, it should throw a denied entry.

Example output

type=AVC msg=audit(1676192040.183:404404): avc:  denied  { bind } for  pid=128555 comm="fping" lport=1 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=rawip_socket permissive=0

Now we have verified that the issue is SELinux permissions related. We can create a module to allow it.

audit2allow -a -M fping_http < /var/log/audit/audit.log

And apply the module with

semodule -i fping_http.pp

You may need to do this a couple times. Check the audit log again to see if anything new shows up. Notice the slight difference in this error compared to the above error.

# tail -f /var/log/audit/audit.log | grep denied
type=AVC msg=audit(1676192613.121:404409): avc: denied { node_bind } for pid=153257 comm="fping" scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=rawip_socket permissive=0

We’ll create a new module for this and apply it

audit2allow -a -M node_http < /var/log/audit/audit.log
semodule -i node_http.pp

Not sure that is the best way to fix the problem. But it appears that SELinux is keeping Apache “httpd” from running fping which is why we need to create and load the modules.

Enable Logging for firewalld

Enabling logging on firewall rules can be beneficial for tracking why a certain rule is not behaving as you intended.

Enabling logging is relatively straight forward.

  • Enable Firewall Logging
  • Check Logs
  • Disable Firewall Logging (Optional)

Enable Firewall Logging

Quickest way to enable logging is to run

sudo firewall-cmd --set-log-denied=all

This changes the options in the /etc/firewalld/firewalld.conf config file. Options include all, unicast, broadcast, multicast, and off

Enable Log option for firewalld

The command also reloads the firewall so manually restarting the firewall is necessary.

Checking Logs

You can use dmesg to view the failed attempts or you can follow the messages log and filter to just show the rejects

sudo tail -f /var/log/messages | grep -i REJECT

You can now try to access the server or run a test to trigger a log event. In my case I tried initiating a SSH connection.

Oct  1 16:32:10 localhost kernel: FINAL_REJECT: IN=eno1 OUT= MAC=f8:ab:98:12:fe:11:a1:ec:a6:00:67:3e:97:00 SRC=192.168.1.1 DST=192.168.88.2 LEN=60 TOS=0x08 PREC=0x40 TTL=59 ID=43080 DF PROTO=TCP SPT=38192 DPT=22 WINDOW=52240 RES=0x00 SYN URGP=0

Interesting bits are bolded. Our destination port it 22 “ssh” and our source address is 192.168.1.1. If I want this IP to access the server, I’ll need to add the 192.168.1.1 IP range in the allowed IP ranges.

Disable Logging (Optional)

After you have finished troubleshooting your problem, you may want to turn the logging feature off so you don’t fill up the logs with failed entries.

You can turn it off with

sudo firewall-cmd --set-log-denied=off

We can verify that logging is off by running

sudo firewall-cmd --get-log-denied 

If the firewall logging option is off it will return “off”

The following site has some more information and alternative ways

https://www.cyberciti.biz/faq/enable-firewalld-logging-for-denied-packets-on-linux/

Troubleshooting Backup Errors on WHM / cPanel

Below are some helpful locations of files, logs etc for troubleshooting backup errors on WHM

WHM backup logs
Change date to the correct date. Should be one log per day or I guess every time a backup runs.

/usr/local/cpanel/logs/cpbackup/{date}.log

View WHM backup config

cat /var/cpanel/backups/config

View WHM remote destination config(s)
Replace *** with the appropriate name.

cat /var/cpanel/backups/***.backup_destination

Rysnc.pm file

May need to modify this file to increase time out limits if you are having issues with time out errors for backups.

/usr/local/cpanel/Cpanel/Transport/Files/Rsync.pm

This link has some more info https://forums.cpanel.net/threads/what-commands-does-cpanel-run-over-ssh-to-do-rsync-backups.671777/

Check Access Logs for website

Typically on a cPanel host your access logs are kept in

/usr/local/apache/domlogs/username/incredigeek.com

Where username is your cPanel username and incredigeek.com is your website.

To view the logs you can use tail -f to follow the log.

tail -f /usr/local/apache/domlogs/username/incredigeek.com

You can also use grep to search the logs.

grep "text to search" /usr/local/apache/domlogs/username/incredigeek.com

cPanel Logs

How to view the logs

There are multiple ways to view log files, here are some common ways.

tail the log, shows the 10 most recent log entries.

tail /var/log/messages

tail the log and keep monitoring it for new entries.

tail -f /var/log/messages

Find specific info in log file

cat /var/log/messages | grep texttosearch

cPanel Log Paths

Main log

/var/log/messages

Access logs

/usr/local/cpanel/logs/access_log

Access logs for a specific domain

/home/user/access-logs/domainname.com

Account Transfers/miscellaneous logs

/var/cpanel/logs

Auditing Log (Account creation and deletions)

/var/cpanel/accounting.log

Backup Logs

/usr/local/cpanel/logs/cpbackup

CPHULKD Log

/usr/local/cpanel/logs/cphulkd.log

DNSAdmin, DNS Clustering

/usr/local/cpanel/logs/dnsadmin_log

Task Queue Processing Daemon

/usr/local/cpanel/logs/queueprocd.log

DBMapping

/usr/local/cpanel/logs/setupdbmap_log

Easy Apache Build logs

/usr/local/cpanel/logs/easy/apache/

Error logs

/usr/local/cpanel/logs/error_log
/var/log/cpanel

License log

/usr/local/cpanel/logs/license_log

local database modifications

/usr/local/cpanel/logs/build_locale_database_log

Login errors CPSRVD

/usr/local/cpanel/logs/login_log

Bandwidth History

/var/cpanel/bandwidth/{USERNAME}

Service Status Logs

/var/log/chkservd.log

Tailwatch log

/usr/local/cpanel/logs/tailwatch_log

Update Analysis Reporting

/usr/local/cpanel/logs/updated_analysis/{TIMESTAMP}.log

Update log UPCP

/var/cpanel/updatelogs/updated.{TIMESTAMP}.log

cPanel Email Logs

 

Horde log

/var/cpanel/horde/log/

RoundCube

/var/cpanel/roundcube/log/

Squirrel Mail

/var/cpanel/squirrelmail/

Panic log

/usr/local/cpanel/logs/panic_log

Delivery and receipt log

/var/log/exim_mainlog

Incoming mail queue

/var/spool/exim/input/

Log of messages rejected based on ACLS or other policies

/var/log/exim_rejectlog

Unexpected/Fatal error log

/var/log/exim_paniclog

IMAP, POP login attempts, transactions, fatal errors and spam scoring

/var/log/maillog

Mailman

/usr/local/cpanel/3rdparty/mailmain/logs

MySQL

MySQL error log

/var/lib/mysql/{SERVER_NAME}.err

MySQL slow query log (if enabled in my.cnf)

/var/log/slowqueries