Tag Archives: log

Using Auditd to monitor changes to Linux

Posted on by
Reply

Install and enable auditd with

sudo dnf install auditd
sudo systemctl enable auditd
sudo systemctl start auditd

Add a file or directory to monitor with

auditctl -w /etc/passwd -k password

-w is watch path
-k is a filter key we can use later to search through logs

Now we can search with ausearch

ausearch -k password

Using Preconfigured Rules

There are already some preconfigured rules in /usr/share/audit/sample-rules/

We can copy those to /etc/auditd/rules.d/ and use them. 

cd /usr/share/audit/sample-rules/
cp 10-base-config.rules 30-stig.rules 31-privileged.rules 99-finalize.rules /etc/audit/rules.d/
augenrules --load

Note on the 31-privileged.rules file. You’ll need to run the commands in the file which will create a new file. Then we can copy that to “/etc/auditd/rules.d/”

find /bin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' > priv.rules
#find /sbin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#find /usr/bin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#find /usr/sbin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#filecap /bin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /sbin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /usr/bin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /usr/sbin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules

And Copy priv.rules to /etc/audit/rules.d/31-privileged.rules. Overwrite the file there if needed.

cp ./priv.rules /etc/audit/rules.d/31-privileged.rules

Load the rules.

augenrules --load

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/auditing-the-system_security-hardening

Log rotation for rsyslog using fixed size

Posted on by
Reply

We’ll follow the documentation from here.

Changes are made to the /etc/rsyslog.conf config file.

For this example, we will be configuring our named.log file to not exceed 50MiB, and then we’ll have a rotated log “.1” that is also 50MiB. Total it should not exceed 100BMiB.

First we need to create an out channel, and then we assign the out channel to a logging channel. We also need a script that rotates the logs.

Create the Output Channel

$outchannel log_rotation,/var/log/named.log, 52428800,/home/user/log_rotation.sh

Assign Output Channel to Logging Channel

On our line that is logging named, at the end add :$log_rotation

Example:

local0.*                   /var/log/named.log:$log_rotation

Script to Rotate Log

Somewhere on the system, create a rotate.sh script. Name it whatever you want, just be sure the path and name in the rsyslog.conf is the same.

Add the following one line to move the current log to a rotate log. 

mv -f /var/log/named.log /var/log/named.log.1

As the log fills up and hits ~50MiB, the named.sh script will run which rotates(moves) the log file to logfile.log.1. This will keep our usage for named.log to 100MiB.

Server Logs not Showing up in LibreNMS

Posted on by
Reply

The problem: Linux servers have been configured to send their local syslogs to LibreNMS, but are not showing up under the LibreNMS -> DEVICE -> Logs-> Syslog

After a bit of troubleshooting, found that the issue is the hostname being sent with the logs is different than what LibreNMS has for the device. It appears that some Linux distributions will or can use an abbreviated system hostname. There is a section in the LibreNMS docs about this

https://docs.librenms.org/Extensions/Syslog/#matching-syslogs-to-hosts-with-different-names

We can either do what the docs say, or we can set the host name in the rsyslog.conf file on each of the servers.

Log into the server and open up

sudo vi /etc/rsyslog.conf

At the very top, add the following line to set the hostname

$LocalHostName host.server_name_fqdn.com

Save the file and restart rsyslog

sudo systemctl restart rsyslog

Refresh the page to verify the logs are showing up in LibreNMS.

If you are still having issues, you may want to check the following

  1. SELinux on LibreNMS SELinux Audit Commands and Links, Setup LibreNMS as Syslog Server
  2. Firewall on LibreNMS FrDual Zones in Firewalld (Public/Private or External/Internal), Install LibreNMS on CentOS
  3. Read the documentation entirely through

Enable Syslog for PowerDNS Recursor

Posted on by
Reply

  1. Enable Logging in PowerDNS Recursor Config
  2. Edit Systemd Unit File for PowerDNS to Allow Syslog
  3. Enable Logging in rsyslog Config File

The following links were helpful in setting things up.

https://doc.powerdns.com/recursor/running.html
https://www.reddit.com/r/linuxadmin/comments/9lc4jl/logging_queries_in_pdnsrecursor/

Enable logging in PowerDNS Recursor Config

First we need to find the line that says “disable-syslog” and uncomment/change it to 

disable-syslog=no

Next find the line that says “quiet” and uncomment/change it to 

quiet=no

Some other lines you may want to check and change

logging-facality=1
loglevel=6

Edit Systemd Unit File for PowerDNS to allow Syslog

Next we need to modify the Systemd unit file to allow PowerDNS Recursor to log to syslog.

systemctl edit --full pdns-recursor.service

On the ExecStart Line, remove the part that says

--disable-syslog

The resulting line should look something like

[Service]
ExecStart=/usr/sbin/pdns_recursor --socket-dir=%t/pdns-recursor --socket-dir=%t/pdns-recursor --daemon=no --write-pid=no --log-timestamp=no

Save the file.

Enable Logging in rsyslog Config File

Edit the rsyslog file 

sudo vim /etc/rsyslog.conf

Add the following line

local1.*        /var/log/pdns_recursor.log

This should now log all of the PowerDNS Recursor log info to “/var/log/pdns_recursor.log”

Restart the rsyslog and PowerDNS Recursor service

sudo systemctl restart rsyslog
sudo systemctl restart pdns-recursor

You should now see DNS request in the log file.

tail /var/log/pdns_recursor.log

They should also show up in the “/var/log/messages”

Enable Logging for firewalld

Posted on by
Reply

Enabling logging on firewall rules can be beneficial for tracking why a certain rule is not behaving as you intended.

Enabling logging is relatively straight forward.

  • Enable Firewall Logging
  • Check Logs
  • Disable Firewall Logging (Optional)

Enable Firewall Logging

Quickest way to enable logging is to run

sudo firewall-cmd --set-log-denied=all

This changes the options in the /etc/firewalld/firewalld.conf config file. Options include all, unicast, broadcast, multicast, and off

Enable Log option for firewalld

The command also reloads the firewall so manually restarting the firewall is necessary.

Checking Logs

You can use dmesg to view the failed attempts or you can follow the messages log and filter to just show the rejects

sudo tail -f /var/log/messages | grep -i REJECT

You can now try to access the server or run a test to trigger a log event. In my case I tried initiating a SSH connection. 

Oct  1 16:32:10 localhost kernel: FINAL_REJECT: IN=eno1 OUT= MAC=f8:ab:98:12:fe:11:a1:ec:a6:00:67:3e:97:00 SRC=192.168.1.1 DST=192.168.88.2 LEN=60 TOS=0x08 PREC=0x40 TTL=59 ID=43080 DF PROTO=TCP SPT=38192 DPT=22 WINDOW=52240 RES=0x00 SYN URGP=0

Interesting bits are bolded. Our destination port it 22 “ssh” and our source address is 192.168.1.1. If I want this IP to access the server, I’ll need to add the 192.168.1.1 IP range in the allowed IP ranges.

Disable Logging (Optional)

After you have finished troubleshooting your problem, you may want to turn the logging feature off so you don’t fill up the logs with failed entries.

You can turn it off with 

sudo firewall-cmd --set-log-denied=off

We can verify that logging is off by running

sudo firewall-cmd --get-log-denied

If the firewall logging option is off it will return “off”

The following site has some more information and alternative ways

https://www.cyberciti.biz/faq/enable-firewalld-logging-for-denied-packets-on-linux/

Check Access Logs for website

Posted on by
Reply

Typically on a cPanel host your access logs are kept in

/usr/local/apache/domlogs/username/incredigeek.com

Where username is your cPanel username and incredigeek.com is your website.

To view the logs you can use tail -f to follow the log.

tail -f /usr/local/apache/domlogs/username/incredigeek.com

You can also use grep to search the logs.

grep "text to search" /usr/local/apache/domlogs/username/incredigeek.com

cPanel Logs

Posted on by
Reply

How to view the logs

There are multiple ways to view log files, here are some common ways.

tail the log, shows the 10 most recent log entries.

tail /var/log/messages

tail the log and keep monitoring it for new entries.

tail -f /var/log/messages

Find specific info in log file

cat /var/log/messages | grep texttosearch

cPanel Log Paths

Main log

/var/log/messages

Access logs

/usr/local/cpanel/logs/access_log

Access logs for a specific domain

/home/user/access-logs/domainname.com

Account Transfers/miscellaneous logs

/var/cpanel/logs

Auditing Log (Account creation and deletions)

/var/cpanel/accounting.log

Backup Logs

/usr/local/cpanel/logs/cpbackup

CPHULKD Log

/usr/local/cpanel/logs/cphulkd.log

DNSAdmin, DNS Clustering

/usr/local/cpanel/logs/dnsadmin_log

Task Queue Processing Daemon

/usr/local/cpanel/logs/queueprocd.log

DBMapping

/usr/local/cpanel/logs/setupdbmap_log

Easy Apache Build logs

/usr/local/cpanel/logs/easy/apache/

Error logs

/usr/local/cpanel/logs/error_log
/var/log/cpanel

License log

/usr/local/cpanel/logs/license_log

local database modifications

/usr/local/cpanel/logs/build_locale_database_log

Login errors CPSRVD

/usr/local/cpanel/logs/login_log

Bandwidth History

/var/cpanel/bandwidth/{USERNAME}

Service Status Logs

/var/log/chkservd.log

Tailwatch log

/usr/local/cpanel/logs/tailwatch_log

Update Analysis Reporting

/usr/local/cpanel/logs/updated_analysis/{TIMESTAMP}.log

Update log UPCP

/var/cpanel/updatelogs/updated.{TIMESTAMP}.log

cPanel Email Logs

 

Horde log

/var/cpanel/horde/log/

RoundCube

/var/cpanel/roundcube/log/

Squirrel Mail

/var/cpanel/squirrelmail/

Panic log

/usr/local/cpanel/logs/panic_log

Delivery and receipt log

/var/log/exim_mainlog

Incoming mail queue

/var/spool/exim/input/

Log of messages rejected based on ACLS or other policies

/var/log/exim_rejectlog

Unexpected/Fatal error log

/var/log/exim_paniclog

IMAP, POP login attempts, transactions, fatal errors and spam scoring

/var/log/maillog

Mailman

/usr/local/cpanel/3rdparty/mailmain/logs

MySQL

MySQL error log

/var/lib/mysql/{SERVER_NAME}.err

MySQL slow query log (if enabled in my.cnf)

/var/log/slowqueries