-w is watch path -k is a filter key we can use later to search through logs
Now we can search with ausearch
ausearch -k password
Using Preconfigured Rules
There are already some preconfigured rules in /usr/share/audit/sample-rules/
We can copy those to /etc/auditd/rules.d/ and use them.
cd /usr/share/audit/sample-rules/
cp 10-base-config.rules 30-stig.rules 31-privileged.rules 99-finalize.rules /etc/audit/rules.d/
augenrules --load
Note on the 31-privileged.rules file. You’ll need to run the commands in the file which will create a new file. Then we can copy that to “/etc/auditd/rules.d/”
The problem: Linux servers have been configured to send their local syslogs to LibreNMS, but are not showing up under the LibreNMS -> DEVICE -> Logs-> Syslog
After a bit of troubleshooting, found that the issue is the hostname being sent with the logs is different than what LibreNMS has for the device. It appears that some Linux distributions will or can use an abbreviated system hostname. There is a section in the LibreNMS docs about this
Create docker compose file with the following options
vi docker-compose.yml
Change the TZ to your time zone. If you have issues with the graphs, most likely something is off with the time zone between this container and Grafana/LibreNMS server
A note on SSL/TLS certificates. If you have an SSL certificate for LibreNMS, you can use it for grafana. If you run into issues, try copying the cert (fullchain.pem, privkey.pem) to /etc/grafana/
In Grafana, go to Configuration -> Data Sources -> Add data source
Set Name for Data Source
URL should be https://your_librenms_url/api/v0
Add Custom HTTP Header
Header field should be “X-Auth-Token”
Value field should contain the API token we created in LibreNMS
Save and Test If you receive any errors, refer to the Troubleshooting part at the end.
Adding LibreNMS API Data Source in Grafana
Add RRDReST Data Source
In Grafana, go to Configuration -> Data Sources -> Add data source
Set Name for Data Source
URL needs to be your docker container IP address (Steps above)
Save and Test (Should return “Unprocessable Entity”)
Adding RRDReST API Data Source in Grafana
Import Dashboard into Grafana
Now we need a dashboard to present our data.
Go to Create -> Import
Upload JSON file (Download from here or PasteBin )
Under RRDReST API , select our RRDReST Data Source
Under LibreNMS API , select our LibreNMS Data Source
Click Import
You should now be able to view your dashboard and use the drop down menus to select devices
Grafana viewing bandwidth on device being monitored by LibreNMS
Troubleshooting
There were a couple of issues I ran into while trying to get everything working together.
RRDReST shows 404 Not Found
Issue: When trying to run RRDReST with uvicorn, I was never able to access the rrd files, even the test rrd files that are included when installing RRDReST. I am guessing it is either a permisions issue, or something unable to access the files. Work around: Install RRDReST via Docker container.
Error Running uvicorn RRDReST
Error Adding LibreNMS API
Issue: Get a “JSON API: Bad Request” when trying to set up the LibreNMS API Data Source in Grafana.
Work around: Install a valid SSL Certificate and set up a DNS record so you can access LibreNMS at librenms.yourdomain.com.
More info: I would assume that “Skip TLS Verify” would work with or without a valid certificate, but it would not work for me. There are potentially some other options with modifying how Nginx or Apache is set up that would get this working. If you setup Grafana to use a SSL certificate, you may need to copy the certificate files (fullchain.pem, privkey.pem) to /etc/grafana/ and run “chown root:grafana *.pem” to let grafana have access to the files.
Recently I started getting the following error while running ./validate.php.
$ sudo -u librenms ./validate.php
====================================
Component | Version
--------- | -------
LibreNMS | 21.8.0-41-g0a76ca4
DB Schema | 2021_08_26_093522_config_value_to_medium_text (217)
PHP | 7.4.20
Python | 3.6.8
MySQL | 10.5.10-MariaDB
RRDTool | 1.4.8
SNMP | NET-SNMP 5.7.2
====================================
[OK] Composer Version: 2.1.6
[OK] Dependencies up-to-date.
[OK] Database connection successful
[FAIL] Database: incorrect column (notifications/datetime)
[FAIL] Database: incorrect column (users/created_at)
[FAIL] We have detected that your database schema may be wrong, please report the following to us on Discord (https://t.libren.ms/discord) or the community site (https://t.libren.ms/5gscd):
[FIX]:
Run the following SQL statements to fix.
SQL Statements:
SET TIME_ZONE='+00:00';
ALTER TABLE `notifications` CHANGE `datetime` `datetime` timestamp NOT NULL DEFAULT '1970-01-02 00:00:00' ;
ALTER TABLE `users` CHANGE `created_at` `created_at` timestamp NOT NULL DEFAULT '1970-01-02 00:00:01' ;
Reading online it sounds like some of the recent changes are causing the issue. Looks fairly easy to resolve though.
First we’ll need to get a MySQL prompt. We’ll do that by running
mysql -u librenms -p librenms
It’ll ask use for the librenms user’s mysql password.
Once we have the MySQL prompt we can just copy and paste the commands in.
SET TIME_ZONE='+00:00';
ALTER TABLE `notifications` CHANGE `datetime` `datetime` timestamp NOT NULL DEFAULT '1970-01-02 00:00:00' ;
ALTER TABLE `users` CHANGE `created_at` `created_at` timestamp NOT NULL DEFAULT '1970-01-02 00:00:01' ;
Type ‘quit’ to exit MySQL and lets run the validate script again.
You may get the following alert in LibreNMS. Basically you need to install python 3 to keep things up to date.
Python 3 is required to run LibreNMS as of May, 2020. You need to install Python 3 to continue to receive updates. If you do not install Python 3 and required packages, LibreNMS will continue to function but stop receiving bug fixes and updates.
Install Python 3
Install Python 3 with yum, or apt if you are on a Debian based distro.
All the following commands are run from the “/opt/librenms” directory
$ sudo ./daily.sh
Re-running /opt/librenms/daily.sh as librenms user
Updating to latest codebase FAIL
error: Your local changes to the following files would be overwritten by merge:
html/js/lang/de.js
html/js/lang/en.js
Please, commit your changes or stash them before you can merge.
...
$
Checking with validate.php shows that you can run githup-remove to fix it
$ sudo ./validate.php
[WARN] Your install is over 24 hours out of date, last update: Mon, 23 Feb 2020 05:43:12 +0000
[FIX]:
Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.
[WARN] Your local git contains modified files, this could prevent automatic updates.
[FIX]:
You can fix this with ./scripts/github-remove
Modified Files:
html/js/lang/de.js
...
$
Run “sudo ./scripts/github-remove -d”
$ sudo ./scripts/github-remove -d Are you sure you want to delete all modified and untracked files? [y/N] y $
You should also be able to remove the files individually if the above command does not work.
Run validate again to make sure it checks out good.
