Tag Archives: grafana

Setting up HTTPS SSL/TLS Certificate for Grafana

Posted on by
Reply

Prerequisites

  1. Grafana Installed (Install guide)
  2. SSL/TLS Certificate

In this example, the server is already using Let’s Encrypt to create the certificate for a LibreNMS server. So all we are doing is copying the certificate to a Grafana directory, putting the correct permissions on it, and updating the Grafana config file to use the certificate.

Steps

  1. Copy Certificate to Grafana Directory
  2. Configure Grafana Config File
  3. Automate Certificate Copy to Grafana Directory

Copy Certificate files

In the following commands, change librenms.incredigeek.com to the directory that Let’s Encrypt is using for your fully qualified domain name (FQDN). Usually it is just your FQDN, but could also have -0001 or something appended to the end.

cp -f /etc/letsencrypt/live/librenms.incredigeek.com/privkey.pem 
/etc/grafana/ 
cp -f /etc/letsencrypt/live/librenms.incredigeek.com/fullchain.pem /etc/grafana/ 
chown root:grafana /etc/grafana/*.pem
chmod 640 /etc/grafana/*.pem Enable grafana on system bootup

In the above, we are copying the privkey.pem and fullchain.pem to /etc/grafana. We are then setting the correct owner/permissions on the files so that the Grafana service can read the certificate.

Configure Grafana Config File

This is super easy. Open up the Grafana config file in /etc/grafana.ini

vi /etc/grafana.ini

Find the following variables and configure them like so

protocol = https
cert_file = /etc/grafana/fullchain.pem
cert_key = /etc/grafana/privkey.pem

Restart Grafana

systemctl restart grafana-server.service

You should now have a working SSL certificate for the site.

Automate Certificate Copy

Let’s Encrypt certificates need to be updated frequently. This means that we should automate the above steps to avoid any down time. After all, a monitoring tool with down time defeats the purpose of monitoring.

We’ll need to create a root crontab 

sudo crontab -e

Add the following changing out the FQDN to your FQDN. 

0 0 1 * * cp -f /etc/letsencrypt/live/librenms.incredigeek.com/privkey.pem /etc/grafana/ && cp -f /etc/letsencrypt/live/librenms.incredigeek.com/fullchain.pem /etc/grafana/ && chown root:grafana /etc/grafana/*.pem && chmod 640 /etc/grafana/*.pem

This is set to run once a month. Change if desired. Also change out librenms.incredigeek.com with your FQDN.

Note about domain name and IP addresses. Let’s Encrypt will not create a certificate for an IP address. You should be using a domain name instead (i.e. networkmonitoring.yourdomain.com) If the certificate is installed, and you access it via the IP address, you will receive a HTTPS error in your browser.

Setting up Grafana on LibreNMS

Posted on by
Reply

Thanks to the guys who put together the information at the following links.

https://wadman.co.nz/2021/01/02/Viewing-LibreNMS-data-in-Grafana/
https://www.reddit.com/r/LibreNMS/comments/ojc8cc/how_to_almost_natively_integrate_librenms_and/

I ran into some issues trying to get this to work. So here are some of my notes. I already had a LibreNMS installation set up.

  1. Install RRDReST
    1. Install Docker
    2. Configure Docker compose file
  2. Configure LibreNMS API User and Key
  3. Set up and Configure Grafana
    1. Install Grafana
    2. Install JSON Data Source Plugin
    3. Add LibreNMS API Data Source
    4. Add RRDReST Data Source
    5. Import Dashboard into Grafana
  4. View graphs

Installing RRDReST

NOTE FOR CENTOS 8, ALMALINUX 8 and 9
The steps for installing RRDReST are slightly different. Check out the following post.
https://www.incredigeek.com/home/setting-up-rrdrest-on-centos-8-or-almalinux-9/

I had issues installing RRDReST. I am guessing it had to do with it accessing files. I was able to install it in a docker container.

  1. Install Docker
  2. Create a compose file
  3. Install container

Install docker

sudo yum install -y docker docker-compose
sudo systemctl enable docker

Create docker compose file with the following options

vi docker-compose.yml

Change the TZ to your time zone. If you have issues with the graphs, most likely something is off with the time zone between this container and Grafana/LibreNMS server

version: "3.5"

services:

  rrdrest:
    image: michaelwadman/rrdrest:latest
    container_name: rrdrest
    restart: always
    volumes:
      - "/opt/librenms:/opt/librenms"
    environment:
      - TZ=America/Denver

Save the file and start and setup the container with 

sudo docker-compose up -d

You will need your docker container IP address to setup the connection in Grafana

sudo docker exec -it rrdrest ip addr | grep eth0

Congratulations. You should now have a RRDReST docker container that will auto start on system boot and has the correct time zone.

Configure LibreNMS API User and Key

  1. Create a Grafana user in LibreNMS. (Settings Gear -> Manage Users -> Add Users)
    You could technically skip this step and use an existing user.
  2. Create API token for the newly created user (Setting Gear -> API -> API Settings)
Create a Grafana user in LibreNMS
Setup API Key for Grafana in LibreNMS

Set up and Configure Grafana

Basic steps are as follows

  1. Install Grafana
  2. Install JSON Data Source Plugin
  3. Configure Data Sources
    1. LibreNMS API
    2. RRDRest API
  4. Import Dashboard into Grafana

Install Grafana

There is not anything special with installing Grafana on the same server as LibreNMS. You can follow the official guide to install it

https://grafana.com/docs/grafana/latest/installation/

After Grafana is installed, install the JSON API data source. You can do this using the grafana-cli

grafana-cli plugins install marcusolsson-json-datasource

A note on SSL/TLS certificates. If you have an SSL certificate for LibreNMS, you can use it for grafana. If you run into issues, try copying the cert (fullchain.pem, privkey.pem) to /etc/grafana/

You’ll most likely need to change owner

sudo chown root:grafana /etc/grafana/*.pem

And maybe the file permissions.

sudo chmod 640 /etc/grafana/*.pem

Install JSON Data Source Plugin

This is fairly straight forward.

grafana-cli plugins install marcusolsson-json-datasource

Add LibreNMS API Data Source

In Grafana, go to Configuration -> Data Sources -> Add data source

  • Set Name for Data Source
  • URL should be https://your_librenms_url/api/v0
  • Add Custom HTTP Header
    • Header field should be “X-Auth-Token”
    • Value field should contain the API token we created in LibreNMS
  • Save and Test
    If you receive any errors, refer to the Troubleshooting part at the end.
Adding LibreNMS API Data Source in Grafana

Add RRDReST Data Source

In Grafana, go to Configuration -> Data Sources -> Add data source

  • Set Name for Data Source
  • URL needs to be your docker container IP address (Steps above)
  • Save and Test (Should return “Unprocessable Entity”)
Adding RRDReST API Data Source in Grafana

Import Dashboard into Grafana

Now we need a dashboard to present our data.

  • Go to Create -> Import
  • Upload JSON file (Download from here or PasteBin )
  • Under RRDReST API , select our RRDReST Data Source
  • Under LibreNMS API , select our LibreNMS Data Source
  • Click Import

You should now be able to view your dashboard and use the drop down menus to select devices

Grafana viewing bandwidth on device being monitored by LibreNMS

Troubleshooting

There were a couple of issues I ran into while trying to get everything working together.

RRDReST shows 404 Not Found

Issue: When trying to run RRDReST with uvicorn, I was never able to access the rrd files, even the test rrd files that are included when installing RRDReST. I am guessing it is either a permisions issue, or something unable to access the files.
Work around: Install RRDReST via Docker container.

Error Running uvicorn RRDReST

Error Adding LibreNMS API

Issue: Get a “JSON API: Bad Request” when trying to set up the LibreNMS API Data Source in Grafana.

Work around: Install a valid SSL Certificate and set up a DNS record so you can access LibreNMS at librenms.yourdomain.com.

More info: I would assume that “Skip TLS Verify” would work with or without a valid certificate, but it would not work for me. There are potentially some other options with modifying how Nginx or Apache is set up that would get this working. If you setup Grafana to use a SSL certificate, you may need to copy the certificate files (fullchain.pem, privkey.pem) to /etc/grafana/ and run “chown root:grafana *.pem” to let grafana have access to the files.

Grafana LibreNMS API JSON API: Bad Request