Recovering LibreNMS from crashed XenServer VM

Had a LibreNMS instance crash, or the VM crashed, not bootable anymore.  Was able to boot it up on a CentOS iso with rescue mode, which gave me access to the files.  So the idea is to manually copy off the LibreNMS files and LibreNMS database and import them to a new LibreNMS instance

Issues

  • Need to access the the system files
  • MySQL doesn’t start in a chroot environment, so no way to do a mysqldump —  (Has to do with systemv or something)

Steps

  1. Create new LibreNMS VM with a new instance of LibreNMS installed
  2. Gain access to the crashed system
  3. Copy over LibreNMS MySQL databases to new LibreNMS instance
  4. Copy over LibreNMS files “/opt/librenms” to new LibreNMS instance
  5. Clean up.  Set users on directories, check SELinux etc.

1. Installing New LibreNMS VM

  • Guide for Ubuntu/Debian distro’s here
  • Guide for for Fedora/CentOS/RedHat here

2. Gaining access to crashed VM

There can be a couple of ways to gain access to a crashed XenServer VM.  One of the easiest ways is to boot up in recovery mode and go through the installers rescue mode.  Guide to boot up XenServer VM in recovery mode here.

The installer rescue mode should detect the OS and mount everything.  If not you should be able to mount the root partition manually.

Once booted up, you’ll need to enable network access if your going to use sftp or scp to copy files.  There are a few different ways to do this

  • Run “dhclient” to pull an address via DHCP
  • Set a static IP address
    • Guide for Ubuntu/Debian distro’s here
    • Guide for fedora/CentOS/RedHat here

3.Copy LibreNMS Mysql Database

Backup the LibreNMS MySQL database directory

tar czvf librenms_mysql.tgz /var/lib/mysql

Use scp or sftp to copy it to the new LibreNMS instance

scp librenms_mysql.tgz user@new_LibreNMS_ip

Now on the new LibreNMS instance we need to run the following few commands

systemctl stop mariadb
rm -rf /var/lib/mysql/*
tar xzvf librenms_mysql.tgz -C /
chown -R mysql:mysql /var/lib/mysql

4. Copy LibreNMS directory

Backup the LibreNMS directory

tar czvf librenms.tgz /opt/librenms

Use scp or sftp to copy it to the new LibreNMS instance

scp librenms.tgz user@new_LibreNMS_ip

Now on the new LibreNMS instance we need to run the following few commands

rm -rf /opt/librenms/*
tar xzvf librenms.tgz -C /
chown -R librenms:librenms /opt/librenms

5. Clean up

Disable SELinux if you have not already.  Guide here

Restart apache, and start mysql.  If your on Ubuntu, the services are named apache2 and mysql

systemctl restart httpd
systemctl restart mariadb

That should get it working, if not try a reboot.

Special notes

The whole MySQL directory needs to be copied, there are innodb files that will keep MySQL from starting if they are not copied.

There is some good info here
https://stackoverflow.com/questions/1795176/how-to-change-mysql-data-directory

Auto mount CIFS mount point on system startup on Ubuntu

Install CIFS utils

sudo apt-get install -y cifs-utils

You can manually test it with the following command.  Change the ip address, mount points, username, and password.

mount.cifs /192.168.1.102/mount/point /mnt -o user=john,pass=password3,uid=john

Note that specifying the uid in the options, allows the user to add, delete, and modify the files and folders of that specific mount point.

To auto mount on system startup, add the following line to /etc/fstab.  Change the appropriate lines.

//192.168.1.102/mount/point   /mnt  auto   user=john,pass=password3,uid=john   0   0

You can test it by mounting everything in fstab

sudo mount -a

Install UniFi Video 3.8.5 on Ubuntu or Debian

You can run all the commands from the terminal, or ssh into the server

See here if you need to setup SSH on the server.

Install prerequisites

sudo apt-get install mongodb mongodb-server openjdk-8-jre-headless jsvc

Download UniFi Video installer

Note the Debian package works on Ubuntu, and has been tested on the latest Ubuntu

wget https://dl.ubnt.com/firmwares/ufv/v3.8.5/unifi-video.Debian7_amd64.v3.8.5.deb

Install package

sudo dpkg -i unifi-video.Debian7_amd64.v3.8.5.deb

Login to the UniFi Video controller using your web browser and going to the following address to finish configuring the NVR.

https://your-server-address:7443

Install SSH Server on Linux (Debian, Ubuntu, Fedora, CentOS, RedHat)

Debian / Ubuntu

sudo apt-get install -y openssh-server

RPM based Distros, Fedora / CentOS / RedHat

sudo dnf install -y openssh-server

or use yum

sudo yum install -y openssh-server

Start ssh service

sudo systemctl start sshd

By default the SSH service should start when the system starts, but if not try the following command to enable the service on boot up.

Debian / Ubuntu

systemctl enable ssh

Fedora, CentOS, RedHat

systemctl enable sshd

Change SSH port

Not necessary, but it is a good idea to change the default ssh port.  To change the port edit the sshd file.

vi /etc/ssh/sshd_config

If you change the port, you’ll need to allow it in the firewall (firewalld, iptables) and if SELinux is enabled, semanage.

Setup SNMP v3 on Debian or Ubuntu

All the following commands should work on Ubuntu, or just about any other Debian based Linux distro.  If you have a firewall on the server, you’ll need to allow UDP on port 161.

Install SNMP

Install snmp, snmpd, and libsnmp.

sudo apt-get -y install snmp snmpd libsnmp-dev

Stop the snmpd service so we can add a user

sudo service snmpd stop

Add SNMP v3 user

  • Change AuthPassword to your Authentication password
  • Change CryptoPassword to your Crypto Password
  • Change privUser to your private users username
sudo net-snmp-config --create-snmpv3-user -ro -A AuthPassword -X CryptoPassword -a MD5 -x AES privUser

Change System Location, System Contact, and allow SNMP on all interfaces

Open up the SNMP config file usually in /etc/snmp/snmpd.conf

vi /etc/snmp/snmpd.conf

Search for “sysLocation”  and change to whatever your system location is.

Search for “sysContact” and change it.  It should be right underneath sysLocation.

Now allow SNMP on all interfaces.  Find the following line and comment it out.

agentAddress udp:127.0.0.1:161

Add a # to the beginning.

#agentAddress udp:127.0.0.1:161

Now find this line (should be a couple lines down from the line you just commented out)

#agentAddress udp:161,udp6:[::1]:161

and uncomment it

agentAddress udp:161,udp6:[::1]:161

That will enable it so you can read the SNMP info using the servers IP address, as opposed to being limited to localhost.

Start the SNMP service and Test

Start the SNMP service

service snmpd start

Test with

snmpwalk -v3 -a MD5 -A AuthPassword -X CryptoPassword -l authNoPriv -u privUser localhost

 

Install LibreNMS on Ubuntu with HTTPS

The goal of this guide is to install LibreNMS on an Ubuntu Server with a self signed certificate.  Most of the steps are copied out of the LibreNMS Documentation found here.

Install required packages

sudo apt install apache2 composer fping git graphviz imagemagick libapache2-mod-php7.0 mariadb-client mariadb-server mtr-tiny nmap php7.0-cli php7.0-curl php7.0-gd php7.0-json php7.0-mcrypt php7.0-mysql php7.0-snmp php7.0-xml php7.0-zip python-memcache python-mysqldb rrdtool snmp snmpd whois

Create LibreNMS user

sudo useradd librenms -d /opt/librenms -M -r
sudo usermod -a -G librenms www-data

Install LibreNMS

cd /opt
sudo git clone https://github.com/librenms/librenms.git librenms

Configure MySQL

sudo systemctl restart mysql
sudo mysql -uroot -p

Run the following MySQL commands to create the LibreNMS user. Change password to your own password.

CREATE DATABASE librenms CHARACTER SET utf8 COLLATE utf8_unicode_ci;
CREATE USER 'librenms'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON librenms.* TO 'librenms'@'localhost';
FLUSH PRIVILEGES;
exit

Edit following file

sudo vi /etc/mysql/mariadb.conf.d/50-server.cnf

Add the following inside the [mysqld] section

innodb_file_per_table=1
sql-mode=""
lower_case_table_names=0

Restart MySQL

sudo systemctl restart mysql

Configure PHP

Edit the two files and set the time zone, date.timezone.  Example “America/New_York”

sudo vi /etc/php/7.0/apache2/php.ini
sudo vi /etc/php/7.0/cli/php.ini

Then run these commands

sudo a2enmod php7.0
sudo a2dismod mpm_event
sudo a2enmod mpm_prefork
sudo phpenmod mcrypt

Generate Self signed certificate

Enable ssl in apache

sudo a2enmod ssl

Generate Cert

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/localhost.key -out /etc/ssl/certs/localhost.crt

Configure Apache

Edit the following config file

sudo vi /etc/apache2/sites-available/librenms.conf

Add the following

<VirtualHost *:443>
 DocumentRoot /opt/librenms/html/
 ServerName librenms.example.com
 SSLEngine on
 SSLCertificateFile /etc/ssl/certs/localhost.crt
 SSLCertificateKeyFile /etc/ssl/private/localhost.key
 CustomLog /opt/librenms/logs/access_log combined
 ErrorLog /opt/librenms/logs/error_log
 AllowEncodedSlashes NoDecode
 <Directory "/opt/librenms/html/">
 Require all granted
 AllowOverride All
 Options FollowSymLinks MultiViews
 </Directory>
</VirtualHost>

Run the following commands

sudo a2ensite librenms.conf
sudo a2enmod rewrite
sudo systemctl restart apache2

Configure snmpd

sudo cp /opt/librenms/snmpd.conf.example /etc/snmp/snmpd.conf

Set the SNMP community string in the following file

sudo vi /etc/snmp/snmpd.conf

Then run these commands

sudo curl -o /usr/bin/distro https://raw.githubusercontent.com/librenms/librenms-agent/master/snmp/distro
sudo chmod +x /usr/bin/distro
sudo systemctl restart snmpd

Setup Crontab

sudo cp /opt/librenms/librenms.nonroot.cron /etc/cron.d/librenms

Copy logrotate config

sudo cp /opt/librenms/misc/librenms.logrotate /etc/logrotate.d/librenms

Set permissions

mkdir -p /opt/librenms/rrd /opt/librenms/logs
sudo chown -R librenms:librenms /opt/librenms
sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs
sudo setfacl -R -m g::rwx /opt/librenms/rrd /opt/librenms/logs

Web Installer

Restart apache

sudo systemctl restart apache2

Finish the install by going to

https://your-server/install.php

Change “your-server” to your server’s ip address, or hostname.  Since we created a self signed certificate, you’ll need to accept the https error.

Validate

Back on the command line run the php validation script

sudo /opt/librenms/validate.php

Finally log into your new LibreNMS instance by going to

https://your-server

Change “your-server” to your server’s IP address or hostname.

Ubuntu 16.04 Login Screen Issues

This post is not meant to be a how to, but rather a compilation commands that could be helpful in trying to fix issues with the login screen after an upgrade.

These commands should be executed from a prompt, you can get to one using the ctrl+alt+f1 (you can replace f1 with any of the function keys from 1-7), or you can boot up from recovery, or you could mount the drive and chroot into it.

Reinstall Desktop and Login Manager

sudo apt-get install --reinstall lightdm ubuntu-desktop

If your using using Gnome Shell

sudo apt-get install --reinstall gdm3 gnome-shell

Reconfigure lightdm, set it as the default login manager

sudo dpkg-reconfigure lightdm

Replace GDM with LightDM

sudo apt-get remove gdm3 
sudo apt-get install lightdm

Installing GDM (Gnome Login Manager)

sudo apt-get install gdm3

Remove Intel video and Plymouth

sudo apt-get remove xserver-xorg-video-intel plymouth

Reboot, get to a prompt and reinstall login manager, replace lightdm with whatever one your using

sudo apt-get install --reinstall lightdm

Reinstall Intel video and Plymouth

sudo apt-get install xserver-xorg-video-intel plymouth

Install Ubuntu Gnome Desktop

This seems to solve some of the issues with GDM3 not starting up.

sudo apt-get install ubuntu-gnome-desktop

 

Allow SSH access from a specific host using hosts.allow and hosts.deny on Linux

This is just a quick write on the hosts.allow and deny files.  You can lookup “spawn” and/or “twist” for some advanced usage.

 

So to limit an IP address, or a IP range access to SSH, do the following

Deny all incoming request for SSH

Edit the “hosts.deny” file

vi /etc/hosts.deny

add the following line

sshd : ALL

Now edit “hosts.allow” and allow the client IP, or IP range to access SSH

vi /etc/hosts.allow

add the following line to allow a single IP

sshd : 192.168.1.182

If you want to allow the whole subnet, then replace the above line with this one

sshd : 192.168.1.

hosts.allow overrides hosts.deny.  So you deny everything and then allow exceptions.

How to View Installed Programs in linux

Debian Based Distros

The below command should work for Debain, Ubuntu, and the Raspberry Pi Raspbian.

dpkg --get-selections

 

RPM Based Distros

The following works on Fedora, CentOS, ReHat.

rpm -qa

 

The above commands return all of the packages installed on a system.  If you want to look for a specific program or package you can use grep to filter the results.

rpm -qa | grep program

or

dpkg --get-selections | grep program

How to Install Flash on Linux

Installing flash on Ubuntu is pretty straight forward.  Type the following command into a terminal and press enter and away you go.

sudo apt-get install flashplugin-installer

On Fedora need to do the following

(64-bit)

sudo yum install http://linuxdownload.adobe.com/adobe-release/adobe-release-x86_64-1.0-1.noarch.rpm -y

(32-bit)

sudo yum install http://linuxdownload.adobe.com/adobe-release/adobe-release-i386-1.0-1.noarch.rpm -y

And install the Adobe Flash web browser plugin.  The first command imports the GPG key for the Adobe Flash plugin repository and the second command installs the plugin itself.

sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-adobe-linux 
sudo yum install flash-plugin -y