Category Archives: Command Line

Recovering LibreNMS from crashed XenServer VM

Posted on by
Reply

Had a LibreNMS instance crash, or the VM crashed, not bootable anymore.  Was able to boot it up on a CentOS iso with rescue mode, which gave me access to the files.  So the idea is to manually copy off the LibreNMS files and LibreNMS database and import them to a new LibreNMS instance

Issues

  • Need to access the the system files
  • MySQL doesn’t start in a chroot environment, so no way to do a mysqldump —  (Has to do with systemv or something)

Steps

  1. Create new LibreNMS VM with a new instance of LibreNMS installed
  2. Gain access to the crashed system
  3. Copy over LibreNMS MySQL databases to new LibreNMS instance
  4. Copy over LibreNMS files “/opt/librenms” to new LibreNMS instance
  5. Clean up.  Set users on directories, check SELinux etc.

1. Installing New LibreNMS VM

  • Guide for Ubuntu/Debian distro’s here
  • Guide for for Fedora/CentOS/RedHat here

2. Gaining access to crashed VM

There can be a couple of ways to gain access to a crashed XenServer VM.  One of the easiest ways is to boot up in recovery mode and go through the installers rescue mode.  Guide to boot up XenServer VM in recovery mode here.

The installer rescue mode should detect the OS and mount everything.  If not you should be able to mount the root partition manually.

Once booted up, you’ll need to enable network access if your going to use sftp or scp to copy files.  There are a few different ways to do this

  • Run “dhclient” to pull an address via DHCP
  • Set a static IP address
    • Guide for Ubuntu/Debian distro’s here
    • Guide for fedora/CentOS/RedHat here

3.Copy LibreNMS Mysql Database

Backup the LibreNMS MySQL database directory

tar czvf librenms_mysql.tgz /var/lib/mysql

Use scp or sftp to copy it to the new LibreNMS instance

scp librenms_mysql.tgz user@new_LibreNMS_ip

Now on the new LibreNMS instance we need to run the following few commands

systemctl stop mariadb
rm -rf /var/lib/mysql/*
tar xzvf librenms_mysql.tgz -C /
chown -R mysql:mysql /var/lib/mysql

4. Copy LibreNMS directory

Backup the LibreNMS directory

tar czvf librenms.tgz /opt/librenms

Use scp or sftp to copy it to the new LibreNMS instance

scp librenms.tgz user@new_LibreNMS_ip

Now on the new LibreNMS instance we need to run the following few commands

rm -rf /opt/librenms/*
tar xzvf librenms.tgz -C /
chown -R librenms:librenms /opt/librenms

5. Clean up

Disable SELinux if you have not already.  Guide here

Restart apache, and start mysql.  If your on Ubuntu, the services are named apache2 and mysql

systemctl restart httpd
systemctl restart mariadb

That should get it working, if not try a reboot.

Special notes

The whole MySQL directory needs to be copied, there are innodb files that will keep MySQL from starting if they are not copied.

There is some good info here
https://stackoverflow.com/questions/1795176/how-to-change-mysql-data-directory

How to chroot into a Linux drive

Posted on by
Reply

Chrooting can be super useful for changing things like the root password, repairing grub bootloader etc., things that require the system to be mounted.  Typically if your chrooting into an OS drive you can boot up on a Live Linux distro, or use the Rescue feature on some Linux installers.

Mount main drive

Change sdb2 to the root partition of your drive.

mount /dev/sdb2 /mnt

Mount other stuff

mount -t proc none /mnt/proc
mount -o bind /dev /mnt/dev
mount -o bind /sys /mnt/sys
mount -o bind /run /mnt/run

If you want to, you can mount the boot partition.  Change sdb1 if your boot partition is something else.  For efi, you may need to mount the boot partition then mount the efi partition inside /boot/efi.

mount /dev/sdb1 /mnt/boot

Chroot

 chroot /mnt

You should now be inside the chroot environment.  To exit the chroot, hit control+d or type exit.

If your having issues resolving DNS refer to this post.

Change timezone in CentOS, Fedora, RedHat

Posted on by
Reply

The file /etc/localtime is a symbolic link to the timezone.  All the timezones are listed in /usr/share/zoneinfo/

Replace America/New_York with the appropriate timezone.

ln -sf /usr/share/zoneinfo/America/New_York /etc/localtime

You can view the current time zone with the following command

timedatectl

or with date

date +"%Z %z"

Add, List, and Delete iptable rules

Posted on by
Reply

Add iptable rule

The following rule rejects access to port 22 on all devices except ones on the 192.168.1.0/24 network.  Note the “!”.  This command can be useful for a WHM/cPanel server to limit ssh access.

iptables -A INPUT ! -s 192.168.1.0/24 -p tcp --dport 22 -j REJECT

List iptable rules with line numbers

iptables -L --line-numbers

Example output

root@localhost [~]# iptables -L --line-numbers
Chain INPUT (policy ACCEPT)Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
1 REJECT tcp -- !192.168.1.11 anywhere tcp dpt:ssh reject-with icmp-port-unreachable

Chain FORWARD (policy ACCEPT)
num target prot opt source destination

Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
1 ACCEPT tcp -- anywhere anywhere multiport dports smtp,urd,submission owner GID match mailman
2 cpanel-dovecot-solr all -- anywhere anywhere

Chain cpanel-dovecot-solr (1 references)
num target prot opt source destination
1 ACCEPT tcp -- anywhere anywhere multiport sports 8984,7984 owner UID match cpanelsolr

Remove iptable rule

To delete a rule use the -D option with the Chain and the line number.  So to delete the first rule in the example output above, we would specify the INPUT chain and the the line number 1

 iptables -D INPUT 1

 

Auto mount CIFS mount point on system startup on Ubuntu

Posted on by
Reply

Install CIFS utils

sudo apt-get install -y cifs-utils

You can manually test it with the following command.  Change the ip address, mount points, username, and password.

mount.cifs /192.168.1.102/mount/point /mnt -o user=john,pass=password3,uid=john

Note that specifying the uid in the options, allows the user to add, delete, and modify the files and folders of that specific mount point.

To auto mount on system startup, add the following line to /etc/fstab.  Change the appropriate lines.

//192.168.1.102/mount/point   /mnt  auto   user=john,pass=password3,uid=john   0   0

You can test it by mounting everything in fstab

sudo mount -a

Install SSH Server on Linux (Debian, Ubuntu, Fedora, CentOS, RedHat)

Posted on by
Reply

Debian / Ubuntu

sudo apt-get install -y openssh-server

RPM based Distros, Fedora / CentOS / RedHat

sudo dnf install -y openssh-server

or use yum

sudo yum install -y openssh-server

Start ssh service

sudo systemctl start sshd

By default the SSH service should start when the system starts, but if not try the following command to enable the service on boot up.

Debian / Ubuntu

systemctl enable ssh

Fedora, CentOS, RedHat

systemctl enable sshd

Change SSH port

Not necessary, but it is a good idea to change the default ssh port.  To change the port edit the sshd file.

vi /etc/ssh/sshd_config

If you change the port, you’ll need to allow it in the firewall (firewalld, iptables) and if SELinux is enabled, semanage.

Repair UniFi Video database

Posted on by
Reply

Stop the unifi-video service

sudo service unifi-video stop

Delete the journal with

rm -rf /usr/lib/unifi-video/data/db/journal/*

Switch to the unifi-video user

sudo su unifi-video

Repair the database

mongod --dbpath /var/lib/unifi-video/db --repair

Exit the user

exit

Start the unifi-video service

service unifi-video start

Linux add mount point to fstab

Posted on by
Reply

You can use the Linux /etc/fstab to automatically mount hard drives on system boot up.  In the file you should see all your default system mount points, to add another hard drive or mount point, just create a new line at the bottom of the file and put in the following info

/dev/drive          /mount/point    filesystem   options    0      0

Example:

/dev/sdb1           /mnt/    ext4       rw,defaults         0      0

You can also use the UUID of the drive. You can find the UUID by running the following command

sudo blkid

In the fstab file just replace the /dev/drivename with the UUID

UUID=ba84c923-4413-090a-441d-6e12f32991b3         /mnt    ext4  rw   0      0

Upgrade Firmware on Ubiquiti Airmax Equipment from the Command Line/SSH

Posted on by
Reply

Upgrading the firmware via the command line is super easy.  Basic steps are

  1. Upload firmware file to radio using ftp, scp, or download directly to radio using wget
  2. Move the firmware the /tmp and rename to fwupdate.bin
  3. Upgrade the firmware by running
ubntbox fwupdate.real -m fwupdate.bin

More Detail explanation

Downloading Firmware to Radio

There are a couple of ways to get the firmware uploaded to the radio

  1. Download from Ubiquiti’s website and upload via ftp, scp, filezilla or like
  2. Download directly to the radio using wget

Using wget

ssh into the radio.  Change username and ip address as needed.

ssh ubnt@192.168.1.20

cd to the /tmp directory

cd /tmp

Find the firmware file on Ubiquiti’s website, accept the terms, copy the link and paste the link in the terminal after wget.  Replace the below link with the appropriate firmware link.

wget https://dl.ubnt.com/firmwares/XC-fw/v8.4.2/WA.v8.4.2.35930.171017.1722.bin

If you run into issues with wget, try using curl.

curl -k -o fwupdate.bin -L https://ui.com/downloads/firmwares/XC-fw/v8.7.13/WA.v8.7.13.47729.240606.1144.bin

Here is what the options mean:
-k mean insecure, which allows us to download via https.  There is a certificate error if not used.
-o fwupdate.bin, saves the output file as fwupdate.bin
-L means follow redirects

Installing Firmware

Rename firmware

mv *.bin fwupdate.bin

Start the upgrade

ubntbox fwupdate.real -m fwupdate.bin

The radio will now upgrade and reboot

Another Method.  Using the ubntmod.sh script

Another way to upgrade a radios firmware from the command line is to use the UBNTMOD tool.  More info on the UBNTMOD script is available here.

Make sure you have the firmware downloaded to your computer and run ubntmod.sh with the “-U” upgrade option.

Example,

bob@localhost:~$ ./ubntmod.sh -i 192.168.1.20 -U WA.v8.4.2.35930.171017.1722.bin