Not able to remove ZenPack – Zenoss 5

Ran into an issue where I was not able to list the installed ZenPacks and Zenoss was having issues starting. I think it was due to a ZenPack having issues and not being installed properly.

Drop into the container

serviced service shell -i -s FixZenpack zope bash

Change user and cd to ZenPack directory

su zenoss
cd /opt/zenoss/ZenPacks

Once there, remove the offending ZenPack. Replace ZENPACK.egg with the ZenPack

rm -rf ./ZENPACK.egg

Example:

rm -rf ./ZenPacks.zenoss.MikroTik-1.1.1.egg

Exit out of the container.

exit
exit

You can also hit ctrl+d twice.
Commit the Container

 serviced service shell -i -s FixZen zope bash

Restart the Zenoss.Core Service

service service restart Zenoss.Core

How To remove a ZenPack from Zenoss 5

To remove a ZenPack on Zenoss 5.1 and later, you are going to need to stop the zenoss services, restart a couple services that are needed to install the ZenPack, install the ZenPack, and then restart the Zenoss service.

Stop Zenoss.Core

serviced service stop Zenoss.Core

Make sure the service is stopped by running the following command.  When it reports back the Zenoss is Stopped, then continue.

 if ( serviced service status Zenoss.Core | awk '{print $3}' | grep -q Stopped) ; then echo "Zenoss is not running!" ; else echo "Zenoss is not Stopped!" ; fi

Or you can run this command, just make sure it says it is stopped.

serviced service status zenoss.core

Create a Snapshot

serviced service snapshot Zenoss.core

Start the following three services

serviced service start Infrastructure
zeneventserver
Zope

serviced service start Infrastructure
serviced service start zeneventserver
serviced service start Zope

List the ZenPacks that are currently installed.

serviced service run zope zenpack list

Remove the ZenPack.  Replace “ZenPack.comunity.Zenpack” with the zenpack name from the previous command.

serviced service run zope zenpack-manager uninstall ZenPack.community.Zenpack

How To use the Zenoss Toolbox

This is all contained in the Zenoss Upgrade guide.

Attach to Zope

serviced service attach zope/0
su zenoss

You can run the following commands to check on zenoss

zenossdbpack
findposkeyerror
zenrelationscan
zencatalogscan

zenossdbpack : Quickly scans the Zope Object Database (ZODB) to provide a preliminary indication of the health of the database, and to determine whether the database needs to be compressed with “zenossdbpack” before upgrading.
findposkeyerror : Checks objects and their relationships, and provides options for fixing errors.
zenrelationscan : Checks only ZenRelations between objects.
zencatalogscan : Checks ZODB object catalogs, which speed up web interface access

How To Install the Zenoss Tool Box

Skip to the bottom if you just want to see all of the commands.

Make a temporary directory

mkdir /tmp/toolbox
cd /tmp/toolbox

Download the tool box

wget https://github.com/zenoss/zenoss.toolbox/archive/master.zip
chmod -R 777 /tmp/toolbox

Drop to a shell and switch user

serviced service shell -i -s ToolBox zope bash
su zenoss

Install

easy_install /mnt/pwd/master.zip

Exit and commit container

exit
exit
serviced snapshot commit ToolBox

Restart Zope

serviced service restart zope

 

All the commands:

mkdir /tmp/toolbox && cd /tmp/toolbox
wget https://github.com/zenoss/zenoss.toolbox/archive/master.zip
chmod -R 777 /tmp/toolbox
serviced service shell -i -s ToolBox zope bash
su zenoss
easy_install /mnt/pwd/master.zip
exit
exit
serviced snapshot commit ToolBox
serviced service restart zope

How to Install ZenPack on Zenoss 5.1

To install a ZenPack on Zenoss 5.1 and later, you are going to need to stop the zenoss services, restart a couple services that are needed to install the ZenPack, install the ZenPack, and then restart the Zenoss service.

Stop Zenoss.Core :

serviced service stop 

Make sure the service is stopped by running the following command.  When it reports back the Zenoss is Stopped, then continue.

 if ( serviced service status Zenoss.Core | awk '{print $3}' | grep -q Stopped) ; then echo "Zenoss is not running!" ; else echo "Zenoss is not Stopped!" ; fi

Or you can run this command, just make sure it says it is stopped.

serviced service status zenoss.core

Create a Snapshot

serviced service snapshot Zenoss.core

Start the following three services

serviced service start Infrastructure
zeneventserver
Zope


serviced service start Infrastructure
serviced service start zeneventserver
serviced service start Zope

Change directory to tmp, make sure you put the zenpack in /tmp

cd /tmp

Install the Zenpack

serviced service run zope zenpack-manager install *.egg

Example:

serviced service run zope zenpack-manager install ZenPacks.zenoss.MikroTik-1.1.1.egg

Restart the Zenoss Service

serviced service restart zenoss.core

Log into Zenoss and check it.

Zenoss 5 RabbitMQ not starting

The following is some commands to try and troubleshoot RabbitMQ not starting in Zenoss 5.

Connecting to the RabbitMQ container

serviced service attach $(serviced service list | grep -i rabbitmq | awk '{print $2}')

Check the service

You can check the RabbitMQ service by running “rabbitmqctl status”

[root@764399e5hhba /]# rabbitmqctl status
Status of node rabbit@rbt0 ...
Error: unable to connect to node rabbit@rbt0: nodedown

DIAGNOSTICS
===========

attempted to contact: [rabbit@rbt0]

rabbit@rbt0:
  * unable to connect to epmd (port 4369) on rbt0: nxdomain (non-existing domain)


current node details:
- node name: rabbitmqct22222@764399e5hhba
- home dir: /var/lib/rabbitmq
- cookie hash: yy3+awwOpeaaaa12wdf42ff==

[root@764399e5hhba /]# 

As you can see the node is down so RabbitMQ is not able to start.

Try to ping rbt0 (RabbitMQ connects to rbt0 so if it can’t resolve, then it can’t start)

ping rbt0

If you get a “ping: unknown host rbt0” then add the following to /etc/hosts.  Change the IP address to the IP address of the container.  You can run “ip addr” or ifconfig to get the IP.

172.20.0.11  rbt0

Or if your interface is eth0, you can run this command.

echo "$(ifconfig eth0 | grep "inet " | awk '{print $2}')  rbt0"  >> /etc/hosts

Then run “rabbitmqctl status” again.  The service auto starts, so it may take a minute, but you should see something similar to the following. (Not that I cut some of the text out.)

[root@764399e5hhba /]# rabbitmqctl status
 Status of node rabbit@rbt0 ...
 [{pid,4629},
 {running_applications,
 ... CUT TEXT ...
 {uptime,5}]
 ...done.
 [root@764399e5hhba /]#

 

Checking vhost

List the RabbitMQ vhosts with “rabbitmqcl list_vhosts”

[root@764399e5hhba /]# rabbitmqctl list_vhosts
 Listing vhosts ...
 /
 /zenoss
 ...done.

If you run rabbitmqctl list_vhosts and don’t see /zenoss, then add it

rabbitmqctl add_vhost /zenoss
rabbitmqctl set_permissions -p /zenoss zenoss '.*' '.*' '.*'

If the vhost is up then you can try deleting the zenoss vhosts and readding it.

[root@764399e5hhba /]#
rabbitmqctl delete_vhost /zenoss
rabbitmqctl add_vhost /zenoss
rabbitmqctl set_permissions -p /zenoss zenoss '.*' '.*' '.*'

The only downside to theses changes is that once the service gets restarted the changes you made inside the container will be lost, There should be a way to update the container so that the changes are persistent.

Zenoss 5 backup Script

#!/bin/bash

# Zenoss 5 backup script
# 
#

# CHANGE THESE VARIABLES 
# Zenoss 5 Backup log file
log="/root/zenbackup.log"
# Backup directory
bdir="/backup/zenbackup/"

echo "Starting Zenoss backup Script"

echo "Starting backup to ${bdir}." $(date) >> ${log}
echo "Starting Backup to ${bdir}." $(date)

# Stop Zenoss Service
serviced service stop Zenoss.core

wait

# Wait for Zenoss to stop and then continue the backup
while [ -ne `serviced service status Zenoss.core | grep Stopped` ]
do
        echo "Waiting for Zenoss to Stop"
        sleep 5
done

# Backup
echo "Starting Zenoss Backup"
serviced backup ${bdir} 
echo "Backup Finished"

wait    

# Start Zenoss
echo "Starting Zenoss"
serviced service start Zenoss.core

echo "Finished backup." $(date) >> ${log}
echo "Finished backup." $(date)

exit

Restore Zenoss 5 from backup

What you need.

  1. Zenoss backup
  2. Access to Control Center
  3. Ability to edit /etc/default/serviced

First upload you Zenoss 5 backup file, to the “/opt/serviced/var/backps/” directory on the new Zenoss server.  Create the directory if it does not exist.

Once the backup file is on the New Zenoss server you will see it listed in the Control Panel’s Backup/Restore tab, you can try to restore it, but if it fails you will need to do the following.

Edit the serviced file “/etc/default/serviced” and uncomment the following line. i.e.(Delete the hash symbol at the beginning of the line).

# SERVICED_REGISTRY=1

Now go to the Control Center, and click restore to start restoring your backup.

It’ll take a little bit to pull the docker images, and get everything setup.  In the mean time you could setup your hosts file so you can access Zenoss once it is restored.  Refer to this guide under “Setting up name resolution”

Once the Control Center is finished restoring Zenoss, go to the Applications tab, and start the Zenoss Core.

Set up ssh keys in zenoss 5

Here are the basic steps of how to setup ssh keys for zenoss.

  1. Log into the zenoss server
  2. Open up docker container
  3. Change to zenoss user
  4. Generate ssh keys
  5. Upload ssh keys
  6. Exit out of container
  7. Commit container

Example:

[root@zenoss ~]# serviced service shell -s AddingSSHkey -i zope bash
I0709 3:02:47.791788 01773 server.go:341] Connected to the control center at port 192.168.1.10:4979
I0709 3:02:48.127949 01773 server.go:435] Acquiring image from the dfs...
I0709 3:02:48.131438 01773 server.go:437] Acquired!  Starting shell
Trying to connect to logstash server... 127.0.0.1:5042
Connected to logstash server.
[root@321feeg2253a /]# su zenoss
[zenoss@321feeg2253a /]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/zenoss/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/zenoss/.ssh/id_dsa.
Your public key has been saved in /home/zenoss/.ssh/id_dsa.pub.
The key fingerprint is:
12:ab:14:d5:54:09:d3:1f:f7:12:21:ae:hd:16:a5:1b zenoss@321feeg2253a
The key's randomart image is:
+--[ DSA 2048]----+
|     =====F      |
|    S+== + AA    |
|  A=+=++  +      |
| AB= .. + =      |
|  ++S S  +.      |
|   ..    -       |
|                 |
|                 |
|                 |
+-----------------+
[zenoss@321feeg2253a /]# ssh-copy-id admin@192.168.1.10
The authenticity of host '192.168.1.10 (192.168.1.10)' can't be established.
RSA key fingerprint is 12:ab:14:d5:54:09:d3:1f:f7:12:21:ae:hd:16:a5:1b.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
admin@192.168.1.10's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'admin@192.168.1.10'"
and check to make sure that only the key(s) you wanted were added.

[zenoss@321feeg2253a /]# exit
exit
[root@321feeg2253a /]# exit
exit
[root@zenoss ~]# serviced snapshot commit AddingSSHkey
0sdj2jj412waawjideow120x_isjriw19-121200
[root@zenoss ~]# exit