Searching for devices in UniFi via command line / MongoDB

While the UniFi controller is nice and everything, it does make it hard to see if a device is already adopted. At least if you have a ton of sites. Fortunately, we can search the database directly to find out if a UniFi is already adopted and which site it is assigned to.

Connect to Mongo DB

First we need to connect to MongoDB. And then we need to use the ace database.

mongo -port 27117
use ace

List all the devices on the controller

This command will list all the devices on the controller. Regardless of which site they are assigned to.

db.device.find({}, { site_id:"", ip : "", name :"", mac:""})

Example output

{ "_id" : ObjectId("563a4d94e4b054e5376fc600"), "mac" : { "_id" : ObjectId("563a4d94e4b054e5376fc600"), "mac" : "44:d9:e7:34:d1:08", "ip" : "192.168.1.200", "name" : "Main_WiFi", "site_id" : "39485e9abf0e9a047bcded96" }
{ "_id" : ObjectId("9873b39ed1f5d30a6738abe"), "mac" : "44:d9:e7:01:a3:d4", "ip" : "192.168.1.201", "name" : "Testing_Wifi", "site_id" : "39485e9abf0e9a047bcded96" }

Each UniFi will have a “site_id”. You can use that ID to figure out which site it is assigned to.

List all the sites on the controller

db.site.find()

Example output

{ "_id" : ObjectId("39485e9abf0e9a047bcded96"), "name" : "default", "desc" : "Testing Site", "attr_hidden_id" : "default", "attr_no_delete" : true, "anonymous_id" : "83ae20ba-2948-458e-fd0a-1320583ecb04" }

Using our “site_id” from above, we see that the Testing_Wifi device is assigned to the “Testing Site” on the controller.

Something else to look at would be to use the UniFi controller API.

https://ubntwiki.com/products/software/unifi-controller/api

Using ADB to Pull APKs off Device

ADB Help for pull and shell

It is sometimes helpful to pull an APK from a working device so you can install it on a different device. These commands should work on an emulator, phone, tablet, or other Android device. You just need to be able to connect with ABD.

  • Connect to device with ADB
  • View installed apps
  • Find path for APK
  • Pull/Download APK

View Installed Apps

This will display a list of all the installed packages.

adb shell pm list packages 

Find path for specific App/APK

Replace com.android.apk with the app of interest.

adb shell pm path com.android.apk

Pull APK to local machine

Pull/Download the APK of interest to your local machine with the following command. Change the path “/data/app/…” to the path returned from the previous command.

adb shell pull /data/app/info/base.apk

You can view the following link for more information.

https://stackoverflow.com/questions/4032960/how-do-i-get-an-apk-file-from-an-android-device

Advanced Tricks

What if you need to get an APK off a secondary profile, or would like to download all the APKs off a system? And what about split APKs?

Multiple User Profiles

Run the following command to list the users.

adb shell pm list users

Example return

Users:
        UserInfo{0:User:a41} running
        UserInfo{11:User:439} running

In this case our second user id is 11. To get a list of APKs installed for our second user we would specify the –user= option

adb shell pm list packages --user=11

To get the path for the app we would run it with

adb shell pm path --user=11 com.android.apk

Split APKs

Split APKs can be slightly more difficult to manage, mainly due to the fact that there are multiple APKs to keep track of.

When you run the “pm path” command, it should return multiple APKs. Use the pull command like normal, but download each APK.

You’ll need to use a split APK installer to install all the APKs.

PowerShell script for Pulling/Downloading all APKs on Device

The following PowerShell script will download all APKs for a specific user and put them in their own folders.

  • Copy the contents to a .ps1 file
  • Enable ps1 execution policy if not already enabled
  • Run PowerShell script.

This script will pull all the APKs off of a device and put them in the current folder.
It will also download split APKs.

# adbapkbackup uses adb to get a list of all the APKs you have on a phone and then
# Creates folders for each app and downloads the APKs for those apps.

# Copy and save code as a ps1 file

# Enable ps1 scripts to run on your computer by launching an Admin promopt and running
# set-executionpolicy remotesigned

# If you are in a secondary profile, add and/or modify
#  "--user 15"
# to your user id
# adb shell pm list users

# If in secondary profile, add "--user 15" after packages before > apklist.txt
adb shell pm list packages --user 15 > apklist.txt

 $apks = ((Get-Content .\apklist.txt)) -replace 'package:',''

 ForEach ($apk in $apks) {
    echo "APK is $apk"
    md $apk
    # If in secondary profile, add "--user 15" after path, before $file
    adb shell pm path $apk
    $filepath = ((adb shell pm path --user 15 $apk | % {$_.replace("package:","")}))
    ForEach ($lapk in $filepath | % {$_.replace("package:","")}) {
        echo "pulling $lapk $apk"
        adb pull $lapk $apk
    }
 }

Identify “Magic Bytes” for Mikroitk Backup File

the “magic bytes” are the first few bytes of a file that can tell you what format the file is.

https://en.wikipedia.org/wiki/List_of_file_signatures

Mikrotik Backup file magic bytes

File typeMagic bytes
RC4 EncryptedEFA89172
AES EncryptedEFA89173
Not Encrypted88ACA1B1

Using the above list, we can view a Mikrotik .backup file in a hex editor like GHex or dump it with xxd.

Setting up Grafana on LibreNMS

Thanks to the guys who put together the information at the following links.

https://wadman.co.nz/2021/01/02/Viewing-LibreNMS-data-in-Grafana/
https://www.reddit.com/r/LibreNMS/comments/ojc8cc/how_to_almost_natively_integrate_librenms_and/

I ran into some issues trying to get this to work. So here are some of my notes. I already had a LibreNMS installation set up.

  1. Install RRDReST
    1. Install Docker
    2. Configure Docker compose file
  2. Configure LibreNMS API User and Key
  3. Set up and Configure Grafana
    1. Install Grafana
    2. Install JSON Data Source Plugin
    3. Add LibreNMS API Data Source
    4. Add RRDReST Data Source
    5. Import Dashboard into Grafana
  4. View graphs

Installing RRDReST

NOTE FOR CENTOS 8, ALMALINUX 8 and 9
The steps for installing RRDReST are slightly different. Check out the following post.
https://www.incredigeek.com/home/setting-up-rrdrest-on-centos-8-or-almalinux-9/

I had issues installing RRDReST. I am guessing it had to do with it accessing files. I was able to install it in a docker container.

  1. Install Docker
  2. Create a compose file
  3. Install container

Install docker

sudo yum install -y docker docker-compose
sudo systemctl enable docker

Create docker compose file with the following options

vi docker-compose.yml

Change the TZ to your time zone. If you have issues with the graphs, most likely something is off with the time zone between this container and Grafana/LibreNMS server

version: "3.5"

services:

  rrdrest:
    image: michaelwadman/rrdrest:latest
    container_name: rrdrest
    restart: always
    volumes:
      - "/opt/librenms:/opt/librenms"
    environment:
      - TZ=America/Denver

Save the file and start and setup the container with

sudo docker-compose up -d

You will need your docker container IP address to setup the connection in Grafana

sudo docker exec -it rrdrest ip addr | grep eth0

Congratulations. You should now have a RRDReST docker container that will auto start on system boot and has the correct time zone.

Configure LibreNMS API User and Key

  1. Create a Grafana user in LibreNMS. (Settings Gear -> Manage Users -> Add Users)
    You could technically skip this step and use an existing user.
  2. Create API token for the newly created user (Setting Gear -> API -> API Settings)
Create a Grafana user in LibreNMS
Setup API Key for Grafana in LibreNMS

Set up and Configure Grafana

Basic steps are as follows

  1. Install Grafana
  2. Install JSON Data Source Plugin
  3. Configure Data Sources
    1. LibreNMS API
    2. RRDRest API
  4. Import Dashboard into Grafana

Install Grafana

There is not anything special with installing Grafana on the same server as LibreNMS. You can follow the official guide to install it

https://grafana.com/docs/grafana/latest/installation/

After Grafana is installed, install the JSON API data source. You can do this using the grafana-cli

grafana-cli plugins install marcusolsson-json-datasource

A note on SSL/TLS certificates. If you have an SSL certificate for LibreNMS, you can use it for grafana. If you run into issues, try copying the cert (fullchain.pem, privkey.pem) to /etc/grafana/

You’ll most likely need to change owner

sudo chown root:grafana /etc/grafana/*.pem

And maybe the file permissions.

sudo chmod 640 /etc/grafana/*.pem

Install JSON Data Source Plugin

This is fairly straight forward.

grafana-cli plugins install marcusolsson-json-datasource

Add LibreNMS API Data Source

In Grafana, go to Configuration -> Data Sources -> Add data source

  • Set Name for Data Source
  • URL should be https://your_librenms_url/api/v0
  • Add Custom HTTP Header
    • Header field should be “X-Auth-Token”
    • Value field should contain the API token we created in LibreNMS
  • Save and Test
    If you receive any errors, refer to the Troubleshooting part at the end.
Adding LibreNMS API Data Source in Grafana

Add RRDReST Data Source

In Grafana, go to Configuration -> Data Sources -> Add data source

  • Set Name for Data Source
  • URL needs to be your docker container IP address (Steps above)
  • Save and Test (Should return “Unprocessable Entity”)
Adding RRDReST API Data Source in Grafana

Import Dashboard into Grafana

Now we need a dashboard to present our data.

  • Go to Create -> Import
  • Upload JSON file (Download from here or PasteBin )
  • Under RRDReST API , select our RRDReST Data Source
  • Under LibreNMS API , select our LibreNMS Data Source
  • Click Import

You should now be able to view your dashboard and use the drop down menus to select devices

Grafana viewing bandwidth on device being monitored by LibreNMS

Troubleshooting

There were a couple of issues I ran into while trying to get everything working together.

RRDReST shows 404 Not Found

Issue: When trying to run RRDReST with uvicorn, I was never able to access the rrd files, even the test rrd files that are included when installing RRDReST. I am guessing it is either a permisions issue, or something unable to access the files.
Work around: Install RRDReST via Docker container.

Error Running uvicorn RRDReST

Error Adding LibreNMS API

Issue: Get a “JSON API: Bad Request” when trying to set up the LibreNMS API Data Source in Grafana.

Work around: Install a valid SSL Certificate and set up a DNS record so you can access LibreNMS at librenms.yourdomain.com.

More info: I would assume that “Skip TLS Verify” would work with or without a valid certificate, but it would not work for me. There are potentially some other options with modifying how Nginx or Apache is set up that would get this working. If you setup Grafana to use a SSL certificate, you may need to copy the certificate files (fullchain.pem, privkey.pem) to /etc/grafana/ and run “chown root:grafana *.pem” to let grafana have access to the files.

Grafana LibreNMS API JSON API: Bad Request

LibreNMS – Package not found: The ‘command_runner>=’

The Problem

Running the ./validate.php script returns the following error

[FAIL]  Python3 module issue found: 'Required packages: ['PyMySQL!=1.0.0', 'python-dotenv', 'redis>=3.0', 'setuptools', 'psutil>=5.6.0', 'command_runner>=1.3.0']
Package not found: The 'command_runner>=1.3.0' distribution was not found and is required by the application
'
        [FIX]:
        pip3 install -r /opt/librenms/requirements.txt

Running the [FIX] throws an error saying gcc failed with exit status 1.

The Solution

Fortunately this issue is easy to resolve.

First we need to install python3-devel

sudo yum install python3-devel

Next, as the librenms user, run the pip command to install the requirements.

pip3 install --user -U -r /opt/librenms/requirements.txt

Run ./validate.php to verify that everything is working.

librenms validate.php results

UniFi UNVR – Recover from Failed USB Drive

Some of the UniFi UNVR’s have system files on a USB drive. There seem to be a number of the drives failing recently, rendering the UNVR inoperable. Fortunately it is easy to replace. The following steps should preserve the video recordings.

https://community.ui.com/questions/UNVR-stopped-responding-just-white-flashing-light/a051f869-8349-4a2a-a72a-ce3a8aa8c759

Steps to recover UNVR

  1. Power off the UNVR
  2. Remove the USB drive (use a heat gun or screw driver to break the glue that is holding the USB drive)
  3. Install new USB drive
  4. Temporarily remove UNVR HDDs (this may not be necessary, but rather be safe then sorry.)
  5. Boot UNVR with new USB drive. (Give it a little time to format and copy contents to the new USB drive. Should not take more then 30 minutes.)
  6. Setup the UNVR like it was before
  7. Power off the UNVR again
  8. Reinstall the HDD’s
  9. Power on the UNVR
  10. Log in and reconfigure the users

Note on Recovery

You could potentially mount the failed or failing USB drive on a Linux machine copy off a UniFi backup. Unfortunately, the UniFi Protect backup does not preserve the users. Just the video groups. You will probably have to resend invites to users.

Note on the video storage drives

It sounds like the UniFi Protect system will try to read the drives and if it can preserve the data or read the raid information it will try to use that. That is what it sounds like at least from the forums. More info on drive management.

https://help.ui.com/hc/en-us/articles/360037340954-UniFi-Hard-drive-compatibility-and-management#h_01F8QE56P7EY1P9FMTZHFZD463

Enable Syslog for PowerDNS Recursor

  1. Enable Logging in PowerDNS Recursor Config
  2. Edit Systemd Unit File for PowerDNS to Allow Syslog
  3. Enable Logging in rsyslog Config File

The following links were helpful in setting things up.

https://doc.powerdns.com/recursor/running.html
https://www.reddit.com/r/linuxadmin/comments/9lc4jl/logging_queries_in_pdnsrecursor/

Enable logging in PowerDNS Recursor Config

First we need to find the line that says “disable-syslog” and uncomment/change it to

disable-syslog=no

Next find the line that says “quiet” and uncomment/change it to

quiet=no

Some other lines you may want to check and change

logging-facality=1
loglevel=6

Edit Systemd Unit File for PowerDNS to allow Syslog

Next we need to modify the Systemd unit file to allow PowerDNS Recursor to log to syslog.

systemctl edit --full pdns-recursor.service

On the ExecStart Line, remove the part that says

--disable-syslog

The resulting line should look something like

[Service]
ExecStart=/usr/sbin/pdns_recursor --socket-dir=%t/pdns-recursor --socket-dir=%t/pdns-recursor --daemon=no --write-pid=no --log-timestamp=no

Save the file.

Enable Logging in rsyslog Config File

Edit the rsyslog file

sudo vim /etc/rsyslog.conf

Add the following line

local1.*        /var/log/pdns_recursor.log

This should now log all of the PowerDNS Recursor log info to “/var/log/pdns_recursor.log”

Restart the rsyslog and PowerDNS Recursor service

sudo systemctl restart rsyslog
sudo systemctl restart pdns-recursor

You should now see DNS request in the log file.

tail /var/log/pdns_recursor.log

They should also show up in the “/var/log/messages”

Enable Remote Logging in Rsyslog on Linux (Debian, Ubuntu, CentOS, Fedora)

Edit the rsyslog file

sudo vi /etc/rsyslog.conf

add the following. Change out the IP address for your remote syslog server IP address.

The syntax for this has changes. Newer recommended way is

*.*  action(type="omfwd" target="192.0.10.5" port="514" protocol="tcp"

Old Syntax

## rsyslog Server
*.*     @10.0.10.5:514

https://www.rsyslog.com/sending-messages-to-a-remote-syslog-server/

Restart the rsyslog service and check your remote log server.

systemctl restart rsyslog

If you are having issues viewing logs in LibreNMS, try adding the hostname in the /etc/rsyslog.conf file

$LocalHostName server.hostname.com

Update Matrix Server that is utilizing the matrix-docker-ansible-deploy scripts

Upgrading is fairly straightforward.

https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/maintenance-upgrading-services.md

cd into the git directory and run

git pull

That will download any new files.

Run the ansible command, you can have it ask you for the password to use.

ansible-playbook -i inventory/hosts setup.yml --tags=setup-all --ask-pass --ask-become-pass

It should give you a report if anything failed.

Checking Postgres version for Matrix-docker-ansible-deploy

https://github.com/spantaleev/matrix-docker-ansible-deploy

When using the above to run a matrix server, it can be confusing how to verify and check which version of Postgres you are running. Fortunately this is really easy to check.

Run

sudo matrix-postgres-cli --version

And you should get the version of Postgres that is being used.

psql (PostgreSQL) 9.6