Ever wanted to monitor Linode's or DigitalOcean's status page? Or how about Zoom's now that its so popular in 2020?

You have the option to subscribe to each of them, but who wants to get be subscribed to 50 different services, all with different notifications. Personally, I prefer to have all my monitoring happen within PRTG, even for services I am not hosting.

In comes the StatusPage.io sensor. This sensor allows you to parse the JSON endpoints of a status page site, and report the values to PRTG via a custom PowerShell script.

Installation

The script is hosted on my custom prtg sensors github repo: https://github.com/SoarinFerret/prtg-custom-sensors/tree/master/statuspage-io

• Copy the script Get-StatusPageData.ps1 to C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML
• For the lookup in PRTG to work you need to copy the file custom.statuspage.status.ovl to your PRTG installation folder C:\Program Files (x86)\PRTG Network Monitor\lookups\custom\ of your core server and reload the lookups
  (Setup/System Administration/Administrative Tools -> Load Lookups).

Usage

By default, all it needs is the URI parameter to be populated. The URI should end with /api/v2/components.json, so for example: https://status.linode.com/api/v2/components.json

However, there are a couple of other parameters to use. -PrependGroupNames will add the groups every item is included under. The other one is the -Name parameter, which supports wildcards so you can filter the items you want to see. For example, if you would like to only monitor items from the Dallas datacenter for Linode, just use "*Dallas*".

If you have any issues, go ahead a file an issue on GitHub.

This is a small PowerShell script I wrote to retrieve the key value pairs that Hyper-V exposes to Linux VMs. To view the most up to date version of this script, go to this GitHub link, otherwise look at the initial public code at the bottom of this blog post.

What is Hyper-V Key-Value Pair Data Exchange?

Simply put, Hyper-V Key-Value Pair Data Exchange is simply a way of transferring data between Hyper-V and a VM, both to and from. In Windows, this is exposed as registry keys in HKLM:\Software\Microsoft\Virtual Machine.

I am definitely not the most informed on this, if you would like to learn more, please see the following links:

• Eric Siron on Altaro (bonus: he wrote a C++ program to read AND write to the kvp_pools in Linux!)
• Microsoft's Documenation

How my script works

This script doesn't really do anything fancy. In linux, these values are exposed via a file. My script primarily targets the content in /var/lib/hyperv/.kvp_pool_3. In fact, if you wanted to access them without a script, you could just by using cat in bash or Get-Content in PowerShell.

$ cat /var/lib/hyperv/.kvp_pool_3
HostNameHyperV.ad.example.comHostingSystemEditionId8HostingSystemNestedLevel0HostingSystemOsMajor10HostingSystemOsMinor0HostingSystemProcessorArchitecture9HostingSystemProcessorIdleStateMax0HostingSystemProcessorThrottleMax100HostingSystemProcessorThrottleMin5HostingSystemSpMajor0HostingSystemSpMinor0PhysicalHostNameHYPERVPhysicalHostNameFullyQualifiedHyperV.ad.example.comVirtualMachineDynamicMemoryBalancingEnabled0VirtualMachineId5A535AFF-1708-4F6B-9B23-E506DF8CC5C5VirtualMachineNamekvp-testingtu18.04

But, that is literally just a long string with no delimiter. Or is it?

Each KVP in the file is stored with a 512 byte key and 2048 byte value. Everything after the actual content is null content basically. So, we can use that to parse the data.

Essentially, my script reads the file as a byte stream, and parses out 512 bytes for the key, 2048 for the value, and then starts on the next one. I store this in a hashtable, and then return it as PSObject (because objects are awesome).

Here is a sample output:

PS /home/serveradmin> Get-LinuxHypervKvpValues

HostingSystemOsMinor                        : 0
HostingSystemProcessorThrottleMax           : 100
HostingSystemProcessorIdleStateMax          : 0
PhysicalHostName                            : HYPERV
HostingSystemEditionId                      : 8
HostingSystemOsMajor                        : 10
HostingSystemNestedLevel                    : 0
HostingSystemSpMajor                        : 0
HostName                                    : HyperV.ad.example.com
VirtualMachineDynamicMemoryBalancingEnabled : 0
HostingSystemSpMinor                        : 0
VirtualMachineName                          : kvp-testingtu18.04
HostingSystemProcessorThrottleMin           : 5
HostingSystemProcessorArchitecture          : 9
PhysicalHostNameFullyQualified              : HyperV.ad.example.com
VirtualMachineId                            : 5A535AFF-1708-4F6B-9B23-E506DF8CC5C5

In addition, I added the ability to send a PSSession to the script, so I could get info about remote VMs without making sure to copy this function over.

The Script

Function Get-LinuxHypervKvpValues {
    <#
    .SYNOPSIS
    Retrive Hyper-V KVP Data Exchange values from a Linux based Hyper-V virtual machine.

    .DESCRIPTION
    Hyper-V provides key value pairs to VMs to send VM/Host info in a safe way. This function retrieves those key value pairs and returns them as a PowerShell object.

    .PARAMETER Path
    Location of the kvp_pool you would like to access. They are usually named .kvp_pool_x, where x is an integer between 0 and 4.

    .PARAMETER Session
    Optional parameter for a PSSession to remotely retrieve the KVP values.

    .EXAMPLE
    Get-LinuxHypervKvpValues -Session (New-PSSession -Hostname 192.168.1.1 -Username serveradmin)

    .NOTES
    Cody Ernesti
    github.com/soarinferret

    .LINK
    https://github.com/Soarinferret/PowerShell
    https://blog.kanto.cloud/retrieving-linux-hyper-v-kvps-in-powershell

    #>

    Param(
        [ValidateScript({Test-Path $_ -PathType 'Leaf'})] 
        [String]$Path = "/var/lib/hyperv/.kvp_pool_3",

        [Parameter(Mandatory=$false)]
        [ValidateScript({$_.State -eq "Opened"})] 
        [PsSession]$Session
    )
    function get-kvp ($KvpPath){
        $KEY_LENGTH = 512
        $VALUE_LENGTH = 2048

        $KVP_POOL = Get-Content $KvpPath -AsByteStream 

        $properties = @{}
        for($y = 0; $y -lt $KVP_POOL.Length; $y = $y + $KEY_LENGTH + $VALUE_LENGTH){

            $properties.add(
                # Key
                $([System.Text.Encoding]::UTF8.GetString($KVP_POOL[$y..$($y+$KEY_LENGTH -1)]) -replace "`0", ""),
                
                # Value
                $([System.Text.Encoding]::UTF8.GetString($KVP_POOL[$($y+$KEY_LENGTH)..$($y+$VALUE_LENGTH -1)]) -replace "`0", "")
            )
        }
        return New-Object PSObject -Property $properties
    }

    if($Session -and $Session.State -eq "Opened"){
        Invoke-Command -Session $Session -ScriptBlock ${function:get-kvp} -ArgumentList $Path
    }else{
        get-kvp $Path
    }
}

This post was updated on 2/15/2021 with an updated config to bypass the Single Logout issues.

In the last few weeks, v0.28 was released for BookStack, bringing lots of awesome new features and bug fixes, like their baseline API.

However, my favorite addition is the inclusion of SAML2 as a built-in authentication option. Looking through the code, they are taking advantage of the onelogin/php-saml library, which is very popular in a lot of other projects.

BookStack Setup

Not a lot of setup involved, simple edit your .env file with the following values:

## SAML Config
# Set authentication method to be saml2
AUTH_METHOD=saml2

# Set the display name to be shown on the login button.
# (Login with <name>)
SAML2_NAME=ADFS

# Name of the attribute which provides the users email address
SAML2_EMAIL_ATTRIBUTE=mail

# Name of the attribute to use as an ID for the SAML user.
SAML2_EXTERNAL_ID_ATTRIBUTE=http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn

# Name of the attribute(s) to use for the users display name
# Can have mulitple attributes listed, separated with a '|' in which
# case those values will be joined with a space.
# Example: SAML2_DISPLAY_NAME_ATTRIBUTES=firstName|lastName
# Defaults to the ID value if not found.
SAML2_DISPLAY_NAME_ATTRIBUTES=displayName

# Identity Provider entityID URL
SAML2_IDP_ENTITYID=http://sts.example.com/adfs/services/trust

# Auto-load metatadata from the IDP
# Setting this to true negates the need to specify the next three options
SAML2_AUTOLOAD_METADATA=false

# Identity Provider single-sign-on service URL
# Not required if using the autoload option above.
SAML2_IDP_SSO=https://sts.example.com/adfs/ls/

# Identity Provider single-logout-service URL
# Not required if using the autoload option above.
# Not required if your identity provider does not support SLS.
#SAML2_IDP_SLO=null

# Identity Provider x509 public certificate data.
# Not required if using the autoload option above.
SAML2_IDP_x509="MIIC2...."

ADFS Setup

I do not use ADFS with a GUI, so I don't have screenshots of what the ADFS Management MMC would show. I do however have the PowerShell and the claims rules you need.

Simply copy the claims rules into a file, and use that file in the PowerShell command provided below.

Claims Rules

@RuleTemplate = "LdapClaims"
@RuleName = "User Attributes"
c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname", Issuer == "AD AUTHORITY"]
 => issue(store = "Active Directory", types = ("http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn", "mail", "groups", "displayName"), query = ";userPrincipalName,otherMailbox,tokenGroups,displayName;{0}", param = c.Value);

@RuleName = "Transform UPN to Name ID"
c:[Type == "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn"]
 => issue(Type = "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier", Issuer = c.Issuer, OriginalIssuer = c.OriginalIssuer, Value = c.Value, ValueType = c.ValueType, Properties["http://schemas.xmlsoap.org/ws/2005/05/identity/claimproperties/format"] = "urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress");

PowerShell

Add-AdfsRelyingPartyTrust -Name Bookstack `
           -MetadataUrl https://docs.example.com/saml2/metadata `
           -IssuanceAuthorizationRules '@RuleTemplate = "AllowAllAuthzRule" => issue(Type = "http://schemas.microsoft.com/authorization/claims/permit", Value = "true");'`
           -IssuanceTransformRulesFile C:\bookstack-claimrules.txt

But, this also allows me to show different setups that may vary from the norm. For example, I am using external SSL termination on a completely different host. These are the methods I will go over:

• Rancher 2.0 with self-signed cert
• Rancher 2.0 with your own cert
• Rancher 2.0 using Let's Encrypt
• Rancher 2.0 using External SSL Termination
• Rancher 2.0 with SSL termination using nginx-proxy and Let's Encrypt

Bonus: All of these already include the necessary docker volume for persistent storage!

This does not include the steps for installing docker, or docker-compose. I recommend using the documentation provided on https://docs.docker.com, and in addition, take a look at the Docker version requirements on Ranchers website here.

Rancher 2.0 with self-signed cert

version: '3'

services:
  rancher:
    image: rancher/rancher:latest
    restart: unless-stopped
    ports:
      - '443:443'
      - '80:80'
    volumes:
      - rancher-vol:/var/lib/rancher

volumes:
  rancher-vol:

Rancher 2.0 with your own cert

Make sure if you use this, you update the certificate volumes to the location of your pem files!

version: '3'

services:
  rancher:
    image: rancher/rancher:latest
    restart: unless-stopped
    ports:
      - '443:443'
      - '80:80'
    volumes:
      - rancher-vol:/var/lib/rancher
      - ./full_chain.pem:/etc/rancher/ssl/cert.pem:ro
      - ./privatekey.pem:/etc/rancher/ssl/key.pem:ro

volumes:
  rancher-vol:

If you are using a self-signed certificate, make sure you add the following line to your volumes with the location of your cacerts.pem file:

      - ./cacerts.pem:/etc/rancher/ssl/cacerts.pem:ro

Rancher 2.0 using Let's Encrypt

For this one to work, Rancher needs to be sitting on a machine with a public IP, or ports 80 and 443 forwarded to it. Be sure to update your domain and double-check your public DNS record is pointed to your public IP address.

version: '3'

services:
  rancher:
    image: rancher/rancher:latest
    restart: unless-stopped
    ports:
      - '80:80'
    volumes:
      - rancher-vol:/var/lib/rancher
    command: --acme-domain rancher.example.com

volumes:
  rancher-vol:

Rancher 2.0 with External SSL Termination

Since the SSL termination is happening on a different host or load balanecer, we only need to expose port 80.

version: '3'

services:
  rancher:
    image: rancher/rancher:latest
    restart: unless-stopped
    ports:
      - '80:80'
    volumes:
      - rancher-vol:/var/lib/rancher
    command: --no-cacerts

volumes:
  rancher-vol:

Rancher 2.0 with SSL termination using Nginx-Proxy and Let's Encrypt

This one may seem a little weird. You may be thinking, "If rancher already has built-in Let's Encrypt support, why would I use this?" To which I would reply, what if you'd like SSL termination to more than one service, on the same host?

version: '3'

services:
  nginx-proxy:
    restart: always
    image: jwilder/nginx-proxy:alpine
    container_name: nginx-proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/etc/nginx/vhost.d"
      - "/usr/share/nginx/html"
      - "certs:/etc/nginx/certs:ro"
      - "/var/run/docker.sock:/tmp/docker.sock:ro"
      - "./custom_nginx_settings.conf:/etc/nginx/conf.d/custom_nginx_settings.conf"

  nginx-letsencrypt:
    restart: always
    image: jrcs/letsencrypt-nginx-proxy-companion
    container_name: nginx-letsencrypt
    depends_on:
      - nginx-proxy
    volumes_from:
      - nginx-proxy
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "certs:/etc/nginx/certs:rw"

  rancher:
    image: rancher/rancher:latest
    restart: unless-stopped
    expose:
      - '80'
    volumes:
      - rancher-vol:/var/lib/rancher
    command: --no-cacerts
    environment:
      - "VIRTUAL_HOST=rancher.example.com"
      - "LETSENCRYPT_HOST=rancher.example.com"

# other proxied services below here

volumes:
  rancher-vol:

Using this script, you can set assign a IP to a Windows VM with ease:

Get-VMNetworkAdapter -VMName Win2016 | `
     Set-VMNetworkConfiguration.ps1 -IPAddress 192.168.1.2 `
                                    -Subnet 255.255.255.0 `
                                    -DNSServer 192.168.1.1 `
                                    -DefaultGateway 192.168.1.1

If you try the same thing with an Ubuntu VM without any initial setup, it will fail. I will provide the steps necessary to make it work below on Ubuntu 16.04 Server and Ubuntu 18.04 Server. This document gives the steps for Ubuntu 16.04, but doesn't provide the full steps for Ubuntu 18.04.

Ubuntu 16.04

Ubuntu 16.04 is quite simple, just a few lines and a full shutdown:

sudo apt update
sudo apt install linux-azure -y
sudo shutdown now

Its important to note, you have to do a full shutdown, otherwise it will not reflect the changes.

Ubuntu 18.04

Ubuntu 18.04 is slightly more complicated. Ubuntu 18.04 by default ships with netplan.io, which is incompatible with the linux-azure module. It will show the netplan IP address in Hyper-V, but it will not update the IP address. To do so, you will need to reinstall ifupdown, resolvdconf, and then remove netplan.io

# install ifupdown, resolvvond, and linux-azure
sudo apt update
sudo apt install ifupdown resolvconf linux-azure -y

# add a default config
cat <<EOF >/etc/network/interfaces
source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp
EOF

# remove netplan and shutdown
sudo apt purge netplan.io -y
sudo shutdown now

Modified PowerShell Script Gist

Here is the gist promised from above:

No, this guide is for proxying non-dockerized services through the same reverse proxy, and the same docker-compose.yml file. I made a small alpine container just for this purpose. You can find it on GitHub and Docker Hub.

For example, I have a VM running docker, with nginx-proxy performs proxy services for a Ghost blog and Grafana. The file would look something like this:

version: '3'

services:
  nginx-proxy:
    restart: always
    image: jwilder/nginx-proxy:alpine
    container_name: nginx-proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/etc/nginx/vhost.d"
      - "/usr/share/nginx/html"
      - "/var/run/docker.sock:/tmp/docker.sock:ro"
  
  ghost:
    image: ghost:latest
    restart: always
    environment:
      url: https://blog.example.com
      VIRTUAL_HOST: blog.example.com
  
  grafana:
    image: grafana/grafana:latest
    restart: always
    environment:
      VIRTUAL_HOST: monitor.example.com

Now lets say I have a PRTG server running on a separate VM I want to add to the proxy service. You could add the following to the bottom of the docker-compose.yml file:

...
  prtg:
    image: soarinferret/iptablesproxy:latest
    restart: always
    cap_add:
      - NET_ADMIN
      - NET_RAW
    environment:
      SERVERIP: 192.168.0.5
      SERVERPORT: 80
      HOSTPORT: 80
      VIRTUAL_HOST: prtg.kanto.cloud
    expose:
      - '80'

Now after doing running docker-compose up -d, the new container will spin up, and simply forward all traffic destined to port 80 on the container to port 80 of 192.168.0.5.

Ok, technically Veeam created the checkpoints on my VHDS due to a misconfiguration on my part. And I didn't notice that they were there for a few months. Oops. Until one day, I got alerted from PRTG saying that the cluster shared volume (csv) on the cluster was running out of space, however the volume on the guest cluster was still showing up with plenty of space.

And then I saw it. 3.5 TBs of data taking up 5 TBs of my available space.

So I did some research. Sure, some people have complained about similar issues. Their solution? Copy all the data off onto a new data storage, and try again, or revert back to a VHDX. That wasn't really an option for me. So I started to do some more digging in how a VHDS works

How a VHDS works

Quoted by Microsoft:

VHD Set files are a new shared Virtual Disk model for guest clusters in Windows Server 2016. VHD Set files support online resizing of shared virtual disks, support Hyper-V Replica, and can be included in application-consistent checkpoints.

... That definition makes it sound pretty cool. But also gives off the idea that VHDS are a new fancy thing. In reality, at its core, it looks like its simply a binary file with a .vhds extension that has a pointer to a VHDX with a .avhdx file extension.

Don't believe me? Opening up the .vhds file in notepad will be hectic, but if you dig around towards the bottom of the file, you will see the reference to the .avhdx file.

The Problem

So the real problem is I have a VHDS pointing to an .avhdx, however the .avhdx file it is pointed to has a bunch of parent .avhdx files that I need to merge together. Here is what an example directory (sorry, no productions screenshots here!) reflecting the problem looks like:

Also, due to my basic understanding of how a VHDS is working behind the scenes, we have to take into account some potential problems:

• The VHDS might somehow be aware of the other checkpoints (though after some tests I doubt it!)
• Depending on the size of the checkpoints and speed of the underlying storage, it could take a VERY long time to merge
• This will break any replication going on, and a re-seed will be necessary

The Solution

For every problem, a solution exists. And that solution probably can be written in PowerShell!

Disclaimer: Your results may vary. TEST THIS IN A LAB FIRST! Also, double-check your backups. And then triple-check them. I do not take any fault if you lost all of your data. Merging checkpoints is dangerous, and if done wrong, can and will loose all your data.

Our tasks

Here are the basic tasks we will need to accomplish:

1. Power down the VMs referencing this VHDS file, and pause any replication going on
2. Sort the .avhdx files based on the parent-child relationship
3. Merge the files without breaking the parent-child relationship
4. This step has a few possiblities.
   a. According to this article, you can convert a vhdx to a vhds using Convert-VHD. Since the .avhdx file being used is technically just a VHDX, we could rename it and run the PowerShell conversion. However, when converting, it creates a DUPLICATE of the original VHDX file for the conversion, even if you have -DeleteSource specified when running the command. If you have a 5TB volume, you will need 5TBs of scratch space.
   b. If you are confident that the VHDS does not have any knowledge of the checkpoints, then you can rename the final merged file to the name of the .avhdx file the VHDS references
   c. If you wanted to be tricky, you could create a new VHDS, delete the newly created .avhdx file, and rename our merged file to that.

Step 1

I'm leaving this to you to handle!

Step 2: Sorting the AVHDX Files

The explanation for each step of this is in the comments of the code!

# Get the VHD files
$disks = Get-VHD ".\temp*" | select -Property Path,ParentPath | ? Path -NotLike "*.vhds"

# Create an Arraylist. Better than arrays for performance reasons
$list = New-Object System.Collections.ArrayList($null)

# Add the Highest Parent VHDX first
$list.add($($disks | where {$_.ParentPath -eq ""}).Path)

# Cycle through the rest, and add them to the array as we find the parent
while($list.Count -lt $disks.Count )
{   
    forEach($x in $disks)
    {
        if($x.ParentPath -eq $list[$list.count-1]){
            $list.add($($x.Path))
        }
    }
}

Step 3: Merging the AVHDX Files

So, in case you did not know this, you actually do NOT have to merge every single VHDX file to the previous parent. If you select the Parent of Parent VHD files, and the lowest child, it will merge all of the inbetween files too.

Merge-VHD -Path $list[$list.Count - 1] -DestinationPath $list[0]

Step 4

Okay, pick your poison on this one.

Step 4a: Converting the file

I must remind you again, if storage is an issue with this, keep that in mind.

# Create the avhdx files new name, and backup the path
$path = Split-Path $list[0]
$baseFileName = (Split-Path $list[0] -Leaf).Substring(0, (Split-Path $list[0] -Leaf).indexof('_'))

# Rename the file
Rename-Item $list[0] "$baseFileName.vhdx"

# Perform the conversion
Convert-VHD "$path\$baseFileName.vhdx" "$path\$baseFileName.vhds"

Step 4b: Renaming the file

Rename-Item $list[0] $(Split-Path $list[$list.Count - 1])

Step 4c: New VHDS

# Create the new VHDS with the same size as previous VHDS
New-VHD .\new.vhds -SizeBytes $(Get-VHD $list[0]).Size

# Rename newly created avhdx to .old
$newAvhdxName = Split-Path $(Get-VHD .\new_*).Path -Leaf
Rename-item .\$newAvhdxName "$newAvhdxName.old"

# Rename old avhdx to new avhdx
Rename-Item $list[0] $newAvhdxName

Results

Well, I ended up testing all three of these in a lab environment, and all of them worked as far as I can tell.

To fix my actual problem though, I ended up going with Option C. I didn't have enough scratch space to perform option A, and I just didn't quite feel comfortable with running Option B.

Good luck and good day to you!