Tinkering with Azure Anomaly Detector

I’ve fancied having a play around with Azure Anomaly Detector (which is part of Azure Cognitive Services) for some time, I’d never really had a good use-case or excuse to do this…..until now!

I recently created a script for my Raspberry Pi that regularly checks the latency and speed of my Internet connection and writes this to a CSV file. I wrote about this here, my primary reason for doing this was to help me diagnose some random connectivity issues that I was experiencing – although ironically since creating this script my Internet connection has been super-reliable!

It got me thinking that I could take the data collected by this script and run it through Anomaly Detector to automate analysis and identify specific times of the day that my Internet connection speed deviated from the norm, which sounded far more appealing than cracking open Excel and doing this manually 😀.

I put together the PowerShell script below (which is also available on GitHub), this takes the CSV file created by my Internet speed test script, extracts the relevant data, formats this into JSON and submits to Azure Anomaly Detector for analysis, I opted to perform Batch rather than Streaming detection as I didn’t need to analyse the data real-time (now that would be overkill!), the differences between Batch and Streaming detection are explained here. I’m also using the Univariate API as this doesn’t require any experience with ML.

I opted to call the REST endpoint directly using this sample as inspiration, the script does the following:

  • Creates a JSON representation of the data from an input CSV file – “SpeedTestAnomaly.csv” in the format required by Anomaly Detector, an example JSON for reference can be found here. I’ve also uploaded a sample input file to GitHub, I’m only using two values from this input file – the date/time and download speed.
  • Submits this to Anomaly Detector – I’m using the maxAnomalyRatio and sensitivity settings from the sample (0.25 and 95 respectively). I used hourly for granularity as I only test my Internet connection once per hour.
  • Returns the expected and actual results for each test and indicates if the results were flagged as an anomaly (Red = Anomaly, Green = OK)

If you do want to re-use this script, you’ll need to update the $AnomalyURI and $APIKey variables.

$JSON = @"
{ 
    "series": [
    ],
   "maxAnomalyRatio": 0.25,
   "sensitivity": 95,
   "granularity": "hourly"
  }
"@
$NonJSON = $JSON | ConvertFrom-Json

$Output = Get-Content ./SpeedTestAnomaly.csv
Foreach ($Line in $Output)
{
  $DL = $Line.split(",")[2]  
  $Date = $Line.split(",")[0]
  $Add = New-Object -TypeName psobject -Property @{timestamp = $Date;value = $DL}
  $NonJSON.series += $Add
}

$JSON = $NonJSON | ConvertTo-Json

$AnomalyURI = "https://PREFIX.cognitiveservices.azure.com/anomalydetector/v1.0/timeseries/entire/detect"
$APIKey = "KEY"

$Result = Invoke-RestMethod -Method Post -Uri $AnomalyURI -Header @{"Ocp-Apim-Subscription-Key" = $APIKey} -Body $JSON -ContentType "application/json" -ErrorAction Stop

$i = 0
Foreach ($Anomaly in $Result.isAnomaly)
{
  if ($Anomaly -eq "True") 
  {
    Write-Host "Expected Value: " $Result.expectedValues[$i] "Actual Value: " $NonJSON.series[$i] -ForegroundColor Red
  }
  else 
  {
    Write-Host "Expected Value: " $Result.expectedValues[$i] "Actual Value: " $NonJSON.series[$i] -ForegroundColor Green
  }
  
  $i ++
}

Below is an extract from the input file (SpeedTestAnomaly.csv), I’m only using Column A (date/time) and Column C (download speed – mbps)

Below is the output of the script, this details the expected and actual values for each hourly test and highlights those tests that have a result that has been identified as an anomaly (in red), you can see there are three examples where anomalies have been detected in my Internet connection speed over the course of a couple of days.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s