Latency Analysis using Grafana and raintank-probe

Real time and historical latency analysis in a Telco environment is an important troubleshooting tool that can help the NOC and Ops teams to identity networking issues. Issues like fiber cuts, errors in backbone links, wrong configuration in routing protocols and link saturations can affect customers applications running over those links. Being able to proactively identify issues before customers applications are affected due to latency or packet loss is one of the most important activities of NOC teams.

Typical monitoring tools (Opensource based open Nagios or commercial ones like Solarwinds, Zabbix, etc) are centralized and usually only provide ICMP analysis from one or two geographic locations which is not enough when trying to identify latency between the different markets or POPs.

There are several commercial and open source solutions that allow you to deploy internal probes in such a way that you can send ICMP tests every 60 seconds and send alarms via email or web hooks to a chat application like slack or Hangouts chats. An interesting one is Worldping which is a cloud based solution that can be integrated as plugin with Grafana and have the following advantages:

• Cloud based
• You don’t have to deploy an internal VM or public cloud VM and install Linux and Grafana
• Grafana upgrades
• Hosted Grafana supports community plugins which means that can be used not only for Worldping
• Additional to ICMP tests the probe supports DNS, HTTP and HTTPS
• You can use external probes additional to the private ones

Steps to deploy a hosted Grafana with Worldping plugin:

1. Create an account in grafana.net
2. Install and enable Worldping plugin the the hosted Grafana
3. Install raintank in the internal probes. You can use any Linux server, either a vm or baremetal server or use a Raspberry Pi with Raspbian. Make sure that Go is installed
4. Create API Key
5. Configure key in internal probes under probe.ini and start daemon
6. Configure Internal probes in Worldping (Should appear automatically if the probe list is key was configured correctly)
7. Configure Endpoints in Worldping
8. Configure alarms
9. Enjoy and share dashboards with your team

Other commercial and open source alternatives:

Thousand eyes

PRTG

Vaping (Interesting alternative to Smokeping based on Python)