Prometheus

As explained by TechWorld with Nana Prometheus Architecture explained and https://medium.com/techno101/how-to-send-a-mail-using-prometheus-alertmanager-7e880a3676db

Prometheus Server

  • Time Series Database (TSDB)

  • Data Retrieval Worker

  • Web Server/API/UI

Metrics

  • HELP {Describing text }

  • TYPE {Counter, gauge, histogram}

The data retrieval worker pulls /metrics e.g. http://host.name:9090/metrics.

Exporter

Exports metrics from target to own /metrics endpoint.

E.g. exporters for

  • databases (mysql, postgresql)

  • linux server

  • build tools

List of prometheus official exporters

  • The node_exporter (e.g. apt install prometheus-node-exporter) exports all linux system related information.

  • The blackbox_exporter can probe remote locations for reachability.

  • Custom exporters can be written with client libs .

Configuration

  • Rules and endpoints can be configured in prometheus.yaml

  • scrape_interval, evaluation_interval can be configure globally and per job

  • Rules are in $filename.rules files and can be included

Alertmanager

  • Is queried using PromQL

  • We can use this to setup alerts

Configuration

alert rules are configured in alert.rules

./promtool check rules alert.rules

⚠️ Prometheus itself does not send the actual alert messages to users, this is the responsibility of the Alertmanager (deployed independently). When an alert reaches the Firing state, Prometheus notifies the Alertmanager, which in turn will route the alert to the right channel (e.g. Slack, Pagerduty…) and the right people.

https://devconnected.com/monitoring-disk-i-o-on-linux-with-the-node-exporter/