Prometheus Monitoring - Part 4

In previous post we discussed about PromQL and Alerts
https://sunlnx.blogspot.com/2020/11/promotheus-monitoring-part-3.html

In this article, we will discuss these

• Prometheus Alert Manager
• Configure SMTP local on Prometheus Server
• Test Alerts

Prometheus Alert Manager

The AlertManager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.

Install the Prometheus Alert Manager

sudo apt install prometheus-alertmanager
sudo service prometheus-alertmanager status
ps -u prometheus

Note that the service is running on port 9093

Visit http://[your domain name or ip]:9093/

Block Port 9093 for external requests

iptables -A INPUT -p tcp -s localhost --dport 9093 -j ACCEPT
iptables -A INPUT -p tcp --dport 9093 -j DROP
iptables -L

iptables-save > /etc/iptables/rules.v4
iptables-save > /etc/iptables/rules.v6

tail -4 prometheus.yml
- job_name: alertmanager
      static_configs:
        - targets: ['localhost:9093']

verify the config and restart the service. Once they are successful, it must display the target in the Prometheus UI.

Configure SMTP for Alerts

Setup a simple local SMTP server which can only send emails from localhost.

sudo apt install mailutils
sudo vim /etc/postfix/main.cf

Go to the End of the line,
inet_interfaces = loopback-only
inet_protocols = ipv4

sudo systemctl restart postfix

Make sure your forward and reverse looks fine, otherwise it is very likely that email providers don't think this would be a valid email and you won't receive any emails.
Once verified, fire below command from the prometheus server,

echo "This is the body" | mail -s "This is the subject" -a "FROM:admin@yourdomainname" your@email-address

check your mail account, you would have received an email..

configure the Alert Manager process to send emails when the alerting rules fire and resolve.
cd /etc/prometheus
cp alertmanager.yml alertmanager_orig.yml
cat >  alertmanager.yml
[ctrl-d]
Add the below contents and configure your alerts

route:

  receiver: smtp-local
receivers:
  - name: 'smtp-local'
    email_configs:
    - to: 'sunlnx@gmail.com'
      from: 'promoalertadmin@devtestlabs.in'
      require_tls: false
      #auth_username: 'alertmanager'
      #auth_password: 'password'
      #auth_secret: 'secret'
      #auth_identity: 'identity'
      smarthost: localhost:25
      send_resolved: true

  Now, you would have received an alert as the state is in Firing.  

Source mentioned in the email are w.r.t to localhost and we would configure it to use the prometheus source, to change it
sudo vim /etc/default/prometheus
ARGS="--web.enable-admin-api --web.external-url=https://example.com"

restart your prometheus server to take effect.

systemctl restart prometheus

Thanks.