In this article, we will discuss these
- Prometheus Alert Manager
- Configure SMTP local on Prometheus Server
- Test Alerts
Prometheus Alert Manager
The AlertManager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.
Install the Prometheus Alert Managersudo apt install prometheus-alertmanager
sudo service prometheus-alertmanager status
ps -u prometheus
Note that the service is running on port 9093
Visit http://[your domain name or ip]:9093/
Block Port 9093 for external requests
iptables -A INPUT -p tcp --dport 9093 -j DROP
iptables-save > /etc/iptables/rules.v4
iptables-save > /etc/iptables/rules.v6tail -4 prometheus.yml
- job_name: alertmanager
- targets: ['localhost:9093']
verify the config and restart the service. Once they are successful, it must display the target in the Prometheus UI.
Configure SMTP for Alerts
Setup a simple local SMTP server which can only send emails from localhost.
sudo apt install mailutils
sudo vim /etc/postfix/main.cf
Go to the End of the line,
inet_interfaces = loopback-only
inet_protocols = ipv4
sudo systemctl restart postfix
Make sure your forward and reverse looks fine, otherwise it is very likely that email providers don't think this would be a valid email and you won't receive any emails.
Once verified, fire below command from the prometheus server,
echo "This is the body" | mail -s "This is the subject" -a "FROM:admin@yourdomainname" your@email-address
check your mail account, you would have received an email..
configure the Alert Manager process to send emails when the alerting rules fire and resolve.
cp alertmanager.yml alertmanager_orig.yml
cat > alertmanager.yml
Add the below contents and configure your alerts
- name: 'smtp-local'
- to: 'firstname.lastname@example.org'
Now, you would have received an alert as the state is in Firing.
sudo vim /etc/default/prometheus
restart your prometheus server to take effect.