Prometheus Monitoring - part 1

Prometheus Monitoring
We shall be installing and configuring the Prometheus on the Ubuntu server.  you can either create a local box or from any of the cloud providers.

Arch Diagram

Prerequisites
I have set up this monitoring tool using Ubuntu 20.04 LTS Server with root access. You can use other operating systems, such as Centos, but since it was already installed for some demo purpose I wanted to configure this.

Installations
Prometheus package installed both Prometheus and the Prometheus Node to be installed.

sudo apt-get update
sudo apt install prometheus -y
sudo service prometheus status
sudo service prometheus-node-exporter status

Prometheus should now be running.
ps -u prometheus

You can visit it at http://[your ip address]:9090

Pointing your 'A' Domain name 
If your Prometheus server is accessible from the internet, you want it to look more professional to clients, login to your domain name provider, and add an A Name record that points to the IP address of the new Prometheus server.

Reverse Proxy Prometheus with Nginx
One option to help secure our Prometheus server is to put it behind a reverse proxy so that we can later add SSL and an Authentication layer over the default unrestricted Prometheus web interface.

sudo apt install nginx -y
sudo vim /etc/nginx/sites-enabled/prometheus

server {
    listen 80;
    listen [::]:80;
    server_name  prometheus.YOUR-DOMAIN-NAME;

    location / {
        proxy_pass           http://localhost:9090/;
    }
}

Save and test the new configuration has no errors
nginx -t

http://YOUR-DOMAIN-NAME
Visiting your ip address directly will still show the default Nginx welcome page. you can remove

rm /etc/nginx/sites-enabled/default

restart nginx,
sudo service nginx restart
sudo service nginx status

Add SSL to Prometheus Reverse Proxy
We will now add transport encryption to the Prometheus web user interface.
Certbot will install a LetsEncrypt SSL certificate for free. Ensure your domain name has propagated before running CertBot.

sudo snap install --classic certbot
sudo certbot --nginx

<snip>

.

.
Follow the prompts and select the domain name I want to secure.
.

.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations! You have successfully enabled https://prometheus.YOUR-DOMAIN-NAME
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
<snip>


Add Basic User Authentication Prometheus UI
Everything is great so far, but anybody in the world with the internet access and the URL can visit my Prometheus server and see my data.
To solve this problem, we will add user authentication.

cd /etc/nginx
sudo apt install apache2-utils
htpasswd -c /etc/nginx/.htpasswd admin

Nginx Prometheus config file, 

sudo vim /etc/nginx/sites-enabled/prometheus
server {
    ...

    #addition authentication properties
    auth_basic  "Protected Area";  <=============== append
    auth_basic_user_file /etc/nginx/.htpasswd; <=== append

    location / {
        proxy_pass           http://localhost:9090/;
    }

    ...
}

restart nginx,
sudo service nginx restart
sudo service nginx status

when you try to open your Prometheus server, it would prompt for your basic authentication.

you would still be able to access the IP:9090 of the Prometheus server and hence we block ports from external connections.

iptables -A INPUT -p tcp -s localhost --dport 9090 -j ACCEPT
iptables -A INPUT -p tcp --dport 9090 -j DROP
iptables -A INPUT -p tcp -s localhost --dport 9100 -j ACCEPT
iptables -A INPUT -p tcp --dport 9100 -j DROP
iptables -L

To save rules permanently,
sudo apt install iptables-persistent
iptables-save > /etc/iptables/rules.v4
iptables-save > /etc/iptables/rules.v6

you have now successfully installed Prometheus server on your machine.