Linux Server Performance Monitoring with Netdata

Netdata is an open-source real-time Linux server performance monitoring tool with a beautiful web front-end. It allows you to monitor CPU, RAM usage, disk I/O, network traffic, Postfix, among many others.  Written in the C programming language, netdata is super fast and resource-efficient.

Linux Server Performance Monitoring with Netdata

Netdata Features:

  • It helps you instantly diagnose slowdowns and anomalies in your infrastructure with thousands of metrics, interactive visualizations, and insightful health alarms.
  • 1s granularity – Netdata updates system statistics per second.
  • Linux kernel insights via eBPF
  • Parse Apache and Nginx web server logs to show you request processing time, upstream response time, and many other performance statistics.
  • Collect database health and performance metrics (MySQL/MariaDB, PostgreSQL, MongoDB, etc), including Galera Cluster.
  • Fast and lightweight – By default it uses only 1% CPU of a single core.
  • and more.

In this tutorial, we are going to look at how to install netdata on Debian/Ubuntu and Redhat/CentOS/Fedora servers. We will also discuss how to enable password authentication on the netdata web interface so that only authorized users can have access to it.

Note: If you run a mail server with iRedMail, then you don’t have to follow this tutorial, because iRedMail automatically installed it for you. You can access Netdata web interface at https://mail.example.com/netdata/. You will need to enter your postmaster account and password.

Step 1: Install netdata on Linux Server

Netdata is included in many Linux distributions’ repositories. However, it’s probably not the latest version. To get the latest version, you can use the official netdata script to install the software. Simply run the following command on your Linux system.

bash <(curl -Ss https://my-netdata.io/kickstart.sh)

It might ask you to enter your password if you are not root.

install netdata on linux server

Then it will try to install dependencies, if they are not already installed on your system. Next, it gives you a nice summary about where files will be installed to your system. Press Enter to start building and installation.

netdata ubuntu

Once it’s installed, it should be automatically started and enabled auto start on system boot. As you can see with systemctl status.

systemctl status netdata

Sample output:

* netdata.service - Real time performance monitoring
     Loaded: loaded (/lib/systemd/system/netdata.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-07-10 16:44:51 SAST; 18s ago
    Process: 1059965 ExecStartPre=/bin/mkdir -p /var/cache/netdata (code=exited, status=0/SUCCESS)
    Process: 1059977 ExecStartPre=/bin/chown -R netdata:netdata /var/cache/netdata (code=exited, status=0/SUCCESS)
    Process: 1059978 ExecStartPre=/bin/mkdir -p /var/run/netdata (code=exited, status=0/SUCCESS)
    Process: 1059979 ExecStartPre=/bin/chown -R netdata:netdata /var/run/netdata (code=exited, status=0/SUCCESS)
   Main PID: 1059980 (netdata)
      Tasks: 49 (limit: 38335)

If you can see the following lines in the output, don’t panic. Your installation is fine.

ebpf.plugin[1060201]: PROCFILE: Cannot open file '/etc/netdata/apps_groups.conf'
ebpf.plugin[1060201]: Cannot read process groups configuration file '/etc/netdata/apps_groups.conf'. Will try '/usr/lib/netdata/conf.d/apps_groups.conf'

Netdata by default listens on port 19999. Now enter server-ip:19999 in your browser address bar to access the netdata web interface. It doesn’t have an authentication mechanism. Anyone who knows your IP address can have access.

Linux-server-performance-monitoring-netdata-dashboard-overview

If your server has firewall enabled, then you need to open TCP port 19999. For instance, if you use the UFW firewall on Debian/Ubuntu, then run the following command.

sudo ufw allow 19999/tcp

If you use Firewalld on RHEL/CentOS/Alma Linux/Rocky Linux, then run the following commands.

sudo firewall-cmd --permanent --add-port=19999/tcp

sudo systemctl reload firewalld

Troubleshooting

If Netdata fails to install on your system, and you see the following error messages.

Makefile:3001: recipe for target 'all' failed
make: *** [all] Error 2
 FAILED   

 FAILED  

 ABORTED  netdata-installer.sh exited with error 

Then you can try installing Netdata with the deb or RPM package.

Debian/Ubuntu

RHEL/CentOS/Alma Linux/Rocky Linux/Fedora

curl -s https://packagecloud.io/install/repositories/netdata/netdata/script.rpm.sh | sudo bash

sudo dnf install netdata

OpenSUSE

curl -s https://packagecloud.io/install/repositories/netdata/netdata/script.rpm.sh | sudo bash

sudo dnf install netdata

Step 2: Set Up Reverse Proxy

To access the web interface through domain name instead of IP address and port number, we can set up a reverse proxy for netdata with Nginx or Apache. This also allows us to enable HTTPS later.

Nginx

Install Nginx on the Linux server.

Debian/Ubuntu

sudo apt install nginx

Redhat/CentOS/Fedora

sudo dnf install nginx

OpenSUSE

sudo zypper install nginx

Arch Linux/Manjaro

sudo pacman -S nginx

After Nginx is installed, create a virtual host config file for netdata under /etc/nginx/conf.d/ directory.

sudo nano /etc/nginx/conf.d/netdata.conf

Put the following text into the file. Replace the red-colored text with your actual domain name, and don’t forget to set DNS A record for this subdomain.

upstream backend {
   server 127.0.0.1:19999;
   keepalive 64;
}

server {
   listen 80;
   server_name netdata.example.com;

   location / {
     proxy_set_header X-Forwarded-Host $host;
     proxy_set_header X-Forwarded-Server $host;
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     proxy_pass http://backend;
     proxy_http_version 1.1;
     proxy_pass_request_headers on;
     proxy_set_header Connection "keep-alive";
     proxy_store off;
   }
}

Save and close this file. Then test Nginx configuration.

sudo nginx -t

If the config test is successful, reload Nginx.

sudo systemctl reload nginx

Now the netdata web interface is available at http://netdata.example.com.

Apache

Install Apache on the Linux server.

Debian/Ubuntu

sudo apt install apache2

Redhat/CentOS/Fedora

sudo dnf install httpd

OpenSUSE

sudo zypper install apache2

Arch Linux/Manjaro

sudo pacman -S apache

After Apache is installed, create a virtual host config file for netdata.

sudo nano /etc/apache2/sites-available/netdata.conf

or

sudo nano /etc/httpd/conf.d/netdata.conf

Put the following text into the file. Replace the red-colored text with your actual domain name, and don’t forget to set DNS A record for this subdomain.

<VirtualHost *:80>
    ProxyRequests Off
    ProxyPreserveHost On
    
    ServerName netdata.example.com

    <Proxy *>
        Require all granted
    </Proxy>

    ProxyPass "/" "http://localhost:19999/" connectiontimeout=5 timeout=30 keepalive=on
    ProxyPassReverse "/" "http://localhost:19999/"

    ErrorLog ${APACHE_LOG_DIR}/netdata-error.log
    CustomLog ${APACHE_LOG_DIR}/netdata-access.log combined
</VirtualHost>

Save and close the file. To use Apache as a reverse proxy, we need to enable the proxy modules and the header module.

sudo a2enmod proxy proxy_http rewrite headers proxy_wstunnel

Then enable this virtual host.

sudo a2ensite netdata.conf

Restart Apache

sudo systemctl restart apache2

Now you can access Netdata web interface using the domain name netdata.example.com.

Step 3: Listen on Localhost Only

By default, netdata listens on the public IP address. Now that netdata can be accessed via the Nginx reverse proxy, it’s a good security measure to make netdata listen only on 127.0.0.1 .  Open the netdata config file.

sudo nano /etc/netdata/netdata.conf

Go to the [web] section and find the following line (line 67).

# bind to = *

Remove the # sign and set its value to 127.0.0.1.

bind to = 127.0.0.1

Save and close the file. Then restart netdata for the change to take effect.

sudo systemctl restart netdata

Please note that if you set the bind to value to the IPv6 address ::1. Then in Nginx virtual host config file, you should also specify an IPv6 address in the upstream section like below.

upstream backend {
   server [::1]:19999;
   keepalive 64;
}

Step 4: Enable HTTPS

It’s highly recommended that you use TLS to encrypt HTTP traffic. We can enable HTTPS by installing a free TLS certificate issued from Let’s Encrypt. Run the following command to install Let’s Encrypt client (certbot)

Debian/Ubuntu

sudo apt install certbot

RHEL/CentOS/Alma Linux/Rocky Linux

sudo dnf install certbot

OpenSUSE

sudo zypper install certbot

Arch Linux

sudo pacman -S certbot

If you use Nginx, then you also need to install the Certbot Nginx plugin.

sudo apt install python3-certbot-nginx

Next, run the following command to obtain and install TLS certificate.

sudo certbot --nginx --agree-tos --redirect --hsts --staple-ocsp --email [email protected] -d netdata.example.com

If you use Apache, then you need to install the Certbot Apache plugin.

sudo apt install python3-certbot-apache

Next, run the following command to obtain and install TLS certificate.

sudo certbot --apache --agree-tos --redirect --hsts --staple-ocsp --email [email protected] -d netdata.example.com

Where

  • --nginx: Use the nginx plugin.
  • --apache: Use the Apache plugin
  • --agree-tos: Agree to terms of service.
  • --redirect: Force HTTPS by 301 redirect.
  • --hsts: Add the Strict-Transport-Security header to every HTTP response. Forcing browser to always use TLS for the domain. Defends against SSL/TLS Stripping.
  • --staple-ocsp: Enables OCSP Stapling. A valid OCSP response is stapled to the certificate that the server offers during TLS.

The certificate should now be obtained and automatically installed.

Step 5: Enable Password Authentication

If you installed netdata on a production Linux server, it’s important to enable access control so only authorized users can see what applications are running on your system.

Nginx

Generate a password file with the following command. Replace the red-colored text with your preferred username and password. The password will be created at /etc/nginx/password.

printf "yourusername:$(openssl passwd -crypt 'yourpassword')" | sudo tee -a /etc/nginx/passwords

If you see the following warning message, don’t panic. Your password is fine.

Warning: truncating password to 8 characters

Then edit the Nginx virtual host config file for netdata.

sudo nano /etc/nginx/conf.d/netdata.conf

Add the auth directives in server section. auth_basic enables basic password authentication. auth_basic_user_file directive specifies the password file.

server {
.....

auth_basic "Protected";
auth_basic_user_file /etc/nginx/passwords;

....

Save and close the file. Then reload Nginx.

sudo systemctl reload nginx

Now your browser will ask you to enter the username and password.

Apache

Generate a password file with the following command. Replace the red-colored text with your preferred username and password. The password will be created at /etc/apache2/password.

printf "yourusername:$(openssl passwd -crypt 'yourpassword')" | sudo tee -a /etc/apache2/passwords

If you see the following warning message, don’t panic. Your password is fine.

Warning: truncating password to 8 characters

Then edit the Apache virtual host config file for netdata.

sudo nano /etc/apache2/sites-enabled/netdata-le-ssl.conf

Change the <Proxy *>...</Proxy> section to the following.

    <Proxy *>
        AllowOverride None
        AuthType Basic
        AuthName "Protected site"
        AuthUserFile /etc/apache2/passwords
        Require valid-user
    </Proxy>

Save and close the file. Then restart Apache.

sudo systemctl restart apache2

Now your browser will ask you to enter the username and password.

Netdata Linux Server Performance Monitoring Screenshot Tour

CPU Usage

netdata monitor CPU usage

RAM Usage

Linux server performance monitoring RAM usage

Disk I/O

Linux server performance monitoring disk io

Network Traffic

Linux server performance monitoring network traffic

Memory De-duplication

If kernel memory de-duper (called Kernel Same-page Merging, or KSM) is available on your system, you can enable it to save 40-60% of netdata memory. To enable KSM, run the following command as root (sudo won’t work).

echo 1 >/sys/kernel/mm/ksm/run

echo 1000 >/sys/kernel/mm/ksm/sleep_millisecs

How to Enable Email Alert

Edi the health alarm notify config file with the following command.

sudo /etc/netdata/edit-config health_alarm_notify.conf

Find the following line.

DEFAULT_RECIPIENT_EMAIL="root"

By default, email alerts are sent to the root user on localhost. Change it to your email address.

DEFAULT_RECIPIENT_EMAIL="[email protected]"

Save and close the file. Then restart Netdata.

sudo systemctl restart netdata

If you use iRedMail, then email alerts are disabled and you will see the following line in this file.

SEND_EMAIL="NO"

Change it to

EMAIL_SENDER="[email protected]"

# enable/disable sending emails
SEND_EMAIL="YES"

# if a role recipient is not configured, an email will be send to:
DEFAULT_RECIPIENT_EMAIL="[email protected]"

Save and close the file. Then restart Netdata.

sudo systemctl restart netdata

You should install Postfix to send email alerts.

Debian/Ubuntu

sudo apt install postfix

RHEL/CentOS/Alma Linux/Rocky Linux

sudo dnf install postfix

OpenSUSE

sudo zypper install postfix

How to Uninstall Netdata

The uninstall script is available at /usr/libexec/netdata/netdata-uninstaller.sh.

How to Update Netdata

The update script is available at /usr/libexec/netdata/netdata-updater.sh. So when a new version comes out, run the following command.

sudo /usr/libexec/netdata/netdata-updater.sh

Fortunately, you don’t need to do it manually. A cron job (/etc/cron.daily/netdata-updater) is added by Netdata to automatically update the software daily.

How to Monitor Nginx Web Server Performance

In order to collect Nginx Performance metrics, you need to configure Nginx stub_status.

sudo nano /etc/nginx/conf.d/stub_status.conf

Add the following files to this file.

server {
     listen 127.0.0.1:80;
     server_name 127.0.0.1;
     location /nginx_status {
        stub_status on;
	allow 127.0.0.1;
	deny all;
     }
}

Save and close the file.Then reload Nginx for the changes to take effect.

sudo systemctl reload nginx

Now you can see the Nginx metrics in Netdata, which is accessible via the web log nginx menu on the right sidebar.

netdata web log nginx

Netdata can collect information from Nginx log files. However, the default log format gives us limited information. If you really care about application performance, you should create a custom log format to show the request time, upstream response time, cache hits, etc.

Open the nginx.conf file.

sudo nano /etc/nginx/nginx.conf

Add the following lines in the http {...} context, above the include directives. Here we are creating a custom Nginx log format called netdata that includes information about request_time, and upstream_response_time, measured in seconds with millisecond resolution.

log_format netdata '$remote_addr - $remote_user [$time_local] '
              '"$request" $status $body_bytes_sent '
              '$request_length $request_time $upstream_response_time '
              '"$http_referer" "$http_user_agent"';

Save and close the file. Then open your Nginx virtual host configuration file. For example,

sudo nano /etc/nginx/conf.d/linuxbabe.com.conf

Your Nginx virtual host configuration file might be under /etc/nginx/sites-enabled/ directory. I like to use the /etc/nginx/conf.d/ directory.

In the server {...} context, add the following two lines to enable access log and error log. The access log is using the netdata format and error log uses the warn log level.

access_log /var/log/nginx/linuxbabe.com.access.log netdata;
error_log /var/log/nginx/linuxbabe.com.error.log warn;

Save and close the file. Then reload Nginx for the changes to take effect.

sudo systemctl reload nginx

Next, edit the Netdata web_log.conf file.

sudo /etc/netdata/edit-config python.d/web_log.conf

Scroll down to the Nginx log section. Add the following lines.

yourdomain.com:
  name: 'yourdomain'
  path: '/var/log/nginx/yourdomain.com.access.log'

netdata monitor individual virtual host

Save and close the file. Then grant read permission to the netdata user.

sudo setfacl -R -m u:netdata:rx /var/log/nginx/

Restart Netdata

sudo systemctl restart netdata

How to Monitor Apache Performance

First, run the following command to create the Netdata apache.conf file. You don’t have to edit anything in this file. Simply press Ctrl+X to exit.

sudo /etc/netdata/edit-config python.d/apache.conf

Next, edit the main Apache configuration file.

sudo nano /etc/apache2/apache2.conf

Find the LogFormat section and add two new log formats.

LogFormat "%h %l %u %t \"%r\" %>s %O %I %D \"%{Referer}i\" \"%{User-Agent}i\"" vhost_netdata
LogFormat "%h %l %u %t \"%r\" %>s %O %I %D \"%{Referer}i\" \"%{User-Agent}i\"" netdata

Save and close the file. Then edit your Apache virtual host file and use the following CustomLog setting.

CustomLog "/var/log/apache2/yourdomain.com.access.log" netdata

Save and close the file. Next, edit the Netdata web_log.conf file.

sudo /etc/netdata/edit-config python.d/web_log.conf

Scroll down to the Apache log section. Add the following lines.

yourdomain:
   name: 'yourdomain'
   path: '/var/log/apache2/yourdomain.com.access.log'

Save and close the file. Grant read permission to the netdata user.

sudo setfacl -R -m u:netdata:rx /var/log/apache2/

Then restart Apache and Netdata.

sudo systemctl restart apache2 netdata

If Netdata doesn’t show performance metrics for your Apache virtual host, check if the web_log.conf file has a typo or a syntax error.

Install Apache JMeter

Apache JMeter is an open-source load testing tool, available for Linux, macOS, and Windows. You can use it to test your website performance under various load scenarios.

Wrapping Up

Hope you like this Linux server performance monitoring tool. Comments, questions or suggestions are always welcome. As always, if you found this post useful,  subscribe to our free newsletter to get more tips and tricks 🙂

Rate this tutorial
[Total: 10 Average: 4.7]

15 Responses to “Linux Server Performance Monitoring with Netdata

  • Dennis Keefe
    5 years ago

    Great tutorial, thank you!

  • Stenfrank
    3 months ago

    Thanks!

  • Thanks for the tutorial – lots of good info available in Netdata.

  • Rage Quit
    3 months ago

    For each server do I need to install NetData or is there a client I can install to see everything feeding back into the primary?

  • Rage Quit
    3 months ago

    Okay, found out after install that you can enable cloud monitoring. Very cool!!! Your content is always A++ !!!

  • If you have some privacy concerns about the data they are collecting, you should change the line

    bash <(curl -Ss https://my-netdata.io/kickstart.sh)

    to

    wget https://my-netdata.io/kickstart.sh
    chmod u+x kickstart.sh
    bash ./kickstart.sh --disable-telemetry
    

    And if your server ist 64 Bit only you should replace kickstart.sh with kickstart-static64.sh

  • Hello, very nice tutorial as usual.

    But i have a question, there is something that I wasn’t able to do. I have changed the lines between … into the file /etc/apache2/sites-enabled/netdata.conf but doesn’t ask me any password. Of course I have generated the file password with the username and the password

    Thanks

    • Between the lines “…

    • I’m having the same issue with the password authentication. It is not asking me for a password.

    • Edit the Apache HTTPS virtual host file, not the HTTP virtual host file.

      sudo nano /etc/apache2/sites-enabled/netdata-le-ssl.conf
      • Hello, yes I did that last night. After that is asking me the password but even if I use the correct username and password it is telling that I use the wrong password so after that I generated the password using the command htpasswd and now it is working good

  • Sébastien
    3 months ago

    Thanks a lot for this !

    Just a little add : if you want to access Netdata from a non-root directory (https://my.server/netdata/) you can add the following locations in your server block :

    location /netdata {
        return 301 /netdata/;
    }
    location ~ /netdata/(?<ndpath>.*) {
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_http_version 1.1;
        proxy_pass_request_headers on;
        proxy_set_header Connection "keep-alive";
        proxy_store off;
        proxy_pass http://backend/$ndpath$is_args$args;
        gzip on;
        gzip_proxied any;
        gzip_types *;
    }
  • I’ve installed this on an old Dell T610 running OpenMediaVault, with no conflicts with the OMV Dashboard, it works perfectly. The online real time monitoring is well done too.

    Your tutorial was easy to follow, thank you.

Leave a Comment

  • Comments with links are moderated by admin before published.
  • Your email address will not be published.
  • Use <pre> ... </pre> HTML tag to quote the output from your terminal/console.
  • Please use the community (https://community.linuxbabe.com) for questions unrelated to this article.
  • I don't have time to answer every question. Making a donation would incentivize me to spend more time answering questions.

The maximum upload file size: 2 MB. You can upload: image. Links to YouTube, Facebook, Twitter and other services inserted in the comment text will be automatically embedded. Drop file here