LEMP Stack Performance Monitoring With Nginx Amplify on Ubuntu 20.04/18.04

Nginx Amplify is a free, flexible, and powerful way to analyze the load and performance of Nginx and Nginx Plus web servers. It’s easy to set up and use. Nginx Amplify also comes with MySQL/MariaDB, PHP-FPM plugins so you can monitor a full LEMP stack application. (LEMP stands for Linux, Nginx, MySQL/MariaDB, PHP.)

Nginx Amplify is developed by Nginx, Inc, the company behind Nginx web server. It’s a SaaS-based solution. You install the Nginx Amplify agent on your server and it will collect and send metrics to the SaaS service. The Nginx Amplify agent is open-source and lightweight. It collects many metrics, including:

  • System metrics (CPU, RAM usage, network traffic, disk usage, disk I/O, disk latency, etc)
  • Nginx metrics (connections, requests, HTTP status, response time, traffic, and more)
  • MySQL/MariaDB metrics (connections, select queries, insert queries, update queries, slow queries, etc)
  • PHP-FPM metrics (connections, connections queue, slow requests, etc)

Nginx Amplify can also

  • Use static analyzer to help you improve Nginx configuration.
  • Alert you of abnormal behavior

Installing Nginx Amplify on Your Ubuntu 18.04/20.04 Linux Server

The Nginx Amplify agent is open-source and there’s an install script that you can use to install Nginx Amplify on the following Linux Distributions:

  • Debian 10, Debian 11
  • Ubuntu 20.04, Ubuntu 18.04
  • RHEL/CentOS 8

First, register an account at https://amplify.nginx.com

nginx amplify setup

Verify your email address. Next, you will need to install the Nginx Amplify Agent on your Linux server. SSH into your server and login as root. The Nginx Amplify website is very nice to give the commands you need to run to install Nginx Amplify, so just copy and run these commands on your server.

Note that you need to run these commands as root.

nginx amplify install script

Download the install script.

install nginx amplify

Run the install script. (Each Amplify account has a unique API key.)

nginx amplify ubuntu 18.04

Once it’s installed, the agent will automatically start. You can check its status with:

systemctl status amplify-agent

Output:

 amplify-agent.service - LSB: Stop/start nginx-amplify-agent
   Loaded: loaded (/etc/init.d/amplify-agent; generated)
   Active: active (running) since Wed 2019-02-27 03:37:05 UTC; 1min 17s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 9306 ExecStop=/etc/init.d/amplify-agent stop (code=exited, status=0/SU
  Process: 9341 ExecStart=/etc/init.d/amplify-agent start (code=exited, status=0/
    Tasks: 3 (limit: 2361)
   CGroup: /system.slice/amplify-agent.service
           └─9407 amplify-agent

Hint: press Q to gain back control of the terminal after running the above command.

To enable auto start at boot time, run

sudo systemctl enable amplify-agent

Configuring Stub_status in Nginx

Amplify agent can collect system metrics out-of-the-box. In order to collect Nginx metrics, you need to configure Nginx stub_status. Open the nginx.conf file.

sudo nano /etc/nginx/nginx.conf

Add the following server block in the http {...} context.

server {
     listen 127.0.0.1:80;
     server_name 127.0.0.1;
     location /nginx_status {
        stub_status on;
	allow 127.0.0.1;
	deny all;
     }
}

Save and close the file. Then reload Nginx for the changes to take effect.

sudo systemctl reload nginx

Now Amplify agent can start collecting Nginx metrics.

nginx amplify upstream response time

Creating Custom Nginx Log Format

Nginx Amplify agent can also collect information from Nginx log files. However, the default log format gives us limited information. If you really care about application performance, you should create a custom log format to show the request time, upstream response time, cache hits, etc.

Open the nginx.conf file.

sudo nano /etc/nginx/nginx.conf

Add the following lines in the http {...} context, above the include directives. Here we are creating a custom Nginx log format called apm (application performance monitoring) that includes information about request_time, upstream_response_time, upstream_connect_time and upstream_header_time. All are measured in seconds with millisecond resolution.

log_format apm '"$time_local" client=$remote_addr '
               'method=$request_method request="$request" '
               'request_length=$request_length '
               'status=$status bytes_sent=$bytes_sent '
               'body_bytes_sent=$body_bytes_sent '
               'referer=$http_referer '
               'user_agent="$http_user_agent" '
               'upstream_addr=$upstream_addr '
               'upstream_status=$upstream_status '
               'request_time=$request_time '
               'upstream_cache_status="$upstream_cache_status" '
               'upstream_response_time=$upstream_response_time '
               'upstream_connect_time=$upstream_connect_time '
               'upstream_header_time=$upstream_header_time';

Save and close file. Then open your Nginx virtual host configuration file. For example,

sudo nano /etc/nginx/conf.d/linuxbabe.com.conf

Your Nginx virtual host configuation file might be under /etc/nginx/sites-enabled/ directory. I like to use the /etc/nginx/conf.d/ directory.

In the server {...} context, add the following two lines to enable access log and error log. The access log is using the apm format and error log uses the warn log level.

access_log /var/log/nginx/linuxbabe.com.access.log apm;
error_log /var/log/nginx/linuxbabe.com.error.log warn;

Save and close the file. Then reload Nginx.

sudo systemctl reload nginx

Now I can see the upstream response time and other performance related metrics in the Amplify graphs. A good upstream response time is below 0.5 seconds. Otherwise your visitors can feel your website is slow. If you can tune your server to get upstream response time below 0.2 seconds, your website will be very snappy.

nginx amplify performance monitoring

How do you find out which HTTP request takes long time? You can search the Nginx access log for the word pattern “request_time=1“, which shows you the requests that take more than 1 second to complete. Hint: If you set up Nginx FastCGI cache, the request_time will be reduced to 0 seconds for cached pages.

Possible causes of long request time:

  • MySQL/MariaDB database is slow.
  • The server is under heavy load. (Use the htop command to check load averages.)

If Amplify agent isn’t reporting Nginx metrics, check that Amplify agent runs as the same user as the Nginx worker process. You may also need to check if there’s a new version of Amplify-agent available.

How to Exclude Your Own IP Address in Nginx Access Log

Sometimes, your activity can skew the Nginx metrics. You can exclude your own IP address in the access log to avoid that. First, add the following lines in the http context of Nginx configuration file.  Replace 12.34.56.78 with your own IP address. If the HTTP request comes from your own IP address, Nginx will set the value of $log_ip variable to 0.

map $remote_addr $log_ip {
    "12.34.56.78" 0;
    default 1;
}

Then change the access_log directive as follows.

access_log /var/log/nginx/linuxbabe.com.access.log apm if=$log_ip;

Save and close the file. Then text nginx configuration and reload.

sudo nginx -t
sudo systemctl reload nginx

Note: If you use the WordPress Popular Posts plugin on your WordPress site, then there will be lots of HTTP POST requests sent from visitors’ browsers to your web server, and this can also skew the request_time in Nginx Amplify. To minimize this impact, you can go to Settings -> WordPress Popular Posts -> Tools and set log views from to logged-in users only. This will also reduce your server load.

How to Monitor MariaDB/MySQL Database Server

First, we need to create a user in MariaDB/MySQL to collect metrics. Log into MariaDB/MySQL monitor.

sudo mysql -u root

Create a new user for Amplify agent. Replace your_password with your preferred password. (It’s recommended to choose a password that is different from the MariaDB/MySQL root password.)

create user 'amplify-agent'@'localhost' identified by 'your_password';

Exit MariaDB/MySQL server.

exit;

Next, we need to enable the Amplify MariaDB/MySQL plugin in the Amplify configuariton file.

sudo nano /etc/amplify-agent/agent.conf

In the [extensions] section, you can see that mysql plugin is disabled by default.

[extensions]
phpfpm = True
mysql = False

Change False to True to enable this plugin.

[extensions]
phpfpm = True
mysql = True

Then in [mysql] section, find the following line.

password = amplify-agent

Replace the default password with the password you set for the amplify-agent user. Save and close the file. Then restart Amplify agent for the changes to take effect.

sudo systemctl restart amplify-agent

Now Amplify agent starts collecting MariaDB/MySQL metrics.

nginx amplify mariadb performance monitoring

How to Enable MariaDB Slow Query Log

The slow query log can show you which queries are taking long time to give a response. It’s an important tool for optimizing MariaDB performance. To enable the MariaDB slow query log, edit the server configuration file.

sudo nano /etc/mysql/mariadb.conf.d/50-server.cnf

Find the following 4 lines:

#slow_query_log_file = /var/log/mysql/mariadb-slow.log
#long_query_time = 10
#log_slow_verbosity = query_plan,explain
#log-queries-not-using-indexes

Remove the # symbol and change the value of long_query_time to a low number like 1 second. (You can specify a smaller value like 0.5.)

slow_query_log_file = /var/log/mysql/mariadb-slow.log
long_query_time = 1
log_slow_verbosity = query_plan,explain
log-queries-not-using-indexes

Also, add the following line to enable the slow query log.

slow_query_log = 1

Save and close the file. Restart MariaDB for the changes to take effect.

sudo systemctl restart mariadb

or

sudo systemctl restart mysql

Now Nginx Amplify can show you slow queries in the graphs.

nginx amplify mariadb slow query log

If your MySQL/MariaDB is slow, the first thing you need to do is to check QPS (Queries per second), which is a metric that monitors the load on the database server. As you can see from the following screenshot, one of my MariaDB database servers is under heavy load (around 1000 queries per second).

mysql queries per second

You can also check QPS from command line.

sudo mysqladmin status

Sample output:

Uptime: 405276 Threads: 11 Questions: 389965946 Slow queries: 4536289 Opens: 103692 Flush tables: 3 Open tables: 3999 Queries per second avg: 962.223

Enable Slow Query Log in MySQL 8

If you use MySQL 8 instead of MariaDB, then edit the server config file.

sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf

Add the following lines at the end of this file.

slow_query_log_file = /var/log/mysql/slow-query.log
long_query_time = 1
log-queries-not-using-indexes = ON
slow_query_log = 1

Save and close the file. Restart MySQL server for the changes to take efffect.

sudo systemctl restart mysql

How to Monitor PHP-FPM

The Amplify agent PHP-FPM plugin is enabled by default, but we need to enable PHP-FPM status in order to collect metrics. Edit the PHP-FPM pool configuration file.

Ubuntu PHP7.4

sudo nano /etc/php/7.4/fpm/pool.d/www.conf

Find the following line and remove the semicolon to enable PHP-FPM status.

;pm.status_path = /status

It’s also a good idea to enable the PHP slow log, which can show you which requests are taking too long. Find the following line:

;slowlog = log/$pool.log.slow

Remove the semicolon and set the log path.

slowlog = /var/log/php-fpm/slow.log

Then find the following line.

;request_slowlog_timeout = 0

Remove the semicolon and set the timeout seconds like 1s.

request_slowlog_timeout = 1s

Save and close the file. Create a directory for PHP-FPM logs.

sudo mkdir -p /var/log/php-fpm/

Restart PHP-FPM for the changes to take effect.

sudo systemctl restart php7.4-fpm

Now Amplify agent starts collecting PHP-FPM metrics including the slow requests log.

nginx amplify php-fpm monitoring

Note that if you install a new version of PHP-FPM, like PHP8.0-FPM, you need to configure this again in /etc/php/8.0/fpm/pool.d/www.conf file.

The Amplify Overview Page

On the overview page, you can see the overall application health score and 5 key metrics for your system.

  • Total requests
  • HTTP 5xx errors
  • Request time (You need to create custom log format as described earlier to show this metric.)
  • How much bandwidth Nginx uses
  • CPU usage

Nginx amplify overview page

The application health score is affected by the number of HTTP 5xx errors. If there’s no HTTP 5xx errors, then the score will be 100%.

If you click the server icon, you can see how much CPU and RAM has been used on your system and how much Nginx is using.

nginx amplify ubuntu system metrics

Graphs

In the Graphs page, you can see predefined system, Nginx, MariaDB/MySQL and PHP-FPM metrics.

nginx amplify graphs

Nginx Amplify Dashboards

Amplify dashboards allow you to create custom graphs and charts and group them on one page. For example, you can assemble all performance related graphs on one page, or maybe you want to display all metric for a particular URL.

nginx amplify dashboard

If you have configured Nginx FastCGI cache, you can also create graphs for Nginx cache hit ratio and cache expiration in the dashboard.

Analyzer

One of the useful tools in the analyzer is static analysis. Amplify can parse your Nginx configuration file and make performance, security and reliability recommendations. You can use it to identify mistakes and improve configurations.

nginx amplify static analysis

As you can see, there are 5 warnings for my Nginx server, so I click the Open link to see how to adjust my Nginx configurations.

nginx amplify config analysis

After making the adjustments, wait a few minutes and the warning will be gone.

Alerts

By default, if the Amplify agent stops working, an alert will be sent to your email address. On the alerts page, you can set various alerts for your server. For example, I can tell Amplify to send me an email if the upstream response time is above 1 second for the past 2 minutes.

Nginx amplify alert

If your server runs of out RAM, then it could kill MySQL/MariaDB process. You can create an alert rule to monitor the MySQL/MariaDB uptime like below. If the uptime is below 120 seconds over the past 2 minutes, then send an email notification.

nginx amplify mysql.global.uptime

You may also want to receive alert email when there’s a 5xx error on your server.

nginx amplify alert nginx.http.status.5xx

Amplify Agent Stops Working

Often I found that Amplify agent stops sending performance metrics for unknown reason. I create a Cron job to restart Amplify-agent once a day to solve this problem.

sudo crontab -e

Add the following line.

@daily systemctl restart amplify-agent

MariaDB Monitoring with Galera Cluster

It seems the Amplify agent will stop sending MariaDB metrics if its in a Galera cluster.

Linux Server Performance Monitoring

Nginx Amplify is useful for monitoring Nginx, MySQL/MariaDB, and PHP performance, but how about the performance of the underlying hardware/operating system? How do you know if your disks are slow? You can use Netdata to peek into those details.

Conclusion

I hope this article helped you install and use Nginx Amplify to monitor performance of Nginx or LEMP stack applications on Ubuntu 18.04 or 20.04. You might also want to read:

As always, if you found this post useful, then subscribe to our free newsletter to get more tips and tricks. Take care 🙂

Rate this tutorial
[Total: 4 Average: 5]

7 Responses to “LEMP Stack Performance Monitoring With Nginx Amplify on Ubuntu 20.04/18.04

  • Rupeshpanneerselvam
    5 years ago

    nice article..

  • vanhussen
    5 years ago

    If I have more than one application under /etc/nginx/conf.d/
    so I must add:

    access_log /var/log/nginx/linuxbabe.com.access.log apm;
    error_log /var/log/nginx/linuxbabe.com.error.log warn;

    for all my *.conf file?

    becouse my file in conf.d is:

    -rw-r--r--  1 www-data www-data 487354 Oct  8 21:23 nextcloud.access
    -rw-r--r--  1 www-data www-data      0 Oct  8 21:00 nextcloud.access.log
    -rw-r--r--  1 www-data www-data   1422 Oct  8 21:13 nextcloud.error
    -rw-r--r--  1 www-data www-data      0 Oct  8 21:00 nextcloud.error.log
    -rw-r--r--  1 www-data www-data 146520 Oct  8 18:35 passbolt.access
    -rw-r--r--  1 www-data www-data 212049 Oct  8 18:35 passbolt.error
    -rw-r--r--  1 www-data www-data   5972 Oct  8 16:20 shlink.access
    -rw-r--r--  1 www-data www-data      0 Oct  8 16:01 shlink.error

    Please answer my questions. Thank you

    • Xiao Guoan (Admin)
      5 years ago

      Yes, you need to enable access_log and error_log for in all .conf files under /etc/nginx/conf.d/. The access log format should be apm;

  • Vasily Podguzov
    4 years ago

    Thanks a lot for the great article, and although I used Amplify during long time. Some features made by author with love. I mean don’t write localhost in log, right alert on unresponsive agent and of course percentage of the health, caused 50x errors. I’ve really thought that it caused by access speed. Also wanna try fastcgi cache analysis. It will be nice if developers can add request statistics by unique IP and may be for WAF. Thanks.

  • As of MariaDB 10.6.10, this tutorial will not fully work.
    @Linuxbabe, you might want to add these changes to your tutorial for Ubuntu Users using the latest version of MariaDb and not myqsl.

    For those running MariaDB (not mysql) version 10.6.10, on Ubuntu server, you need to make some changes for Amplify to pickup MariaDB, otherwise, it won’t work with MariaDB.

    First run:

    systemctl status mariadb
    

    which should return something with this line…

    Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
    

    This line shows the location of the service file at

    /lib/systemd/system/mariadb.service

    Open that file in a text editor.

    sudo nano /lib/systemd/system/mariadb.service
    

    Find the line starting with…

    ExecStart=
    

    and change the path pointing to ‘mariadb’ to ‘mysqld’ which is

    /usr/sbin/mysqld

    .

    So your full

    ExecStart=

    line in

    /lib/systemd/system/mariadb.service

    should look like the following…

    ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION
    

    Save and close the file, then run the following commands:

    sudo systemctl daemon-reload
    sudo systemctl restart mariadb
    sudo systemctl restart amplify-agent
    

    NOW, your MariaDB should now be up and running in Amplify.

    • There is a problem with this configuration. Once MariaDB updates, it overwrites the manual changes you made with these above instructions i have written. After some time digging through github posts and stack exchange posts, I have found the proper solution to get these changes to MariaDB to be permanent even after an update or upgrade. I have written the answer in a short tutorial here:

      www.danran.rocks/2023/06/the-trick-to-monitoring-maria-database-mariadb-mysql-in-nginx-amplify-on-ubuntu-server-20-04-for-raspberry-pi/
      

      Hopefully this helps others!

  • On Ubuntu 20.04, if you are using Modsecurity with Amplify, You need to add some Modsecurity rule exclusions for Amplify to properly collect all metrics. Here are my modsecurity rule exclusions to get Amplify fully working:

    # Allow AMPLIFY AGENT Supervisor and Bridge_Manager (receiver.amplify.nginx.com) to communicate with Amplify Web Page.
    # 2024-02-10: Leave this uncommented to allow amplify metrics to work.
    SecRule REQUEST_URI "@streq /nginx_status" \
        "id:1050,\
        phase:1,\
        pass,\
        nolog,\
        ctl:ruleRemoveById=920350,\
        ctl:ruleRemoveById=920300,\
        ctl:ruleRemoveById=949110"
    #    ctl:ruleRemoveById=920450,\
    #    ctl:ruleRemoveById=949110"
    

    Hopefully this helps users using Modsecurity with Amplify. Thanks for the tutorials Linuxbabe!

Leave a Comment

  • Comments with links are moderated by admin before published.
  • Your email address will not be published.
  • Use <pre> ... </pre> HTML tag to quote the output from your terminal/console.
  • Please use the community (https://community.linuxbabe.com) for questions unrelated to this article.
  • I don't have time to answer every question. Making a donation would incentivize me to spend more time answering questions.

The maximum upload file size: 2 MB. You can upload: image. Links to YouTube, Facebook, Twitter and other services inserted in the comment text will be automatically embedded. Drop file here