LEMP Stack Performance Monitoring with Nginx Amplify on Ubuntu 18.04/16.04

Nginx Amplify is a free, flexible and powerful way to analyze the load and performance of Nginx and Nginx Plus web servers. It’s easy to set up and use. Nginx Amplify also comes with MySQL/MariaDB, PHP-FPM plugins so you can monitor a full LEMP stack application. (LEMP stands for Linux, Nginx, MySQL/MariaDB, PHP.)

Nginx Amplify is developed by Nginx, Inc, the company behind Nginx web server. It’s a SaaS based solution. You install the Nginx Amplify agent on your server and it will collect and send metrics to the SaaS service. The Nginx Amplify agent is open-source and lightweight. It collects many metrics, including:

  • System metrics (CPU, RAM usage, network traffic, disk usage, disk I/O, disk latency, etc)
  • Nginx metrics (connections, requests, HTTP status, response time, traffic, and more)
  • MySQL/MariaDB metrics (connections, select queries, insert queries, update queries, slow queries, etc)
  • PHP-FPM metrics (connections, connections queue, slow requests, etc)

Nginx Amplify can also

  • Use static analyzer to help you improve Nginx configuration.
  • Alert you of abnormal behavior

Installing Nginx Amplify on Your Ubuntu 18.04/16.04 Linux Server

The Nginx Amplify agent is open-source and there’s an install script that you can use to install Nginx Amplify on the following Linux Distributions:

  • Debian 8, Debian 9
  • Ubuntu 16.04, Ubuntu 18.04
  • RHEL/CentOS 6, RHEL/CentOS 7

First, register an account at https://amplify.nginx.com

nginx amplify setup

Verify your email address. Next, you will need to install the Nginx Amplify Agent on your Linux server. SSH into your server and login as root. The Nginx Amplify website is very nice to give the commands you need to run to install Nginx Amplify, so just copy and run these commands on your server.

Note that you need to run these commands as root.

nginx amplify install script

Download the install script.

install nginx amplify

Run the install script. (Each Amplify account has a unique API key.)

nginx amplify ubuntu 18.04

Once it’s installed, the agent will automatically start. You can check its status with:

systemctl status amplify-agent

Output:

 amplify-agent.service - LSB: Stop/start nginx-amplify-agent
   Loaded: loaded (/etc/init.d/amplify-agent; generated)
   Active: active (running) since Wed 2019-02-27 03:37:05 UTC; 1min 17s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 9306 ExecStop=/etc/init.d/amplify-agent stop (code=exited, status=0/SU
  Process: 9341 ExecStart=/etc/init.d/amplify-agent start (code=exited, status=0/
    Tasks: 3 (limit: 2361)
   CGroup: /system.slice/amplify-agent.service
           └─9407 amplify-agent

Hint: press Q to gain back control of the terminal after running the above command.

To enable auto start at boot time, run

sudo systemctl enable amplify-agent

Configuring Stub_status in Nginx

Amplify agent can collect system metrics out-of-the-box. In order to collect Nginx metrics, you need to configure Nginx stub_status. Open the nginx.conf file.

sudo nano /etc/nginx/nginx.conf

Add the following server block in the http {...} context.

server {
     listen 127.0.0.1:80;
     server_name 127.0.0.1;
     location /nginx_status {
        stub_status on;
	allow 127.0.0.1;
	deny all;
     }
}

Save and close the file. Then reload Nginx for the changes to take effect.

sudo systemctl reload nginx

Now Amplify agent can start collecting Nginx metrics.

nginx amplify upstream response time

Creating Custom Nginx Log Format

Nginx Amplify agent can also collect information from Nginx log files. However, the default log format gives us limited information. If you really care about application performance, you should create a custom log format to show the request time, upstream response time, cache hits, etc.

Open the nginx.conf file.

sudo nano /etc/nginx/nginx.conf

Add the following lines in the http {...} context, above the include directives. Here we are creating a custom Nginx log format called apm (application performance monitoring) that includes information about request_time, upstream_response_time, upstream_connect_time and upstream_header_time. All are measured in seconds with millisecond resolution.

log_format apm '"$time_local" client=$remote_addr '
               'method=$request_method request="$request" '
               'request_length=$request_length '
               'status=$status bytes_sent=$bytes_sent '
               'body_bytes_sent=$body_bytes_sent '
               'referer=$http_referer '
               'user_agent="$http_user_agent" '
               'upstream_addr=$upstream_addr '
               'upstream_status=$upstream_status '
               'request_time=$request_time '
               'upstream_cache_status="$upstream_cache_status" '
               'upstream_response_time=$upstream_response_time '
               'upstream_connect_time=$upstream_connect_time '
               'upstream_header_time=$upstream_header_time';

Save and close file. Then open your Nginx virtual host configuration file. For example,

sudo nano /etc/nginx/conf.d/linuxbabe.com.conf

Your Nginx virtual host configuation file might be under /etc/nginx/sites-enabled/ directory. I like to use the /etc/nginx/conf.d/ directory.

In the server {...} context, add the following two lines to enable access log and error log. The access log is using the apm format and error log uses the warn log level.

access_log /var/log/nginx/linuxbabe.com.access.log apm;
error_log /var/log/nginx/linuxbabe.com.error.log warn;

Save and close the file. Then reload Nginx.

sudo systemctl reload nginx

Now I can see the upstream response time and other performance related metrics in the Amplify graphs. A good upstream response time is below 0.5 seconds. Otherwise your visitors can feel your website is slow. If you can tune your server to get upstream response time below 0.2 seconds, your website will be very snappy.

nginx amplify performance monitoring

If Amplify agent isn’t reporting Nginx metrics, check that Amplify agent runs as the same user as the Nginx worker process. You may also need to check if there’s a new version of Amplify-agent available.

How to Exclude Your Own IP Address in Nginx Access Log

Sometimes, your activity can skew the Nginx metrics. You can exclude your own IP address in the access log to avoid that. First, add the following lines in the http context of Nginx configuration file.  Replace 12.34.56.78 with your own IP address. If the HTTP request comes from your own IP address, Nginx will set the value of $log_ip variable to 0.

map $remote_addr $log_ip {
    "12.34.56.78" 0;
    default 1;
}

Then change the access_log directive as follows.

access_log /var/log/nginx/linuxbabe.com.access.log apm if=$log_ip;

Save and close the file. Then text nginx configuration and reload.

sudo nginx -t
sudo systemctl reload nginx

How to Monitor MariaDB/MySQL Database Server

First, we need to create a user in MariaDB/MySQL to collect metrics. Log into MariaDB/MySQL monitor.

sudo mysql -u root

Create a new user for Amplify agent. Replace your_password with your preferred password. (It’s recommended to choose a password that is different from the MariaDB/MySQL root password.)

create user 'amplify-agent'@'localhost' identified by 'your_password';

Exit MariaDB/MySQL server.

exit;

Next, we need to enable the Amplify MariaDB/MySQL plugin in the Amplify configuariton file.

sudo nano /etc/amplify-agent/agent.conf

In the [extensions] section, you can see that mysql plugin is disabled by default.

[extensions]
phpfpm = True
mysql = False

Change False to True to enable this plugin.

[extensions]
phpfpm = True
mysql = True

Then in [mysql] section, find the following line.

password = amplify-agent

Replace the default password with the password you set for the amplify-agent user. Save and close the file. Then restart Amplify agent for the changes to take effect.

sudo systemctl restart amplify-agent

Now Amplify agent starts collecting MariaDB/MySQL metrics.

nginx amplify mariadb performance monitoring

How to Enable MariaDB Slow Query Log

The slow query log can show you which queries are taking long time to give a response. It’s an important tool for optimizing MariaDB performance. To enable the MariaDB slow query log, edit the server configuration file.

sudo nano /etc/mysql/mariadb.conf.d/50-server.cnf

Find the following 3 lines:

#slow_query_log_file = /var/log/mysql/mariadb-slow.log
#long_query_time = 10
#log-queries-not-using-indexes

Remove the # symbol and change the value of long_query_time to a low number like 1 second. (You can specify a smaller value like 0.5.)

slow_query_log_file = /var/log/mysql/mariadb-slow.log
long_query_time = 1
log-queries-not-using-indexes

Also, add the following line to enable the slow query log.

slow_query_log = 1

Save and close the file. Restart MariaDB for the changes to take effect.

sudo systemctl restart mariadb

or

sudo systemctl restart mysql

Now Nginx Amplify can show you slow queries in the graphs.

nginx amplify mariadb slow query log

How to Monitor PHP-FPM

The Amplify agent PHP-FPM plugin is enabled by default, but we need to enable PHP-FPM status in order to collect metrics. Edit the PHP-FPM pool configuration file.

Ubuntu PHP7.2

sudo nano /etc/php/7.2/fpm/pool.d/www.conf

Find the following line and remove the semicolon to enable PHP-FPM status.

;pm.status_path = /status

It’s also a good idea to enable the PHP slow log, which can show you which requests are taking too long. Find the following line:

;slowlog = log/$pool.log.slow

Remove the semicolon and set the log path.

slowlog = /var/log/php-fpm/slow.log

Then find the following line.

;request_slowlog_timeout = 0

Remove the semicolon and set the timeout seconds like 1s.

request_slowlog_timeout = 1s

Save and close the file. Create a directory for PHP-FPM logs.

sudo mkdir /var/log/php-fpm/

Restart PHP-FPM for the changes to take effect.

sudo systemctl restart php7.2-fpm

Now Amplify agent starts collecting PHP-FPM metrics including the slow requests log.

nginx amplify php-fpm monitoring

Note that if you install a new version of PHP-FPM, like PHP7.3-FPM, you need to configure this again in /etc/php/7.3/fpm/pool.d/www.conf file.

The Amplify Overview Page

On the overview page, you can see the overall application health score and 5 key metrics for your system.

  • Total requests
  • HTTP 5xx errors
  • Request time (You need to create custom log format as described earlier to show this metric.)
  • How much bandwidth Nginx uses
  • CPU usage

Nginx amplify overview page

The application health score is affected by the number of HTTP 5xx errors. If there’s no HTTP 5xx errors, then the score will be 100%.

If you click the server icon, you can see how much CPU and RAM has been used on your system and how much Nginx is using.

nginx amplify ubuntu system metrics

Graphs

In the Graphs page, you can see predefined system, Nginx, MariaDB/MySQL and PHP-FPM metrics.

nginx amplify graphs

Nginx Amplify Dashboards

Amplify dashboards allow you to create custom graphs and charts and group them on one page. For example, you can assemble all performance related graphs on one page, or maybe you want to display all metric for a particular URL.

nginx amplify dashboard

If you have configured Nginx FastCGI cache, you can also create graphs for Nginx cache hit ratio and cache expiration in the dashboard.

Analyzer

One of the useful tools in the analyzer is static analysis. Amplify can parse your Nginx configuration file and make performance, security and reliability recommendations. You can use it to identify mistakes and improve configurations.

nginx amplify static analysis

As you can see, there are 5 warnings for my Nginx server, so I click the Open link to see how to adjust my Nginx configurations.

nginx amplify config analysis

After making the adjustments, wait a few minutes and the warning will be gone.

Alerts

By default, if the Amplify agent stops working, an alert will be sent to your email address. In the alerts page, you can set various alerts for your server. For example, I can tell Amplify to send me an email if the upstream response time is above 1 second for the past 2 minutes.

Nginx amplify alert

Amplify Agent Stops Working

Often I found that Amplify agent stops sending performance metrics for unknown reason. I create a Cron job to restart Amplify-agent once a day to solve this problem.

sudo crontab -e

Add the following line.

@daily systemctl restart amplify-agent

Conclusion

I hope this article helped you install and use Nginx Amplify to monitor performance of Nginx or LEMP stack applications on Ubuntu 18.04 or 16.04. As always, if you found this post useful, then subscribe to our free newsletter to get more tips and tricks. Take care 🙂

Rate this tutorial
[Total: 3 Average: 5]

3 Responses to “LEMP Stack Performance Monitoring with Nginx Amplify on Ubuntu 18.04/16.04

  • Rupeshpanneerselvam
    5 months ago

    nice article..

  • vanhussen
    1 month ago

    If I have more than one application under /etc/nginx/conf.d/
    so I must add:

    access_log /var/log/nginx/linuxbabe.com.access.log apm;
    error_log /var/log/nginx/linuxbabe.com.error.log warn;

    for all my *.conf file?

    becouse my file in conf.d is:

    -rw-r--r--  1 www-data www-data 487354 Oct  8 21:23 nextcloud.access
    -rw-r--r--  1 www-data www-data      0 Oct  8 21:00 nextcloud.access.log
    -rw-r--r--  1 www-data www-data   1422 Oct  8 21:13 nextcloud.error
    -rw-r--r--  1 www-data www-data      0 Oct  8 21:00 nextcloud.error.log
    -rw-r--r--  1 www-data www-data 146520 Oct  8 18:35 passbolt.access
    -rw-r--r--  1 www-data www-data 212049 Oct  8 18:35 passbolt.error
    -rw-r--r--  1 www-data www-data   5972 Oct  8 16:20 shlink.access
    -rw-r--r--  1 www-data www-data      0 Oct  8 16:01 shlink.error

    Please answer my questions. Thank you

    • Yes, you need to enable access_log and error_log for in all .conf files under /etc/nginx/conf.d/. The access log format should be apm;

Leave a Comment

  • Comments with links are moderated by admin before published.
  • Your email address will not be published.
  • Use <pre> ... </pre> HTML tag to quote the output from your terminal/console.
  • Please use the community (https://community.linuxbabe.com) for questions unrelated to this article.
  • If you ask me more than 5 questions, I expect you to make a donation, or I would stop answering your questions.