Run Apache, Nginx & HAProxy on Same Server (Debian, Ubuntu, CentOS)
If you’re a server admin, you probably have a web server of your choice like Apache, or Nginx. Apache is a well-known web server since the 1990s. Nginx was first developed in 2004 and quickly gained traction due to its lightweight memory footprint and fast processing speed for static HTML files.
Both Apache and Nginx support virtual hosting, which means you can host multiple websites or web applications on the same server. However, you will encounter situations in which you have an existing web server running, but a particular web application requires using a different web server. Port 80 or 443 on the public IP address can be used by only one process. If Apache is using the port, then Nginx can’t use (or bind to) it. So what can you do?
You can configure Nginx as a reverse proxy to Apache, so Nginx can redirect HTTP requests to Apache. In my experience, I found that this isn’t always the best way because it has once caused weird problems that I cannot troubleshoot. Instead, I prefer to use HAProxy as a reverse proxy for both Nginx and Apache. HAProxy is a free, open-source high availability load balancer and proxy server for TCP and HTTP-based applications.
Run Apache, Nginx and HAProxy on the Same Server
Here’s how it works.
- Nginx listens on 127.0.0.1:80 and 127.0.0.1:443
- Apache listens on 127.0.0.2:80 and 127.0.0.2:443
- HAProxy listens on port 80 and 443 of the public IP address. It redirects HTTP request on port 80 to port 443. When a request arrives on port 443, it will choose between Nginx and Apache back end by analyzing the SNI (server name indication) header in the HTTPS request.
Actually, Cloudflare (a CDN provider) is also using the SNI header to determine how to route HTTPS requests to origin servers.
Step 1: Stop Nginx and Apache
To stop Nginx on Debian, Ubuntu and CentOS, run
sudo systemctl stop nginx
To stop Apache on Debian/Ubuntu, run
sudo systemctl stop apache2
To stop Apache on CentOS, run
sudo systemctl stop httpd
Step 2: Change the Listen Port in Nginx
We need to make Nginx listen on 127.0.0.1:80. Open your Nginx configuration files in /etc/nginx/conf.d/ or /etc/nginx/sites-enabled/ and find the following line.
listen 80;
Change it to
listen 127.0.0.1:80;
If https is enabled on the Nginx server block, then also find
listen 443 ssl;
And change it to
listen 127.0.0.1:443 ssl;
The Nginx main configuration file /etc/nginx/nginx.conf might include a default virtual host listening on port 80 or 443, so you might need to edit this file too.
Restart Nginx for the changes to take effect.
sudo systemctl restart nginx
Step 3: Change the Listen Port in Apache
We need to make Apache listen on 127.0.0.2:80.
Debian/Ubuntu
On Debian and Ubuntu, edit the /etc/apache2/ports.conf file.
sudo nano /etc/apache2/ports.conf
Change
Listen 80 Listen 443
To
Listen 127.0.0.2:80 Listen 127.0.0.2:443
Save and close the file. Also go to /etc/apache2/sites-enabled/ directory, edit the virtual host files. Change
<VirtualHost *:80>
To
<VirtualHost 127.0.0.2:80>
If there are SSL virtual host, then also change
<VirtualHost *:443>
To
<VirtualHost 127.0.0.2:443>
Restart Apache.
sudo systemctl restart apache2
CentOS
On CentOS, edit the /etc/httpd/conf/httpd.conf file.
sudo nano /etc/httpd/conf/httpd.conf
Find
Listen 80
Change it to
Listen 127.0.0.2:80
Save and close the file. Then go to /etc/httpd/conf.d/ directory, edit the virtual host files. Change
<VirtualHost *:80>
To
<VirtualHost 127.0.0.2:80>
If there are SSL virtual host, then also change
<VirtualHost *:443>
To
<VirtualHost 127.0.0.2:443>
In the /etc/httpd/conf.d/ssl.conf file, there’s
Listen 443 https
Change it to:
Listen 127.0.0.2:443 https
Save and close the file. Restart Apache for the changes to take effect.
sudo systemctl restart httpd
Step 4: Configure HAProxy
Install HAProxy on your distro.
Debian/Ubuntu
sudo apt install haproxy
CentOS
sudo dnf install haproxy
Edit HAProxy configuration file.
sudo nano /etc/haproxy/haproxy.cfg
Add the following code snippet at the end of the file, which will make HAPorxy listen on port 80 of the public IP address and redirect HTTP requests on port 80 to port 443. Replace 12.34.56.78 with your server’s public IP address.
frontend http
bind 12.34.56.78:80
mode http
redirect scheme https code 301
Now we also need to add an HTTPS front end.
frontend https
bind 12.34.56.78:443
mode tcp
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
Then define the Nginx and Apache back ends. The check parameter tells HAProxy to perform health checks on the back-end by sending a TCP packet.
backend nginx
mode tcp
option ssl-hello-chk
server nginx 127.0.0.1:443 check
backend apache
mode tcp
option ssl-hello-chk
server apache 127.0.0.2:443 check
You can define a default back end with:
default_backend nginx
We will use SNI header in the HTTPS request to redirect to the correct back end. For example, if Nginx is serving domain1.com and Apache is serving domain2.com, then you add the following two lines. You can also use sub-domains, as long as they are different.
use_backend nginx if { req_ssl_sni -i domain1.com }
use_backend apache if { req_ssl_sni -i domain2.com }
Note that the default_backend and use_backend directives should be placed above the backend definitions.
frontend http
bind 12.34.56.78:80
mode http
redirect scheme https code 301
frontend https
bind 12.34.56.78:443
mode tcp
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
default_backend nginx
use_backend nginx if { req_ssl_sni -i domain1.com }
use_backend apache if { req_ssl_sni -i domain2.com }
backend nginx
mode tcp
option ssl-hello-chk
server nginx 127.0.0.1:443 check
backend apache
mode tcp
option ssl-hello-chk
server apache 127.0.0.2:443 check
In the configuration above, we utilized the SNI (Server Name Indication) feature in TLS to differentiate HTTPS traffic.
- When domain1.com is in the TLS Client Hello, HAProxy redirects traffic to the
nginxbackend. - When domain2.com is in the TLS Client Hello, HAProxy redirects traffic to the
apachebackend.
If the client doesn’t specify the server name in TLS Client Hello, then HAproxy will use the default backend (nginx).
Save and close the file. Then restart HAproxy.
sudo systemctl restart haproxy
Now Apache, Nginx, and HAProxy are able to run on the same server.
How to Forward Client’s IP address to Backend
By default, Apache and Nginx can only see HAProxy’s IP address. To get the client’s real IP address, make sure you added the send-proxy-v2 option in the HAProxy’s back end definition like below.
backend nginx
mode tcp
option ssl-hello-chk
server nginx 127.0.0.1:443 send-proxy-v2 check
backend apache
mode tcp
option ssl-hello-chk
server apache 127.0.0.2:443 send-proxy check
Note: Nginx backend uses the send-proxy-v2 directive; Apache backend uses the send-proxy protocol. Technically they are the same. Apache doesn’t add a version number.
We also need to add some configuration in Nginx and Apache to make it work, otherwise your website will be inaccessible.
Nginx
Add proxy_protocol in the Nginx listen directive like below.
listen 127.0.0.2:443 ssl http2 proxy_protocol;
Then add the following two directives in the Nginx http { } block in /etc/nginx/nginx.conf file.
set_real_ip_from 127.0.0.1; real_ip_header proxy_protocol;
Save and close the file. Then reload Nginx.
sudo systemctl reload nginx
Note: If your website runs behind Cloudflare CDN, then you should change real_ip_header proxy_protocol; to real_ip_header CF-Connecting-IP; in order to show the client’s real IP address. You can also add this real_ip_header directive to the individual server block file to override the global configuration in /etc/nginx/nginx.conf file.
Finally, restart HAProxy.
sudo systemctl restart haproxy
Apache
If you use Apache on Debian/Ubuntu, you need to enable the remoteip module. (This module is enabled on CentOS by default.)
sudo a2enmod remoteip
Then add the following 3 lines in your Apache virtual host configuration file.
RemoteIPProxyProtocol On RemoteIPHeader X-Forwarded-For RemoteIPTrustedProxy 127.0.0.1
Like this.
<VirtualHost 127.0.0.2:443> ServerName www.example.com RemoteIPProxyProtocol On RemoteIPHeader X-Forwarded-For RemoteIPTrustedProxy 127.0.0.1
Note: If you have multiple Apache virtual hosts, then you only need to add the above lines in one virtual host. It will be enabled for all virtual hosts. Don’t add the above lines to all virtual hosts, or you might get the RemoteIPProxyProtocol: Unsupported Command Error.
Save and close the file. Then we also need to change the combined log format. Edit the Apache main configuration file.
sudo nano /etc/apache2/apache2.conf
or
sudo nano /etc/httpd/conf/httpd.conf
Find the following line.
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
Replace it with:
LogFormat "%a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
Save and close the file. Then restart Apache for the changes to take effect.
sudo systemctl restart apache2
or
sudo systemctl restart httpd
Note that the RemoteIPProxyProtocol On directive is only available in Apache 2.4.31 and newer. To check your Apache version, run
sudo apache2 -v
or
sudo httpd -v
Ubuntu 18.04 ships with Apache 2.4.29. If your Apache version doesn’t meet this requirement, then you should remove the send-proxy-v2 in the HAProxy back end definition. CentOS 8 ships with Apache 2.4.37.
Finally, restart HAProxy.
sudo systemctl restart haproxy
How to Add New Virtual Hosts
If you have Apache and Nginx web servers, then you need only two backends: one for Apache and one for Nginx.
When you need to add another domain (or sub-domain), you don’t need to add a new backend in HAProxy. Instead, you should create a new virtual host for the new domain (or sub-domain) in Apache or Nginx, then use the use_backend directive in HAProxy to redirect traffic to the correct backend.
frontend http
bind 12.34.56.78:80
mode http
redirect scheme https code 301
frontend https
bind 12.34.56.78:443
mode tcp
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
default_backend nginx
use_backend nginx if { req_ssl_sni -i domain1.com }
use_backend nginx if { req_ssl_sni -i sub.domain1.com }
use_backend apache if { req_ssl_sni -i domain2.com }
use_backend apache if { req_ssl_sni -i sub.domain2.com }
backend nginx
mode tcp
option ssl-hello-chk
server nginx 127.0.0.1:443 check
backend apache
mode tcp
option ssl-hello-chk
server apache 127.0.0.2:443 check
I recommend using apache and nginx as the backend names like in the above code, so you know which web server is hosting which application.
Obtaining New Let’s Encrypt SSL Certificate
http-01 challenge
First, you need to understand how to use the Certbot apache and nginx authenticator to obtain TLS certificate.
- Properly Enable HTTPS on Apache with Let’s Encrypt on Ubuntu
- Properly Enable HTTPS on Nginx with Let’s Encrypt on Ubuntu
The Certbot apache and nginx authenticator use http-01 challenge, which works on TCP port 80. Because HAProxy is using TCP port 80, so the Certbot http-01 challenge is likely to fail. However, you can make it work with the following method.
- Enable SSL host in Apache or Nginx with a self-signed certificate or a certificate for another domain.
- Then use the Certbot apache and nginx authenticator as usual.
This method works because the Certbot http-01 challenge can be redirected from TCP port 80 to TCP port 443, and Certbot accepts invalid certificates on port 443.
dns-01 challenge
You can also use the certbot dns-01 challenge, which works by creating a temporary TXT record for your domain to certify that you actually own this domain, so it can bypass TCP port 80 and TCP port 443. The disadvantage of this method is that not all domain registrars are supported.
First, you need to install Certbot DNS plugin. They are several DNS plugins available in the Debian and Ubuntu software repository, which you can find with
apt search python3-certbot-dns
Output:
python3-certbot-dns-cloudflare/bionic,bionic 0.23.0-1 all Cloudflare DNS plugin for Certbot python3-certbot-dns-digitalocean/bionic,bionic 0.23.0-1 all DigitalOcean DNS plugin for Certbot python3-certbot-dns-dnsimple/bionic,bionic 0.23.0-1 all DNSimple DNS plugin for Certbot python3-certbot-dns-google/bionic,bionic 0.23.0-1 all Google DNS plugin for Certbot python3-certbot-dns-rfc2136/bionic,bionic 0.23.0-1 all RFC 2136 DNS plugin for Certbot python3-certbot-dns-route53/bionic,bionic 0.23.0-1 all Route53 DNS plugin for Certbot
On CentOS 8, you can find Certbot DNS plugins from the EPEL repository. Note that you need to install Certbot from the default CentOS repository.
sudo dnf install certbot
Find DNS plugins from the EPEL repository.
sudo dnf install epel-release dnf search certbot-dns
Output:
python3-certbot-dns-ovh.noarch : OVH DNS Authenticator plugin for Certbot python3-certbot-dns-nsone.noarch : NS1 DNS Authenticator plugin for Certbot python3-certbot-dns-gehirn.noarch : Gehirn Infrastructure Service DNS Authenticator plugin for Certbot python3-certbot-dns-google.noarch : Google Cloud DNS Authenticator plugin for Certbot python3-certbot-dns-linode.noarch : Linode DNS Authenticator plugin for Certbot python3-certbot-dns-luadns.noarch : LuaDNS Authenticator plugin for Certbot python3-certbot-dns-rfc2136.noarch : RFC 2136 DNS Authenticator plugin for Certbot python3-certbot-dns-route53.noarch : Route53 DNS Authenticator plugin for Certbot python3-certbot-dns-cloudxns.noarch : CloudXNS DNS Authenticator plugin for Certbot python3-certbot-dns-dnsimple.noarch : DNSimple DNS Authenticator plugin for Certbot python3-certbot-dns-cloudflare.noarch : Cloudflare DNS Authenticator plugin for Certbot python3-certbot-dns-cloudflare.noarch : Cloudflare DNS Authenticator plugin for Certbot python3-certbot-dns-dnsmadeeasy.noarch : DNS Made Easy DNS Authenticator plugin for Certbot python3-certbot-dns-sakuracloud.noarch : Sakura Cloud DNS Authenticator plugin for Certbot
Install one of these plugins according to which DNS hosting service you are using. I’m using Cloudflare, so I will use Cloudflare as an example. Run the following command to install the python3-certbot-dns-cloudflare package on Debian/Ubuntu.
sudo apt install python3-certbot-dns-cloudflare
Then create a configuration file for Cloudflare.
sudo nano /etc/letsencrypt/cloudflare.ini
We need to add our Cloudflare account email address and API key in this file.
# Cloudflare API credentials used by Certbot dns_cloudflare_email = [email protected] dns_cloudflare_api_key = 0123456789abcdef0123456789abcdef01234567
You can find your Cloudflare API key at https://dash.cloudflare.com/profile. Note that the Certbot Cloudflare plugin does not currently support Cloudflare’s “API Tokens”, so ensure you use the “Global API Key” for authentication.
Save and close the file. The API key bypasses two-factor authentication of Cloudflare, so you should only allow root user to read this file.
sudo chmod 600 /etc/letsencrypt/cloudflare.ini
Now run certbot.
sudo certbot --agree-tos -a dns-cloudflare -i nginx --redirect --hsts --staple-ocsp --email [email protected] -d www.your-domain.com,your-domain.com
In the above command, we specified that we will use dns-cloudflare as the authenticator to obtain new TLS certificate and use the nginx plugin to create the HTTPS server block. If you use Apache, then replace nginx with apache.
This command will ask you to enter the path of the .ini file, so enter /etc/letsencrypt/cloudflare.ini and press the Enter key.
After the SSL certificate is obtained and installed in the web server configuration file. You need to make it listen on 127.0.0.1:443 or 127.0.0.2:443. For example, if you use Nginx, then find
listen 443 ssl
And change it to
listen 127.0.0.1:443 ssl http2
Also, change port 80 to
listen 127.0.0.1:80;
Save and close the file. Then restart your web server.
Common Errors
1 Wrong Domain Name
If you have multiple Nginx virtual hosts on a server, and when you type one domain name, the web browser takes you to another domain name hosted on the same server, it could be that one of your Nginx virtual host file name doesn’t end with the .conf file name extension, so Nginx didn’t load this virtual host. It also could be that you have set AAAA record for the domain name, but you didn’t configure Nginx to serve the domain name in IPv6.
The above also applies to Apache web server.
2 Network Protocol Violation
If you set the send-proxy-v2 header in HAProxy, but didn’t enable proxy_protocol in Nginx or Apache, then your website will display the following error.
The site has experienced a network protocol violation that cannot be repaired.
Remember to restart Nginx/Apache after making changes to configuration file.
3 PR_CONNECT_RESET_ERROR
If you enable proxy_protocol in Nginx or Apache, but didn’t set the send-proxy-v2 header in HAProxy, then your website will display this error.
secure connection failed. PR_CONNECT_RESET_ERROR
And you will find the following error message in Nginx or Apache error log.
broken header while reading PROXY protocol
Remember to restart Nginx/Apache after making changes to configuration file.
Also note that if you enable proxy protocol in one Apache/Nginx virtual host file, then it’s also implicitly enabled for all other virtual hosts. And I haven’t found a way to disable proxy_protocol for individual virtual host. Since you are running Apache and Nginx on the same server, a workaround is that you only enable proxy protocol in Nginx and don’t enable it in Apache. If a virtual host don’t need to get the client’s real IP address, configure it to use Apache.
Update: It occurred to me that you can set up a new Apache/Nginx virtual host on another loopback address like 127.0.0.4. In this way, the new virtual host won’t enable proxy protocol.
4 PR_END_OF_FILE_ERROR
If you enabled proxy protocol in Apache/Nginx, but you try to access the virtual host directly (bypassing HAProxy), then you will see the following error.
PR_END_OF_FILE_ERROR
5 SSL_ERROR_SYSCALL
If you use the curl command:
curl -I https://example.com
And you encounter the following error
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to example.com:443
It’s probably because you didn’t set backend definition for this domain.
6 backend ‘apache’ has no server available!
If you see this error in HAProxy log (/var/log/haproxy.log), then you need to check your HAProxy backend definition.
backend apache
mode tcp
option ssl-hello-chk
server apache 127.0.0.2:443 send-proxy-v2 check
You can remove the check option to fix this error.
backend apache
mode tcp
option ssl-hello-chk
server apache 127.0.0.2:443 send-proxy-v2
Sometimes you need to include the check option, but it can also create problems. That’s a bit weird.
Wrapping Up
I hope this tutorial helped you run Apache, Nginx and HAProxy on the same server. As always, if you found this post useful, then subscribe to our free newsletter to get more tips and tricks. Take care 🙂
Thank you. Most useful tutorial.
I want to send all internet request through one pc in house. Are there any possible way to do
Hi,
Thanks a lot,
Very useful article
Thank you very much, this helped me out a lot!
Is it possible to apply this to IPv6 the same way? Could you tell the steps necessary to do that?
1. Configure HAProxy to listen on port 80 and 443 of the public IPv6 address.
frontend http bind 12.34.56.78:80 bind [2607:f8b0:4006:810::200e]:80 mode http redirect scheme https code 301 frontend https bind 12.34.56.78:443 bind [2607:f8b0:4006:810::200e]:443 mode tcp tcp-request inspect-delay 5s tcp-request content accept if { req_ssl_hello_type 1 }2. If your Nginx listens on port 80 or 443 of the public IPv6 address, configure Nginx to listen on port 443 of the IPv6 loopback addresses
::1.In you Nginx virutal hosts, change
listen [::]:443tolisten [::1]:443. You can go to your Nginx virtual host directory (/etc/nginx/cond.d/ or /etc/nginx/sites-enabled/) and run the following command to replace them.Also change
listen [::]:80tolisten [::1]:80Test Nginx configuration.
Then restart Nginx.
::1is the only one IPv6 loopback address, so we can’t configure Apache to listen on::2(You can’t ping::2). However, HAProxy can listen to requests on IPv6 interface and forward requests to IPv4 backends, so simply restart HAProxy and you are done.Thanks a lot! This works pretty well 🙂
I’m giving this a try now on Digital Ocean.
I tried earlier and failed with a redirect loop so I’m going to start fresh on a test droplet to track what I’m doing wrong on a test server. Already reverted everything back.
The base will be the 22.04 default Haproxy in front of Apache2 and Nginx
I’ll try and share how to get Nginx hosting an instance of the ghost blogging software with this configuration type as well. That might add a wrinkle because it’s a JavaScript webserver.
This article helped me a lot.
Thank you for sharing.