Nginx haproxy and keepalived

  Network
 

In this article I show three different reverse proxy architectures based on nginx for implementing a high availability reverse proxy environment. As load balancer two different solutions are proposed: one with haproxy, an application load balancer; another with keepalived, a layer 3 load balancer.

Before starting, let me explain what is a reverse proxy.

Reverse proxy is a proxy that forward any http request on behalf of a client. It permit to manage different virtual servers that pass the web traffic to backend servers hosted on application server like Tomcat, Jboss, WebSphere, WebLogic, etc.

Every virtual server is recognized from host http header. Example: when browsing http://web-whois.nic.it/, a http request is sent to ip address resolved via DNS:

header http host

The host header permits to reverse proxy to filter the request and to manage as configured: a lot of websites can be managed by one only public ip.

It becomes the contact point between IT and Internet and Its configuration is vital for the IT security.

For all that, the reverse proxy should be located in the DMZ network: reachable from internet but closed to internet and to database network.

After this introduction about reverse proxy meaning, let’s start with the first solution.

Simple Active/Standby Reverse Proxy Architecture

This first simple solution with Nginx and KeepAlived  permit to have a high availability architecture as frontend for all the application servers.

The architecture has been implemented in my laboratory and is showed below:

 

Simple Active/Standby Reverse Proxy Architecture

In this solution there are two nginx configured in active standby for avoiding single point of failure. High Availability is implemented by keepalived daemon: it permit to manage virtual IP in a network. It’s the free response to cisco’s hsrp. In the application server I installed two apache server with simple php application to generate a cookie.

Behind Nginx is present a Firewall: there are different free solution. A simple iptables can be useful for the scope. If you want more sophisticated control you should use more complex firewall like Pfsense that permit to have ips/ids functionality integrated. How to configure the firewall is out of scope of this article.

All the linux systems used in the laboratory are Centos 7 and deployed in a oracle virtualbox environment.

Let’s start the installation with keepalived. On both nginx nodes.

[root@nginx01 ~]# yum install keepalived

The configuration is very simple:

[root@nginx01 ~]# more /etc/keepalived/keepalived.conf
vrrp_script chk_http_port {
script “/usr/bin/killall -0 nginx”
interval 2
weight 10
}
vrrp_instance lb_nginx {
interface enp0s3
state Master
virtual_router_id 111
priority 85
authentication {
auth_type PASS
auth_pass Security
}
track_script {
chk_http_port
}
virtual_ipaddress {
192.168.1.200 dev enp0s3
}
}

For slave node the state is SLAVE and the priority is 80.

The priority of master is 85+10=95. The priority of slave is 80+10=90. If the nginx goes down, the priority of master becames 85+0=85 and the slave becames active. Communication between nodes is in multicast.

This is explained below:

Keep Alived01

Let’s install the nginx Software in both nodes:

[root@nginx01 yum.repos.d]# yum -y install epel-release
[root@nginx01 yum.repos.d]# yum -y install nginx

You can visit official nginx site for more information about nginx’s installation: http://nginx.org/en/docs/install.html.

Nginx can be started in this way.

[root@nginx01 log]# systemctl restart nginx

Now we create our site on both nginx:

[root@nginx01 log]# more /etc/nginx/conf.d/www.example.com
server {
listen 0.0.0.0:80;
server_name www.example.com;
access_log /var/log/nginx/www.example.com_access.log main;
error_log /var/log/nginx/www.example.com_error.log main;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
allow all;
proxy_pass myapp1;
}
}
upstream myapp1 {
ip_hash;
server 192.168.1.113;
server 192.168.1.114;
}

The configuration is simple to understand. All the webtraffic that with the URI starting with ‘/’, it means all, is balanced to two backend servers. The two proxy_set_header add the virtual host and the remote ip as http header in the request that is proxied to upstream server (backend servers). The remote ip is very useful information: it permit to application server to know the remote ip and implement statistical service.

An easy way to maintain affinity between a user and a server is to use user’s IP address: this is called Source IP affinity. It is used in this case. IP’s hash is used to decide the upstream server that must receive the traffic.

Let’s do some test. From nginx we make some HEAD to VIP 192.168.1.200. We put in the /etc/hosts the following map:

192.168.1.200 www.example.com

[root@nginx01 log]# curl -I http://www.example.com

As you can see below the nginx load balances the HEAD request always to app01 node: this why affinity policy is configured with haship directive. The same ip is balanced in the same node. The HEAD’s answers is HTTP 403 but it is not important for our scope.

Curl Nginx simple

You can think that an ip address, by NAT, can hide a lot of clients and this is why source IP affinity is not good method to use  to “stick” a user to a server.

Unfortunately nginx doesn’t have good functionality for load balancing. You should use nginx plus: nginx with commercial modules that implements application load balancing, application health checks, activity monitoring and on-the-fly reconfiguration. This is the reason to use the next solution.

Complex Active/Standby Reverse Proxy Architecture

Load balancing is very important in a reverse proxy and fortunately there is a free solution: haproxy.

Haproxy is application load balancing. The load balancing is at application layer. It means that a new socket is created to upstreams servers. This is not fast like a classical ip load balancing: keepalived or other commercial products like F5 or CSM Module Cisco that load balance the ip packet without creating new sockets. But it provides a lot of features: it’s a good reason to prefer to another free solution like keepalived. We continue to use keepalived for managing the virtual ip address and haproxy for load balancing.

For configuring keepalived as Load Balancer you can read this good article: https://docs.oracle.com/cd/E37670_01/E41138/html/section_xsx_wl2_4r.html.

Let’s continue to install Haproxy and Keepalived on two new nodes.

[root@haproxy01 ~]#yum install haproxy
[root@haproxy01 ~]#yum install keepalived

The keepalived’s configuration is like before. You should change the router ID and the new VIP.

The haproxy’s configuration is:

[root@haproxy01 ~]#vi /etc/haproxy/haproxy.conf
frontend HTTP-Service *:8080
mode http
default_backend HTTP-Service
backend HTTP-Service
balance roundrobin
option tcp-check
cookie JSESSIONID prefix nocache
#Rewrite Header. Add secure Flag on the http header
http-response replace-value Set-Cookie (.*) \1;\ HttpOnly;
http-response replace-value Set-Cookie (.*) \1;\ Secure
server App01 192.168.1.113:80 check port 80 cookie app01 weight 100
server App02 192.168.1.114:80 check port 80 cookie app02 weight 100

The configuration above load balances the traffic to two applications servers. It permit stick session using the JSESSION ID cookie provided from applications. Every cookie is returned to client adding the extension App01~.

A simple tcp-check is implemented: haproxy tries to open a tcp socket to port 80 of applications server. If it works, the application is considered up, otherwise down. The balancing is roundrobin. Two flags are added to Set-Cookie returned from application servers: read it for more information http://resources.infosecinstitute.com/securing-cookies-httponly-secure-flags/. You can know more about haproxy reading the haproxy official site: http://www.haproxy.org/.

The configuration’s nginx must be changed in order to balance only to VIP of haproxy: in our case is 192.168.1.201:8080.

The new architecture implemented in my laboratory is:

Complex Active/Standby Reverse Proxy Architecture

Now we try to reach our application servers:

root@ngin01:/etc# curl -v –silent http://www.example.com/ 2>&1| grep Set-Cookie
< Set-Cookie: JSESSIONID=App01~i12KJBKMJKJ1EKJ21213KJ; HttpOnly; Secure
root@ngin01:/etc# curl -v –silent http://www.example.com/ 2 2>&1| grep Set-Cookie
< Set-Cookie: JSESSIONID=App02~p45KJF23JKJ1EKJ21213KJ; HttpOnly; Secure

As you can see the balancing is round robin; the HttpOnly; Secure has been added to Set-Cookie, and the JSESSIONID has been modified. The next request from browsers will have as starting cookie App01~ or App02~ and haproxy will have all the information for right stick session. The cookie are generated from a simple php application running in apache.

[root@app01 html]# yum install php
[root@app01 html]# systemctl restart httpd

[root@app01 html]# more index.php
<?php
$value = ‘p45KJF23JKJ1EKJ21213KJ’;
setcookie(“JSESSIONID”, $value);
?>

This architecture is scalable and easy to manage. For improving security it’s better to move application server in a new network: they connect to database and they should be more protected.

This solution has a limit: only one nginx server can be used. If we need to have more nginx for managing a lot of virtual server, what kind of architecture should we use? The answer it the next.

Active/Active Reverse Proxy Architecture

This solution adds a load balancers layer for forwarding the traffic to nginx server: It permit to split the traffic to different nginx and to have a solution more scalable.

We could use haproxy load balancer, but the priority in this layer is the performace : it’s better to use faster load balancing. For that reason we move the focus to keepalived.

Keepalived can also be used as virtual router and layer 3 load balancer. It’s the perfect for our case.

In the architecture we introduced two new servers: keepalived01 and keepalived02.

The configuration of master keepalived is the below. The slave has different priority.

[root@keepalived01 ~]# more /etc/keepalived/keepalived.conf
vrrp_instance lb_FW {
interface enp0s8
state Master
virtual_router_id 22
priority 85
authentication {
auth_type PASS
auth_pass Security
}
virtual_ipaddress {
192.168.2.201 dev enp0s8
}
}
vrrp_instance lb_nginx {
interface enp0s3
state Master
virtual_router_id 111
priority 85
authentication {
auth_type PASS
auth_pass Security
}
virtual_ipaddress {
192.168.1.200 dev enp0s3
}
}
server 192.168.1.200 80 {
delay_loop 6
lb_algo rr
lb_kind NAT
protocol tcp
real_server 192.168.1.2 80 {
TCP_CHECK {
connect_timeout 10
}
}
real_server 192.168.1.3 80 {
TCP_CHECK {
connect_timeout 10
}
}
}

There is a virtual ip 192.168.1.200 and two real server load balanced in round robin. We can add more nginx for managing more virtual hosts. On nginx there is no need to use keepalived. We continue to use keepalived on haproxy for managing virtual ip address.

There is another virtual ip (192.168.2.201) on a new vlan beetwen firewall and heepalived and two new systems: keepalived01 and keepalived02. These systems become virtual router for the VLAN 192.168.1.0/24. Every node in this vlan has as default gateway the virtual ip 192.168.1.200 managed by keepalived.

The new architecture proposed in my laboratory is:

Active/Active Reverse Proxy Architecture

Don’t forget  to configure the keepalived systems as router:

[root@keepalived01 ~]# sysctl -w net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1

Following tcpdump on keepalived after connection request to VIP 192.168.1.200:80 from host 192.168.2.10

virtual server keepalived

 

The important thing to understand is that in this case there a layer 3 balancing. This the meaning of packets.

  1. The first packet is the SYN sent from 192.168.2.10 to 192.168.1.200.
  2. The second packet is the forward of the packet below to nginx real server 192.168.1.3. Keepalived perform it. As you can see the source packet original is not changed.
  3. The nginx answers to SYN with SYN-ACK.
  4. Keep alived forwards the SYN-ACK to client masquerating the nginx server.
  5. The ACK is sent from client.
  6. Keepalived forwards the ACK to nginx server.

It’s possible to move the firewall on keepalived systems using firewalld or iptable: try it and good luck.

For knowing more about  keepalived  virtual server configurationplease read this good article http://www.linuxvirtualserver.org/docs/ha/keepalived.html.

I hope that this article about different Reverse Proxy Architectures with nginx has been appreciated: please leave a comment.

2 thoughts on - Nginx haproxy and keepalived

    • Hi,

      192.168.1.4/5 are the ip addresses of haproxy used in Complex Active/Standby Architecture.
      A virtual ip address configured by keep alived must be configured for having a active/standby haproxy solution.

      Stefano

       

Leave a Reply to admin Cancel reply