Security with nginx and haproxy

  Network, Security, System
 

Nginx is a robust and fast reverse proxy. Haproxy is a fast application load balancer. Together can be used for publishing to internet web services in security way.

For this reason this article has the objective to explain how to secure web services using nginx and haproxy.

The haproxy, in addition to application load balancer functionality, has a native language to work better with http header rewrite and moreover it implements the sticky session functionality present only in nginx commercial product.

Using nginx and haproxy is possible to manage too a warning web page useful during maintenance activity.

This is the scenario involved in the article.

security with nginx and haproxy

security with nginx and haproxy

The linux distribution used is Centos 7.2.

The apache with php are running in two docker containers (ubuntu based, see https://medium.com/dev-tricks/apache-and-php-on-docker-44faef716150#.l1r622qav) for avoiding to create two virtual linux systems.

The laburatory implemented can be tested at the following url https://nikto.sysandnetsecurity.com/index.php. The haproxy gui is reachable at https://nikto.sysandnetsecurity.com/proxy/. The user account is admin, the password is stegri.

Let’s start with nginx and naproxy security configuration.

Nginx security configuration

After installing nginx, a new virtual server called nikto.sysandnetsecurity.com is configured for forwarding all the interested uri to haproxy that is configured for balancing all the traffic to two apache back end servers.

A secure web server must have ssl configured and redirect the http request to https. The virtual server in listening on port 80 redirects to https returning a HTTP 301 in this way:

[root@nikto conf.d]# pwd
/etc/nginx/conf.d
[root@nikto conf.d]# vi nikto.sysandnetsecurity.com.conf
server {
server_name nikto.sysandnetsecurity.com;
listen 80;
access_log /var/log/nginx/nikto.sysandnetsecurity.com.log;
error_log /var/log/nginx/nikto.sysandnetsecurity.com.log;
return 301 https://$host$request:uri;
}

For SSL is absolutely to avoid: non-authenticated Diffie-Hellman, RC4 cipher, DES and null-encryption ciphers. A very good SSL configuration is that (rating A+ at https://www.ssllabs.com/)  (see http://www.securityandit.com/security/penetration-testing/)

[root@nikto conf.d]#vi  nikto.sysandnetsecurity.com.conf
server {
listen 443;
server_name nikto.sysandnetsecurity.com;
access_log /var/log/nginx/nikto.sysandnetsecurity.com.log;
error_log /var/log/nginx/nikto.sysandnetsecurity.com.log;
ssl_certificate /etc/letsencrypt/live/nikto.sysandnetsecurity.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/nikto.sysandnetsecurity.com/privkey.pem;
ssl on;
ssl_session_timeout 5m;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; #libssl > 1.0
ssl_ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH!aNULL:!MD5:!kEDH;
ssl_session_cache shared:TLSSL:16m;}

It’s best practice to intercept the back end errors returning to client a standard error. This solution is more elegant and avoid to make known to internet important info about software used.

For this reason, a new piece of configuration is added:

[root@nikto conf.d]#vi  nikto.sysandnetsecurity.com.conf
location = /error_page.html {
root /usr/share/nginx/html;
internal;
}
location / {
proxy_intercept_errors on;
error_page 500 501 502 503 504 505 506 507 /error_page.html;
proxy_redirect ~^http:(.*)$ https:$1;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_buffering off;}

Le’t go to configure the virtual server for proxying the http request to haproxy. It’s good practice to separate internal from external traffic for applying, for example, different balancing policy, or for closing the site to internet only.

The new  configuration becomes:

location / {
proxy_intercept_errors on;
error_page 500 501 502 503 504 505 506 507 /error_page.html;
proxy_redirect ~^http:(.*)$ https:$1;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
if ($remote_addr ~* “^(10.)*$”) {
proxy_pass http://INTERNAL-back-end-server;
}
proxy_pass http://EXTERNAL-back-end-server;
upstream INTERNAL-back-end-server{
server nikto.sysandnetsecurity.com:7001;
}
upstream EXTERNAL-back-end-server{
server nikto.sysandnetsecurity.com:7002;
}

Three new http header have been added in the request proxied to haproxy:

  1. proxy_set_header Host $host: The Host Header is proxied as received from client.
  2. proxy_set_header X-Real-IP: The remote ip is useful for troubleshooting and for monitoring.
  3. proxy_set_header X-Forwarded-For: Useful for the same reasons.

Before passing to haproxy configuration, new http security headers can be returned to client.

For mitigating clickjacking attack:

add_header add_header X-Frame-Options SAMEORIGIN;

For forcing the client to use https::

add_header add_header Strict-Transport-Security: max-age=3456000;

For mitigating xss attack:

add_header X-XSS-Protection “1; mode=block”;

The final configuration for the virtual servers became:

[root@nikto conf.d]#vi  nikto.sysandnetsecurity.com.conf
server {
listen 443;
server_name nikto.sysandnetsecurity.com;
access_log /var/log/nginx/nikto.sysandnetsecurity.com.log;
error_log /var/log/nginx/nikto.sysandnetsecurity.com.log;
ssl_certificate /etc/letsencrypt/live/nikto.sysandnetsecurity.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/nikto.sysandnetsecurity.com/privkey.pem;
ssl on;
ssl_session_timeout 5m;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; #libssl > 1.0
ssl_ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH!aNULL:!MD5:!kEDH;
ssl_session_cache shared:TLSSL:16m;
location = /error_page.html {
root /usr/share/nginx/html;
internal;
}
location /warning_page.html {
root /usr/share/nginx/html;
allow all;
}
location / {
proxy_intercept_errors on;
error_page 500 501 502 503 504 505 506 507 /error_page.html;
proxy_redirect ~^http:(.*)$ https:$1;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
add_header X-Frame-Options SAMEORIGIN;
add_header Strict-Transport-Security: max-age=3456000;
add_header X-XSS-Protection “1; mode=block”;
allow all;
if ($remote_addr ~* “^(10.)*$”) {
proxy_pass http://INTERNAL-back-end-server;
}
proxy_pass http://EXTERNAL-back-end-server;
}}
server {
server_name nikto.sysandnetsecurity.com;
listen 80;
access_log /var/log/nginx/nikto.sysandnetsecurity.com.log;
error_log /var/log/nginx/nikto.sysandnetsecurity.com.log;
return 301 https://$host;
}
upstream EXTERNAL-back-end-server{
server nikto.sysandnetsecurity.com:7001;
}
upstream INTERNAL-back-end-server{
server nikto.sysandnetsecurity.com:7002;}

For more information about these new headers see http://www.securityandit.com/security/penetration-testing/.

For managing a warning_page to publish for maintenance activity, a equal virtual server in listening on port 88 is created and contacted by haproxy when the all the active servers in the balancing pool are down or in maintenance state.

[root@nikto conf.d]# vi nikto_88.sysandnetsecurity.com.conf
server {
listen 0.0.0.0:88;
server_name nikto.sysandnetsecurity.com;
return 301 https://$host/warning_page.html;
access_log /var/log/nginx/nikto.sysandnetsecurity.combined;
error_log /var/log/nginx/nikto.sysandnetsecurity.com.error.log error;
root /usr/share/nginx/html;}

Don’t forget to have in /usr/share/nginx/html two html pages: error and warning page.

It’s good approach to redirect http traffic for other virtual server not managed to a default virtual server that answers always with a HTTP 401:

[root@nikto conf.d]# pwd
/etc/nginx/conf.d
[root@nikto conf.d]# vi aaa.sysandnetsecurity.com.conf
server {
listen 443;
server_name aaa.sysandnetsecurity.com;
access_log /var/log/nginx/aaa.sysandnetsecurity.com.log;
error_log /var/log/nginx/aaa.sysandnetsecurity.com.log;
ssl_certificate /etc/letsencrypt/live/nikto.sysandnetsecurity.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/nikto.sysandnetsecurity.com/privkey.pem;
ssl on;
ssl_session_timeout 5m;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; #libssl > 1.0
ssl_ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH!aNULL:!MD5:!kEDH;
ssl_session_cache shared:TLSSL:16m;
location / {
return 401;
}
server {
listen 80;
access_log /var/log/nginx/aaa.sysandnetsecurity.com.log;
error_log /var/log/nginx/aaa.sysandnetsecurity.com.log;
server_name aaa.sysandnetsecurity.com;
return 301 https://$host$request_uri;}

The default access log can be used for monitoring the attack attempts and for example for categorize them in a gui environment by kibana (see http://www.securityandit.com/security/ids-with-pfsense-suricata-and-kibana/).

Let’s go now now to configure haproxy.

Haproxy security configuration

After installing haproxy, two new balancing pools must be created: the first, in listening on 7001 port, for internet access; the other, in listening on 7002 port, for internal access.

For forcing the cookie to be used only in https session and not in java script the “HttpOnly; Secure” field is added in the cookie’s end.

The balancing pool has a third application server in backup state called warning : it is actived when the first two application servers are down or in maintenance state.

The stick session is performed putting in the head of cookie  app01~ or app02~. These prepend strings are sent to client but not to to back-end servers. The process is trasparent for them.

The configuration of haproxy is the following:

[root@nikto conf.d]# cd /etc/haproxy/
[root@nikto haproxy]# vi haproxy.cfg
frontend HTTP-Service-Internet *:7001
mode http
default_backend HTTP-Service-Internet
backend HTTP-Service-Internet
balance roundrobin
option tcp-check
cookie PHPSESSID prefix nocache
#Rewrite Header. Add secure Flag on the http header
http-response replace-value Set-Cookie (.*) \1;\ HttpOnly;
http-response replace-value Set-Cookie (.*) \1;\ Secure
server App01 172.17.0.1:8080 check port 8080 cookie app01 weight 100
server App02 172.17.0.1:8081 check port 8080 cookie app02 weight 100
server warning 164.132.193.215:88 backup
frontend HTTP-Service-Intranet *:7002
mode http
default_backend HTTP-Service-Intranet
backend HTTP-Service-Intranet
balance roundrobin
option tcp-check
cookie PHPSESSID prefix nocache
#Rewrite Header. Add secure Flag on the http header
http-response replace-value Set-Cookie (.*) \1;\ HttpOnly;
http-response replace-value Set-Cookie (.*) \1;\ Secure
server App01 172.17.0.1:8080 check port 8080 cookie app01 weight 100
server App02 172.17.0.1:8081 check port 8080 cookie app02 weight 100
server warning 164.132.193.215:88 backup

Following some test to demonstrate how the service is balanced to two back end servers. You can test directly using https://nikto.sysandnetsecurity.com/index.php.

[root@nikto conf.d]# curl -v –insecure https://nikto.sysandnetsecurity.com/index.php
* About to connect() to nikto.sysandnetsecurity.com port 443 (#0)
* Trying 127.0.0.1…
> GET /index.php HTTP/ 1.1
> User-Agent: curl/7.29.0
> Host: nikto.sysandnetsecurity.com
> Accept: */*
< HTTP/ 1.1 200 OK
< Server: nginx/1.6.3
< Date: Thu, 23 Jun 2016 08:14:36 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 0
< Connection: keep-alive
< Set-Cookie: PHPSESSID=app01~e6fvbknrcmujorettfo6bot684; path=/; HttpOnly;; Secure
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security:: max-age=3456000
< X-XSS-Protection: 1; mode=block
[root@nikto conf.d]# curl -v –insecure https://nikto.sysandnetsecurity.com/index.php
* About to connect() to nikto.sysandnetsecurity.com port 443 (#0)
* Trying 127.0.0.1…
* Connected to nikto.sysandnetsecurity.com (127.0.0.1) port 443 (#0)
> GET /index.php HTTP/ 1.1
> User-Agent: curl/7.29.0
> Host: nikto.sysandnetsecurity.com
> Accept: */*
< HTTP/ 1.1 200 OK
< Server: nginx/1.6.3
< Date: Thu, 23 Jun 2016 08:14:41 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 0
< Connection: keep-alive
< Set-Cookie: PHPSESSID=app02~2huufirthd4cei64pr4hf5m2u5; path=/; HttpOnly;; Secure
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security:: max-age=3456000
< X-XSS-Protection: 1; mode=bloc
[root@nikto conf.d]# curl -v –insecure –cookie “PHPSESSID=app02~2huufirthd4cei64pr4hf5m2u5;” https://nikto.sysandnetsecurity.com/index.php
* About to connect() to nikto.sysandnetsecurity.com port 443 (#0)
* Trying 127.0.0.1…
* Connected to nikto.sysandnetsecurity.com (127.0.0.1) port 443 (#0)
> GET /index.php HTTP/ 1.1
> User-Agent: curl/7.29.0
> Host: nikto.sysandnetsecurity.com
> Accept: */*
> Cookie: PHPSESSID=app02~2huufirthd4cei64pr4hf5m2u5;
< HTTP/ 1.1 200 OK
< Server: nginx/1.6.3
< Date: Thu, 23 Jun 2016 08:14:58 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 0
< Connection: keep-alive
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security:: max-age=3456000
< X-XSS-Protection: 1; mode=block

After putting in maintanance mode the two application server by haproxy gui https://nikto.sysandnetsecurity.com/haproxy (admin/stegri), the warning redirect page is returned.

root@kali:~# curl -v –insecure https://nikto.sysandnetsecurity.com/index.php
* Hostname was NOT found in DNS cache
* Trying 164.132.193.215…
* Connected to nikto.sysandnetsecurity.com (164.132.193.215) port 443 (#0)
> GET /index.php HTTP / 1.1
> User-Agent: curl/7.38.0
> Host: nikto.sysandnetsecurity.com
> Accept: */*
< HTTP/ 1.1 301 Moved Permanently
* Server nginx/1.6.3 is not blacklisted
< Server: nginx/1.6.3
< Date: Thu, 23 Jun 2016 08:48:50 GMT
< Content-Type: text/html
< Content-Length: 184
< Connection: keep-alive
< Location: https://nikto.sysandnetsecurity.com/warning_page.html
< Strict-Transport-Security: max-age=31536000
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security:: max-age=3456000
< X-XSS-Protection: 1; mode=block

Conclusions

Nginx and haproxy permit to configure a secure and scalable reverse proxying system.

High availability is given using keepalived as explained in http://www.securityandit.com/system-and-network/nginx-haproxy-and-keepalived/.

It’s also possible to have more WAN internet provider with nginx and haproxy that balance to back end servers. The internet traffic can be balanced by DNS. Very scalable architecture.

Don’t hesitate to contact me for any suggestion or problem.

LEAVE A COMMENT