Skip to content

Nginx

Jump to bottom
arnaud gaboury edited this page Sep 6, 2018 · 17 revisions

Nginx

nginx (pronounced "engine X"), is a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server, written by Igor Sysoev in 2005. Nginx is the only supported web server for GitLab.

Installation

We will use Docker to install and manage the application.

Configuration

Nginx uses server blocks to accomplish the functionality found in Apache's virtual hosts. Think of server blocks as specifications for individual web sites that your server can host. Most Linux distro will use /etc/nginx/sites-availables directory to store hosts. Fedora will instead use /etc/nginx/conf.d. All additional web services .conf files shall be placed in this directory.

These files are included in alphabetical order, you need to keep that in mind if you don't specify any server as default_server, because first one would be the default.

nginx root, called webroot as well, has been moved to /db/www/nginx/html

WARNING: most http services installed by Fedora have by default permissions set to apache:apache. This has to be changed to nginx:nginx. Please check permissions on directory content /var/lib.

directory tree

The below scheme is the directory tree, which tends to group related things in one only place

$ tree -a /etc/nginx
/etc/nginx
├── conf.d
│   ├── 00-gzip.conf
│   ├── 04-ssl.conf
│   ├── 10-thetradinghall.conf
│   ├── 20-psqlstat.conf
│   ├── 30-wiki.conf
│   ├── 40-phppgadmin.conf
│   └── php-fpm
├── default.d
│   ├── fastcgi.conf.default
│   ├── fastcgi_params.default
│   ├── mime.types.default
│   ├── nginx.conf.default
│   ├── php.conf
│   ├── php.default
│   ├── scgi_params.default
│   └── uwsgi_params.default
├── fastcgi.conf
├── fastcgi_params
├── .htaccess
├── koi-utf
├── koi-win
├── mime.types
├── nginx.conf
├── scgi_params
├── ssl
│   ├── phppgadmin.ssl.conf
│   ├── psql.ssl.conf
│   ├── rstudio.ssl.conf
│   ├── thetradinghall.ssl.conf
│   └── wiki.ssl.conf
├── uwsgi_params
└── win-utf

Nginx.conf

The main configuration file is located at /etc/nginx/nginx.conf.

NOTE: best practice is to keep this main configuration file as simple as possible. All added settings and subdomains will be placed in /etc/conf.d directory. This file is common to all server blocks.

/etc/nginx/nginx.conf
------------------------
user nginx;
worker_processes 8;
error_log  /storage/log/nginx/error.log;
pid /run/nginx.pid;
worker_rlimit_nofile 8192;


events {
    worker_connections 2048;
}


http {# every lines below and before server section are common to all blocks
      # Configuration specific to HTTP and affecting all virtual servers


        include /etc/nginx/mime.types;
        include /etc/nginx/fastcgi.conf;
        include /etc/nginx/conf.d/*.conf;
        #include /etc/nginx/ssl/ssl.conf;

        index index.html index.htm index.php;

        types_hash_max_size 2048;
        default_type application/octet-stream;
        log_format   main '$remote_addr - $remote_user [$time_local]  $status '
        '"$request" $body_bytes_sent "$http_referer" '
        '"$http_user_agent" "$http_x_forwarded_for"';
        access_log /storage/log/nginx/access.log;

        server_names_hash_bucket_size 128;

        sendfile            on;
        tcp_nopush          on;
        tcp_nodelay         on;
        keepalive_timeout   70;
        #gzip               on;
}

PHP

The php location is writen in /etc/nginx/conf.d/php-fpm. To add the location in any block needing PHP, just write an include. Must be an absolute path

        location ~ \.php$ {# pass the PHP scripts to FastCGI server listening on the  php-fpm socket

                fastcgi_pass unix:/run/php5-fpm.sock;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;
                fastcgi_param HTTPS on;

        }

server block

Below is an example of the phppgadmin server.

/etc/nginx/conf.d/40-phppgadmin.conf
-------------------------------------
server{
        listen 80;
        listen 443 ssl;
        server_name phppgadmin.thetradinghall.com;
        root /usr/share/phpPgAdmin;
        include /etc/nginx/ssl/phppgadmin.ssl.conf;
        error_log  /storage/log/phppgadmin/error.log;
        access_log /storage/log/phppgadmin/access.log;

        include /etc/nginx/conf.d/php-fpm;

        location /.well-known/acme-challenge/(.*) {# needed by letsencrypt --webroot
            add_header Content-Type application/jose+json;
        }
}

Log levels

The error_log directive can be configured to log more or less information as required. The level of logging can be any one of the following:

  • emerg: Emergency situations where the system is in an unusable state.
  • alert: Severe situation where action is needed promptly.
  • crit: Important problems that need to be addressed.
  • error: An Error has occurred. Something was unsuccessful.
  • warn: Something out of the ordinary happened, but not a cause for concern.
  • notice: Something normal, but worth noting has happened.
  • info: An informational message that might be nice to know.
  • debug: Debugging information that can be useful to pinpoint where a problem is occurring.

Test configuration

Test configuration file /etc/nginx/nginx.conf with global directives for PID and quantity of worker processes:

$ nginx -t -c /etc/nginx/nginx.conf -g "pid /var/run/mynginx.pid; worker_processes 2;"

Configure HTTPS

To configure an HTTPS server, the ssl parameter must be enabled on listening sockets in the server block, and the locations of the server certificate and private key files should be specified.

TLS/SSL

It is largely recommended to disable SSL3, but Nginx by default do not use SSL3.

Getting a free certificate is now possible with letsencrypt. Fedora has an available package.

1- check Nginx has been built with SSL support:

# /usr/sbin/nginx -V
nginx version: nginx/1.8.0
built by gcc 5.1.1 20150422 (Red Hat 5.1.1-1) (GCC)
built with OpenSSL 1.0.1k-fips 8 Jan 2015
.................

2- create the etc/letsencrypt/cli.ini. This allows to store some settings.

rsa-key-size = 4096
server = https://acme-v01.api.letsencrypt.org/directory
logs-dir = /storage/log/letsencrypt
email = [email protected]
text = True
debug = True
agree-tos = True
installer = nginx

Running the command with the certonly subcommands will install certificates and keys in /etc/letsencrypt directory. The needed keys are stored in /etc/letsencrypt/live/. Then, they have to be added per URL in /etc/nginx/ssl directory.

3- create needed directories

On all server root paths, a .well-known sub-directory has to be created. Verify it is owned by root:root. Example for thetradinghall.com

# mkdir /db/nginx/data/www/.well-known

4- nginx configuration file modifications

The below lines have to be added in all server configuration files to allow ACME to access automatically the previously created directory.

location /.well-known/acme-challenge/(.*) {
      add_header Content-Type application/jose+json;
}

5- run the below command for every server. Example for mediawiki:

# letsencrypt --renew-by-default certonly --webroot --webroot-path /usr/share/mediawiki -d wiki.thetradinghall.com

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/wiki.thetradinghall.com/fullchain.pem. Your
   cert will expire on 2016-03-15. To obtain a new version of the
   certificate in the future, simply run Let's Encrypt again.

6- write the /etc/nginx/ssl/ssl.conf

server_tokens off;
ssl_session_timeout 4h;
ssl_session_cache shared:SSL:10m;
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/pki/tls/certs/dhparams.pem;
ssl_protocols TLSv1.1 TLSv1.2;
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;
ssl_session_tickets on;

ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';

ssl_certificate /etc/letsencrypt/live/thetradinghall.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/thetradinghall.com/privkey.pem;
ssl_session_timeout 10m;
ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;

7- add ssl settings for each server. Create a /etc/nginx/ssl/MyServer.conf, add the two lines regarding the path of the ssl certificate, then add an include /etc/nginx/ssl/MyServer.conf in each of server nginx configuration file.

8- test your configuration

Visit SSLabs test page, enter your URL and wait for the test to complete.

9- display content of pem certificate:

$ openssl x509 -in /path/to/cert -text -noout

letsencrypt systemd

We use systemd service file to start the service, and timer files for to renew the certificate. Below is the settings for psqlstat.thetradinghll.com.

start the service

/etc/systemd/system/letsencrypt.psql.service
-------------------------------------------
[Unit]
Description=install let's encrypt certificate for psqlstat
Requires=nginx.service
After=nginx.service

[Service]
ExecStart=/usr/bin/certbot certonly --webroot -w/storage/psqlReport -d psql.thetradinghall.com

** NOTE**

You can add as many domains/subdomains in a single service file using the -w and -d options.

renew certificate

To test if everything is alright, you can run the following command and see the output: # certbot renew --dry-run. If no errors, then run the following service twice a day.

/etc/systemd/system/letsencrypt.renewal.service
--------------------------------------------
[Unit]
Description=Renew Let's encrypt certificates

[Service]
ExecStart=/usr/bin/certbot renew --quiet
/etc/systemd/system/letsencrypt.renewal.timer
----------------------------------------------
[Unit]
Description=Run letsencrypt.renewal.service twice a display

[Timer]
OnUnitActivateSec=12hours

Nginx management

Start and enable nginx

% sudo systemctl status nginx
● nginx.service - The nginx HTTP and reverse proxy server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2015-05-08 16:00:03 CEST; 5s ago
  Process: 13222 ExecStart=/usr/sbin/nginx (code=exited, status=0/SUCCESS)
  Process: 13221 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
 Main PID: 13223 (nginx)
   CGroup: /system.slice/system-systemd\x2dnspawn.slice/systemd-nspawn@poppy.service/system.slice/nginx.service
           ├─13223 nginx: master process /usr/sbin/nginx
           └─13224 nginx: worker process

May 08 16:00:03 poppy systemd[1]: Starting The nginx HTTP and reverse proxy server...
May 08 16:00:03 poppy nginx[13221]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
May 08 16:00:03 poppy nginx[13221]: nginx: configuration file /etc/nginx/nginx.conf test is successful
May 08 16:00:03 poppy systemd[1]: Started The nginx HTTP and reverse proxy server.

Nginx optimization

Enable compression

It allows nginx to compress files and deliver them to clients (e.g. browsers) that can handle compressed content which most modern browsers do.

configuration file

/etc/nginx/conf.d/00-gzip.conf
--------------------------------

gzip on;
gzip_static on;
gzip_disable "msie6";
gzip_http_version 1.1;
gzip_vary on;
gzip_comp_level 6;
gzip_proxied any;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript text/x-js;
gzip_buffers 16 8k;

verify

Install Live http header plugin for Firefox and verify the Accept-Encoding: gzip,deflate header is here.

enable cache

A web cache sits in between a client and an “origin server”, and saves copies of all the content it sees.

configuration file

/etc/nginx/conf.d/00-cache.conf
----------------------------------

open_file_cache max=5000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;

Clustering

Clustering refers to connecting two or more computers together such that they look like a single (virtual) machine to clients. In general, this is achieved by placing a load balancer in front of the cluster to accept client requests and distribute them across the member nodes in the cluster. Clustering is a cost-effective way to improve a website or application’s performance, reliability, and scalability using commodity hardware.

In the most common configuration, all servers in a cluster host the same content and run the same applications. In this case, simple load-balancing algorithms are sufficient; one of the simplest is round robin, in which requests are distributed sequentially across the nodes in the cluster.

Load balancing

Load balancing across multiple application instances is a commonly used technique for optimizing resource utilization, maximizing throughput, reducing latency, and ensuring fault-tolerant configurations.

Deploy Nginx in a cluster with Rancher

Network

Our first deployed workload on Rancher is Nginx.

  1. Create a new namespace to regroup all apps regarding http. We call it web-http. It will hep us to differenciete later with some front end applications.

  2. Deploy a new workload on two pods, with 80 for container port on every node. Workload is called mywebserver.

# kubectl -n web-http describe service mywebserver 
Name:                     mywebserver
Namespace:                web-http
Labels:                   <none>
Annotations:              field.cattle.io/targetWorkloadIds=["deployment:web-http:mywebserver"]
                          targetWorkloadIdNoop=true
Selector:                 workload.user.cattle.io/workloadselector=deployment-web-http-mywebserver
Type:                     NodePort
IP:                       10.43.154.211
Port:                     tcp80  80/TCP
TargetPort:               80/TCP
NodePort:                 tcp80  32521/TCP
Endpoints:                10.42.0.60:80,10.42.1.25:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

NOTE:

  • a service mywebserver has been created and is listed in the service discovery page. The 10.43.154.211 IP is our cluster IP.

  • a random port of 32521 has been assigned to listen request. This port will stay the same as long as the app exists.

  • our nginx welcome page can be viewed with CLI

$ curl http://10.52.11.199:32521
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
....

Configuration file

High availability

NGINX can be configured into a high‑availability cluster to ensure application availability. The cluster consists of an active‑passive pair of NGINX instances: a master that actively processes traffic and a backup that monitors the health of the master, becoming active and taking over if the master fails.

Ressources

Clone this wiki locally