Originally posted by apeschel on Aug. 19, 2015, 6:08 p.m.
Abstract
A question was raised recently about what the impact of hosting extremely large amounts of SSL certificates via Nginx would be. This document provides an explanation of how Nginx manages SSL, provides data about the impact of hosting large numbers of certificates, and provides an analysis of what the data means.
How Nginx Manages SSL Certificates
Logically, each SSL certificate will be associated with a separate host. In Nginx, separate hosts are managed via VHosts. A VHost is a collection of the IP, Port, Hostname, and other relevant data about a host. Nginx collects the information relevant to a VHosts into a ngx_http_core_srv_conf_t struct. These structs are contained within the server_names_hash, which uses the hostname string as a key.
Nginx defers SSL management to OpenSSL. It stores certificates in OpenSSL's SSL_CTX global context structure, and uses the OpenSSL API for any SSL-related interaction between the client and server.
When a connection is made to an nginx server, nginx looks up the given hostname in its server names hash, to find the relevant structure containing the information for that VHost. Once the struct is found, it can use the OpenSSL API to establish the SSL connection with the correct certificate for the requested domain.
There is an upper limit on the number of VHosts that Nginx can fit into its server hash. This upper limit is configurable via two settings:
server_names_hash_bucket_size
server_names_hash_max_size
For optimal performance, server_names_hash_bucket_size should be set to the same size as your processor's cache line. In Linux, you can find this value at /sys/devices/system/cpu/cpu0/cache/index*/coherency_line_size. This leaves you with server_names_hash_max_size as the value that should be modified to increase the VHosts maximum.
For more information about how Nginx manages this stuff, refer to the following Nginx docs:
Testing and Analyzing Nginx Performance
Setup
A simple Nginx host was set up in Digital Ocean for testing. The tests were run on a 2CPU / 2GB instance.
For each iteration, all VHosts were running SSL on the same IP:Port pair, utilizing SNI to do so. Each VHost was configured with an SSL certificate, a corresponding private key, and a unique static file to host. All SSL certificates were generated using a 4096-bit RSA key.
siege was used for stress testing. All the hostnames were put into urls.txt, which is what siege uses to determine what hosts to send traffic to. Siege was configured to use 100 concurrently running clients, with each client attempting to retrieve 50 pages from hosts listed in urls.txt. This totals to 5000 total pages requested from the Nginx host, spread across all the VHosts running on it.
It was noted to me after initially writing this article that siege did not support SNI. It seemed unlikely that SNI support on the client side would have any impact on the results. In the interest of accuracy, I worked with the siege maintainer to add support for SNI and reran the tests. As expected, there was no impact on the results.
The goal of this testing was to determine the memory and performance overhead of hosting a large number of SSL certificates. This means the servers do not very closely reflect what an actual production host would look like. This is a deliberate attempt to isolate only the dependent variables for testing.
Summarized Results
|Num Certs|Memory (KiB)|Time (seconds)|
---------------------------------------
| 1| 3000| 41|
| 2| 3200| 42|
| 100| 9600| 41|
| 1000| 40000| 44|
| 10000| 370000| 43|
Analysis
Nginx took approximately the same amount of time, regardless of the number of VHosts. This is what would be expected from Nginx's use of a hash for managing the VHosts.
Performing a linear regression against the memory data provides us with the following relationship:
m = 3900 + 37 * n
Where
m = Expected Memory Usage (in KiB)
n = Num of VHosts using SSL
This demonstrates that each additional VHost consumes approximately 37 KiB of memory.
Raw Data
The following values are used unless otherwise noted;
server_names_hash_bucket_size: 64
server_names_hash_max_size: 512
No Certificates
root 1740 0.0 0.1 85880 1344 ? Ss 18:36 0:00 nginx: master process /usr/sbin/nginx
www-data 1741 0.0 0.2 86224 2268 ? S 18:36 0:00 \_ nginx: worker process
www-data 1742 0.0 0.2 86224 2276 ? S 18:36 0:00 \_ nginx: worker process
www-data 1743 0.0 0.2 86224 2408 ? S 18:36 0:00 \_ nginx: worker process
www-data 1744 0.0 0.1 86224 2028 ? S 18:36 0:00 \_ nginx: worker process
siege -q -b -c 100 -r 50 http://1.apestest/
Transactions: 5000 hits
Availability: 100.00 %
Elapsed time: 17.87 secs
Data transferred: 1.83 MB
Response time: 0.30 secs
Transaction rate: 279.80 trans/sec
Throughput: 0.10 MB/sec
Concurrency: 83.88
Successful transactions: 5000
Failed transactions: 0
Longest transaction: 4.11
Shortest transaction: 0.05
One Certificate
root 1740 0.0 0.2 85992 3012 ? Ss 18:36 0:00 nginx: master process /usr/sbin/nginx
www-data 2377 0.0 0.1 86276 1904 ? S 19:10 0:00 \_ nginx: worker process
www-data 2378 0.0 0.1 86276 1904 ? S 19:10 0:00 \_ nginx: worker process
www-data 2379 0.0 0.1 86276 1904 ? S 19:10 0:00 \_ nginx: worker process
www-data 2380 0.0 0.3 86276 3116 ? S 19:10 0:00 \_ nginx: worker process
siege -q -b -c 100 -r 50 https://1.apestest/
Transactions: 5000 hits
Availability: 100.00 %
Elapsed time: 40.97 secs
Data transferred: 1.83 MB
Response time: 0.73 secs
Transaction rate: 122.04 trans/sec
Throughput: 0.04 MB/sec
Concurrency: 89.43
Successful transactions: 5000
Failed transactions: 0
Longest transaction: 5.80
Shortest transaction: 0.12
Two Certificates
root 1740 0.0 0.3 86680 3216 ? Ss Aug10 0:00 nginx: master process /usr/sbin/nginx
www-data 22502 0.0 0.1 86680 1992 ? S 21:32 0:00 \_ nginx: worker process
www-data 22503 0.3 0.3 86680 3440 ? S 21:32 0:00 \_ nginx: worker process
www-data 22504 0.0 0.1 86680 1992 ? S 21:32 0:00 \_ nginx: worker process
www-data 22505 0.0 0.1 86680 1992 ? S 21:32 0:00 \_ nginx: worker process
siege -q -b -c 100 -r 50
Transactions: 5000 hits
Availability: 100.00 %
Elapsed time: 41.92 secs
Data transferred: 1.83 MB
Response time: 0.75 secs
Transaction rate: 119.27 trans/sec
Throughput: 0.04 MB/sec
Concurrency: 89.68
Successful transactions: 5000
Failed transactions: 0
Longest transaction: 7.86
Shortest transaction: 0.12
One Hundred Certificates
root 1740 0.0 0.9 92476 9560 ? Ss Aug10 0:00 nginx: master process /usr/sbin/nginx
www-data 30236 0.0 0.8 92476 8156 ? S 13:13 0:00 \_ nginx: worker process
www-data 30237 0.0 0.8 92476 8156 ? S 13:13 0:00 \_ nginx: worker process
www-data 30238 0.0 0.8 92476 8156 ? S 13:13 0:00 \_ nginx: worker process
www-data 30239 0.0 0.8 92476 8156 ? S 13:13 0:00 \_ nginx: worker process
siege -q -b -c 100 -r 50
Transactions: 5000 hits
Availability: 100.00 %
Elapsed time: 40.50 secs
Data transferred: 0.67 MB
Response time: 0.73 secs
Transaction rate: 123.46 trans/sec
Throughput: 0.02 MB/sec
Concurrency: 90.17
Successful transactions: 0
Failed transactions: 0
Longest transaction: 6.91
Shortest transaction: 0.00
One Thousand Certificates
server_names_hash_max_size: 1024
root 912 0.0 1.9 124704 40272 ? Ss Aug17 0:00 nginx: master process /usr/sbin/nginx
www-data 6543 18.2 1.9 124704 40288 ? S 02:54 1:01 \_ nginx: worker process
www-data 6544 17.3 1.9 124704 40288 ? S 02:54 0:58 \_ nginx: worker process
www-data 6545 18.2 1.9 124704 40288 ? S 02:54 1:01 \_ nginx: worker process
www-data 6546 16.3 1.9 124704 40288 ? S 02:54 0:55 \_ nginx: worker process
10079> siege -q -b -c 100 -r 50
Transactions: 5000 hits
Availability: 100.00 %
Elapsed time: 43.83 secs
Data transferred: 1.83 MB
Response time: 0.77 secs
Transaction rate: 114.08 trans/sec
Throughput: 0.04 MB/sec
Concurrency: 87.71
Successful transactions: 5000
Failed transactions: 0
Longest transaction: 17.73
Shortest transaction: 0.13
Ten Thousand Certificates
server_names_hash_max_size: 16384
root 912 0.0 17.8 449596 365304 ? Ss Aug17 0:04 nginx: master process /usr/sbin/nginx
www-data 15625 0.0 17.7 449596 363992 ? S 03:09 0:00 \_ nginx: worker process
www-data 15626 0.0 17.7 449596 363992 ? S 03:09 0:00 \_ nginx: worker process
www-data 15627 0.0 17.7 449596 363992 ? S 03:09 0:00 \_ nginx: worker process
www-data 15628 0.0 17.7 449596 363992 ? S 03:09 0:00 \_ nginx: worker process
siege -q -b -c 100 -r 50
Transactions: 5000 hits
Availability: 100.00 %
Elapsed time: 42.71 secs
Data transferred: 1.83 MB
Response time: 0.74 secs
Transaction rate: 117.07 trans/sec
Throughput: 0.04 MB/sec
Concurrency: 87.15
Successful transactions: 5000
Failed transactions: 0
Longest transaction: 14.30
Shortest transaction: 0.12
Top comments (0)