The majority of container setups need a reverse proxy to redirect incoming requests to the appropriate container. Some time ago I went with a template generator written by my friend, Dotege, this avoids having to expose the docker socket to the web and makes the choice of proxy reasonable flexible, but by default it ships with a template for Haproxy - as a very established proxy, I went with this and this served me quite well for some time.

I have more recently experienced some annoying bugs with haproxy, one of which prevented any containers being pushed to my registry and a more recent one caused an intermittent issue with my site not rendering a background due to a permissions-policy header causing an internal server error. In light of this I took the opportunity to take a look at the alternatives, and as everything is in docker and the configs can be templated as needed, I decided the best way to pick a proxy would be to spin up a few instances of Dotege with different templates and see how they handled running my site with some simple benchmarks and pick one that was nicest to configure whilst being quite performant.

The below results were a 10-second run, with 10 threads using wrk, whilst the configs for these might not be the most optimised examples, I think they’re fairly optimised and definitely typical examples. This should give a look at both the latency and throughput of the proxy, I picked a few big examples and them some less common ones, the results were not exactly as expected.

Haproxy

Avg Stdev Max +/- Stdev
Latency 19.78ms 8.64ms 139.07ms 75.39%
Thread Req/sec 50.95 12.40 180.00 82.14%
Total Req/sec 5193.85

Config  

Centauri

Avg Stdev Max +/- Stdev
Latency 24.52ms 11.16ms 89.78ms 70.00%
Thread Req/sec 40.99 10.62 120.00 63.67%
Total Req/sec 4163.63

Config  

Apache

Avg Stdev Max +/- Stdev
Latency 25.12ms 13.36ms 190.05ms 72.82%
Thread Req/sec 40.29 13.93 140.00 78.02%
Total Req/sec 4097.49

Config  

Nginx

Avg Stdev Max +/- Stdev
Latency 29.65ms 11.71ms 198.40ms 73.94%
Thread Req/sec 33.75 9.17 111.00 75.97%
Total Req/sec 3412.71

Config  

Traefik

Avg Stdev Max +/- Stdev
Latency 42.83ms 30.42ms 455.54ms 90.41%
Thread Req/sec 25.69 9.44 120.00 74.22%
Total Req/sec 2529.13

Static config Dynamic config  

Caddy

Avg Stdev Max +/- Stdev
Latency 50.25ms 18.75ms 159.01ms 68.96%
Thread Req/sec 19.83 6.84 80.00 60.57%
Total Req/sec 2017.68

Config  

Haproxy is an obvious leader here, it’s a very well optimised long-established proxy, with no intentions to be anything else, so seeing it come out on top was no surprise.

Apache and nginx definitely did not come out in the order I expected, I did spend some time optimising nginx as it seemed like an outlier, I did manage to increase its performance by about 50% from my initial config. I have for a long time held the opinion that nginx was a faster alternative to apache when it came to being a webserver and proxy, but after seeing these I have definite re-evaluated these opinions.

Caddy and Traefik seem to be about the same, which makes sense as they’re both fairly standard and modern Go proxies with almost the same goals - although caddy does do an awful lot more than Traefik in terms of web serving.

I had initially started using Caddy, its defaults are very sensible and config file is terse yet flexible, but after running these benchmarks have since made a switch to Centauri (and switched my website from Caddy to Apache!). This is a very new proxy server (written by Chris Smith the author of Dotege), its considerably more performant then the others here and still meets all my requirements and (touch wood) isn’t showing any hard to reproduce bugs in what should be fairly normal web traffic.