Edit Article: Slowloris DOS Mitigation Guide

Documentation[edit]

If an extended news item, enter info here:

== Introduction == [http://ha.ckers.org/slowloris/ Slowloris] is the name of a perl-based HTTP client that can be used as a denial of service against Apache-based HTTP servers and the squid caching proxy server. It operates by repeatedly initiating several hundred valid HTTP requests to the server, and keeping these connections open using a minimal amount of TCP traffic, in order to consume server resources. Once server resources are exhausted, the server will no longer be able to respond to legitimate traffic. Slowloris was written by 'RSnake', and was announced in a [http://ha.ckers.org/blog/20090617/slowloris-http-dos/ ha.ckers.org blog] post on June 17, 2009. As of July 5, 2009, vulnerable HTTP servers and proxies include: *'''Apache HTTP Server''' *'''IBM HTTP Server''' *'''IBM WebSphere Edge Server Caching Proxy''' *'''Squid caching proxy server''' There has been much discussion on the Internet relating to what HTTP servers, HTTP proxies, and network configurations are not affected by Slowloris. It's important to note that, based on our testing, much of the conventional wisdom about supposedly non-vulnerable configurations is misleading at best. One of the primary goals of this document is to dispel some of these myths and provide reliable information on properly mitigating against Slowloris and other similar denials of service. In particular, it's important to note that hardware load balancers typically do not protect against this denial of service without additional configuration, which we detail below. In addition, other supposedly non-vulnerable HTTP servers and proxies can be affected by this denial of service using non-default Slowloris settings. {{fancyimportant|Networks that utilize hardware load balancers and alternative Web servers may still be vulnerable to Slowloris.}} === What Makes Slowloris "Different" === Before we review Slowloris mitigations, let's review what makes Slowloris different from other denials of service. Slowloris is different from typical denials of service in that Slowloris traffic utilizes legitimate HTTP traffic, and does not rely on using special "bad" HTTP requests that exploit bugs in specific HTTP servers. Because of this, existing IPS and IDS solutions that rely on signatures to detect attacks will generally not recognize Slowloris. This means that Slowloris is capable of being effective even when standard enterprise-grade IPS and IDS systems are in place. The second issue that makes Slowloris different is that it is an easy-to-use perl script. While similar denials of service have been documented in security publications, RSnake has provided a "weaponized" ready-to-use version of this denial of service that is trivial to use. Combine these two facts together, plus throw in the fact that Apache is vulnerable against Slowloris, and script kiddies now have an easy way to take down a large portion of the Internet. == Hardware Load Balancers == === Introduction === As we mentioned earlier, conventional wisdom is that if your infrastructure is behind a hardware load balancer, then you are not vulnerable to Slowloris. This is not true. Slowloris can traverse hardware load balancers even if they are properly configured. However, many load balancers can be configured to protect your infrastructure against Slowloris. Here's how to do it. === Load Balancer Mitigation === Many hardware load balancers, including F5 Big-IP, Cisco CSS, Cisco ACE and Citrix NetScaler have a feature generically known as delayed binding, or TCP Splicing. This feature allows the load balancer to allow a TCP three-way handshake between the client and the virtual IP address (a.k.a. the hardware load balancer) configured in front of the Web server(s). After this handshake has been completed, the client will send in the HTTP request header, which the load balancer can inspect to determine what action to perform on the HTTP request. The load balancer can be configured to respond to the client in a number of ways, such as sending a HTTP 302 Redirect, or selecting an appropriate Web server to handle the HTTP Request based upon something identifiable in the HTTP Request header like a cookie, URL or User-Agent. Delayed binding typically causes the load balancer to perform an HTTP Request header completeness check, which means that the HTTP Request will not be sent to the appropriate Web server until the final two carriage return and line feeds are sent by the HTTP client. This is the key bit of information. Basically, delayed binding ensures that your Web server or proxy will never see any of the incomplete requests being sent out by Slowloris. Because of this, delayed binding is a very effective way to protect against Slowloris, but it must be properly configured. We'll look at an example configuration next. === Cisco CSS Configuration === Here is a Cisco CSS configuration that enables delayed binding: <pre> content www_80_rule vip address 10.5.154.200 protocol tcp port 80 add service wwwserver1_80 add service wwwserver2_80 url "/*" active </pre> The second to last line ('url "/*"') is what enables a Layer5 rule that performs delayed binding on the Cisco CSS, ''and without this enabled in the content rule, any Apache based HTTP Server that receives an incomplete HTTP Header will be vulnerable to this exploit''. Other load balancers, such as F5 Big-IP, can be configured to protect against Slowloris using a similar approach. {{fancyimportant|This hardware load balancer mitigation only works with HTTP traffic. Please read below to understand how to protect against SSL-based attacks}} . === Load Balancers and SSL === We've addressed the HTTP side of the house, but what about HTTPS (SSL)? If you have an environment where you must run end to end encryption and are running an Apache-based HTTP server with an SSL Listener, '''then it's vulnerable. A Hardware load balancer placed in front of your Apache-based SSL Listener does not mitigate this vulnerability'''. Here is a typical scenario: *Client attempts HTTPS connection to Web site. *Load balancer 'fronts' the Web site and may be able to detect the SSL Session ID, but cannot inspect SSL traffic, and therefore has no further HTTP Protocol inspection capability. *The Hardware Load Balancer sends the HTTPS connection back to the Apache-based HTTP server, performs a SSL Handshake and then sends the HTTP Data inside the SSL connection. If the HTTP Request employs the exploit, the Apache-based HTTP Server worker threads will be exploited. Since this is done over SSL, you are limited in your ability to inspect and detect the exploit using the hardware load balancer. Since the hardware load balancer must be able to inspect the HTTP Request Header in order to perform delayed binding and protect against Slowloris, it first must be able to see the HTTP Data. To do this, your hardware must be configured to decrypt the SSL and then perform the delayed binding, which we previously covered. This configuration can be accomplished with back-end SSL configuration on a Cisco CSS, or by utilizing a similar feature on a different hardware load balancer. This does require that your load balancer decrypt incoming SSL traffic using encryption offload. === Hardware Load Balancer Summary === Hardware load balancers do not protect against Slowloris by default. However, when configured to perform delayed binding, hardware load balancers can be a very effective protection for HTTP. However, for HTTPS, things become more complex, as the hardware load balancer is limited in its ability to inspect traffic. This problem can be addressed by configuring your load balancer to decrypt SSL traffic so that the HTTP request can be inspected and delayed binding can be performed. == Apache == We've looked at how to configure load balancers, what what about fixing Apache itself? The problem with Apache in its current incarnation is that it can be taken offline by a relatively mild Slowloris attack. Certainly, Apache developers must be working on an imminent fix for this issue -- or are they? Reaction from Apache developers has been mixed. Take for example the response quoted in the [http://ha.ckers.org/blog/20090617/slowloris-http-dos/ ha.ckers.org blog post], where this issue is not taken seriously, as well as the official apache.org Slowloris bug, where this issue was described as "INVALID". Fortunately, some Apache developers are being more honest about the issue, such as in an apache-dev post from Paul Querna, Apache committer, about this issue: <pre> Mitagation is the wrong approach. We all know our architecture is wrong. We have started on fixing it, but we need to finish the async input rewrite on trunk, but all of the people who have hacked on it, myself included have hit ENOTIME for the last several years. Hopefully the publicity this has generated will get renewed interest in solving this problem the right way, once and for all :) </pre> There are a number of efforts to either modify Apache or add an additional Apache module to deal effectively with Slowloris. We have looked at one of these modifications so far. === Anti-slowloris.diff === Andreas Krennmair posted a patch to the apache-dev list on June 21, 2009 called [http://thread.gmane.org/gmane.comp.apache.devel/37773 "anti-slowloris.diff."] This patch applies to only the prefork MPM and is a basic proof of concept of how Apache can be more resilient towards Slowloris attacks. In our testing, we've found this patch to not be fully effective. While it does allow Apache to survive basic Slowloris attacks, Apache still goes offline when a more powerful Slowloris attack is used. === Other Options === There are other options, such as [http://thread.gmane.org/gmane.comp.apache.devel/37773 mod_noloris], which are being developed and may be incorporated into Apache. We have not yet had time to test mod_noloris to determine whether it's effective. === Summary === We have not yet found a defense against Slowloris that can be integrated directly into Apache. == NetFilter == Another option to protect against Slowloris is to utilize something like Linux's netfilter connlimit functionality (available in kernel 2.6.23+) in order to limit the rate of incoming connections coming from a particular host. This can be implemented for HTTP as follows: <pre> iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 100 -j DROP </pre> While this is an effective defense against Slowloris, we do have a number of concerns about this approach. First, the connlimit rate must be very low to protect Apache. This can result in legitimate requests being denied, and is probably not a workable solution for higher-traffic sites. It also presents problems for traffic originating from behind "mega-proxies," as these may easily hit the connection rate limiting cut-off. Since this solution doesn't provide any kind of delayed binding functionality, is also possible to craft the Slowloris attack so that it utilizes multiple hosts, operating at a slower connection rate, and Apache would easily be taken offline. Therefore, we cannot recommend netfilter alone as a complete mitigation for Slowloris. It may work for some low-traffic sites, but does not provide robust protection against a motivated attacker. == Alternate Web Servers == For some organizations, switching to an alternative Web server that is not supposed to be vulnerable to Slowloris may be a viable option to protect against Slowloris. We utilized the [http://www.cherokee-project.com/ Cherokee Web server] for our testing. === Initial Cherokee Tests === We configured Cherokee on Funtoo Linux, and found that it was resilient to Slowloris, when Slowloris was used with default settings. We were even able to ramp up Slowloris quite a bit and were still able to load pages from the Web server. However, as we ramped up Slowloris further, we found that Cherokee began to refuse connections. Upon investigation, we discovered that Cherokee had exhausted its 1024 available file descriptors, limiting the server to 512 simultaneous connections. After raising the default ulimits on the system, Cherokee was able to withstand a very signficant Slowloris attack and still serve pages. However, several thousand file descriptors were consumed on the host system. === Configuring Cherokee for Slowloris === On Gentoo Linux or Funtoo Linux, you can configure Cherokee to be resilient to Slowloris by increasing the number of file descriptors available to the cherokee user. To do this, first raise the default number of file descriptors available to a user by adding the following lines to <span style="color:green">/etc/security/limits.conf</span>: <pre> * soft nofile 4096 * hard nofile 5028 </pre> To apply this change to just the cherokee user, change the <span style="color:green">*</span>'s to <span style="color:green">cherokee</span>. Next, you will need to configure Cherokee to utilize all these file descriptors by modifying <span style="color:green">/etc/cherokee/cherokee.conf</span> or utilizing <span style="color:green">cherokee-admin</span> (Advanced settings.) This can be done by adding the following line to <span style="color:green">/etc/cherokee/cherokee.conf</span> and restarting Cherokee: <pre> server!fdlimit = 4096 </pre> {{fancyimportant|While cherokee-admin states that Cherokee will automatically detect the number of available file descriptors, it didn't seem to do so in our testing.}} == Conclusion == We will continue to update this document as more information is available to us. If you are trying to protect against Slowloris, our current top recommendations are: *'''Test''' your infrastructure to determine its level of resiliency. *'''Don't assume''' that your infrastructure is not vulnerable to Slowloris. *'''Utilize multiple layers of defense''' when possible. *'''Find limits when testing'''. Don't just use default Slowloris settings. In general, a hardware load balancer with SSL encryption offload when ''configured to perform delayed binding'' is a great first defense against Slowloris. This will prevent Apache from seeing the Slowloris traffic. Our one concern about just relying on a hardware load balancer is that it is only one layer of defense. However, one layer of defense may be an acceptable interim solution. If you do not have a hardware load balancer that can be configured to protect against Slowloris, and you are in a position where you are able to switch Web servers, then our recommendation is to deploy an alternative HTTP server such as Cherokee, and configure it so that it can handle a large number of simultaneous connections. Also consider combining Cherokee with Linux netfilter connection rate limiting (for kernels 2.6.23+) with a ''high threshold''. This should provide adequate defense against Slowloris - the Web server will have adequate resources to handle typical Slowloris attacks, and extreme attacks will hit the connection rate limit and be denied. Our one concern with this approach is that it will still result in signficant kernel resources being utilized, since no delayed binding is being performed. This can cause resource problems that could result in performance degradation, and could allow sophisticated multi-host attacks to be effective. And if at all possible, use a combination of approaches, such as a properly- configured hardware load balancer combined with a resilient Web server. And please test your configuration to ensure that your defenses are working properly! :) [[Category:Articles]]

Save page Show preview Show changes Cancel