Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
litespeed_wiki:config:mitigating-ddos-attacks [2018/12/13 20:08]
Lisa Clarke Copyediting
litespeed_wiki:config:mitigating-ddos-attacks [2020/01/07 17:21] (current)
Lisa Clarke [Never set **Use Client IP in Header** to ''Yes'']
Line 3: Line 3:
 LiteSpeed Web Server provides several features aimed at reducing and even eliminating the impact of HTTP-level Denial of Service (DoS) and Distributed Denial of Service (DDoS) attacks. You can either use LSWS built-in features or third party ModSecurity rules such as Owasp, Atomicorp, Comodo and CloudLinux Imunify360. The following LSWS built-in configuration settings will help mitigate such attacks. LiteSpeed Web Server provides several features aimed at reducing and even eliminating the impact of HTTP-level Denial of Service (DoS) and Distributed Denial of Service (DDoS) attacks. You can either use LSWS built-in features or third party ModSecurity rules such as Owasp, Atomicorp, Comodo and CloudLinux Imunify360. The following LSWS built-in configuration settings will help mitigate such attacks.
  
 +===== Enable reCAPTCHA feature =====
 +One of the most effective methods to mitigate DoS and DDoS attacks is to enable the reCAPTCHA feature. reCAPTCHA is a free service from Google that helps protect websites from spam and abuse. A “CAPTCHA” is a turing test to tell human and bots apart. It is easy for humans to solve, but hard for bots and other malicious software to figure out. The reCAPTCHA feature is supported as of LSWS 5.4RC1 and later. Please see [[litespeed_wiki:​config:​recaptcha|these instructions]] to enable reCAPTCHA.
 ===== Enable LiteSpeed Cache ===== ===== Enable LiteSpeed Cache =====
 Enabling LiteSpeed Cache will increase the server'​s capacity to handle heavy traffic. Enabling LiteSpeed Cache will increase the server'​s capacity to handle heavy traffic.
Line 50: Line 52:
  
 The blocked IPs can be found in the real-time-stats report. The blocked IPs can be found in the real-time-stats report.
 +
 +===== Virtual Host-Level Bandwidth Throttling =====
 +
 +LiteSpeed Web Server version 5.0 introduces virtual host-level bandwidth throttling. This can be thought of as an extension of LSWS's **Per Client Throttling settings** explained as above, which allow you to control the amount of stress a single IP can put on your server. Virtual host-level bandwidth throttling allows you to customize bandwidth throttling, in Apache configs, for particular virtual hosts through ''​MaxConnPerClient <​limit_for_connections>'',''​LargeFileLimit [Type] [Minimum Size in kilobytes] [Speed in bytes/​s]'',​ ''​BandWidth [Origin] [Speed in bytes/​s]''​ and ''​MinBandWidth all -1''​.
 +. Please check [[litespeed_wiki:​config:​vhost-level_bw_throttling|here]] for details.
 +
 ===== Use Max Request/​Response Settings ===== ===== Use Max Request/​Response Settings =====
 You can take advantage of the Max Request/​Response Settings to reduce memory usage. You can take advantage of the Max Request/​Response Settings to reduce memory usage.
Line 94: Line 102:
 ===== Order LiteSpeed'​s Advanced Anti-DDoS Setup Service ==== ===== Order LiteSpeed'​s Advanced Anti-DDoS Setup Service ====
 If you need assistance configuring your site to mitigate attacks, check out [[https://​store.litespeedtech.com/​store/​knowledgebase.php?​action=displayarticle&​id=125|LiteSpeed'​s Advanced Anti-DDoS Setup Service]]. LiteSpeed Denial of Service Packet Filter Setup Service will fine-tune your anti-DDoS configuration and set up iptables to automatically block attacking IPs detected by the web server. If you need assistance configuring your site to mitigate attacks, check out [[https://​store.litespeedtech.com/​store/​knowledgebase.php?​action=displayarticle&​id=125|LiteSpeed'​s Advanced Anti-DDoS Setup Service]]. LiteSpeed Denial of Service Packet Filter Setup Service will fine-tune your anti-DDoS configuration and set up iptables to automatically block attacking IPs detected by the web server.
 +
 +This is sufficient for many common attack scenarios. In cases of extreme attacks, this service will not be sufficient, and only custom / hourly support may be appropriate.
 +
 +In order to determine whether the service fits your needs, we will need to identify the type of attack your site is experiencing - for example, whether it is targeting layer 4 (IP/port) or 7 (HTTP/URL), what the scale of the attack is, how many bots are in the attacking botnet, and whether you have layer 3 protection at the firewall level for synflood attacks.
 +
 +For example, LiteSpeed Advanced Anti-DDoS Setup will efficiently protect against Layer 7 HTTP and Layer 4 TCP bot attacksl, but not against Layer 3 SYN Flood attack. ​ SYN Flood send SYN packets with spoofed source IP addresses and require Layer 3 protection at the firewall level. If a TCP connection established,​ it is a Layer 4 attack, but if a TCP connection is not established,​ it is Layer 3.
 +
 +Layer 4 TCP connection floods ​ can be detected and blocked by LiteSpeed Advanced Anti-DDos Setup Layer 4 connection hard limit settings.
 +
 +For large scale attacks, server kernel level settings may need to be adjusted to handle the large amount of HTTP requests during the attack.
 +
 +Generally speaking, LiteSpeed can handle a up to 1000 bots without problem. If bot number are well over 1000, while the LiteSpeed Web Server can handle the concurrent connections,​ typically server memory or PHP execution become bottlenecks. LiteSpeed Web Server can be configured to cache the attacked page, reducing the server/PHP resources ​ and increasing the server overall capacity, but this is an example that is well beyond the scope of this service.
 +
 +When you have a front-end proxy/CDN, the Denial of Service Packet Filter Setup may not work, since it blocks attacking robots at the IP level with iptables. When there is a front proxy, it only sees the IP of the proxy, and it cannot block the proxy IP, as all traffic is coming from that IP. If you have CloudFlare Pro or a similar service already, you may not need Denial of Service Packet Filter Setup Service since they do a similar job.
 +
 +Should you need such advance support services, they may be requested through evaluation and quotation. ​
  
 To order LiteSpeed Advanced Anti-DDos Setup Service, please [[https://​store.litespeedtech.com/​store/​cart.php?​gid=5|visit our store]]. To order LiteSpeed Advanced Anti-DDos Setup Service, please [[https://​store.litespeedtech.com/​store/​cart.php?​gid=5|visit our store]].
  
 +===== Never set Use Client IP in Header to Yes =====
 +To restore real visitor IPs, navigate to **LiteSpeed WebAdmin Console > Configuration > General Settings** and set **Use Client IP in Header** to ''​Trusted IP Only'',​ and add your CDN such as CloudFlare IPs/subnets to the trusted list. Never set **Use Client IP in Header** to ''​Yes'',​ since clients can spoof IPs with the ''​X-Forwarded-For''​ header that is sent to CloudFlare.
 ===== Troubleshooting ===== ===== Troubleshooting =====
  
 +==== Check concurrent connections ====
 +To check the number of concurrent TCP connections,​ run the following command:
 +  netstat -an | grep 80 | grep ESTA | wc 
 +
 +To check concurrent connections sorted by IP, run the following:
 +  netstat -ntu | grep ESTABLISHED | awk '​{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr
 +  ​
 +Please keep in mind that ''​netstat -ntu''​ will list TCP in TIME_WAIT state, which will inflate the number. For the correct concurrent TCP connections counting method, you should only count TCPs in ''​ESTABLISHED''​ state. Hence ''​grep ESTA''​ or ''​grep ESTABLISHED''​ will be required.
 +
 +==== Analysis of IPs from Attacker ==== 
 +
 +Bad IP's can make quick connections,​ and you end up with many ''​time_waits''​ which you won't see when just looking at established.
 +
 +If you don't necessarily count concurrent connections,​ and just want to analyze which IPs might be attackers, you can include ''​time_waits''​ connections. Run the command without ''​grep ESTABLISHED'',​ which gives you the ability to see what IP's just connected and dropped and may need to be blocked:
 +
 +   ​netstat -ntu | awk '​{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n | awk '$1 >= 5 {print $0}'
 +
 +An attacker could make a connection, send requests to an expensive URL, wait a little while, and then close the connection. If the server does not abort the process, the backend will be used up soon, as it will keep serving requests that have been abandoned. The above command will be useful during such a situation. ​
 ==== Check the Banned IP and Reason ==== ==== Check the Banned IP and Reason ====
 If an IP has been banned, but you don't know why, you can check it with SSH. Here is an example of a connection that was banned because it reached the hard limit. If an IP has been banned, but you don't know why, you can check it with SSH. Here is an example of a connection that was banned because it reached the hard limit.
Line 118: Line 162:
  ​[NOTICE] [x.x.x.x reached per client hard connection limit: 1, close connection!  ​[NOTICE] [x.x.x.x reached per client hard connection limit: 1, close connection!
  ​[NOTICE] [x.x.x.x] bot detected for vhost [N/A], reason: OverConnHardLimit,​ close connection!  ​[NOTICE] [x.x.x.x] bot detected for vhost [N/A], reason: OverConnHardLimit,​ close connection!
 +</​code>​
 +or
 +<​code>​
 + ​[NOTICE] [x.x.x.x] bot detected for vhost [N/A], reason: OverConnSoftLimit,​ close connection
 </​code>​ </​code>​
  
Line 139: Line 187:
  
 If ModSecurity blocks a request and LSWS sees the IP as trusted, the request is still served with 403 response, but that IP won't be blacklisted. If an IP is blacklisted,​ LSWS will stop serving future requests from that IP. If ModSecurity blocks a request and LSWS sees the IP as trusted, the request is still served with 403 response, but that IP won't be blacklisted. If an IP is blacklisted,​ LSWS will stop serving future requests from that IP.
 +
 +
 +Trusted IP can be either set on server level or virtual host level through .htaccess ​
 +
 +=== Set Trusted IP on Server Level ===
 +In LSWS Admin Console Server → Security → Access Control → Allowed List, you can set Trusted IP there with trailing “T”.
 +
 +=== Set Trusted IP on Virtual Host Level .htaccess === 
 +Since LSWS 5.4RC1, LSWS has virtual host trusted IP support, where you may use ''​Trusted 1.2.3.4, 5.6.7.8''​ in the Virtual Host document root .htaccess to unblock a blocked IP and make that IP trusted for that vhost. This is not the same as the **Trusted IP** configured by Admin at server level. It has no effect on bandwidth. The main effect of adding it in .htaccess is to take that IP off of the blacklist and disable WordPress Protect and reCAPTCHA when accessing that specific virtual host. 
  
 ==== Drop or Deny ==== ==== Drop or Deny ====
 What if ModSecurity does a drop (TCP FIN) rather than deny for a trusted IP? The trusted list only has an effect on the "​drop"​ action, but not on the "​deny"​ action. A trusted IP won't be added to blacklist, but trust status has no effect on other actions. What if ModSecurity does a drop (TCP FIN) rather than deny for a trusted IP? The trusted list only has an effect on the "​drop"​ action, but not on the "​deny"​ action. A trusted IP won't be added to blacklist, but trust status has no effect on other actions.
  • Admin
  • Last modified: 2018/12/13 20:08
  • by Lisa Clarke