CSF

CSF is famous firewall developed for linux. here is the website: http://configserver.com/cp/csf.html

Installing CSF

To install it in linux, just download their package, unpack and run installation script:

wget --no-check-certificate https://download.configserver.com/csf.tgz;tar xvfz csf.tgz;rm -rf csf.tgz;mv csf csfinstall;cd csfinstall;sh install.sh
After done, run test script:

perl /etc/csf/csftest.pl
If you receive any errors like:

Testing ipt_state/xt_state...FAILED [FATAL Error: iptables: Unknown error 4294496895] - Required for csf to function
Testing ipt_limit/xt_limit...FAILED [FATAL Error: iptables: Unknown error 42948476295] - Required for csf to function
it means you need to add these modules from linux kernel using "modprobe"

like this:

modprobe ipt_state
modprobe xt_state
modprobe ipt_limit
modprobe xt_limit

If you are on VPS(virtual private server) and above command did not made errors to disappear, you will probably need to contact your hosting company to enable this for you on root server. After tried above modprobe commands and they FAILED missing modules, then contact your VPS provider and they need to:

A) vzctl set CTID --netfilter stateful --save --setmode restart (http://internetlifeforum.com/security-protection/1711-csftest-pl-failed-%5Bfatal-error-iptables-unknown-error-required-csf-funct/)

B) Option a (bad) - Stop your VPS using command "vzctl stop vpsid" and then do command:

vzctl set yourvpsid --iptables ipt_REJECT --iptables ipt_tos --iptables ipt_TOS --iptables ipt_LOG --iptables ip_conntrack --iptables ipt_limit --iptables ipt_multiport --iptables iptable_filter --iptables iptable_mangle --iptables ipt_TCPMSS --iptables ipt_tcpmss --iptables ipt_ttl --iptables ipt_length --iptables ipt_state --iptables iptable_nat --iptables ip_nat_ftp --save
to add these modules to the VPS. Then they need to restart OpenVZ VPS on root server by "vzctl restart vpsid"

C) option b - it may also work to add following line into VM config file end: vi /etc/sysconfig/vz-scripts/YOURVMID.conf

IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp ipt_state iptable_nat ip_nat_ftp"
Run test script again ( perl /etc/csf/csftest.pl ) on VPS or (vzctl exec vmid perl /etc/csf/csftest.pl) on host server to see whether you have all modules available on server where you want to install CSF. I got error "Testing xt_connlimit...FAILED [Error: iptables: Invalid argument. Run `dmesg' for more information.] - Required for CONNLIMIT feature", but i ignored and still everything appears to work.
If you got further errors, try also reading this topic.

If you receive errors like:

Testing ipt_state/xt_state...FAILED [FATAL Error: iptables: No chain/target/match by that name.] - Required for csf to function
Testing xt_connlimit...FAILED [Error: iptables: No chain/target/match by that name.] - Required for CONNLIMIT feature
Testing iptable_nat/ipt_REDIRECT...FAILED [Error: iptables v1.4.21: can't initialize iptables table `nat': Table does not exist (do you need to insmod?)] - Required for MESSENGER feature
Testing iptable_nat/ipt_DNAT...FAILED [Error: iptables v1.4.21: can't initialize iptables table `nat': Table does not exist (do you need to insmod?)] - Required for csf.redirect feature
and you are on the VPS (OpenVZ), the administrator of the VPS host node server may need to enable nat for the VPS:
vzctl set VPSIDHERE --netfilter full --save --setmode restart
and restart VPS:
vzctl restart VPSIDHERE

--------------

Once you sort out all above errors, CSF should have all necessary iptables kernel modules
Then do "csf -r" command to see if there are any errors.

If you receive errors like:

# csf -r
*WARNING* URLGET set to use LWP but perl module is not installed, reverting to HTTP::Tiny
Flushing chain `INPUT'
Flushing chain `FORWARD'
Flushing chain `OUTPUT'
Flushing chain `PREROUTING'
Flushing chain `POSTROUTING'
Flushing chain `OUTPUT'
Error: /sbin/ifconfig (ifconfig binary location) -v does not exist!, at line 2784
Maybe you are running CentOS 7+, install ifconfig: yum install net-tools

If you receive errors like:

*WARNING* Binary location for [HOST] [/usr/bin/host] in /etc/csf/csf.conf is either incorrect, is not installed or is not executable
install bind-utils: yum install bind-utils

If you receive errors like:

*WARNING* URLGET set to use LWP but perl module is not installed, reverting to HTTP::Tiny
install these perl tools: yum install perl-libwww-perl net-tools perl-LWP-Protocol-https

CSF is installed now:

After CSF installation, you should add your IP into the whitelist and ignore list:
echo "youriphere" >> /etc/csf/csf.allow
echo "youriphere" >> /etc/csf/csf.ignore
then restart CSF by command "csf -r" or "service csf restart" and then check CSF configuration file:

vim /etc/csf/csf.conf

CSF has tens of various security, anti bot, anti bruteforce rules. Default options are recommended. Important is to adjust content of the variables TCP_IN, TP_OUT. If you use some non standard ports, like hypervm (8888), webmin (10000), kloxo (7778), cpanel (2083), whm (2087), you may want to add their ports right there so it is not blocked.

Important are also variables CONNLIMIT and PORTFLOOD
It limit number of connections one IP can do to certain port or to all ports (per time interval or concurrently)
I have mine weberver set like:
PORTFLOOD = "22;tcp;5;300,80;tcp;20;5"
(one IP can connect port 22 maximum 5 times in 300 seconds, rest connections dropped, etc. note semicolon and colon dfference)
CONNLIMIT = "22;3,80;20"
(one IP can have maximum 3 new/SYN concurrent connections to the port 22)

After editing, restart CSF by command "csf -r" and watch the output. If there are no clear errors or advices, you may try disabling CSF test mode by editing config file mentioned above and on the top you will find setting: TESTING = "1" , change it to 0. Save file and restart CSF.

Make sure your services like websites works after enabling CSF.

This command can be done to disable watching slow suspicious processes, it can flood server by many mails when processes works for longer time:
find /etc/csf -name 'csf.conf' -type f -print0 | xargs -0 sed -i 's|PT_LIMIT = "60"|PT_LIMIT = "0"|g';csf -r;service lfd restart
How to uninstall, remove Config Server Firewall?

cd /etc/csf;sh uninstall.sh

Installing DDoS Deflate (another layer of protection, might not be needed or compatible because SCF has same function?)

here is how: http://internetlifeforum.com/security-protection/1132-how-manage-ddos-deflate-deny-ddos-attack/#post1675

Installing Fail2ban (another layer of protection, might not be needed or compatible because SCF has same function?)

here is how: http://internetlifeforum.com/security-protection/1136-installing-fail2ban-linux-prevent-various-bad-bots-attacks-bruteforce/

Installing DenyHosts (alternative to Fail2Ban in regards to ssh bruteforce detection/blocking)

here is how: http://internetlifeforum.com/security-protection/7796-how-install-configure-denyhosts-centos/

UWF

If you are on Ubuntu and CSF dont works, one can also try UWF (firewall). Search google on how to install. Here are the rules that can help mitigate DoS attacks.

Add into /etc/ufw/before.rules (put each part where it belongs)
Code:
### Add those lines after *filter near the beginning of the file
:ufw-http - [0:0]
:ufw-http-logdrop - [0:0]
 
 
 
### Add those lines near the end of the file
 
### Start HTTP ###
 
# Enter rule
-A ufw-before-input -p tcp --dport 80   -j ufw-http
-A ufw-before-input -p tcp --dport 443  -j ufw-http
 
# Limit connections per Class C
-A ufw-http -p tcp --syn -m connlimit --connlimit-above 50 --connlimit-mask 24 -j ufw-http-logdrop
 
# Limit connections per IP
-A ufw-http -m state --state NEW -m recent --name conn_per_ip --set
-A ufw-http -m state --state NEW -m recent --name conn_per_ip --update --seconds 10 --hitcount 20 -j ufw-http-logdrop
 
# Limit packets per IP
-A ufw-http -m recent --name pack_per_ip --set
-A ufw-http -m recent --name pack_per_ip --update --seconds 1  --hitcount 20  -j ufw-http-logdrop
 
# Finally accept
-A ufw-http -j ACCEPT
 
# Log-A ufw-http-logdrop -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW HTTP DROP] "
-A ufw-http-logdrop -j DROP
 
### End HTTP ###
full UWF rules tutorial: http://blog.lavoie.sl/2012/09/protect-webserver-against-dos-attacks.html