Apache - SYN attack, Unknown requests – Unknown IP Ranges – Not able to stop – Help us!

Hi All,

For the last 6 months our site has been under severe brute force, syn flood attack. They keep bombarding a single URL of the server and it is xml file. They are not attacking any other URL.

e.g. http://www.example.com/rss123/attackedfilename.xml

We have removed the xml page from our site but still they keep on sending requests, this is for the last 6 months non stop.

The IP has been changed just to see and they are sending several thousand requests per second. The requests come from different IPS and different ranges, so you can not even block the IP’s. They seem to be coming from a legitimate IP’s.

Due to this I have had to pay for an extremely expensive server which holds 8 GB of RAM and quad core processor etc, however, even with this the server server still reaches critical level, just because these requests are eating up my resources.

Our technical team has been working on all aspects of apache server security, external modules, firewall, hardware firewall from beginning but still we are not able to stop them.

We have installed following modules.

1) mod_security
2) mod_evasive
3) Firewall

We have worked with the hosting company and their technical team leader, he installed the best CISCO hardware firewall and tried to stop them, but in vain.

We have checked our server to see if anything from our site is causing the request, no extra file uploaded on to the server. For example if some file has been upload or some text has been added to the file (checked if we’ve been hacked). Even though we checked for any hacks, I am still wondering if there is something we do not know about. Can a hack lead to huge amounts of traffic?

We need some help to stop these attacks. We have searched a lot and have found that sites that get attacked like this have only one option is to shut down till it stops. I really hope that will not be the case for us. Please let us know if any one has any ideas to deal with this.

We are willing to try any small suggestion which might help from coding to scripting to modules to firewall. So please provide suggestion/solutions and way to get us out of this.

Also could it be our own part of php code which can do this? We are ready to check every php file to make sure it does not have any line of code which can be dangerous?

Thank you for your help in advance! Help!

Regards,

Sam

I just did some reading on the syn attack (http://www.cisco.com/web/about/ac12...ng_attacks.html actually some interesting reading) and it details a few ways to help protect against syn flooding, however nothing it appears is a cure-all solution. It also talks about syn spoofing in that the packets are sent to the server from one person containing a different return ip address so your server is essentially sending back packets to IPs that won't reply. SYN cookies seem easy to implement on a server and should help out a lot. Generally the server will send out the syn-ack packet then discard the syn request. only if the other computer responds, the server will reconstruct that syn request and allow it through. If the request is from a spoofed IP (as it sounds it is) the initiating computer will not respond. The article clarifies this stuff a little more.

Other than that, I would think some sort of packet filtering might work as well. I would think if the attack is a request to one specific file, you could filter to look for that file name and deny all those requests at the firewall to prevent the server from even seeing them. However I have limited work with a firewall (I work over an intranet where the firewall is only needed to setup vpn connections and external connections to/from the internet) so I could be wrong with this plus it would be trivial to change the file and essentially restarting the attack.

Also, I doubt that there is some php file that is causing this. The attack itself works by essentially not responding to a TCP handshake. This should all be completed by the TCP stack before php even gets notified of an incoming request.

here is another article I found that talks about how to implement syn cookies to help alleviate the attacks.
http://www.securityfocus.com/infocus/1729

Hi @ IAmALlama,

Yeah, we have been working on this SYN Cookie stuff. Regarding, your second thought, we have tried this stuff even with the hardware firewall where any request coming for this URL will be discard, however, its not that easy.

May be we missed something, so, we are going to look at this option again just to make sure if its possible.

For PHP, i have doubt also. We have checked all the major code and there is nothing suspicious we have found, if you have idea for any php code which can cause such things, please let me know and I ll recheck entire site for the code.

I will keep the thread updated with my latests and please keep sending your suggestions when u get.

Thanks

Sam

Hi,

I just had a long meeting with my technical team and following are the updates.

1) SYN cookies is already Enable but not making much difference.
2) I will provide the packet details soon for the request to give the extra idea for the attack.
3) Following are the details I have got from my linux admin. This will help you to trace the issue in better way.
Problem: Apache SYN_RECV

OS - RHEL5
kernels - 2.6.18-92.1.22.el5-x86_64
2.6.18-92.el5-x86_64

rpms:-
kernel-devel-2.6.18-92.el5
kernel-headers-2.6.18-92.1.22.el5
kernel-devel-2.6.18-92.1.22.el5
kernel-2.6.18-92.1.22.el5
kernel-2.6.18-92.el5

OS Type:
cat /etc/issue
Red Hat Enterprise Linux Server release 5.2 (Tikanga)
> cat /proc/version
Linux version 2.6.18-92.1.22.el5 (mockbuild@hs20-bc2-5.build.redhat.com) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Fri Dec 5 09:28:22 EST 2008

We are providing 403 code for the URL request. Following is part of Access Log

94.70.118.139 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; FunWebProducts; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; CT2077543_4.5.188.7)"
89.216.230.148 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; sr; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10"
82.81.54.226 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB5; CT2077543_4.5.191.15)"
85.229.15.86 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6; .NET CLR 2.0.50727; CT2088752_4.5.188.7)"
92.237.189.17 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10"
84.106.127.218 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; eSobiSubscriber 2.0.4.16; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; InfoPath.2; .NET CLR 3.5.30729; .NET CLR 3.0.30618; CT2088433_4.5.188.7)"
87.93.30.98 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30618; CT2088347_4.5.188.7)"
93.86.61.247 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.0" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.2; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; CT2077543_4.5.188.7)"
91.152.228.27 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; Creative ZENcast v2.00.13; CT2088347_4.5.188.7)"
94.69.164.32 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; FunWebProducts; .NET CLR 1.1.4322; .NET CLR 2.0.50727; CT2088700_4.5.191.15)"
82.201.180.177 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB6; CT2077543_4.5.189.28)"
83.248.2.230 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SIMBAR={46BC3752-9118-483D-8E88-CD3E89FCD192}; GTB6; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; CT2088752_4.5.188.7)"
99.235.137.30 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6; .NET CLR 2.0.50727; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; CT2077543_4.5.191.15)"
216.155.136.84 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.0.04506; .NET CLR 1.1.4322; CT2077543_4.5.188.7)"
217.123.166.205 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB5; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.2; .NET CLR 3.5.30729; .NET CLR 3.0.30618; CT2088433_4.5.188.7)"
86.96.227.88 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; CT2077543_4.5.188.7)"
203.115.189.77 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; CT2077543_1.5.48.2; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10"
203.82.79.102 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; CT2088347_4.5.188.7)"
88.195.52.126 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 3.5.30729; .NET CLR 3.0.30618; CT2088347_4.5.189.24)"
77.81.114.171 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6; SIMBAR={B471FCBA-22ED-11DE-91A3-00196693641D}; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; CT2077543_4.5.191.15)"
92.84.250.65 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; CT2077543_4.5.191.15)"


netstat:

tcp 0 0 domain.com:http 85-156-91-20.elisa-mo:55168 SYN_RECV
tcp 0 0 domain.com:http 220.255.7.227:27183 SYN_RECV
tcp 0 0 domain.com:http 5e03cbc4.bb.sky.com:51086 SYN_RECV
tcp 0 0 domain.com:http 79.126.234.198:18139 SYN_RECV
tcp 0 0 domain.com:http 78.148.175.148:11115 SYN_RECV
tcp 0 0 domain.com:http 83-154-143-68.rev.lib:61479 SYN_RECV
tcp 0 0 domain.com:http ABTS-North-Static-248:54775 SYN_RECV
tcp 0 0 domain.com:http 90-230-131-95-no130.tb:1134 SYN_RECV
tcp 0 0 domain.com:http static-host119-73-6-2:49538 SYN_RECV
tcp 0 0 domain.com:http 222.127.130.238:gtp-control SYN_RECV
tcp 0 0 domain.com:http acl1-1571bts.gw.smartbr:g5m SYN_RECV
tcp 0 0 domain.com:http athedsl-282427.home.o:60002 SYN_RECV
tcp 0 0 domain.com:http CPE-58-166-77-138.nsw:60067 SYN_RECV
tcp 0 0 domain.com:http C-59-101-99-107.syd.c:51097 SYN_RECV
tcp 0 0 domain.com:http ti0111a380-2667.bb.on:60993 SYN_RECV
tcp 0 0 domain.com:http 92.81.2.242:60451 SYN_RECV
tcp 0 0 domain.com:http 118.100.120.248:pserver SYN_RECV
tcp 0 0 domain.com:http triband-del-59.178.84:50140 SYN_RECV
tcp 0 0 domain.com:http cpc4-leds5-0-0-cust82brpd SYN_RECV
tcp 0 0 domain.com:http ALyon-153-1-8-78.w86-:59494 SYN_RECV
tcp 0 0 domain.com:http 120.28.199.183:3comnetman SYN_RECV
tcp 0 0 domain.com:http h248.4.16.98.dynamic.:60758 SYN_RECV
tcp 0 0 domain.com:http 89.211.205.59:64217 SYN_RECV
tcp 0 0 domain.com:http CPE-124-187-26-30.qld:ff-sm SYN_RECV
tcp 0 0 domain.com:http frw.Gloworld.com:59104 SYN_RECV
tcp 0 0 domain.com:http 220.255.7.182:winpoplanmess SYN_RECV
tcp 0 0 domain.com:http srisaionline180.excell:1232 SYN_RECV
tcp 0 0 domain.com:http CPE-60-230-16-150.vic:52611 SYN_RECV
tcp 0 0 domain.com:http 203.82.91.102:41318 SYN_RECV
tcp 0 0 domain.com:http 69.171.165.50:32454 SYN_RECV
tcp 0 0 domain.com:http dsl-TN-static-195.:corbaloc SYN_RECV
tcp 0 0 domain.com:http 210.186.66.179:49330 SYN_RECV
tcp 0 0 domain.com:http ABTS-North-Dinuexpansion3 SYN_RECV
tcp 0 0 domain.com:http c122-106-133-46.livrp:49273 SYN_RECV
tcp 0 0 domain.com:http 173.subnet125-1:nssalertmgr SYN_RECV
tcp 0 0 domain.com:http 121.246.52.30.dynamic:63977 SYN_RECV
tcp 0 0 domain.com:http mobile-3G-dyn-BC-179-1:4464 SYN_RECV
tcp 0 0 domain.com:http crd48.neoplus.adsl.t:aminet SYN_RECV


Following we have done till now is mentioned below for the configurations.

###############
sysctl.conf

##############
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536

# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
# Enable IP spoofing protection, turn on Source Address Verification
net.ipv4.conf.all.rp_filter = 1
# Enable TCP SYN Cookie Protection
net.ipv4.tcp_syncookies = 1

# 65536 seems to be the max it will take
net.ipv4.ip_conntrack_max = 1048576
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 87380 8388608
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_window_scaling = 1


#############
fwsnort, bfd burnintest chkrootkit ddos faf lsm nobody_check sim apf

#############
modsecurity-apache

LoadModule evasive20_module /usr/lib64/httpd/modules/mod_evasive20.so

<IfModule mod_evasive20.c>
DOSHashTableSize 3097
DOSPageCount 3
DOSSiteCount 50
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 30
</IfModule>

LoadModule security_module /usr/lib64/httpd/modules/mod_security.so



<IfModule mod_evasive20.c>
DOSHashTableSize 3097
DOSPageCount 3
DOSSiteCount 50
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 30
</IfModule>

####################
Again,

Hope this helps you to see the issue in details. I have also put the latest configurations to keep site going on.

If it may be where server has some maleware or spyware, please let me know the solution and we can scan the pc for that too. Besides these, please let me know for any suggestion you think will be helpful.

Thanks again for your time and have a wonderful day.

Sam

Hi,

Have updated the firewall for few IP rules but in vain. Also 2-3 questions that have come to us are can it be a Virus or any malware which causes this?

We have the antivirus installed on our dedicated server and its saying server is fine, however, if needed we can try for new ones no matter for paid ones.

I am still doubting for any php code which can cause such things but I am not able to find anything regarding this.

however, we are trying and of anyone has any suggestion, please share them with us.

Thanks

Sam

Unfortunately this is getting beyond me. Most of my work is dealing with an intranet server for my company, no outside access.

Now I doubt there is anything with your server that is causing this at all. The SYN attack starts from someone elses computer sending a request to your server for a file. There is a 3 way TCP handshake that goes on.
1) The client starts that handshake by making the file request.
2) Your server opens a SYN connection and sends a packet back saying hi and leaves that connection open until the client recieves the packet.
3) Usually this would be the client recieving the packet and signaling they got it.

This marks the handshake as completed. What the SYN attack does is gives the server a fake IP address to send the packet in step 2 above. This leaves the SYN connection open until your server times out the connection. You could also try to allow more SYN connections, lower the timeout...etc. However this is very easy to get past by just sending more requests.

The SYN cookies generally will add an extra step after 3. In step 2 it changes the packet it sends back to include a temporary key in the packet header. Then it closes the connection instead of waiting for it to time out. This frees up the available SYN connections by not leaving them open. Then after step 3 (after the client has recieved the server response) the client continues their connection and sends back the packet the server sent out with the temp key. The server then knows that the computer is the one who made the initial request and can reconnstruct the previous connection and complete it.

Because the connection has to be initiated by a client, I doubt that it would have anything to do with your server other than being the target of said attack. I mean I guess it could be possible that your server is opening requests to itself with fake IPs, but I honestly doubt it.

Hi,

Thanks for all your valuable details. Yes, even now I have started to think in that way that there is something in my server which is helping this attack.

I am checking the server once again to see nothing unintentional crap is on the server; however, you want me to check anything specific in the beginning?

Sam

Hi,

I checked entire server again to see any crap is there on the server. I am not able to find anything unintentional.

Anyone has idea regarding the Firewall which drops request at entry point for specific URL request? Currently we have tried are IP and pattern based only to slow down the attack, however, they are being smarter and keep generating new bunch of IP address.

Sam

Hi all,

Scanned the server with rootkit antispyware, no infection found. Regarding the firewall, put on BFD firewall over APF, still requests are not getting down.

Also IP table is getting full of new ips and it is keeping network and site slow. Please advice for next steps to improve the performace.

Sam