Monday, August 3, 2009

Bot’s attack unsuccessful. But frustrating for sure!!

Beginning at approximately 4:20pm CST on 7.30.09, the calan servers came under attack.
The attack, was likely the result of an automated software program commonly called a Botnet or just Bot for short.

http://en.wikipedia.org/wiki/Botnet

The good news is our system security was not breached. However, the attack itself had the side effect of overwhelming our data base’s Event Log file. I am sure most of you have heard the term, denial of service attack. While not the primary intent of the attack in this particular instance, by overwhelming the Log file it had a similar side effect. The Event Log file is normally less than 200 MB at any point in time. In fact, we have a daily maintenance procedure to “truncate” the file to retain maximum operating performance.

During the height of the attack the Log file grew to over 28 GB. That is 20,000 times larger than normal! Throughout the worst of the attack, we were in real time, attempting to contain this growth. While able to shrink the file on several occasions and provide more memory to arrest some of the latency being generated, in the end, the rate of growth simply outpaced all of our efforts and the system’s db began to lock-up around 10:40am CST, Friday morning. This resulted in what was effectively a loss of service, except of short periods of time, from 11:00am CST until 2:15pm CST on Friday. At this time the system was restored to operation inside normal parameters.

Over the weekend the db was taken off-line much of Saturday afternoon for maintenance. While there is still work to be done, the system is up, operating and no data loss was detected. New automated monitoring scripts are being installed to alert tech support earlier to indictors that may represent a similar reoccurrence. The IP addresses of the source of the attack have been identified, blocked and reported. However, as one of our techs noted, “people sophisticated enough to create these types of Bots are also smart enough to spoof their IP addresses.”

While we can only offer our apologies for any inconvenience that this incident may have caused. We are pleased to report that our securities we not breached and no data loss has been detected.