|
AffiliateBeginnersGuide | [an error occurred while processing this directive] |
» Home » website statistic » Bad bots hunt - how to find bad bots
How to find bad bots in website logWeb log statistic software is not that perfect. It is good for a fast look, and to get overall picture of number of visitors visiting your website. But, how many of them are real visitors? Many bots are hiding using "normal" User Agent, referral strings, asking for images and css files, different IPs... to avoid detection. As mentioned in How to block bad bots , there are a few categories of bots (web robots, web spiders, crawlers...) visiting your website: Search engine spiders, waste of bandwidth bots and bad bots. Search engine spidersSearch engine bots (spiders) are important for website. Google, Yahoo and MSN/LIVE/BING bots, coming from their IPs, are bots you want on your website (including some other search engines). Spider detection: All three search engine spiders above could be recognized by User Agent string. When User Agent is detected, next is to check IP (or resolved IP). When this is correct, you can be sure that is the real search engine spider. Not in all cases. IP could be forged, but that is not that often used to worry about. More often you will find User Agent string forged. That is why is good to know IP ranges of search engine spiders. Some IP and resolved IPs you can use to detect search engines web spiders:
** Fake Google spiders spotted from 66.249.16.* (Google IPs are from 66.249.31.xxx) These IPs are for example only, for better detection you need to use longer IP, i.e. 65.55.252.* for MSN, to be sure that is not some another spider. Best is to check WhoIs to get IP range. Bad bots scanner - DIYBelow are examples how to filter good bots and real visitors, to make bad bots detection easier.
For these tasks, you can use any programming language, code is not complicated. Personally,
I am using Perl scripts (under Windows), first, because I am not professional programmer,
second, due to that, I am modifying those scripts all the time, and I don't like to
waste time for some fancy forms. With Perl, it could be easier and faster. These examples (scripts) are used on six website logs,
all tasks are automated and data shared. DNS cache:
WhoIs cache:
Formula to convert IP address to IP number is on this page: Convert IP to Country. Here is an example for using WhoIs cache: Unknown IP is converted to IP number, looking for a match in WhoIs cache database (between Low and High IP number). This is done to avoid separate WhoIs requests for every similar IP. Note: In some WhoIs info, there is CIDR displayed only, without IP address range. To get IP address range from CIDR, since formula is little complicated, I am using subnets.pm (perl module). Search for "convert CIDR to IP range" or similar keywords to find function for your programming language. Bad bots hunt - Stage 1 - filtering good botsThis example above (search engine spiders IPs) shows how to recognize good bots and how to use that data to filter them. You should have a list of well known IPs. This list includes search engine spiders and other well known bots (other search engines, services, AVirus link scanners). IP addresses listed should be shorter (111.222.333. instead of 11.222.333.444). First step is to pull out all IPs from weblog, adding them to the temporary file
(list of shorted IPs, duplicates deleted). Next step is comparing list of well
known IPs with that temporary IPs list, deleting all known IPs from temporary file (IP exists in both files). We will take each unknown IP from that temporary IP list and perform scan on website log. You should use daily website log, not only due to speed, some filters / conditions (number of requested pages for example) could fail if monthly log is used. Good bot or visitor could become bad bot in that case. If you are in hurry to find who is abusing your website, break that monthly log to daily logs (using dates in monthly log) and scan one at once. Practice is required to fine tune filters/conditions. Bad bots hunt - Stage 2 - filtersThis example scan was performed on daily website log of website with static (.html) pages and protected cgi-bin directory (disallowed in robots.txt). This website is also example of that classic structure. I hope that you know behaviour of real visitor. His/her browser is requesting one page at the time, including images and css files. For example, let's take one IP from IP list and scan web log for it. First, we need to match that shorted IP with beginning of logline, to make it faster. When that IP is found, line is parsed and data added to one variable (list of lines with log data), and at the same time filters are checked: Simple filters (for one log line) are listed below:
Simple filters (for all log lines with that IP - all hits) are listed below:
More complicated (cloaked bots):
There are more filters (conditions) you can use, depending of structure of your website. Like you can see above, there are "possible" and "suspicious". More conditions (combination) should be met to confirm that some IP is the bad bot. That is also the reason why requested pages are listed in report. Every detected bad IP (including UserAgent) is added to corresponding file (base). Those bases are used later to check returning IPs, and to find what to block by IP. For example, one base contains web bots IPs, second new bots IPs, third bad bots, next - image scrapers, attackers and spammers, and so on.. Bad bots can be added to one database, it is then easy to spot IPs (ranges, when sorted) used for SPAM (bad proxies) Here is an optional step. When bad IP is found, DNS cache and WhoIs are scanned. If nothing was found there, IP2Country database is checked to get Country. Anyway, result is HTML report where some HTML codes are added (colors, bolds). Below are parts of report from this website log, just for this text :-) As you see, this one is already blocked. (error 500). Looking for well known scripts to inject code residing on other hacked website. No php here, sorry...
Pattern for match here is (=http) and NOT (yourdomain) While above is only Country displayed (WhoIs not found), below is one example with WhoIs output. This below is an example of bad programmed robot (see that version 1.0, I will add beta, too). Not requesting robots.txt, requesting script from protected directory. It is marked as scraper, due to many requests, but not in alert state. Time frame is also O.K., at least 2 seconds. In case that robots.txt was requested, and script (disallowed in robots.txt) is requested too, this bot will be marked different (bad bot).
There are more similar examples. Say, robots.txt is requested from one IP, while pages requests are coming from another IP (obeys robots.txt). In this case, WhoIs cache helps, showing that is the same Company (ISP). There are more false detections (Google and yahoo proxies, for example), where images are requested from one IP and pages from an other. For example, msnbot-media bot was detected as image scraper (in fact, it is image scraper). With little practice, it is easy to spot difference between false detection and real bad bot Simple Web site log scannersIf this above sounds complicated for you, there are another ways to scan your weblogs . Above script is used to detect bad bots, but you can use simple script/program only to scan, using list of known bad IPs or User Agents. List of proxies you can get from proxy websites, where is possible to download proxy list in *.csv format. List of bad User Agents and IPs you can find in various bot blocking scripts. Also, some websites/blogs are publishing their comment spammers and other bad IPs, so you can find some lists there. All you need then, is a little practice. For suspicious IPs that needs to be looked more closely, you can make simple script/program for scanning monthly weblogs. Another way is to enter IP address (A.B.C.) in Google search form. On that way, it is easy to confirm blog comment spammers IPs, since many blogs (and guestbooks) are recording IPs of visitors. Related:
|
|