[an error occurred while processing this directive]

» Home » website statistic » Analyze web site logs and get Search engine top position -

Top ranking on search engines using web site logs?

Probably many of you are using some kind of tool to check web site (page) search engine position. Yes, there are useful online tools and software as is WebPosition, WebCEO, IBP... where you can check position of some web page, tracking history of position on search engines

The fact is that in web site log, that history is already written, not precise as is in above tools, but enough to see if some web site page is positioned on the first page of SERP (search engine results pages), or it is on second or third...

Why to go to the search engines when search engines are coming to your site? Not exactly search engine, visitor is who is carrying information of your search engine position. Analyzing referrer string, you could know what keyword was used and on what page of SERP is positioned your page for that keyword. We are talking here about raw site log, not statistic you can find in your hosting account. It needs to be downloaded from your server to be analyzed.

Google referrer string

First example (older):
http://www.google.com/search?q=my+keyword&hl=en&start=10

Second example (newer):
http://www.google.com/url?sa=t&source=web&cd=18&q=my+keyword

This web site log entry is stripped down to the referrer string only. For sure, that log line shows visitor's IP, time, landing page and browser used. In above (first) example, your site page is positioned on the second page (start=10) on Google SERP (search engine results page) for keyword "my keyword".
Second example (latest format of google referrer string - from 2009) shows exact SERP position for that keyword.(cd=18)

Missing keyword parameter (q= is empty, only position is shown) from google referrer happens when visitor is logged in some google service, and it has something with privacy. Comparing with known position and keyword, you can find what keyword is used.

Yahoo example

http://search.yahoo.com/search?p=my+keyword&b=11

Like above for google referrer, here "b=11" shows that our page is positioned on the second page of search engine results for that keyword.

Bing example

http://www.bing.com/search?q=my+keyword&first=11

For Bing, "first=11" shows that our page is positioned on the second page of search engine results for that keyword.

More info from referrer string

If these parameters are missing from referrer string, your page is positioned on the first page of search results.
Now, there are more factors, helping to better target product to visitors. Search position of your Web page is not always the same. It will be different for different visitors (geotargeting, described below). Additional info could be pulled from search engine referrer string, comparing those strings (same search engine). For example, language, safe browsing, source (web listing,toolbar,custom search..). Above examples of referrer strings are shorted for purpose of this guide.

Geolocation

Sometimes is very important to know in which Country visitor is located. To better target products what are restricted to one or more Countries, check your logs for other search engine domains (subdomains). Search engines are redirecting visitors to their local servers, serving results according to their location.

Clear example is Yahoo, where uk.search.yahoo.com shows that visitor is using Yahoo UK to find your website (web page). If you see more visitors from UK, you should push product for UK visitors, or to get more links to that page from sites based in targeted Country. Same is for Google (google.ca - Canada, google.com.au - Australia)

When visitor is using search.yahoo.com, or google.com or MSN/LIVE, or any other search engine (not local), there is an other way to find from what Country is visitor - from IP address. That is described on IP address to Country page.

Web page position history and automation

Raw site logs are full of pure gold, but we need to separate that gold from an other data. So we need to use little programming to pull out what is interesting for us. This article is focused on search engine position, and here is an example how to get useful data from website log.

It is required that you know simple programming, language is not important. Flat text files are used to store data, to make this simple.

First, in program directory (folder) make subdirectory for every tracked search engine (i.e. google/ for Google).

Second, copy raw website log to that directory or somewhere when program could find it.

In program, we need to define search engine domains (pattern match), folder where we add keywords files for that search engines, and patterns (pairs) to match and extract keyword and position from referrer url.(q=,start=)

Option is, when parsing log lines, referrer strings containing question mark could be added to one separate text file to find unknown search engines, if they are worth to track later.

Now, let's start parsing web log lines. In this example, we are looking for referrers from Google.com Program routine must find google.com (or with wildcards) in referrer field of log line. If google.com is found (in referrers field), next is to get page name (or category/page numbers combined), date (month,day), and complete referrer string.

Next step is, extracted page name (without extension and path) will be name of our text file in "google" directory (folder). Extension of that file could be .txt, and option is to add month to file name if you want to have all searches for one month in one file. So our file looks like "pagename-Jan.txt" or "pagename-1.txt". To avoid confusion wih possible same named pages in different folders, best to use "directory-pagename-Jan.txt". For example, better is to use directory-index-Jan.txt than index-Jan.txt

Next is to add data extracted from web log line to our small database (pagename-Jan.txt)

Every line in file contains pipe or comma separated fields. So, first field is date (with or without month, see page name), and second is referrer string. Why referrer string, and not extracted keyword and position? It is up to you, what field you will add to line, but adding (archiving) referrer string to database, you have an option to check exact position later using that url.

Now when we are finished with parsing and extracting data, it is the time to use that data.

Just fast look, opening that (page name) database in any text viewer, and you know what keyword(s) was used to find that page, when, and where is page positioned (SERP). Position changes are available also (first page, second,etc...). Anyway, with little more programming, better formatted report could be displayed. More fun guaranteed...

Top ranking?

Top ranking for keyword "mp3" is not what we are talking about here, but top ranking for keyword "I want to buy mp3 player" and similar "long tail" keywords are what we are looking for. And top position for promising long tail keyword could mean 1 click = 1 sale, or far better conversion rate than usually.

So, that is not only about tracking page position on search engines, new keywords means new ideas and new optimized pages. And many of those keywords (ideas) are already in your website log. Analyze, optimize...:-)

to top