View Full Version : Using your Logfile Analysis to improve your website


CMA
09-05-2004, 04:53 AM
Good evening to all Guys!
I send this article to seo-guy 2 wekks or so ago. I think the article is not worth to be published on the main page though, but maybe it is nevertheless quite interesting for the seo starters:-)
Please enjoy and comment, I am not a seo and my mother tongue is not english. Some parts were translated..some I wrote knew for you:-)


Nevertheless:

Using Logfile Analysis to improve your Website
Log Files are storing the most important data of your site. They can tell you a lot of important information about your visitors. In this article you learn how to use these information to improve your website.
Important things that a Log File tells you:
* Through which Search Engines they have found your site
*How long they were staying
*What Resolution and Browsers your Visitors are using
* Sites that are linking to you, so called Referer Websites
* Keywords that your site is listed and that bring visitors
A Logfile Analysis can bring up a great load of detailed information about many other aspects. In this article only the important things you can use to improve your website are mentioned and explained.
The important point is that many Hobby-Webmasters and even some SEOs forget about this important issue.
In this Article I will use AwStats as an example of a Logfile Analysis Software. The tricks and information in this article and the effects for SEO and Website Content can also be achieved with other Logfile Analysis Software. In addition to Logfile Analysis it can make sense to view the Logfiles from time to time in an ASCII Text editor (for example VI). The Keywords and Keyword-Phrases that visitors used to come to your website are in the Log Files. You can copy these Queries into your Browser and see the SERPs that your Visitor saw before he clicked on a link and found your website.
One thing that is very important when you choose your Logfile Analysis Software, is the accuracy of it. The most important data if you analyse your Logfiles are your Referrers, Keywords and the visits of Robots.
AwStats Screenshot
In order that all of those that are not analysing there Logfiles at the moment can take a look how a Logfile Analysis Tool looks like. This is a Screenshot of AwStats, a Logfile Analysis Software that is installed directly on the server where your website is hosted. Awstats analyses your Server Logs in Real-Time and provides you with important information.






Detailed Analysis of your Visitors
For a accurate and detailed analysis I would advise to take the Logfile information of the last week. It is easier to analyse trends when you analyse on a broader basis. On days where you changed a lot on your website, for example when you added Backlinks, information or improved the site structure. To analyse the statistics of your website you should spend as much time as necessary. The more time you spend on it the more valuable informations you know about your visitors and what they expect to find on your websites.
Summary
The first point in Logfile Analysis Software is usually the Summary, where you can see the most important points of your analysis. You can use this information for a quick overview. For example you can see the daily unique visitors and Pageviews, and then view the detailed information to see where you can improve your websites.
An important information you can find right at the Top is the time when the first visitor of the day came to your site. This is only a small piece of information, but I can say from my experience that this is nevertheless quite useful.
If you own a small website you can use this information to provide some service fort these first visitors of the Day. The goal when somebody visits a website is to find the information he or she is searching for. The Goal of Webmasters is to get their articles read and linked by interested visitors. To make sure that all these things happen, you should serve your Users. You can’t provide new information or articles manually all the time, but there are many other possibilities for service where you can interact with your visitor.
Possibilities to greet your first surfers of the Day
* Insert a News-Ticker with news from other Sources
* Release new articles with the help of Cronjobs
* For smaller and hobby websites: Greet your visitors according to the time “Good morning” “Still awake?” “Good evening” are only some examples. You can even adjust these greetings according to the topics mentioned at your website.
Advantages of Greeting your Visitors
The visitors fell more comfortable when you greet them personally. They will stay longer on your website because they got the feeling of interaction. If you provide new articles and information then your visitors might come daily just to see what’s new.
The more Possibilities you provide for your users to give you Feedback and interact with you as a webmaster the more Links you might become from interested Webmasters. So you can boost your Link Popularity and Pagerank through providing possibilities for interaction and Feedback. Try to build up a personal contact to your target audience. If you provide great content and are ready to share it, the word will be spread through the net.

Daily Unique Visitors
The number of unique visitors of a website is an interesting aspect for Webmasters. There is a wide range of possibilities to increase the number of daily visits. The most important aspect of this is to provide unique, many and fresh content on your website. Write articles, provide a board, inform about news or write reviews about your products. If you have not got enough visitors on your websites, improve your sites Content and Navigation.
Number of Visits
Visitors that come back often, also referred to as regular guests, are the visitors that really count. Don’t waste your possibility to make your site worth visiting it over and over again. If the number of visits is low, below are some suggestions how you can increase them.
Possibilities to get regular guests:
* Add new content on a daily basis
* Provide News from your relevant topics on your index page

* Provide as many possibilities for interaction as possible. This could include a message board, the possibility to send feedback, contact possibilities or a chat. Work together with other website from the topics you are targeting to provide more information.
Summary
When you achieve the goal to increase the number of daily visitors on your website you have gained a lot. The higher the number of regular visitors is the more independence you gain. Try to reward your regular visitors and authors and take their wishes and ideas serious. This is a rule that’s really important when you are trying to build up a successful community.

Pageviews
The Pageviews are an important point of your Logfile Analysis. In the ideal case the number of Pageviews per visitor should vary from 3 to 10 pages per User or higher. This number is depending or your sites structure and your content. The more sites a visitor looks at and reads the better it is for you. If the user spends a longer time on your websites its probable that he gets interested in further services and information.

Suggestions to boost up your number of Pageviews
* Create a clear navigation that includes your topics, use good internal link structure and link texts. This might be a lot of work and need some thinking, but in the end not only the search engines, but also the user profits.
* Write articles that spread over some pages and provide useful and focused informations. When you are using special terms, explain them in a kind of Glossary.
* Put Links to other pages and comment these Links. When you a providing a good assortment of links, people will link back to you. Always quote your sources and help your friends.
Summary
When you increase the number of Pageviews, you win. Your content gets read more often and your website gains more attention on the web. Use your time to create informative content and a good internal structure, it will pay off soon. Visitors and Search Engines are quite similar according to a website’s content: They both want information, easy to navigate with a clear focus.
Days of the Week
This information might not be interesting for bigger websites with more then 500 visitors daily. For smaller webs and hobby websites it might be helpful. On days where there are not as many visitors (on my websites this is Sunday) try to provide extra content. The better you react to your visitors need and guide them, the more visitors you get. Great Services and articles are spread very fast on the internet.
Countries
When you are analysing your traffic it is important where you get it from. I know it from my own website that it is very hard to handle visitors that are from other countries than your target audiences. The problem is most times the languages. If you get a lot of visitors from other countries that use other languages you think about a translation of the most important information. If you have visitors from a special university or many referrers from large website, suit your content and provide material according to their needs.
Robots/Spider
In these lists of AwStats you can hopefully find the Googlebot and other spiders. Track down the spiders. For the spiders it is important to find something new, this might be external or internal. So try to either enlarge and improve your website’s content, or get new Backlinks from topic relevant website. Normally you should try to do both.
Time spent staying on a website
When you are using AwStats for your Logfile Analysis there is a point where you can see how much time visitors spend on your website. If the majority of your guests only is about 0-30 seconds on your pages then you have to think about what went wrong.
Possibilities to increase the time visitors spend viewing
* Provide fresh and unique content
* Provide a summary on the index page, state clearly what information a visitor finds on this webpage
* Write long articles that take time reading and are read because they are really interesting
* Build up a commented directory with links to articles and topic related websites
* Build Teaser for your most important articles
* Create a good navigation and internal linking structure

Sub-Pages that are not visited
When you are analysing your Logfiles you also see an interesting statistic of the Sub Pages that where not at all, or not often visited. It is very important to improve the quality of these websites.
Reasons why Sub Pages aren’t read
* The site has not yet been indexed by the major search engines
* Poor navigation, site is “hidden” and not directly accessible
* No content on this Sub Page, information not relevant to the guest
Possibilities to improve poorly visited pages
* Place a central link to your poorly visited pages on your index page
* Improve the text content on these Sub-Pages
* Search for topic based link partners
* Link from your own, well visited sub pages

Improve External Factors
For the on the page SEO and improvement of your website the above mentioned points are important. This article focuses on this part of Logfile Analysis. Nevertheless I want to point out further possibilities how to use your Logfiles for Off the Page SEO.
Where your visitors came from
There is a list of referrers that is mentioned in every Logfile Analysis Software. For a webmaster it is important to know where your website is mentioned and where the word about your content has been spread. You should see your partner sites at the top positions. Your main goal should be to get guests send to you from other topic relevant sources that mention you because of your content.
Unfortunately, some Webmasters try to use referrer spamming. When you find Bogus links, mainly FSK 18 content or other Bull***
When you get links without having requested them, be grateful to the webmasters. When it is a site that’s related to yours, provide a link with information or just contact the other webmaster team. Co-Operations and working together can be very effective.
Keywords/ Keyword phrases
When you are doing SEO it is important through which Keywords visitors found you. Sometimes you get a load of guests from Keywords that you didn’t target. When you are analysing your Keywords, your Logfiles and try to improve the Keys where you actually get your visitors from, you get qualified guests. When your visitors found you with Keywords that are not mentioned at your websites, create content and inform them
Summary of this article
The more effort you put into a good Logfile Analysis the better it will get. Improve your website according to your Logfiles and the wishes of your guests. A website isn’t worth anything if it hasn’t got the visitors it was build for. When you improve the number of unique views, your site can grow; you get more links and start becoming a self starter. I wish you all great luck with improving and analysing your websites and Logfiles.

jocelyn
09-05-2004, 08:11 AM
Nice post buddy.

I've been looking at my logs for a long time. In fact, I base most of my tweaking on the information that comes from those logs. Modifying a file that gets lots of hits will make you lose those hits if you are not aware of it when tweaking this page. Logs are often not friendly formatted and changes from one hosting to an other. Server tools are limited to a certain amount of data. For example you get the 250 top referers and often more than half are from your own site. So I made myself a log analiser to get the list of bots and the hits to files. This way I can tell you exactly what a file is found for what search ... I can spot files that don't get hits and tweak them for better search phrases. It's also good to find what obscure searches bring visitors to the pages too. You take this soft away from me and I'm simply lost now.

Why am I not trying to distribute the program I made? I made games and try to sell them through my web site. I don't even make 50$/month from my games sales because most will look for a crack or one dude will buy it and give it's registartion key to his friends. So this website and games project was flawed from the start. I was just too stupid to see it before I put thousands of hours on this stuff. So I'm not jumping the fence with this software quite yet. Also since the logs are not formatted the same way from one host to an other, the program is propriatary and will only work for the user I made it for. I made one for Dilligaf and his will not read my logs like mine will not read his logs. Plus, the raw log file is not always containing the same amount of data. For example I get a cumulative log of all hits from the 1st of the month and dilligaf gets them day per day. His log will be a few megs when mine will get humongous since I get about 10meg of logs per day, so around the end of the month I have like 300mb of logs to look at, no text editor will do this. So I can't just make one that fits all. This propriatary function could prevent people from using a version they did not purchase, but would people pay for this?

So far Dilligaf seems pretty flabbergasted (I like this word) about what he gets from it. I'll be adding more stuff to his version, then give it a better look before I get to test it with someone else. Before I can tell if this is worth working on and not be a waste of time like my games, I must evaluate the time it would take to adapt it to someones log sample and mostly if I'm just gonna be working for peanuts or get something out of this trouble this time.

Again, excellent post

CMA
09-05-2004, 11:17 PM
Thanks for your answer. Concerning logfile software types and data types of software there are such load of information available that I don't know, mnaybe another expert about this can put it into a article.
I am using AwStats, as stated in the article, and find it quite useful, providing me complete lists of referers and keys. Especially the pages that are not or poorly visited are important.. NEVER forget about them, if you make it right you can still get loads of traffic out of them, link them, quote them, and enlarge those sub sites with information:-)
Greetings
David

dilligaf
09-08-2004, 09:13 AM
David, I use weblogpro (or something to that effect) to crunch the logs and present similar information. I think it's been great for the purposes for which it was designed. What got me "flabbergasted" at Joc's tool is that it gives me specific details on the bots on one screen. Where as before I would get which bot, from where and hits, I can now see the path it takes through my site.

Point it to the log file, select the bot and.... more than you ever wanted to know about it's behavior as it rummages through your napsack.

I don't use it for customers per se, but it certainly makes it easier to track SE's. IMO Bot hits = PR

CMA
12-14-2004, 12:36 AM
Good day again guys ;-)
Any more suggestions how do improve my article? If this article did help you in gaining visitors-> please let me know it ;-)
Greetings
David

jocelyn
12-14-2004, 06:46 AM
[QUOTE=CMA]Any more suggestions how do improve my article?[/QUOTE]
Well I tried to get people to say what they would find or try to find in their logs, so I could add it to the log analyser software I'm making, if it was not already included. But I got mostly links to other log analyser. So I gave up and ask my friends through PM instead. Have a look at the thread : http://www.seo-guy.com/forum/thread4733.html , maybe you get a few more input from it.

Cygnus
12-14-2004, 07:49 AM
I may have missed it, but it'd be interesting to show the top paths a user takes...such as index -> contact us -> shopping cart

In my case, I can sort of figure that out by using Joc's program (pretty good by the way) to look at overall file hits and comparing that with search engine hits...its a dirty way of doing it, so that'd be my suggestion...now that the morning fog has lifted from my brain.

Cygnus

CMA
12-14-2004, 09:39 AM
Hello Cygnus!
Yeah thats some other kind of logfile analysis stuff.. pixel analysis tracking user paths.. anybody got serious information about this interesting detail of analysis?
I only look at exit pages very seriously the last days.

jocelyn
12-16-2004, 07:20 AM
[QUOTE=Cygnus]I may have missed it, but it'd be interesting to show the top paths a user takes...such as index -> contact us -> shopping cart[/QUOTE]
You did not miss it. I figured the entry page report would do about this, except it does not show what happend between the entry page and the shopping cart. What I mean is whatever the visitor does in between, was not as relevant to me since the major point is how did he find my site in the first place.

An other problem in doing this is when you have logs like PK_synths that makes your site look like a ghost town. More trafic, more visitors paths, more processing, more time to do the report... if it takes too long to produce the report, it will just make it look like a Mickey Mouse software.

If you send me more details on what you need in a path report, how to present it... I'll add it to the software if I can.

Cygnus
12-16-2004, 07:37 AM
More of a thought than anything.

I'm "assuming" from the entry page report that I can see the major pages entered, categories hit, and product pages selected -- it is more or less the path. The assumption is that they aren't going from landing page -> category 1 page -> product 5 -> category 2 -> product 119 -> shopping cart.

Since I have so many products, it would indeed look a little funny.

jocelyn
12-16-2004, 07:53 AM
The entry page report does this.

Find all hits to the page you ask the report for... find all unique IP's. Dig all hits done by those IP on the site. Then it filters out all internal hits so only the entry page for these IP's are left. Then it does the counting and all. The final result is how did the visitor find and enter the site to go to that page.

If you really need to follow a visitor's path, you could do a IP report to get the whole path of a visitor from the entry to the exit. But if you have 200 IP's to check, then you have to do 200 reports. This is where I see a werry werry long time to do the report with the whole trail path. I can't give details about PK's log, but a report for his site give a txt file of more than half a meg, if you ask for a minimum of 2 hits per search and no more that the top 10 referrers for each keyphrases... so if I wanted to keep the whole trail, he would have to run this one before he goes home to get the report the next morning, if he is lucky... lol