I joined the forum yesterday night and it would be wrong if I were to not enable it to be found via my own search engine which is in New Zealand.
So I have decided to show everyone and give you all a chance to see what a website looks like from a search engines point of view and I will list the way that we see the sites data as it comes back to us from our webspider which is indexing the www.seo-guy.com website right now into our search engine. So that all our users, which are around 650,000 a year will from now on, be able to find it in our results pages.
Everyone who uses this website and forum are looking at making a websites get better scores and appear higher in the results, I would doubt if any of you have ever seen what we search engines have to go on, when we index websites, so this may well be the first glimpse everyone has at being able to see a search engine actually working and how we form our results.
Our usual webspider has the user agent string of the "Linknzbot" and at this time we are developing a new webspider which is in use right now which has a temperary name, We are working on a new webspider that is able to find and index Java code and so far it is giving good results but it is a hard task but we are working at it right now.

So far our webspider has been indexing the website of
http://www.seo-guy.com/ (57 pages) have been found so far. Inside the site we can see that it is divided into directories which are called:
Root
/forum
/link-popularity
/seo-book
/seo-tools
/web-design
Inside the root area we have found and indexed the following eleven files:
-
-about.html
-articles.html
-google-pagerank.html
-international-seo.html
-link-to-us.html
-search-engine-consulting.html
-search-engine-optimization-resources.html
-seo-quote.php
-seo-services.html
-tutorial.html
Then from the forum directory we have found and indexed thirty six files which are:
-
-calendar.php
-coding-and-programming.html
-faq.php
-google.html
-index.php
-internet-marketing-forum.html
-login.php?do=lostpw
-member.php?find=lastposter&f=19
-member.php?find=lastposter&f=7
-member.php?userid=382
-memberlist.php
-misc.php?do=buddylist&focus=1
-newreply.php?do=newreply&noquote=1&p=8772
-private.php?do=newpm&u=1686
-private.php?do=newpm&userid=382
-profile.php?do=addlist&userlist=buddy&u=382
-profile.php?do=editavatar
-profile.php?do=editprofile
-register.php
-search-engine-optimization-forum.html
-search.php?do=finduser&userid=382
-search.php?do=getnew
-search.php?do=process&showposts=0&starteronly=1&exactname=1&searchuser=Web+Gazelle
-search.php?f=4
-sendmessage.php?do=mailmember&u=1686
-seo-chat.html
-seo-discussion.html
-showpost.php?p=8772&postcount=1
-showthread.php?p=82749
-showthread.php?p=82766
-showthread.php?t=1446
-showthread.php?t=1447
-usercp.php
-web-design.html
-web-hosting.html
In the seo-tools directory we found seven pages:
-
-google-dc.php
-link-popularity.php
-se-pos.php
-se-saturation.php
-spider.php
-yahoo-pos.php
This is what we get back from a website.
From the visit we have found one hundred and seven new keywords along with links going off to ninety five other websites. seven have already been indexed before by us and the rest are new ones.
Within this data a few billion calculations have been done and our database is being updated with the new information right now, so the websiteis already able to be found by our users.
The new domain names we found are placed in an area within which we can then go on to index them if we want to.
These were found by the spider as it indexed the forum data and it has looked at all the posts and has taken the domain details from each message posted and later today, I will have alook at them and see if I will index them into the search engine.
This is how we find links to other websites, this is why it is important for you to be linked up by other websites because as our webspiders crawl websites we collect all the links found as the index went on.