WebProNews News Feeds
Search WebProNews:
Search Jayde:  

Jayde.com
Jayde.com:The WebProNews B2B Search Engine.
Submit Your Site

Inside eBusiness

Google's Index Expands and Gmail Goes POP
Graeme Wearden, ZDNet UK

New Service Keeps Track Of Blogger Blather
Gavin O'Malley, MediaDailyNews

Who Has the Largest Index, Google or MSN?
Andy Beal, Search Engine Lowdown

Yahoo's Premium Search Plans
Peter Krasilovsky, PaidContent.org

Microsoft Launches Search Beta: Platform Ho!
John Battelle, Searchblog


Related WebProWorld Posts

Is Yahoo’s Directory Worth Submitting To?
There are numerous reasons for webmasters to submit their sites to various web directories populating the Internet. The positives of having your site listed in a popular directory have been discussed in previous articles and on a multitude of search-related forums.

Can PageRank Be Taken Away
When discussing a site’s PageRank, talk usually focuses on ways of increasing it; "How can I go from a PR 3 to a PR 7" or "If I exchange a link that has a higher PR site, will my site benefit".

Google Results Vs Yahoo Results
According to a Nielsen//Netratings study, Google holds the top spot for search engine destinations. However, if you take a closer look at N//N’s findings, you see that Yahoo and MSN share the second and third positions, respectively.


eBusiness News

MSN Debuts Beta Version of New Search Service
MSN today debuted a beta version of its new MSN Search service, providing consumers with more useful answers to their questions and more control over their search experience, which results in faster access to the information they are seeking online...

Intel Gets New CEO
Intel's board of directors has elected Paul S. Otellini, currently president and chief operating officer (COO), as its next chief executive officer (CEO)...

Gmail Now Offers POP Access
Seemingly answering the cries of Gmail users everywhere, Google has announced a new feature for their growing email service...

Efficiently Monitor Over 100,000 Blogs
CyberAlert launched BlogSquirrel, a new service that enables businesses to monitor efficiently the contents of over 100,000 blogs...

Jason Dowdell

Microsoft Crawling Google Results For New Search Engine?

I was questioned today by a developer who was watching a particular IP address scan his site. The IP was 65.54.188.86 and is registered to Microsoft Corp. located at One Microsoft Way, Redmond, Washington 98052. This visitor was not sending the normal header information associated with a crawler to the web server such as an http robot name or identifying info or even a browser name.

Is Microsoft "using" Google's search results to populate their index? Discuss Microsoft's behavior at WebProWorld.

The behavior it demonstrated made it look like a crawler, especially since it was spidering urls that were no longer in existence (search engine spiders crawl site segments at regular intervals and often come back when an initial crawl left urls uncrawled) and doing so at the rate of 1 page every 3 - 5 seconds. The visitor started their visit at 7:37 am and was still on the site at 12:00 pm.

Correction, the data was there after all, here's the crawler info... msnbot/0.3 (+http://search.msn.com/msnbot.htm)

Here's the kicker

So now you're saying, so what, big deal. But this really is a big deal. It's a big deal not only because the urls this visitor was making requests to don't exist any longer but because the only place these urls can be found is in Google's search results using site:www.sitename.com. A similar query on MSN Search doesn't show the urls at all, even on the beta version of their new Microsoft search engine. But then within just hours of the visitors exit from the site the new same search at Microsoft's new search engine shows all of the urls in question being fully indexed within its results.

Get revenue at www.google.com/usegoogle

My Theory On This Mysterious Microsoft Crawler

The old msn required a fee to be crawled by its spider. But a few months back MSN dropped the fee and said they were going to begin crawling the entire web and doing it without charge. However, that's no easy task. So I believe MSN is using the results from Google and possibly even Yahoo to get all of the pages they've indexed on sites that have a relatively low page count in the current msn search engine.

First off, that's the fastest way to get the relevant pages from a web site. Sure they could just go to the site directly and start crawling but in doing so they're going to get tons of duplicate urls and urls that seem different but point to the same content. Crawling Google's results will eliminate the bandwidth to some extent but will not completely take care of the duplicate content issue their spider will encounter.

Secondly, crawling Google's results can act as a qualitative measure for their new search engine. By creating a baseline number of pages per site when the new Microsoft Search is launched and running a comparison on a regular interval for the next 6 months, they'll be able to determine internally if their engine is finding and indexing the same links and as many links as Google. Call it competitive analysis or whatever you want.

So Microsoft's Screen Scraping?

Obviously my conclusion should be taken as a grain of salt but it's a definite possibility. Microsoft very well could be screen scraping Google (or maybe even using their API, LOL) and crawling the urls it finds. It makes sense from a business case but I wonder if there are any legal issues there. I doubt it. It's like putting garbage out to the curb. Once it's out there it's fair game but I bet Google's lawyers would have more to say than that on the case.

Has anyone out there seen similar behavior on their own sites? Please comment with your qualitative/objective data if so.

Jason's article first appeared on his blog MarketingShift.com.

Comment on this article in WebProWorld.

Jason Dowdell

WebProNews
Articles: 12,861 Contributing Authors: 2,291
Thursday, Nov 11, 2004
Submit Article |  Contact WebProNews |  visit 

Organic SEO – Getting Started With Larger Sites

Jim HedgerBy Jim Hedger

Commercial websites are getting larger. Driven by the rapid evolution of content management systems, shopping carts and e-biz facilitation, and by the increasing sophistication of Internet retailers, "small" business sites averaging 500+ pages have become common.

Some large sites are very well focused and present relatively few problems for SEOs. Most larger sites however list a wide array of products, services and information. The optimization of large retail sites presents multiple issues for SEOs to work through. Achieving product-specific placements for sites featuring numerous products is much more difficult than achieving placements for smaller, more focused sites. Fortunately, good SEOs are good problem solvers and almost every technical problem has a solution.

A placement campaign for a large site is a long-term venture and clients approaching SEOs should be prepared to be involved in the campaign and to exercise patience while waiting for results. It might even take a few days for the SEO to prepare a preliminary plan for the campaign. There are a number of questions and challenges clients should expect from their SEOs.

Read the Full Article

About the Author:
Jim Hedger is the SEO Manager of StepForth Search Engine Placement Inc. Based in Victoria, BC, Canada, StepForth is the result of the consolidation of BraveArt Website Management, Promotion Experts, and Phoenix Creative Works, and has provided professional search engine placement and management services since 1997. http://www.stepforth.com/ Tel - 250-385-1190 Toll Free - 877-385-5526 Fax - 250-385-1198

WebProWorld
Search WebProWorld:  

Rafael Robinson

Two for none?

Our post today comes from Ricardo Zea. He is wanting to know why his site ranks so well in Google's search results, but rank poorly in Yahoo and Ask. I've come across a couple posts that were similar to this one, but it never hurts to help out a friend. Have you ever heard of this or seen it happen? Tell us your thoughts at WebProWorld.

|| Rafael||
   

   

All for one and two for none

By Ricardo Zea

Ok, I have a situation with three kings, Google, Yahoo and Ask, and I hope you can give us a glance to a method which can help us (our company) resolve this case.

Our rankings in Google are really good, in fact, alot better than what we have expected, but for some unkown reason, our site does not appear in Yahoo nor Ask. Now, on Yahoo we come up in place 155 and in Ask we cannot find our link.
  ...Click to read more
Free Newsletters
Part of the of 4 million+ subscribers
WebProNews
WebProWorld
InvestNewz



Send me relevant info on products and services.


:: Quote of the Day ::
Nothing is really work unless you would rather be doing something else.

-James M. Barrie

:: WebPro Question:
Is it better to wait for the search engine spiders or submit by hand?

-madicomputers

Comment

:: News and Events:
Web CEO Case Study & Review

Logo Evaluation

:: Meet the Members:

baldur User: baldur
Rating: WebPro Poster
Joined: 04.09.04
Location: Scotland

Website

View Profile

Advertising Newsletters Corporate Info Site Map Support
© 2004 WebProNews. An email newsletter.
, Inc. 880 Corporate Drive, Lexington, KY 40503
All Rights Reserved. Terms under which this service is provided to you. Read our privacy policy. Contact us.
The WebProNews network includes WebProWorld, Jayde and WebProWire.
Archives About Us Advertise Site Map Submit an article Rafael Robinson WebProNews.com In Affiliation with WebProWorld.com In Allfiliation With Jayde.com eBusiness Enterprise IT Web Development Feedback Edit Your Jayde.com Listing. Get All Your Jayde.com Questions Answered Here