Human-powered search engines like the Open Directory are not listed on this chart because they do not crawl the web to create their listings. See the How Search Engines Work page for an explanation of the differences between crawler-based and human-powered services.
See the Search Engine Features For Searchers page for a summary of how search engines display their results and other information that may be of interest to searchers, rather than search engine marketers and site promoters.
The How Search Engines Work section available to Search Engine Watch members provides more detailed information about the crawlers listed below. This section is just one of the many benefits that Search Engine Watch members receive. To learn more about becoming a member, please visit the membership information page.
This chart covers the crawler of AllTheWeb, AltaVista, Google, Inktomi and Teoma. Some of these crawlers power other search engines, and the relationships are shown on the Search Engine Results page.
Deep Crawl
All crawlers will find pages to add to their web page indexes, even if those pages have never been submitted to them. However, some crawlers are better than others. This section of the chart shows which search engines are likely to do a "deep crawl" and gather many pages from your web site, even if these pages were never submitted. In general, the larger a search engine's index is, the more likely it will list many pages per site. See the Search Engine Sizes page for the latest index sizes at the major search engines.
Frames Support
This shows which search engines can follow frame links. Those that can't will probably miss listing much of your site. However, even for those that do, having individual frame links indexed can pose problem. Be sure to read the Search Engines And Frames page for tips on overcoming the problems with frames and search engines.
robots.txt
The robots.txt file is a means for webmasters to keep search engines out of their sites. Search Engine Watch members have access to the Blocking Crawlers With Robots.txt page, which covers the robots.txt file in more detail. The Web Robots Pages: The Robots Exclusion Protocolpage also provides official information about robots.txt.
Meta Robots Tag
This is a special meta tag that allows site owners to specify that a page shouldn't be indexed. It is explained more on the How HTML Meta Tags Work page and also on the Blocking Crawlers With The Meta Robots Tag page, available to Search Engine Watch members. The Web Robots Pages: The Robots META tagpage also provides official information about robots.txt.
Paid Inclusion
Shows whether a search engine offers a program where you can pay to be guaranteed that your pages will be included in its index. This is NOT the same as paid placement, which guarantees a particular position in relation to a particular search term. The Submitting To Crawlers page provides links to various paid inclusion programs.
Full Body Text
All of the major search engines say they index the full visible body text of a page, though some will not index stop words or exclude copy deemed to be spam (explained further below). Google generally does not index past the first 101K of long HTML pages.
Stop Words
Some search engines either leave out words when they index a page or may not search for these words during a query. These stop words are excluded as a way to save storage space or to speed searches.
Meta Description
All the major crawlers support the meta description tag, to some degree. The ones actually named on the chart are very consistent. If you have a meta description tag on your pages, you'll most likely see the content used in some way.
The How HTML Meta Tags Work page explains how to use the meta description tag, and the Search Engine Display Chart for Search Engine Watch members provides a more detailed breakdown of how crawler-based search engines form descriptions.
Meta Keywords
Shows which search engines support the meta keywords tags, as explained on the How HTML Meta Tags Work page.
No comments:
Post a Comment