Drubby World Wide Web Search

SEO Expert Guide - Search Engines Explained (part 1/10)

Before we explore the world of search engine optimization, it is vital that you know a little about how search engines work and their relative market shares. It will help you to prioritize your activities later!

(a) What are Search Engines and who powers them?

There are essentially four different parts to a typical large search engine; the crawler, the directory, sponsored results and the search engine itself.

Crawlers (e.g. Google) automatically visit web pages to compile their listings, making use of a so-called robot or spider (eg. Googlebot), which follows links from one website to another, ultimately compiling an index of all the pages and sites on the internet. These crawlers provide an index, which can then searched by the search engine. You may find that several or all of the pages on your site are indexed in thisway. Some search engines have their own crawler and others buy-in crawler results from others.

Human-powered directories, such as the Open Directory, rely on submissions from the public, which are reviewed by editors for inlusion in the directory. If you get included in a directory, generally only one page from your site (usually your home - or index - page) will be listed.

Crawled results are combined with sponsored results, supplied by pay-per-click (PPC) advertisers, and the results from human-maintained directories to complete the search engine index. Check out the Search Engine Reationship Chart at Bruce Clay inc. for the latest picture on who powers whom. You will note a couple of things right away. Firstly, the dominance of the Google and Yahoo! crawlers and secondly the importance of DMOZ directory results as a back-door for many search engines.

(b) How do Search Engines find and rank sites?

Search engines do not really search the web directly, but rather an index database of the full text of web pages, which itself is drawn from the billions of web pages on the internet's servers. Search engine databases are selected and built by computer robot programs called spiders.

If a web page is never linked to by any other page, spiders cannot find it, unless the (usually new) site is submitted manually by a human at the search engine's "add URL" page. All search engine companies offer ways to do this.

After spiders find pages, they pass them on to another computer program for "indexing." This program uses an "algorithm" to assess the text, links, and other content in the page for "key words" that might be searched on at the engine. This allows the search engine to order results served by their "relevancy" to the search terms used. As each search engine has a different algorithm, it will index sites in a different way and thus serve up different relevant results.

Some types of pages and links are excluded from most search engines by policy. Others are excluded because search engine spiders cannot access them. Generally, the use of frames, flash graphics and dynamic URLs all get in the way of effective spidering and should thus be avoided.

In addition to indexing pages, most algorithms seek to establish the "authority" of a site. A site which is linked to by many other sites (using keyword-rich anchor text) is assumed to be of greater merit than one with no links at all. This activity is called "ranking" and helps search engines to sort otherwise similar results into ever-more relevant and authoratative results.

(c) Which Search Engines are the most popular?

Based on US analysis in January 2005, the top search engines (by share of total searches at home and work) are as follows:

Google Search - 47%

Yahoo! Search - 21%

MSN Search - 13%

All Others - 19%

These shocking figures do not convey the true dominance of the top players, as you have seen from the interdependence of search provision in section (a) above. You could be searching at AOL (part of the "other" 19%) and viewing Google results, for example.

There is also strong anecdotal evidence that Yahoo! and MSN tend to send more searchers through to their sponsored (or paid) results than do Google (due to the prominence of these results on their results pages). As such, for a typical small webmaster who does not use pay-per-click (PPC) advertising, they might get up to 80% of all their traffic from Google's various sites across the world.

Now you understand the market a little better, you will perhaps understand the obsession many webmasters have with Google! A top-10 position at Google for your key search terms can make your online business fly. If you drop out of that top-10, your business can literally collapse overnight!

Don't forget these key stats as you embark on your optimization journey...

Navigate the guide

Previous : SEO Expert Guide - Index of Contents

Next: SEO Expert Guide - Proposition Development (part 2/10)

David Viney (david@viney.com) is the author of the Intranet Portal Guide; 31 pages of advice, tools and downloads covering the period before, during and after an Intranet Portal implementation.

Read the guide at http://www.viney.com/DFV/intranet_portal_guide or the Intranet Watch Blog at http://www.viney.com/intranet_watch.

MORE ARTICLES:

A Search Engine, Directory, and Classified Submission Software That Actually Works?
For most websites, your profit margins are razor thin and you can?t afford to spend much of anything for advertising.

My Review of Groople- Group Rate Travel Search Engine
Over the years I have planned many trips and far too much time on the phone nailing down details on hotels and airlines for large group trips. I had always wished that booking for groups could be as easy as booking for individuals using online search engines.

How Article Marketing Is Helpful In Search Engine Marketing
Article Marketing has a whole range of benefits that can transform the fate of a website from pauper to prince. First and foremost, it is a great strategy to invite qualified net traffic without having to beg or nudge. The whole idea is very logical. The website submits informative article across free of cost forums, blogs, ezines et al, which in turn may further change hands with other websites out of interest or a nominal amount of fee. Thus by submitting the article to some hundreds or even thousands of websites, the website establishes its web presence. And from there on the article does the rest.

TechPalooza Partners with Popular Shopping Search Engine Sortprice.com
TechPalooza's partnership with Sortprice.com will allow customers the ability to save time and money shopping for the technology products they want and need this holiday season available at TechPalooza.com.

Do It Yourself SEO: Ten Search Engine Optimization Tips (Part 2)
In the first article of this series I outlined five important steps in the optimization of any web page or website. Here are five more tips for those who want to make sure that their web pages are both user and search engine friendly.

Pay Per Click Search Engine Marketing - Starting A Campaign
Pay per click advertising is an on-going and growing arsenal for all entrepreneurs world wide. Whether your have a small business or a large business, buying a search engine placement is definitely the way to go.

On Page Search Engine Optimization Factors ? Do They Matter?
Let's hop into our Internet time machine and take a trip. Today we're headed back in time to 2000 ? in Internet time, that seems like 100 years ago. You?re an SEO expert and you want to write a book on how to get to the top of search engines ? what do you talk about? Meta tags are definitely on the way out. Links help but this is cutting edge stuff ? not the basis of your book. So you talk about keyword density. You say if you want to get ranked highly for a specific term, you need to have that in your title and you need to have it show up towards the top of your page, and in other parts of the page and have your page be keyword dense for your chosen topic. And I?m not talking about 1-3% keyword density ? 100% keyword density for a word ac...

Why You Should NOT Submit Your Site On Search Engines
Before to answer to this question we have to know what is the difference between a search engine and directory. Here is a brief explanation.

Pepperjam Executives to Attend, Exhibit, and Speak at Search-Engine Strategies (SES) - San Jose 2007
Pepperjam, an Inc. 500 Company and industry leading full-service internet marketing agency, announced today that several senior management and company executives will be attending and exhibiting at Search-Engine Strategies (SES) - San Jose 2007. The company also announced that Pepperjam President & CEO, Kristopher B. Jones, will be a featured speaker on a panel entitled, "The SEO Reputation Problem." SES - San Jose is being held at the San Jose Convention Center from August 20 - 23, 2007.

Web Directory Basics, Submissions And The Free Directory Submitter
Seo (search engine optimization) is the most used way to promote your site. A free directory submitter aids in the process tremendously and we'll talk more about that later in this article. It's a known fact; most webmasters give to more time optimizing their site than they devote actually building the website itself. Why? Well, let's think about it, what is the point of building and maintaining a website if your website has little or no traffic? Isnt the entire purpose of building your site to attract visitors and expose them to your sites message? Absolutely! This is the reason search engine optimization is a crucial element to a successful site.

WashingtonVC and BP Holdings Partner to Launch Search Engine Optimization Firm, SEO.com
Partnership Highlights Re-Branding of Web Targeted as SEO.com and Expansion of Search Engine Optimization Services

Search Engine Optimisation ? A Checklist For Success!
By following this checklist you can make sure your business gets the online exposure it deserves 1

Cisco CCNA Candidate FAQ
CCNA FAQQ. What exams do I have to take to get my CCNA?A.

Complete Web-Site Optimization For Search Engines (Part 1)
SEO or search engine optimization strategy now becomes widely popular among online business operators. Nothing strange about it as it allows to substantially increase your gross income, as a result of growing traffic or visitors flow.