On the Internet, a search engine is a coordinated set of programs that includes:
- A spider (also called a "crawler" or a "bot") that goes to every page or representative pages on every Web site that wants to be searchable and reads it, using hypertext links on each page to discover and read a site's other pages
- A program that creates a huge index (sometimes called a "catalog") from the pages that have been read
- A program that receives your search request, compares it to the entries in the index, and returns results to you
An alternative to using a search engine is to explore a structured directory of topics. Yahoo, which also lets you use its search engine, is the most widely-used directory on the Web. A number of Web portal sites offer both the search engine and directory approaches to finding information.
Different Search Engine Approaches
- Major search engines such as Google, Yahoo (which uses Google), AltaVista, and Lycos index the content of a large portion of the Web and provide results that can run for pages - and consequently overwhelm the user.
- Specialized content search engines are selective about what part of the Web is crawled and indexed. For example, TechTarget sites for products such as the AS/400 (http://www.search400.com) and CRM applications (http://www.searchCRM.com) selectively index only the best sites about these products and provide a shorter but more focused list of results.
- Ask Jeeves (http://www.ask.com) provides a general search of the Web but allows you to enter a search request in natural language, such as "What's the weather in Seattle today?"
- Special tools and some major Web sites such as Yahoo let you use a number of search engines at the same time and compile results for you in a single list.
- Individual Web sites, especially larger corporate sites, may use a search engine to index and retrieve the content of just their own site. Some of the major search engine companies license or sell their search engines for use on individual sites.
The last time we looked, the Open Directory Project listed 370 search engines available for Internet users. There are about ten major search engines, each with its own anchor Web site (although some have an arrangement to use another site's search engine or license their own search engine for use by other Web sites). Some sites, such as Yahoo, search not only using their search engine but also give you the results from simultaneous searches of other search indexes. Sites that let you search multiple indexes simultaneously include:
- Yahoo (http://www.yahoo.com)
- search.com (http://search.com)
- EasySearcher (http://www.easysearcher.com)
Yahoo first searches its own hierarchically-structured subject directory and gives you those entries. Then, it provides a few entries from the AltaVista search engine. It also launches a concurrent search for entries matching your search argument with six or seven other major search engines. You can link to each of them from Yahoo (at the bottom of the search result page) to see what the results were from each of these search engines.
A significant advantage of a Yahoo search is that if you locate an entry in Yahoo, it's likely to lead you to a Web site or entire categories of sites related to your search argument.
A search.com search primarily searches the Infoseek index first but also lets you search the other major search engines as well.
EasySearcher lets you choose from either the popular search engines or a very comprehensive list of specialized search engine/databases in a number of fields.
Yahoo, search.com, and EasySearcher all provide help with entering your search phrase. Most Web portal sites offer a quickly-located search entry box that connects you to the major search engines.
For "tips" on entering your search argument, see each search engine, including Yahoo's . It's good to read the information at least once.
By "How to Search," we mean a general approach to searching: what to try first, how many search engines to try, whether to search USENET newsgroups, when to quit. It's difficult to generalize, but this is the general approach we use at whatis.com:
- If you know of a specialized search engine such as SearchNetworking that matches your subject (for example, Networking), you'll save time by using that search engine. You'll find some specialized databases accessible from Easy Searcher 2.
- If there isn't a specialized search engine, try Yahoo. Sometimes you'll find a matching subject category or two and that's all you'll need.
- If Yahoo doesn't turn up anything, try AltaVista, Google, Hotbot, Lycos, and perhaps other search engines for their results. Depending on how important the search is, you usually don't need to go below the first 20 entries on each.
- For efficiency, consider using a ferret that will use a number of search engines simultaneously for you.
- At this point, if you haven't found what you need, consider using the subject directory approach to searching. Look at Yahoo or someone else's structured organization of subject categories and see if you can narrow down a category your term or phrase is likely to be in. If nothing else, this may give you ideas for new search phrases.
- If you feel it's necessary, also search the Usenet newsgroups as well as the Web.
- As you continue to search, keep rethinking your search arguments. What new approaches could you use? What are some related subjects to search for that might lead you to the one you really want?
- Finally, consider whether your subject is so new that not much is available on it yet. If so, you may want to go out and check the very latest computer and Internet magazines or locate companies that you think may be involved in research or development related to the subject.