How Do Search Engines Work – Web Crawlers and their tricks
April 21st, 2010
How Do Search Engines Work – Web Crawlers and their tricks
Lets check how the search engine works as search engine is of prime importance to get your site noticed. We need to give a lot of importance to optimize the site so we get good search engine optimized results.
Search Engines are of two types:-
1) robots called crawlers or spiders.
Spiders are used to index the websites by majority of the search engines. Your entire site is being indexed and reviewed once you submit the website pages to the major search engines. An automated program run by search engines is usually defined as spider. Spider index the site and read the content of your site. Remember “Content is King” – better and fresh the contents, more the importance given by search engines. Spider will collect all the information like meta tags, description and links that your web page have. The spider then returns all that information back to a central depository, where the data is indexed. It will visit each link you have on your website and index those sites as well.
Few spiders do not follow all links and pages on the site, so it is not necessary that more number of pages will be a guarantee to get good results. Good contents and valuable information for the visitor is of prime importance. Spiders visit the site on regular interval of time to check whether there are any other fresh contents that your site is offering to the visitors and importance of those contents.
Example: Excite, Lycos, AltaVista and Google.
Different search engines follow their own set of algorithm to index the site according to its importance. So, it is not necessary that there are similar rankings in all search engines for your web pages.
One of the things that a search engine algorithm scans for is the frequency and location of keywords on a web page, but it can also detect artificial keyword stuffing or spamdexing. Then the algorithms analyze the way that pages link to other pages in the Web. By checking how pages link to each other, an engine can both determine what a page is about, if the keywords of the linked pages are similar to the keywords on the original page.