The robot of the search engine is responsible for scanning of the pages which are posted online. The program automatically reads data from all websites and registers them in a form, clear for the searcher, that subsequently the system displayed the most suitable results for the user.
Functions
All indexed information registers in the general database.
Search the robot – the program which automatically travels around pages of the Internet, requesting the necessary documents and receiving structure of the scanned websites. The robot independently selects pages which should be scanned. In most cases the scanned websites are selected randomly.
Types of bots
Incorrectly functioning robot considerably increases load of network and the server that can become the reason of unavailability of a resource.
Each search engine has several programs which are called robots. Each of them can perform a certain function. For example, at Yandex some robots are responsible for scanning of news feeds of RSS which will be useful at indexation of blogs. There are also programs which are engaged only in search of pictures. Nevertheless the most important is the indexing boat which forms base for carrying out any search. Also there is an auxiliary fast robot intended for search of updates by news feeds and actions.
Procedure of scanning
In a different way the ban on scanning of contents creation of access to the website via the registration panel is.
Visiting the site, the program carries out scanning of the file system regarding availability of files of the instruction of robots.txt. In the presence of the document, reading of the directives stated in the document begins. Robots.txt can prohibit or, on the contrary, resolve, scanning of any given pages and files on the website.
Process of scanning depends on program type. Sometimes robots read out only headings of pages and several paragraphs. In certain cases scanning is carried out according to all document depending on the HTML layout which can also work as means for the indication of key phrases. Some programs specialize in hidden or meta-tags.
Adding in the list
Each webmaster can prohibit scanning of pages the search engine through robots.txt or the META tag. Also the creator of the website can add manually the website to queue of indexing, however adding does not mean at all that the robot will immediately scan the necessary page. To add the website to queue, searchers also provide special interfaces. Adding of the website considerably accelerates indexing process. Also for fast registration in the search engine the systems of web analytics, directories of the websites, etc. can be used.