| (documentation WebCrawler EnglishLanguage "A Web crawler is a ComputerProgram that browses the World Wide Web in a methodical, automated manner or in an orderly fashion. A Web crawler is one type of bot, or software agent. In general, it starts with a list of URLs to visit, called seeds. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit. This list is called the crawl frontier. URLs from the frontier are recursively visited according to a set of policies. Their most common use is in search engines, where, in response to a query, they 'crawl' through the Web and index the significant words on the pages they find.") |
Facebook.kif 831-841 |
|
