web crawler
noun
a computer program that digitally surveys websites, as in order to index web pages for a search engine.
web crawling
-
a word derived from
web crawler.
Example Sentences
Examples are provided to illustrate real-world usage of words in context. Any opinions expressed do not reflect the views of Dictionary.com.
Wayne State University’s Clayton Hayes wrote in a blog post on 2 June that publishers should combat unauthorized web crawling, but that using trap URLs that incorporate DOI-like identifiers compromises the integrity of academic publishing.
From Nature ● Jun. 28, 2016
Frankly, Google and Bing’s search engine’s are ‘smarter’ when you consider the A.I. algorithms involved in web crawling.
From Forbes ● Oct. 17, 2011
Truveo's approach to video search utilizes a proprietary web crawling technology called "Visual Crawling".
From Time Magazine Archive
