Synonyms

web crawler

American

[web-kraw-ler] / ˈwɛbˌkrɔ lər /

Sometimes Web crawler or webcrawler

noun

Digital Technology.

a computer program that digitally surveys websites, as in order to index web pages for a search engine.

Other Word Forms

web crawling noun

Etymology

Origin of web crawler

First recorded in 1990–95

Example Sentences

Examples are provided to illustrate real-world usage of words in context. Any opinions expressed do not reflect the views of Dictionary.com.

Publishers could also try to protect their content from Google by forbidding its web crawler from sharing any content snippets from their sites.

From New York Times • Jun. 1, 2024

The company originally operated a web crawler that searched for GIFs on the internet.

From Slate • Nov. 30, 2021

He rewrote the code for a slow web crawler that Mr. Page had created to understand the relationship between links on different websites.

From New York Times • Aug. 20, 2021

Bumgarner said the domain on Sony Corporation of America's network where the application was located was carefully hidden from view, so a web crawler or casual surfer would not have located it.

From Reuters • May 13, 2011

Definitions and idiom definitions from Dictionary.com Unabridged, based on the Random House Unabridged Dictionary, © Random House, Inc. 2023