web crawler

American  

[web-kraw-ler] / ˈwɛbˌkrɔ lər /

Sometimes Web crawler

Or webcrawler

noun

Digital Technology.

1. a computer program that digitally surveys websites, as in order to index web pages for a search engine.

---

Other Word Forms

• web crawling noun

Etymology

Origin of web crawler

First recorded in 1990–95

Example Sentences

“Hologram is a technology that’s different than this. This is more of an avatar presence, or a telepresence, if you will. Unlike ChatGPT, this is not a web crawler. This is a large language model which has got guardrails on it,” says George Johnson, a member of the Hyperreal technical team.

From Los Angeles Times

Publishers could also try to protect their content from Google by forbidding its web crawler from sharing any content snippets from their sites.

From New York Times

The company’s patent application details its use of a “web crawler” to acquire images, even noting that “online photos associated with a person’s account may help to create additional records of facial recognition data points,” which its machine learning algorithm can then use to find and identify matches.

From The Verge

The company originally operated a web crawler that searched for GIFs on the internet.

From Slate

He rewrote the code for a slow web crawler that Mr. Page had created to understand the relationship between links on different websites.

From New York Times