web crawler

or web·crawl·er

[ web-kraw-ler ]

/ ˈwɛbˌkrɔ lər /

Save This Word!

See synonyms for web crawler on Thesaurus.com

noun Digital Technology.

a computer program that digitally surveys websites, as in order to index web pages for a search engine.

QUIZ

CAN YOU ANSWER THESE COMMON GRAMMAR DEBATES?

There are grammar debates that never die; and the ones highlighted in the questions in this quiz are sure to rile everyone up once again. Do you know how to answer the questions that cause some of the greatest grammar debates?

Question 1 of 7

Which sentence is correct?

His boss, whom he admires, is waiting to meet with him about the big project.

His boss, who he admires, is waiting to meet with him about the big project.

TAKE THE QUIZ TO FIND OUT

Sometimes Web crawl·er .

Also called crawl·er .

Compare bot¹, spider.

Origin of web crawler

First recorded in 1990–95

OTHER WORDS FROM web crawler

web crawling, noun

How to use web crawler in a sentence

He was referring to web censorship behind the Great Firewall.
China’s Internet Is Freer Than You Think|Brendon Hong|December 27, 2014|DAILY BEAST
With its vast web of resources and services, including its support groups, the Center has often helped save these people lives.
The LGBT Center That Changed Our Lives|Justin Jones|December 22, 2014|DAILY BEAST
How has your experience been shaped by digital encounters and the social web?
The Unbearable Whiteness of Protesting|Rawiya Kameir, Judnick Mayard|December 10, 2014|DAILY BEAST
The caller mentioned my work, which focused primarily on consumer products, mobile apps, emerging start-ups, and web trends.
A Female Writer’s New Milestone: Her First Death Threat|Annie Gaus|December 1, 2014|DAILY BEAST

That leads us to wonder why galaxies and their black holes somehow “know” where they are in the cosmic web.
The Black Hole Tango|Matthew R. Francis|November 24, 2014|DAILY BEAST
It was no wonder that he felt quite at home in the duck-pond, which was made for web-footed folk.
The Tale of Grandfather Mole|Arthur Scott Bailey
"It is n't distressingly calm now," said the extra-strong frames—they were called web-frames—in the engine-room.
Kipling Stories and Poems Every Child Should Know, Book II|Rudyard Kipling
The touch of the soft fabric reassured him: it was as soft as though woven of spider's web, and strong as fibres of steel.
Astounding Stories, May, 1931|Various
These threads in the web of industry, which had shone that day for the first time, were the lives of two little children.
The Underworld|James C. Welsh
At first girders had solid or plate webs, but for spans over 100 ft. the web always now consists of bracing bars.
Encyclopaedia Britannica, 11th Edition, Volume 4, Part 3|Various