web crawler

or web·crawl·er

noun Digital Technology.
  1. (sometimes initial capital letter) a computer program that retrieves data from a website, as in order to index web pages for a search engine.
Compare bot1.

Origin of web crawler

First recorded in 1990–95
Also called spider.
Related formsweb crawling, noun
Dictionary.com Unabridged Based on the Random House Unabridged Dictionary, © Random House, Inc. 2018