Spider is a piece of software (also called a Web crawler) designed to follow hyperlinks to their completion, and return information on Internet addresses passed.
This term is commonly used in software engineering, programming, scripting, application design, testing, and development lifecycle work.