Crawling, spidering, indexing, there are many names for it.
At some point in their careers most Perl coders will be asked to write one of one sort or another.
A common problem which comes up while writing crawlers is how to build them to scale.
In this talk I will show how to tie together using gungho crawling framework, a cloud computing environment, and a job-tasking framework to build a highly scalable crawler.
