YAPC::Asia 2008

Gungho and cloud computing, a scalable crawling and processing framework

Crawling, spidering, indexing, there are many names for it.

At some point in their careers most Perl coders will be asked to write one of one sort or another.

A common problem which comes up while writing crawlers is how to build them to scale.

In this talk I will show how to tie together using gungho crawling framework, a cloud computing environment, and a job-tasking framework to build a highly scalable crawler.


Gold Sponsors

Media Sponsors

  • O'Reilly Japan, Inc.
  • Gijutsu-Hyohron Co., Ltd.

Silver Sponsors

  • ActiveState Software Inc.
  • DWANGO Co.,Ltd.
  • endeworks Co.,Ltd.
  • Cybozu Labs, Inc.
  • GaiaX Co. Ltd.
  • cgios technologies Inc.
  • Yahoo Japan Corporation.
  • Labs Xronogram, Inc.

Affiliation with

  • Mozilla Japan

Streaming powered by

  • i-revo