pt. 1. Fundamental concepts and techniques. What's in it for you?
Ideas for webbot projects
Downloading web pages
Basic parsing techniques
Advanced parsing with regular expressions
Automating form submission
Managing large amounts of data
pt. 2. Projects. Price-monitoring webbots
Image-capturing webbots
Link-verification webbots
Search-ranking webbots
Aggregation webbots
FTP webbots
Webbots that read email
Webbots that send email
Converting a website into a function
pt. 3. Advanced technical considerations. Spiders
Procurement webbots and snipers
Webbots and cryptography
Authentication
Advanced cookie management
Scheduling webbots and spiders
Scraping difficult websites with browser macros
Hacking iMacros
Deployment and scaling
pt. 4. Larger considerations. Designing stealthy webbots and spiders
Proxies
Writing fault-tolerant webbots
Designing webbot-friendly websites
Killing spiders
Keeping webbots out of trouble.