scrapelib 1.0.0

Overview

scrapelib is a library for making requests to websites, particularly those that may be less-than-reliable.

scrapelib originated as part of the Open States project to scrape the websites of all 50 state legislatures and as a result was therefore designed with features desirable when dealing with sites that have intermittent errors or require rate-limiting.

As of version 0.7 scrapelib has been retooled to take advantage of the superb requests library.

Advantages of using scrapelib over alternatives like httplib2 simply using requests as-is:

• All of the power of the suberb requests library.
• HTTP(S) and FTP requests via an identical API
• support for simple caching with pluggable cache backends
• request throtting
• configurable retries for non-permanent site failures

Contents

• scrapelib overview
  • Scraper object
  • Response objects
  • Exceptions
• scrapeshell
  • scrapeshell arguments
• cache overview
  • MemoryCache object
  • FileCache object
  • SQLiteCache object
• Migrating from 0.x to 1.x
  • Notable Changes to Responses
  • Examples
• scrapelib changelog
  • 1.0.0
  • 0.10.1
  • 0.10.0
  • 0.9.1
  • 0.9.0
  • 0.8.0
  • 0.7.4
  • 0.7.3
  • 0.7.2
  • 0.7.1
  • 0.7.0
  • 0.6.2
  • 0.6.1
  • 0.6.0
  • 0.5.8
  • 0.5.7
  • 0.5.6
  • 0.5.5
  • 0.5.4
  • 0.5.3
  • 0.5.2
  • 0.5.1
  • 0.5.0
  • 0.4.3
  • 0.4.2
  • 0.4.1
  • 0.4.0
  • 0.3.0
  • 0.2.0

Indices and tables

• Index
• Module Index
• Search Page