custom_settings = { ‘DOWNLOAD_DELAY’: 0.2, ‘CONCURRENT_REQUESTS_PER_IP’: 4, ‘DOWNLOADER_MIDDLEWARES’: {},}
Crawled 60 pages (at 60 pages/min), scraped 0 items
ROBOTSTXT_OBEY = False # 将上面的True注释掉,改为False DOWNLOAD_DELAY = 3 # 时间延长一点