scrapy startproject Redbacktest
cd Redbacktest
创建jobbole爬虫
scrapy genspider jobbole2 blog.jobbole.com
从pycharm中导入后创建main文件
from scrapy.cmdline import execute import sys
sys.path.append("D:\PycharmProjects\Redbacktest")
execute(['scrapy','crawl','jobbole2'])
调试前修改“君子协议”
ROBOTSTXT_OBEY = False
断点调试response是否获取到值