scrapy-环境安装+基本配置

1

pip install scrapy

1

scrapy startproject 项目名称

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

firstBlood/
├── firstBlood
│   ├── __init__.py
│   ├── items.py
│   ├── middlewares.py
│   ├── pipelines.py
│   ├── settings.py
│   └── spiders            # 爬虫文件存放目录
│       └── __init__.py
└── scrapy.cfg

1

scrapy genspider 爬虫名称 起始url

1

scrapy crawl 爬虫名称

1
2
3
4
5
6

USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36'

ROBOTSTXT_OBEY = False

LOG_LEVEL = 'ERROR'