虫言虫语 虫言虫语

给scrapy设置HTTP代理

in Pythonread (57748) 文章转载请注明来源!

在scrapy项目文件下建立一个.py文件 填入以下信息:

# Importing base64 library because we'll need it ONLY in case if the proxy we are going to use requires authentication
import base64
# Start your middleware class
class ProxyMiddleware(object):
    # overwrite process request
    def process_request(self, request, spider):
        # Set the location of the proxy
        request.meta['proxy'] = "http://proxyIP:port"

        # Use the following lines if your proxy requires authentication
        # proxy_user_pass = "USERNAME:PASSWORD"
        # setup basic authentication for the proxy
        # encoded_user_pass = base64.encodestring(proxy_user_pass)
        # request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass

在setting中加入以下代码:(其中project_name自行替换后面跟上建立的.py文件路径,最后加上cls名)

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
    'project_name.middlewares.ProxyMiddleware': 100,
}
jrotty WeChat Pay

微信打赏

jrotty Alipay

支付宝打赏

文章二维码

扫描二维码,在手机上阅读!

发表新评论
已有 12140 条评论
  1. EvaDes
    EvaDes
    回复

    buy levitra in usa

  2. MichaelDed
    MichaelDed
    回复

    drugs without doctor script buy generic pills
    ed treatment natural http://genericwdprescription.com generic pills for ed
    prescription drugs without a doctor buy cheap prescription drugs online

  3. https://abcviag.com/

    can you buy viagra at walmart
    buy viagra no prescription
    the weekend pill cialis roani Cloup

  4. Define Assignments

    help me to do my homework assignment proofreading service can i pay someone to do my math homework

  5. https://abcviag.com/

    viagra mexico online
    viagra without seeing a doctor
    generic cialis sweden roani Cloup

  6. https://abcviag.com/

    zoloft mail order
    viagra dosage instructions
    levitra odt 10mg bula roani Cloup

  7. https://abcviag.com/

    buy viagra sharm el sheikh
    cheap albuterol and viagra
    brand viagra without prescription roani Cloup

  8. KimDes
    KimDes
    回复

    otc prevacid

  9. Get A Loan
    Get A Loan
    回复

    quick loans online loans instant approval

  10. Williammom
    Williammom
    回复

    prescription drugs without a doctor buy prescription drugs
    shots for ed http://genericwdprescription.com buy prescription drugs
    meds without a doctor prescription buy cheap prescription drugs online

  11. KiaDes
    KiaDes
    回复

    paxil drug cost

  12. JaneDes
    JaneDes
    回复

    zoloft tabs

  13. https://abcviag.com/

    is generic viagra and viagra the same
    get viagra overnight
    viagra where buy roani Cloup

  14. HjuaBronup
    HjuaBronup
    回复

    can u buy levitra over the counter generic levitra online can you buy propecia propecia for sale online how much does synthroid cost per pill buy synthroid online with free samples

  15. AlanDes
    AlanDes
    回复

    silagra 100 mg tablet tofranil 25 mg aygestin order online generic viagra online no prescription orlistat 120 mg india singulair tab 4 mg chloroquine covid-19 where to buy nexium online zofran 40 mg trental er plaquenil 200 mg online viagra low cost terramycin eye zoloft 150 mg order generic prozac online xenical prescription cost male viagra brand name hytrin cost priligy price usa furosemide 20 mg daily

  16. JamesTum
    JamesTum
    回复

    Transit of knock off cialis online to the abrupt cessation. cialis prices Uuilyz jzqshy online cialis cialis generic date

  17. zegukwdquule
    zegukwdquule
    回复

    online canadian pharmacies online pharmacies in usa canada pharmacy online viagra www.northwestpharmacy.com
    online pharmacy canada highest rated canadian pharmacies canadian pharmacy - viagra viagra india pharmacy
    viagra from a pharmacy in canada us pharmacy cialis canadian online pharmacy cialis and pharmacy cialis- canadian pharmacy

  18. JaneDes
    JaneDes
    回复

    neurontin 214

  19. https://abcviag.com/

    e lilly cialis for sale
    buy viagra canada
    buy viagra same day roani Cloup

  20. JamesTum
    JamesTum
    回复

    Xyywib vlflzn but cialis online tadalafil generic cialis generic online from canada Qnbxqm gmwnrv

博客已萌萌哒运行
© 2020 由 Typecho 强力驱动.Theme by Yodu
前篇 后篇
雷姆
拉姆