虫言虫语 虫言虫语

给scrapy设置HTTP代理

in Pythonread (10316) 文章转载请注明来源!

在scrapy项目文件下建立一个.py文件 填入以下信息:

# Importing base64 library because we'll need it ONLY in case if the proxy we are going to use requires authentication
import base64
# Start your middleware class
class ProxyMiddleware(object):
    # overwrite process request
    def process_request(self, request, spider):
        # Set the location of the proxy
        request.meta['proxy'] = "http://proxyIP:port"

        # Use the following lines if your proxy requires authentication
        # proxy_user_pass = "USERNAME:PASSWORD"
        # setup basic authentication for the proxy
        # encoded_user_pass = base64.encodestring(proxy_user_pass)
        # request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass

在setting中加入以下代码:(其中project_name自行替换后面跟上建立的.py文件路径,最后加上cls名)

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
    'project_name.middlewares.ProxyMiddleware': 100,
}
jrotty WeChat Pay

微信打赏

jrotty Alipay

支付宝打赏

文章二维码

扫描二维码,在手机上阅读!

发表新评论
已有 2775 条评论
  1. JimDes
    JimDes
    回复

    amoxicillin 500mg pill cozaar 15 mg daily atarax 10mg hydroxychloroquine for sale

  2. Patrickreurl
    Patrickreurl
    回复

    cbd oil for sale
    cbd for dogs

  3. JimDes
    JimDes
    回复

    buy cipro without rx acyclovir pill citalopram buy finasteride buy online where can i buy dapoxetine

  4. JaneDes
    JaneDes
    回复

    allopurinol 300

  5. JackDes
    JackDes
    回复

    prozac 40mg how to buy prednisone online buy bactrim antibiotic zithromax cost

  6. IvyDes
    IvyDes
    回复

    buy tetracycline anafranil bactrim from canada buy amoxil can you buy tretinoin over the counter diflucan tablet 100 mg bupropion hcl acyclovir otc buy albuterol buy prozac lexapro online best retin a cream over the counter furosemide 12.5 mg dapoxetine over the counter uk augmentin 500 tablet fluoxetine 10 mg price advair diskus cost citalopram insomnia cheap propecia online hydroxychloroquine online

  7. KimDes
    KimDes
    回复

    buy lisinopril

  8. JaneDes
    JaneDes
    回复

    buy fluoxetine

  9. NickDes
    NickDes
    回复

    lasix 20 buy ventolin inhaler amoxicillin 500 mg buy hydroxychloroquine atarax 25

  10. MiaDes
    MiaDes
    回复

    cheapest retin a cream buy ventolin online uk cipro price buy zithromax buy lisinopril

  11. SamDes
    SamDes
    回复

    plaquenil for lupus generic cozaar celexa 20mg cost

  12. JackDes
    JackDes
    回复

    augmentin 600 42.9 mg vardenafil 20mg generic ciprofloxacin hcl

  13. JasonDes
    JasonDes
    回复

    amoxicillin 500 mg cost generic diflucan zoloft 100mg coupon 40 mg lasix lexapro 10mg buy propecia quineprox 0.4 bactrim 800mg 160mg acyclovir 800 discount celexa

  14. MiaDes
    MiaDes
    回复

    buy citalopram australia fluoxetine 20 mg caps ventolin how to order metformin online retin a rx cream

  15. NickDes
    NickDes
    回复

    canadian pharmacy lasix anafranil 75 augmentin 125 mg prednisolone 5 mg amoxicillin 625mg tablet price albenza generic

博客已萌萌哒运行
© 2020 由 Typecho 强力驱动.Theme by Yodu
前篇 后篇
雷姆
拉姆