虫言虫语 虫言虫语

给scrapy设置HTTP代理

in Pythonread (1641) 文章转载请注明来源!

在scrapy项目文件下建立一个.py文件 填入以下信息:

# Importing base64 library because we'll need it ONLY in case if the proxy we are going to use requires authentication
import base64
# Start your middleware class
class ProxyMiddleware(object):
    # overwrite process request
    def process_request(self, request, spider):
        # Set the location of the proxy
        request.meta['proxy'] = "http://proxyIP:port"

        # Use the following lines if your proxy requires authentication
        # proxy_user_pass = "USERNAME:PASSWORD"
        # setup basic authentication for the proxy
        # encoded_user_pass = base64.encodestring(proxy_user_pass)
        # request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass

在setting中加入以下代码:(其中project_name自行替换后面跟上建立的.py文件路径,最后加上cls名)

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
    'project_name.middlewares.ProxyMiddleware': 100,
}
jrotty WeChat Pay

微信打赏

jrotty Alipay

支付宝打赏

文章二维码

扫描二维码,在手机上阅读!

发表新评论
已有 52 条评论
  1. EyeDes
    EyeDes
    回复

    metformin 800 lisinopril 12.5 mg 20 mg indocin pill ventolin salbutamol retin a 0.05 cream wellbutrin 25 mg cost for generic valtrex kamagra online pharmacy celebrex 400 mg prices doxycycline drug suhagra 100mg buy online albendazole buy lexapro 100mg price buy propecia online amoxicillin for sale mexico acyclovir 400 mg buy accutane with paypal price of zoloft 50 mg where to buy tretinoin cream albuterol prescription online

  2. JoeDes
    JoeDes
    回复

    generic clomid propecia online no prescription buy tadalafil canada acyclovir 800 mg tablet price in india indocin 75 mg where to buy retin a tretinoin

  3. KimDes
    KimDes
    回复

    cipro 500 mg

  4. JoeDes
    JoeDes
    回复

    diflucan 150 buy suhagra 25 mg online no prescription metformin buy tretinoin

  5. TedDes
    TedDes
    回复

    viagra soft tabs lisinopril 10 mg cephalexin generic capsules indocin otc retin a 0.025

  6. LisaDes
    LisaDes
    回复

    metformin hcl 500

  7. TeoDes
    TeoDes
    回复

    buspar 10mg

  8. LeonardasShemi

    buy cialis in cebu
    https://www.liverichandfree.com/# - generic viagra cialis
    cialis in the morning
    is there a generic for cialis
    que hacer cuando cialis no funciona

  9. PaulDes
    PaulDes
    回复

    cipro 500mg dosage how to get albuterol without prescription retin a 0.025 gel advair diskus how can i get doxycycline over the counter

  10. IvyDes
    IvyDes
    回复

    suhagra 50 mg tablet synthroid mexico pharmacy clomid pills kamagra tablets online india advair diskus indocin 50 tretinoin how to get where to buy cheap tadalafil purchase antabuse accutane cost australia acyclovir 400 mg 40 mg generic cialis buy propecia uk doxy 100 buy amoxicillin 500mg no prescription buy diflucan wellbutrin 142 cheap lexapro online strattera generic buy generic atarax

  11. TedDes
    TedDes
    回复

    buy tadalafil 20 cheap viagra pills usa propranolol 10 mg tablet price

  12. Lesswesty
    Lesswesty
    回复

    Viagra For Man http://buycialisuss.com - cialis price Cheap Zithromax Online Cialis Assunzione Viagra Cialis

博客已萌萌哒运行
© 2020 由 Typecho 强力驱动.Theme by Yodu
前篇 后篇
雷姆
拉姆