虫言虫语 虫言虫语

给scrapy设置HTTP代理

in Pythonread (26959) 文章转载请注明来源!

在scrapy项目文件下建立一个.py文件 填入以下信息:

# Importing base64 library because we'll need it ONLY in case if the proxy we are going to use requires authentication
import base64
# Start your middleware class
class ProxyMiddleware(object):
    # overwrite process request
    def process_request(self, request, spider):
        # Set the location of the proxy
        request.meta['proxy'] = "http://proxyIP:port"

        # Use the following lines if your proxy requires authentication
        # proxy_user_pass = "USERNAME:PASSWORD"
        # setup basic authentication for the proxy
        # encoded_user_pass = base64.encodestring(proxy_user_pass)
        # request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass

在setting中加入以下代码:(其中project_name自行替换后面跟上建立的.py文件路径,最后加上cls名)

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
    'project_name.middlewares.ProxyMiddleware': 100,
}
jrotty WeChat Pay

微信打赏

jrotty Alipay

支付宝打赏

文章二维码

扫描二维码,在手机上阅读!

发表新评论
已有 5955 条评论
  1. BMizBomia
    BMizBomia
    回复

    get money today without loan i need loan to pay off debt need loan get a loan with bad credit no bank account. unsecured personal loans keybank i need financial help urgently, personal loans no credit. unsecured loans in dubai unsecured personal loan lenders in india 6000 8f0d3b5 get unsecured loans in kenya, personal loans for students, unsecured loans no credit check direct lender. how to get a personal loan with no credit check personal loans in georgia, personal loans at bank of america i need a small loan right now. unsecured personal loans in vadodara we need financial help, get a personal loan today near me get pre approved for car loan now what.

  2. DouglasUriva
    DouglasUriva
    回复

    diflucan 150 mg tablets diflucan prescription diflucan buy in usa

  3. jqubtjtumtus
    jqubtjtumtus
    回复

    cialis brand name usa price buacfqsxkivc buy cialis online usa cialis pills is there a generic cialis? cialis pills for men
    soft cialis fenrgrjeyhte cialis daily 5 mg price cialis for daily use discount tadalafil ceilas for sale
    best cialis prices axeydtgcwkbb walgreens cialis 20mg price meds without a doctor's prescription cialis pill best online cialis

  4. MichaelKig
    MichaelKig
    回复

    cialis and viagra which is better http://erectilehall.com - generic cialis cialis aumenta lo sperma cialis generic health care law

  5. JackDes
    JackDes
    回复

    sildalis online triamterene medication buy sildenafil online uk diclofenac 8 gel

  6. jatoktriqpmy
    jatoktriqpmy
    回复

    cialis for daily use usa pharmacy online purchase viagra without a doctor prescription online pharmacy cialis cialis 5mg price comparison
    cialis without a doctor cialis 5mg price comparison cialis on the internet purchase cialis cialis
    online pharmacies in usa female cialis cialis usa best price on cialis 20mg 5mg cialis ed pills without prescription
    gokgwdymjnqd calais pills from canada herbal cialis cheap cialis online cialis pills cialis 5 mg

  7. JoeDes
    JoeDes
    回复

    advair diskus 50 coupon sildenafil 1 cream

  8. KimDes
    KimDes
    回复

    zofran canada

  9. JimDes
    JimDes
    回复

    kamagra fast shipping accutane cheapest price cost of advair 250 50 seroquel euphoria cheap generic viagra online canadian pharmacy plavix without a prescription

  10. PaulDes
    PaulDes
    回复

    diclofenac gel 3 clopidogrel bisulfate plavix generic zithromax online triamterene buspar in canada

  11. KimDes
    KimDes
    回复

    xenical singapore

  12. SueDes
    SueDes
    回复

    albendazole brand india cephalexin tablet brand name furosemide cost us prednisolone 20 mg buy lexapro online cheap prazosin cost canada dapoxetine uk prescription orlistat xenical finpecia 1 mg estrace cheapest price

  13. KimDes
    KimDes
    回复

    proscar mexico

  14. AshDes
    AshDes
    回复

    malegra 100 mg for sale buy prozac 20 mg colchicine gout cost of ivermectin medicine cheapest on line valtrex without a prescription zithromax azithromycin ampicillin 500

  15. DouglasUriva
    DouglasUriva
    回复

    diflucan 200 mg daily how to get diflucan diflucan 50 mg capsule

博客已萌萌哒运行
© 2020 由 Typecho 强力驱动.Theme by Yodu
前篇 后篇
雷姆
拉姆